Skip to content

Metrics Reference

philanthropy.metrics

Donor KPI calculators.

IntervalReport dataclass

Everything needed to judge a set of intervals, not just certify them.

Attributes:

Name Type Description
n int

Rows scored.
coverage float

Fraction of targets inside their interval.
requested_level float

1 - alpha. Compare against the calibrator's attained_level_, which is what a one-rank construction can actually deliver, rather than against this.
score_mean, score_median, score_trimmed_mean float

The interval score aggregated three ways. The mean is the proper score; the other two are there because on gift amounts a handful of donors can carry it. A ranking that flips between them is a ranking of the tail.
median_width float

Median upper - lower.
median_target float

Median y_true.
width_ratio float

median_width / median_target. The regime indicator: near 2 is an interval a gift officer can work from, near 40 is one that is valid and carries no information. inf when the median target is zero.
median_bound_ratio float

Median upper / lower over rows with a positive lower bound, so a clipped-to-zero interval does not read as infinitely wide. nan when no row has one. A ratio of 3,627 is an interval from yhat / 60 to 60 * yhat.
Source code in philanthropy/metrics/_conformal.py 
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
@dataclass(frozen=True)
class IntervalReport:
    """Everything needed to judge a set of intervals, not just certify them.

    Attributes
    ----------
    n : int
        Rows scored.
    coverage : float
        Fraction of targets inside their interval.
    requested_level : float
        ``1 - alpha``. Compare against the calibrator's ``attained_level_``,
        which is what a one-rank construction can actually deliver, rather than
        against this.
    score_mean, score_median, score_trimmed_mean : float
        The interval score aggregated three ways. The mean is the proper score;
        the other two are there because on gift amounts a handful of donors can
        carry it. A ranking that flips between them is a ranking of the tail.
    median_width : float
        Median ``upper - lower``.
    median_target : float
        Median ``y_true``.
    width_ratio : float
        ``median_width / median_target``. The regime indicator: near 2 is an
        interval a gift officer can work from, near 40 is one that is valid and
        carries no information. ``inf`` when the median target is zero.
    median_bound_ratio : float
        Median ``upper / lower`` over rows with a positive lower bound, so a
        clipped-to-zero interval does not read as infinitely wide. ``nan`` when
        no row has one. A ratio of 3,627 is an interval from ``yhat / 60`` to
        ``60 * yhat``.
    """

    n: int
    coverage: float
    requested_level: float
    score_mean: float
    score_median: float
    score_trimmed_mean: float
    median_width: float
    median_target: float
    width_ratio: float
    median_bound_ratio: float

donor_retention_rate(current_donors, prior_donors)

Share of the prior period's donors who gave again this period.

Returns a fraction in [0.0, 1.0]; 0.0 when prior_donors is empty (no base to retain from).

Parameters:

Name Type Description Default
current_donors collection

Donor identifiers seen in the current period.

 required
prior_donors collection

Donor identifiers seen in the prior period.

 required

Returns:

Type Description
float

Retained share in [0.0, 1.0]. Returns 0.0 when prior_donors is empty (no base to retain from).
Source code in philanthropy/metrics/_scoring.py 
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
def donor_retention_rate(
    current_donors: Collection,
    prior_donors: Collection,
) -> float:
    """Share of the prior period's donors who gave again this period.

    Returns a fraction in ``[0.0, 1.0]``; ``0.0`` when ``prior_donors`` is
    empty (no base to retain from).

    Parameters
    ----------
    current_donors : collection
        Donor identifiers seen in the current period.

    prior_donors : collection
        Donor identifiers seen in the prior period.

    Returns
    -------
    float
        Retained share in ``[0.0, 1.0]``. Returns ``0.0`` when
        ``prior_donors`` is empty (no base to retain from).
    """
    current_set = set(current_donors)
    prior_set = set(prior_donors)

    if not prior_set:
        return 0.0

    retained = current_set & prior_set
    return len(retained) / len(prior_set)

donor_acquisition_cost(*, total_fundraising_expense, new_donors_acquired)

Average spend to acquire one new donor.

Returns np.inf when new_donors_acquired is 0 (spend with nothing acquired), so the result is always safe to compare or plot.

Parameters:

Name Type Description Default
total_fundraising_expense float

Total fundraising expense for the period.

 required
new_donors_acquired int

Number of first-time donors acquired in the period.

 required

Returns:

Type Description
float

Expense per new donor. Returns np.inf when new_donors_acquired is 0.
Source code in philanthropy/metrics/_scoring.py 
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
def donor_acquisition_cost(
    *,
    total_fundraising_expense: float,
    new_donors_acquired: int,
) -> float:
    """Average spend to acquire one new donor.

    Returns ``np.inf`` when ``new_donors_acquired`` is 0 (spend with nothing
    acquired), so the result is always safe to compare or plot.

    Parameters
    ----------
    total_fundraising_expense : float
        Total fundraising expense for the period.

    new_donors_acquired : int
        Number of first-time donors acquired in the period.

    Returns
    -------
    float
        Expense per new donor. Returns ``np.inf`` when
        ``new_donors_acquired`` is 0.
    """
    if new_donors_acquired == 0:
        return np.inf

    return total_fundraising_expense / new_donors_acquired

cost_per_dollar_raised(*, total_fundraising_expense, total_raised)

Fundraising expense per dollar of revenue raised.

A headline efficiency KPI: values below ~0.20 are typically healthy for a mature program. Returns np.inf when total_raised is 0 (spend with nothing raised), so the result is always safe to compare or plot.

Parameters:

Name Type Description Default
total_fundraising_expense float

Total fundraising expense for the period.

 required
total_raised float

Total revenue raised in the period.

 required

Returns:

Type Description
float

Expense per dollar raised. Returns np.inf when total_raised is 0.
Source code in philanthropy/metrics/_scoring.py 
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
def cost_per_dollar_raised(
    *,
    total_fundraising_expense: float,
    total_raised: float,
) -> float:
    """Fundraising expense per dollar of revenue raised.

    A headline efficiency KPI: values below ~0.20 are typically healthy for a
    mature program. Returns ``np.inf`` when ``total_raised`` is 0 (spend with
    nothing raised), so the result is always safe to compare or plot.

    Parameters
    ----------
    total_fundraising_expense : float
        Total fundraising expense for the period.

    total_raised : float
        Total revenue raised in the period.

    Returns
    -------
    float
        Expense per dollar raised. Returns ``np.inf`` when ``total_raised``
        is 0.
    """
    if total_raised == 0:
        return np.inf

    return total_fundraising_expense / total_raised

fundraising_roi(*, total_raised, total_fundraising_expense)

Net return on fundraising investment, (raised - expense) / expense.

0.0 means the program broke even; 3.0 means every dollar spent returned three dollars of net revenue. Returns np.inf when total_fundraising_expense is 0 (revenue with no spend).

Parameters:

Name Type Description Default
total_raised float

Total revenue raised in the period.

 required
total_fundraising_expense float

Total fundraising expense for the period.

 required

Returns:

Type Description
float

Net return per dollar of fundraising expense. Returns np.inf when total_fundraising_expense is 0.
Source code in philanthropy/metrics/_scoring.py 
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
def fundraising_roi(
    *,
    total_raised: float,
    total_fundraising_expense: float,
) -> float:
    """Net return on fundraising investment, ``(raised - expense) / expense``.

    ``0.0`` means the program broke even; ``3.0`` means every dollar spent
    returned three dollars of net revenue. Returns ``np.inf`` when
    ``total_fundraising_expense`` is 0 (revenue with no spend).

    Parameters
    ----------
    total_raised : float
        Total revenue raised in the period.

    total_fundraising_expense : float
        Total fundraising expense for the period.

    Returns
    -------
    float
        Net return per dollar of fundraising expense. Returns ``np.inf``
        when ``total_fundraising_expense`` is 0.
    """
    if total_fundraising_expense == 0:
        return np.inf

    return (total_raised - total_fundraising_expense) / total_fundraising_expense

donor_lifetime_value(average_donation, lifespan_years, discount_rate=0.05, retention_rate=None)

Computes the Net Present Value (NPV) of a donor's future giving.

Two modes, and they are different calculations rather than the same one with a substituted lifespan.

Fixed horizon (retention_rate=None). lifespan_years is taken as certain and the result is the NPV of an ordinary annuity::

discount_rate > 0:   NPV = m * (1 - (1 + d) ** -L) / d
discount_rate == 0:  NPV = m * L

Geometric lifetime (retention_rate given). The donor gives once at the end of year 1, then survives each subsequent year with probability r, so the lifetime is geometric on {1, 2, ...} with E[L] = 1 / (1 - r). The expected NPV is then::

discount_rate > 0:   E[NPV] = m / (1 + d - r)
discount_rate == 0:  E[NPV] = m / (1 - r)

Note that this is not the annuity formula evaluated at E[L]. The annuity is concave in L, so by Jensen's inequality NPV(E[L]) >= E[NPV(L)], and substituting the expected lifespan into the annuity therefore overstates lifetime value in every case where d > 0 and 0 < r < 1. The error is one-signed and not small: at r = 0.8, d = 0.05 it is +8.2%, and at r = 0.9, d = 0.10 it is +22.9%. This function computed NPV(E[L]) before version 0.7.0.

The two modes agree where they should: at r = 0 both give m / (1 + d), one gift discounted one year, and the d == 0 branch is the same in both because a sum with no discounting is linear in L.

Parameters:

Name Type Description Default
average_donation float

The average annual donation amount.

 required
lifespan_years float

The fixed number of years the donor is expected to continue giving. Only used if retention_rate is None.

 required
discount_rate float

The discount rate used to compute the net present value of future gifts (e.g., 0.05 for 5%).

 0.05
retention_rate float

The annual retention rate of the donor (e.g., 0.80 for 80%). If provided, the geometric-lifetime expectation above is used and lifespan_years is ignored.

 None

Returns:

Type Description
float

The calculated Net Present Value of the expected donor lifetime value. inf when retention_rate == 1.0 and discount_rate == 0: a donor who never lapses, with no discounting, has unbounded value. With discount_rate > 0 the same donor is a perpetuity worth m / d.

Raises:

Type Description
ValueError

If retention_rate, lifespan_years, or discount_rate is negative, or if retention_rate exceeds 1.

Examples:

>>> round(donor_lifetime_value(1000.0, 5, discount_rate=0.05), 2)
4329.48

An 80% retention rate implies the same 5-year expected lifespan, but the expected NPV is lower than the 5-year annuity, not equal to it:

>>> round(donor_lifetime_value(1000.0, 999, discount_rate=0.05,
...                            retention_rate=0.8), 2)
4000.0
Source code in philanthropy/metrics/_financial.py 
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
def donor_lifetime_value(
    average_donation: float,
    lifespan_years: float,
    discount_rate: float = 0.05,
    retention_rate: float | None = None
) -> float:
    """
    Computes the Net Present Value (NPV) of a donor's future giving.

    Two modes, and they are different calculations rather than the same one with
    a substituted lifespan.

    **Fixed horizon** (``retention_rate=None``). ``lifespan_years`` is taken as
    certain and the result is the NPV of an ordinary annuity::

        discount_rate > 0:   NPV = m * (1 - (1 + d) ** -L) / d
        discount_rate == 0:  NPV = m * L

    **Geometric lifetime** (``retention_rate`` given). The donor gives once at
    the end of year 1, then survives each subsequent year with probability
    ``r``, so the lifetime is geometric on ``{1, 2, ...}`` with
    ``E[L] = 1 / (1 - r)``. The expected NPV is then::

        discount_rate > 0:   E[NPV] = m / (1 + d - r)
        discount_rate == 0:  E[NPV] = m / (1 - r)

    Note that this is **not** the annuity formula evaluated at ``E[L]``. The
    annuity is concave in ``L``, so by Jensen's inequality
    ``NPV(E[L]) >= E[NPV(L)]``, and substituting the expected lifespan into the
    annuity therefore overstates lifetime value in every case where ``d > 0``
    and ``0 < r < 1``. The error is one-signed and not small: at ``r = 0.8``,
    ``d = 0.05`` it is +8.2%, and at ``r = 0.9``, ``d = 0.10`` it is +22.9%.
    This function computed ``NPV(E[L])`` before version 0.7.0.

    The two modes agree where they should: at ``r = 0`` both give
    ``m / (1 + d)``, one gift discounted one year, and the ``d == 0`` branch is
    the same in both because a sum with no discounting is linear in ``L``.

    Parameters
    ----------
    average_donation : float
        The average annual donation amount.
    lifespan_years : float
        The fixed number of years the donor is expected to continue giving.
        Only used if retention_rate is None.
    discount_rate : float, default=0.05
        The discount rate used to compute the net present value of future gifts
        (e.g., 0.05 for 5%).
    retention_rate : float, default=None
        The annual retention rate of the donor (e.g., 0.80 for 80%). If
        provided, the geometric-lifetime expectation above is used and
        ``lifespan_years`` is ignored.

    Returns
    -------
    float
        The calculated Net Present Value of the expected donor lifetime value.
        ``inf`` when ``retention_rate == 1.0`` and ``discount_rate == 0``: a
        donor who never lapses, with no discounting, has unbounded value. With
        ``discount_rate > 0`` the same donor is a perpetuity worth ``m / d``.

    Raises
    ------
    ValueError
        If ``retention_rate``, ``lifespan_years``, or ``discount_rate`` is
        negative, or if ``retention_rate`` exceeds 1.

    Examples
    --------
    >>> round(donor_lifetime_value(1000.0, 5, discount_rate=0.05), 2)
    4329.48

    An 80% retention rate implies the same 5-year expected lifespan, but the
    expected NPV is lower than the 5-year annuity, not equal to it:

    >>> round(donor_lifetime_value(1000.0, 999, discount_rate=0.05,
    ...                            retention_rate=0.8), 2)
    4000.0
    """
    if discount_rate < 0:
        raise ValueError("discount_rate cannot be negative.")

    if retention_rate is not None:
        if retention_rate < 0.0:
            raise ValueError("retention_rate cannot be negative.")
        if retention_rate > 1.0:
            raise ValueError("retention_rate cannot exceed 1.")

        if discount_rate == 0:
            # Undiscounted: m * E[L] = m / (1 - r). Infinite at r == 1.
            if retention_rate == 1.0:
                return float("inf")
            return average_donation / (1.0 - retention_rate)

        # E[NPV] over a geometric lifetime. At r == 1 this is the perpetuity
        # m / d, which the expression already yields.
        return average_donation / (1.0 + discount_rate - retention_rate)

    if lifespan_years < 0:
        raise ValueError("lifespan_years cannot be negative.")

    if discount_rate == 0:
        return average_donation * lifespan_years

    return (
        average_donation
        * (1 - (1 + discount_rate) ** (-lifespan_years))
        / discount_rate
    )

disparate_impact_ratio(y_pred, sensitive_features, pos_label=1)

Four-fifths-rule disparate-impact ratio across protected groups.

Computes min(selection_rate) / max(selection_rate) over the groups in sensitive_features. A value of 1.0 is exact parity; the US EEOC "four-fifths rule" flags a ratio below 0.8 as evidence of adverse impact that warrants investigation.

This is a diagnostic, not a fairness guarantee or legal clearance: a passing ratio does not certify a model as non-discriminatory, and the choice of protected groups and decision threshold materially affects the result.

Parameters:

Name Type Description Default
y_pred array-like of shape (n_samples,)

Binary decisions. Threshold continuous scores first.

 required
sensitive_features array-like of shape (n_samples,)

Protected-group label per sample.

 required
pos_label default=1

Value in y_pred that counts as "selected".

 1

Returns:

Type Description
float

Ratio in [0.0, 1.0]. Returns 1.0 when only one group is present or when no sample in any group is selected (no disparity to measure).

Raises:

Type Description
ValueError

If inputs have mismatched lengths, are empty, or contain missing values.
Source code in philanthropy/metrics/_fairness.py 
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
def disparate_impact_ratio(
    y_pred: Collection,
    sensitive_features: Collection,
    pos_label: Any = 1,
) -> float:
    """Four-fifths-rule disparate-impact ratio across protected groups.

    Computes ``min(selection_rate) / max(selection_rate)`` over the groups in
    ``sensitive_features``. A value of ``1.0`` is exact parity; the US EEOC
    "four-fifths rule" flags a ratio below ``0.8`` as evidence of adverse impact
    that warrants investigation.

    This is a **diagnostic, not a fairness guarantee or legal clearance**: a
    passing ratio does not certify a model as non-discriminatory, and the choice
    of protected groups and decision threshold materially affects the result.

    Parameters
    ----------
    y_pred : array-like of shape (n_samples,)
        Binary decisions. Threshold continuous scores first.
    sensitive_features : array-like of shape (n_samples,)
        Protected-group label per sample.
    pos_label : default=1
        Value in ``y_pred`` that counts as "selected".

    Returns
    -------
    float
        Ratio in ``[0.0, 1.0]``. Returns ``1.0`` when only one group is present
        or when no sample in any group is selected (no disparity to measure).

    Raises
    ------
    ValueError
        If inputs have mismatched lengths, are empty, or contain missing values.
    """
    rates = selection_rate_by_group(y_pred, sensitive_features, pos_label=pos_label)
    values = np.array(list(rates.values()), dtype=float)
    max_rate = values.max()
    if max_rate == 0.0:
        return 1.0
    return float(values.min() / max_rate)

selection_rate_by_group(y_pred, sensitive_features, pos_label=1)

Fraction selected (y_pred == pos_label) within each protected group.

Parameters:

Name Type Description Default
y_pred array-like of shape (n_samples,)

Binary decisions (e.g. "flagged for major-gift outreach"). Threshold continuous scores before calling this.

 required
sensitive_features array-like of shape (n_samples,)

Protected-group label per sample (e.g. race, age band, gender).

 required
pos_label default=1

Value in y_pred that counts as "selected".

 1

Returns:

Type Description
dict

Mapping of group value -> selection rate in [0.0, 1.0].

Raises:

Type Description
ValueError

If inputs have mismatched lengths, are empty, or contain missing values.
Source code in philanthropy/metrics/_fairness.py 
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
def selection_rate_by_group(
    y_pred: Collection,
    sensitive_features: Collection,
    pos_label: Any = 1,
) -> Dict[object, float]:
    """Fraction selected (``y_pred == pos_label``) within each protected group.

    Parameters
    ----------
    y_pred : array-like of shape (n_samples,)
        Binary decisions (e.g. "flagged for major-gift outreach"). Threshold
        continuous scores before calling this.
    sensitive_features : array-like of shape (n_samples,)
        Protected-group label per sample (e.g. race, age band, gender).
    pos_label : default=1
        Value in ``y_pred`` that counts as "selected".

    Returns
    -------
    dict
        Mapping of group value -> selection rate in ``[0.0, 1.0]``.

    Raises
    ------
    ValueError
        If inputs have mismatched lengths, are empty, or contain missing values.
    """
    y_pred = np.asarray(y_pred)
    groups = np.asarray(sensitive_features)
    if y_pred.shape[0] != groups.shape[0]:
        raise ValueError(
            f"y_pred and sensitive_features must be the same length, got "
            f"{y_pred.shape[0]} and {groups.shape[0]}."
        )
    if y_pred.shape[0] == 0:
        raise ValueError("y_pred is empty; nothing to score.")
    if pd.isna(groups).any():
        raise ValueError(
            "sensitive_features contains missing (NaN/None) group labels; a "
            "missing label would silently NaN-out the diagnostic. Drop or "
            "impute those rows before computing fairness metrics."
        )

    selected = y_pred == pos_label
    return {g: float(selected[groups == g].mean()) for g in np.unique(groups)}

gift_concentration_gini(amounts)

Gini coefficient of a set of donor gift amounts.

0.0 is perfect equality (every donor gives the same); values approaching 1.0 mean revenue is concentrated in a few donors.

Parameters:

Name Type Description Default
amounts array-like of shape (n_donors,)

Per-donor total giving. NaN entries are dropped; negatives raise.

 required

Returns:

Type Description
float

Gini coefficient in [0.0, 1.0]. Returns 0.0 for an empty input or an all-zero total (no distribution to measure).

Raises:

Type Description
ValueError

If any gift amounts are negative.
Source code in philanthropy/metrics/_concentration.py 
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
def gift_concentration_gini(amounts: Collection) -> float:
    """Gini coefficient of a set of donor gift amounts.

    ``0.0`` is perfect equality (every donor gives the same); values approaching
    ``1.0`` mean revenue is concentrated in a few donors.

    Parameters
    ----------
    amounts : array-like of shape (n_donors,)
        Per-donor total giving. ``NaN`` entries are dropped; negatives raise.

    Returns
    -------
    float
        Gini coefficient in ``[0.0, 1.0]``. Returns ``0.0`` for an empty input
        or an all-zero total (no distribution to measure).

    Raises
    ------
    ValueError
        If any gift amounts are negative.
    """
    a = _clean_nonneg_amounts(amounts)
    total = a.sum()
    if a.size == 0 or total == 0:
        return 0.0
    a = np.sort(a)
    n = a.size
    index = np.arange(1, n + 1)
    # Mean-absolute-difference form of the Gini coefficient.
    return float((2.0 * np.sum(index * a)) / (n * total) - (n + 1.0) / n)

top_donor_share(amounts, top_fraction=0.1)

Fraction of total revenue contributed by the top top_fraction donors.

Parameters:

Name Type Description Default
amounts array-like of shape (n_donors,)

Per-donor total giving. NaN entries are dropped; negatives raise.

 required
top_fraction float

Slice of donors (ranked by giving, descending) to sum. Must be in (0.0, 1.0]. At least one donor is always counted.

 0.1

Returns:

Type Description
float

Share in [0.0, 1.0]. Returns 0.0 for an empty input or an all-zero total.

Raises:

Type Description
ValueError

If top_fraction is not in (0.0, 1.0], or if any gift amounts are negative.
Source code in philanthropy/metrics/_concentration.py 
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
def top_donor_share(amounts: Collection, top_fraction: float = 0.1) -> float:
    """Fraction of total revenue contributed by the top ``top_fraction`` donors.

    Parameters
    ----------
    amounts : array-like of shape (n_donors,)
        Per-donor total giving. ``NaN`` entries are dropped; negatives raise.
    top_fraction : float, default=0.1
        Slice of donors (ranked by giving, descending) to sum. Must be in
        ``(0.0, 1.0]``. At least one donor is always counted.

    Returns
    -------
    float
        Share in ``[0.0, 1.0]``. Returns ``0.0`` for an empty input or an
        all-zero total.

    Raises
    ------
    ValueError
        If ``top_fraction`` is not in ``(0.0, 1.0]``, or if any gift amounts are negative.
    """
    if not 0.0 < top_fraction <= 1.0:
        raise ValueError("top_fraction must be in (0.0, 1.0].")
    a = _clean_nonneg_amounts(amounts)
    total = a.sum()
    if a.size == 0 or total == 0:
        return 0.0
    a = np.sort(a)[::-1]
    k = max(1, int(np.ceil(top_fraction * a.size)))
    return float(a[:k].sum() / total)

conformal_pvalue(calibration_scores, scores)

Split-conformal p-value of each score against a calibration set.

Small p-values mean the score is high relative to the calibration donors, so conformal_pvalue(...) <= alpha selects the donors whose scores are extreme at level alpha, and the expected selection rate among exchangeable donors is at most alpha. The calibration scores must come from donors held out of training, exchangeable with the ones being scored; reusing training rows breaks the guarantee exactly the way refitting a transformer on test data does. To read alpha as a false-positive rate the calibration set must contain only donors who did not give; see the module docstring.

Parameters:

Name Type Description Default
calibration_scores array-like of shape (n_calibration,)

Scores of held-out donors, higher meaning more likely to give. Must be non-empty and finite; NaN and infinities raise rather than being dropped, because the denominator n + 1 counts them.

 required
scores array-like of shape (n_samples,)

Scores to test. May be a scalar-like sequence of any length, including empty. NaN entries yield NaN p-values.

 required

Returns:

Type Description
ndarray of shape (n_samples,)

P-values in [1 / (n_calibration + 1), 1.0]. Never 0, never above 1.

Raises:

Type Description
ValueError

If calibration_scores is empty, not one-dimensional, or contains non-finite values.

Examples:

>>> import numpy as np
>>> from philanthropy.metrics import conformal_pvalue
>>> calibration = np.arange(9, dtype=float)          # 0 .. 8, n = 9
>>> conformal_pvalue(calibration, [8.5, 4.0, -1.0])
array([0.1, 0.6, 1. ])

A score above every calibration point still gets 1 / (n + 1), not 0:

>>> float(conformal_pvalue(calibration, [1e6])[0])
0.1
Source code in philanthropy/metrics/_conformal.py 
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
def conformal_pvalue(calibration_scores: Collection, scores: Collection) -> np.ndarray:
    """Split-conformal p-value of each score against a calibration set.

    Small p-values mean the score is high relative to the calibration donors,
    so ``conformal_pvalue(...) <= alpha`` selects the donors whose scores are
    extreme at level ``alpha``, and the expected selection rate among
    exchangeable donors is at most ``alpha``. The calibration scores must come
    from donors held out of training, exchangeable with the ones being scored;
    reusing training rows breaks the guarantee exactly the way refitting a
    transformer on test data does. To read ``alpha`` as a false-positive rate
    the calibration set must contain only donors who did not give; see the
    module docstring.

    Parameters
    ----------
    calibration_scores : array-like of shape (n_calibration,)
        Scores of held-out donors, higher meaning more likely to give. Must be
        non-empty and finite; ``NaN`` and infinities raise rather than being
        dropped, because the denominator ``n + 1`` counts them.
    scores : array-like of shape (n_samples,)
        Scores to test. May be a scalar-like sequence of any length, including
        empty. ``NaN`` entries yield ``NaN`` p-values.

    Returns
    -------
    ndarray of shape (n_samples,)
        P-values in ``[1 / (n_calibration + 1), 1.0]``. Never 0, never above 1.

    Raises
    ------
    ValueError
        If ``calibration_scores`` is empty, not one-dimensional, or contains
        non-finite values.

    Examples
    --------
    >>> import numpy as np
    >>> from philanthropy.metrics import conformal_pvalue
    >>> calibration = np.arange(9, dtype=float)          # 0 .. 8, n = 9
    >>> conformal_pvalue(calibration, [8.5, 4.0, -1.0])
    array([0.1, 0.6, 1. ])

    A score above every calibration point still gets ``1 / (n + 1)``, not 0:

    >>> float(conformal_pvalue(calibration, [1e6])[0])
    0.1
    """
    cal = np.asarray(calibration_scores, dtype=float)
    if cal.ndim != 1:
        raise ValueError("calibration_scores must be one-dimensional.")
    if cal.size == 0:
        raise ValueError("calibration_scores must be non-empty.")
    if not np.all(np.isfinite(cal)):
        raise ValueError("calibration_scores must be finite (no NaN or inf).")

    s = np.asarray(scores, dtype=float)
    n = cal.size
    cal_sorted = np.sort(cal)
    # |{i : cal_i >= s}|; 'left' so calibration points equal to s are counted.
    n_ge = n - np.searchsorted(cal_sorted, s, side="left")
    p = (1.0 + n_ge) / (n + 1.0)
    return np.where(np.isnan(s), np.nan, p)

interval_report(y_true, lower, upper, alpha=0.05, trim=0.1)

Coverage, interval score and width-to-target for a set of intervals.

Coverage alone cannot tell a useful interval from [0, inf). This returns the three things that can: the proper score aggregated robustly, the width relative to the amounts being predicted, and the ratio between the bounds.

Parameters:

Name Type Description Default
y_true array-like of shape (n_samples,)

Realised amounts.

 required
lower array-like of shape (n_samples,)

The interval bounds.

 required
upper array-like of shape (n_samples,)

The interval bounds.

 required
alpha float

Miscoverage the intervals were built for; sets both the score's penalty weight and requested_level.

 0.05
trim float

Fraction of rows dropped from each end before score_trimmed_mean. Must be in [0, 0.5).

 0.1

Returns:

Type Description
IntervalReport

Raises:

Type Description
ValueError

On the same input problems as :func:interval_score, or trim outside [0, 0.5).

Examples:

>>> from philanthropy.metrics import interval_report
>>> report = interval_report([10.0, 30.0], [5.0, 5.0], [20.0, 20.0], alpha=0.5)
>>> report.coverage, report.median_width, report.median_target
(0.5, 15.0, 20.0)
>>> report.width_ratio, report.median_bound_ratio
(0.75, 4.0)

The score is heavy-tailed on gift amounts, so read all three:

>>> report.score_mean, report.score_median
(35.0, 35.0)
Source code in philanthropy/metrics/_conformal.py 
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
def interval_report(
    y_true: Collection,
    lower: Collection,
    upper: Collection,
    alpha: float = 0.05,
    trim: float = 0.1,
) -> IntervalReport:
    """Coverage, interval score and width-to-target for a set of intervals.

    Coverage alone cannot tell a useful interval from ``[0, inf)``. This returns
    the three things that can: the proper score aggregated robustly, the width
    relative to the amounts being predicted, and the ratio between the bounds.

    Parameters
    ----------
    y_true : array-like of shape (n_samples,)
        Realised amounts.
    lower, upper : array-like of shape (n_samples,)
        The interval bounds.
    alpha : float, default=0.05
        Miscoverage the intervals were built for; sets both the score's penalty
        weight and ``requested_level``.
    trim : float, default=0.1
        Fraction of rows dropped from **each** end before ``score_trimmed_mean``.
        Must be in ``[0, 0.5)``.

    Returns
    -------
    IntervalReport

    Raises
    ------
    ValueError
        On the same input problems as :func:`interval_score`, or ``trim``
        outside ``[0, 0.5)``.

    Examples
    --------
    >>> from philanthropy.metrics import interval_report
    >>> report = interval_report([10.0, 30.0], [5.0, 5.0], [20.0, 20.0], alpha=0.5)
    >>> report.coverage, report.median_width, report.median_target
    (0.5, 15.0, 20.0)
    >>> report.width_ratio, report.median_bound_ratio
    (0.75, 4.0)

    The score is heavy-tailed on gift amounts, so read all three:

    >>> report.score_mean, report.score_median
    (35.0, 35.0)
    """
    y, lo, hi, a = _interval_inputs(y_true, lower, upper, alpha)
    if not 0 <= trim < 0.5:
        raise ValueError(f"trim must satisfy 0 <= trim < 0.5, got {trim!r}.")

    per_row = _interval_score_per_row(y, lo, hi, a)
    width = hi - lo
    median_width = float(np.median(width))
    median_target = float(np.median(y))
    positive = lo > 0

    return IntervalReport(
        n=int(y.size),
        coverage=float(np.mean((y >= lo) & (y <= hi))),
        requested_level=1.0 - a,
        score_mean=float(np.mean(per_row)),
        score_median=float(np.median(per_row)),
        score_trimmed_mean=_trimmed_mean(per_row, trim),
        median_width=median_width,
        median_target=median_target,
        width_ratio=(
            median_width / median_target
            if median_target != 0
            else (np.inf if median_width > 0 else np.nan)
        ),
        median_bound_ratio=(
            float(np.median(hi[positive] / lo[positive]))
            if positive.any()
            else np.nan
        ),
    )

interval_score(y_true, lower, upper, alpha=0.05)

Mean interval score of a central 1 - alpha interval.

(u - l) + (2/alpha)(l - y)+ + (2/alpha)(y - u)+, averaged. Lower is better. Proper for a central interval, so widening to buy coverage costs more than it gains and the score cannot be gamed.

Parameters:

Name Type Description Default
y_true array-like of shape (n_samples,)

Realised amounts.

 required
lower array-like of shape (n_samples,)

The interval bounds.

 required
upper array-like of shape (n_samples,)

The interval bounds.

 required
alpha float

The miscoverage the interval was built for. This is the score's penalty weight, not a level to be attained, so the caller's requested alpha is the right value here even when the attained level differs.

 0.05

Returns:

Type Description
float

The mean score, in the target's units.

Raises:

Type Description
ValueError

If the three arrays disagree in shape, are empty, alpha is outside (0, 1), or any upper is below its lower.

Examples:

A covered row costs its width; a missed row adds 2 / alpha per dollar of miss:

>>> from philanthropy.metrics import interval_score
>>> interval_score([10.0, 30.0], [5.0, 5.0], [20.0, 20.0], alpha=0.5)
35.0
Source code in philanthropy/metrics/_conformal.py 
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
def interval_score(
    y_true: Collection,
    lower: Collection,
    upper: Collection,
    alpha: float = 0.05,
) -> float:
    """Mean interval score of a central ``1 - alpha`` interval.

    ``(u - l) + (2/alpha)(l - y)+ + (2/alpha)(y - u)+``, averaged. Lower is
    better. Proper for a central interval, so widening to buy coverage costs
    more than it gains and the score cannot be gamed.

    Parameters
    ----------
    y_true : array-like of shape (n_samples,)
        Realised amounts.
    lower, upper : array-like of shape (n_samples,)
        The interval bounds.
    alpha : float, default=0.05
        The miscoverage the interval was built for. This is the score's penalty
        weight, not a level to be attained, so the caller's requested ``alpha``
        is the right value here even when the attained level differs.

    Returns
    -------
    float
        The mean score, in the target's units.

    Raises
    ------
    ValueError
        If the three arrays disagree in shape, are empty, ``alpha`` is outside
        ``(0, 1)``, or any ``upper`` is below its ``lower``.

    Examples
    --------
    A covered row costs its width; a missed row adds ``2 / alpha`` per dollar
    of miss:

    >>> from philanthropy.metrics import interval_score
    >>> interval_score([10.0, 30.0], [5.0, 5.0], [20.0, 20.0], alpha=0.5)
    35.0
    """
    y, lo, hi, a = _interval_inputs(y_true, lower, upper, alpha)
    return float(np.mean(_interval_score_per_row(y, lo, hi, a)))