Skip to content

Metrics Reference

philanthropy.metrics

Donor KPI calculators.

IntervalReport dataclass

Everything needed to judge a set of intervals, not just certify them.

Attributes:

Name Type Description
n int

Rows scored.
coverage float

Fraction of targets inside their interval.
requested_level float

1 - alpha. Compare against the calibrator's attained_level_, which is what a one-rank construction can actually deliver, rather than against this.
score_mean, score_median, score_trimmed_mean float

The interval score aggregated three ways. The mean is the proper score; the other two are there because on gift amounts a handful of donors can carry it. A ranking that flips between them is a ranking of the tail.
median_width float

Median upper - lower.
median_target float

Median y_true.
width_ratio float

median_width / median_target. The regime indicator: near 2 is an interval a gift officer can work from, near 40 is one that is valid and carries no information. inf when the median target is zero.
median_bound_ratio float

Median upper / lower over rows with a positive lower bound, so a clipped-to-zero interval does not read as infinitely wide. nan when no row has one. A ratio of 3,627 is an interval from yhat / 60 to 60 * yhat.
Source code in philanthropy/metrics/_conformal.py 
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
@dataclass(frozen=True)
class IntervalReport:
    """Everything needed to judge a set of intervals, not just certify them.

    Attributes
    ----------
    n : int
        Rows scored.
    coverage : float
        Fraction of targets inside their interval.
    requested_level : float
        ``1 - alpha``. Compare against the calibrator's ``attained_level_``,
        which is what a one-rank construction can actually deliver, rather than
        against this.
    score_mean, score_median, score_trimmed_mean : float
        The interval score aggregated three ways. The mean is the proper score;
        the other two are there because on gift amounts a handful of donors can
        carry it. A ranking that flips between them is a ranking of the tail.
    median_width : float
        Median ``upper - lower``.
    median_target : float
        Median ``y_true``.
    width_ratio : float
        ``median_width / median_target``. The regime indicator: near 2 is an
        interval a gift officer can work from, near 40 is one that is valid and
        carries no information. ``inf`` when the median target is zero.
    median_bound_ratio : float
        Median ``upper / lower`` over rows with a positive lower bound, so a
        clipped-to-zero interval does not read as infinitely wide. ``nan`` when
        no row has one. A ratio of 3,627 is an interval from ``yhat / 60`` to
        ``60 * yhat``.
    """

    n: int
    coverage: float
    requested_level: float
    score_mean: float
    score_median: float
    score_trimmed_mean: float
    median_width: float
    median_target: float
    width_ratio: float
    median_bound_ratio: float

donor_retention_rate(current_donors, prior_donors)

Share of the prior period's donors who gave again this period.

Returns a fraction in [0.0, 1.0]; 0.0 when prior_donors is empty (no base to retain from).

Parameters:

Name Type Description Default
current_donors collection

Donor identifiers seen in the current period.

 required
prior_donors collection

Donor identifiers seen in the prior period.

 required

Returns:

Type Description
float

Retained share in [0.0, 1.0]. Returns 0.0 when prior_donors is empty (no base to retain from).
Source code in philanthropy/metrics/_scoring.py 
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
def donor_retention_rate(
    current_donors: Collection,
    prior_donors: Collection,
) -> float:
    """Share of the prior period's donors who gave again this period.

    Returns a fraction in ``[0.0, 1.0]``; ``0.0`` when ``prior_donors`` is
    empty (no base to retain from).

    Parameters
    ----------
    current_donors : collection
        Donor identifiers seen in the current period.

    prior_donors : collection
        Donor identifiers seen in the prior period.

    Returns
    -------
    float
        Retained share in ``[0.0, 1.0]``. Returns ``0.0`` when
        ``prior_donors`` is empty (no base to retain from).
    """
    current_set = set(current_donors)
    prior_set = set(prior_donors)

    if not prior_set:
        return 0.0

    retained = current_set & prior_set
    return len(retained) / len(prior_set)

donor_acquisition_cost(*, total_fundraising_expense, new_donors_acquired)

Average spend to acquire one new donor.

Returns np.inf when new_donors_acquired is 0 (spend with nothing acquired), so the result is always safe to compare or plot.

Parameters:

Name Type Description Default
total_fundraising_expense float

Total fundraising expense for the period.

 required
new_donors_acquired int

Number of first-time donors acquired in the period.

 required

Returns:

Type Description
float

Expense per new donor. Returns np.inf when new_donors_acquired is 0.
Source code in philanthropy/metrics/_scoring.py 
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
def donor_acquisition_cost(
    *,
    total_fundraising_expense: float,
    new_donors_acquired: int,
) -> float:
    """Average spend to acquire one new donor.

    Returns ``np.inf`` when ``new_donors_acquired`` is 0 (spend with nothing
    acquired), so the result is always safe to compare or plot.

    Parameters
    ----------
    total_fundraising_expense : float
        Total fundraising expense for the period.

    new_donors_acquired : int
        Number of first-time donors acquired in the period.

    Returns
    -------
    float
        Expense per new donor. Returns ``np.inf`` when
        ``new_donors_acquired`` is 0.
    """
    if new_donors_acquired == 0:
        return np.inf

    return total_fundraising_expense / new_donors_acquired

cost_per_dollar_raised(*, total_fundraising_expense, total_raised)

Fundraising expense per dollar of revenue raised.

A headline efficiency KPI: values below ~0.20 are typically healthy for a mature program. Returns np.inf when total_raised is 0 (spend with nothing raised), so the result is always safe to compare or plot.

Parameters:

Name Type Description Default
total_fundraising_expense float

Total fundraising expense for the period.

 required
total_raised float

Total revenue raised in the period.

 required

Returns:

Type Description
float

Expense per dollar raised. Returns np.inf when total_raised is 0.
Source code in philanthropy/metrics/_scoring.py 
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
def cost_per_dollar_raised(
    *,
    total_fundraising_expense: float,
    total_raised: float,
) -> float:
    """Fundraising expense per dollar of revenue raised.

    A headline efficiency KPI: values below ~0.20 are typically healthy for a
    mature program. Returns ``np.inf`` when ``total_raised`` is 0 (spend with
    nothing raised), so the result is always safe to compare or plot.

    Parameters
    ----------
    total_fundraising_expense : float
        Total fundraising expense for the period.

    total_raised : float
        Total revenue raised in the period.

    Returns
    -------
    float
        Expense per dollar raised. Returns ``np.inf`` when ``total_raised``
        is 0.
    """
    if total_raised == 0:
        return np.inf

    return total_fundraising_expense / total_raised

fundraising_roi(*, total_raised, total_fundraising_expense)

Net return on fundraising investment, (raised - expense) / expense.

0.0 means the program broke even; 3.0 means every dollar spent returned three dollars of net revenue. Returns np.inf when total_fundraising_expense is 0 (revenue with no spend).

Parameters:

Name Type Description Default
total_raised float

Total revenue raised in the period.

 required
total_fundraising_expense float

Total fundraising expense for the period.

 required

Returns:

Type Description
float

Net return per dollar of fundraising expense. Returns np.inf when total_fundraising_expense is 0.
Source code in philanthropy/metrics/_scoring.py 
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
def fundraising_roi(
    *,
    total_raised: float,
    total_fundraising_expense: float,
) -> float:
    """Net return on fundraising investment, ``(raised - expense) / expense``.

    ``0.0`` means the program broke even; ``3.0`` means every dollar spent
    returned three dollars of net revenue. Returns ``np.inf`` when
    ``total_fundraising_expense`` is 0 (revenue with no spend).

    Parameters
    ----------
    total_raised : float
        Total revenue raised in the period.

    total_fundraising_expense : float
        Total fundraising expense for the period.

    Returns
    -------
    float
        Net return per dollar of fundraising expense. Returns ``np.inf``
        when ``total_fundraising_expense`` is 0.
    """
    if total_fundraising_expense == 0:
        return np.inf

    return (total_raised - total_fundraising_expense) / total_fundraising_expense

donor_lifetime_value(average_donation, lifespan_years, discount_rate=0.05, retention_rate=None)

Computes the Net Present Value (NPV) of a donor's future giving.

Two modes, and they are different calculations rather than the same one with a substituted lifespan.

Fixed horizon (retention_rate=None). lifespan_years is taken as certain and the result is the NPV of an ordinary annuity::

discount_rate > 0:   NPV = m * (1 - (1 + d) ** -L) / d
discount_rate == 0:  NPV = m * L

Geometric lifetime (retention_rate given). The donor gives once at the end of year 1, then survives each subsequent year with probability r, so the lifetime is geometric on {1, 2, ...} with E[L] = 1 / (1 - r). The expected NPV is then::

discount_rate > 0:   E[NPV] = m / (1 + d - r)
discount_rate == 0:  E[NPV] = m / (1 - r)

Note that this is not the annuity formula evaluated at E[L]. The annuity is concave in L, so by Jensen's inequality NPV(E[L]) >= E[NPV(L)], and substituting the expected lifespan into the annuity therefore overstates lifetime value in every case where d > 0 and 0 < r < 1. The error is one-signed and not small: at r = 0.8, d = 0.05 it is +8.2%, and at r = 0.9, d = 0.10 it is +22.9%. This function computed NPV(E[L]) before version 0.7.0.

The two modes agree where they should: at r = 0 both give m / (1 + d), one gift discounted one year, and the d == 0 branch is the same in both because a sum with no discounting is linear in L.

Parameters:

Name Type Description Default
average_donation float

The average annual donation amount.

 required
lifespan_years float

The fixed number of years the donor is expected to continue giving. Only used if retention_rate is None.

 required
discount_rate float

The discount rate used to compute the net present value of future gifts (e.g., 0.05 for 5%).

 0.05
retention_rate float

The annual retention rate of the donor (e.g., 0.80 for 80%). If provided, the geometric-lifetime expectation above is used and lifespan_years is ignored.

 None

Returns:

Type Description
float

The calculated Net Present Value of the expected donor lifetime value. inf when retention_rate == 1.0 and discount_rate == 0: a donor who never lapses, with no discounting, has unbounded value. With discount_rate > 0 the same donor is a perpetuity worth m / d.

Raises:

Type Description
ValueError

If retention_rate, lifespan_years, or discount_rate is negative, or if retention_rate exceeds 1.

Examples:

>>> round(donor_lifetime_value(1000.0, 5, discount_rate=0.05), 2)
4329.48

An 80% retention rate implies the same 5-year expected lifespan, but the expected NPV is lower than the 5-year annuity, not equal to it:

>>> round(donor_lifetime_value(1000.0, 999, discount_rate=0.05,
...                            retention_rate=0.8), 2)
4000.0
Source code in philanthropy/metrics/_financial.py 
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
def donor_lifetime_value(
    average_donation: float,
    lifespan_years: float,
    discount_rate: float = 0.05,
    retention_rate: float | None = None
) -> float:
    """
    Computes the Net Present Value (NPV) of a donor's future giving.

    Two modes, and they are different calculations rather than the same one with
    a substituted lifespan.

    **Fixed horizon** (``retention_rate=None``). ``lifespan_years`` is taken as
    certain and the result is the NPV of an ordinary annuity::

        discount_rate > 0:   NPV = m * (1 - (1 + d) ** -L) / d
        discount_rate == 0:  NPV = m * L

    **Geometric lifetime** (``retention_rate`` given). The donor gives once at
    the end of year 1, then survives each subsequent year with probability
    ``r``, so the lifetime is geometric on ``{1, 2, ...}`` with
    ``E[L] = 1 / (1 - r)``. The expected NPV is then::

        discount_rate > 0:   E[NPV] = m / (1 + d - r)
        discount_rate == 0:  E[NPV] = m / (1 - r)

    Note that this is **not** the annuity formula evaluated at ``E[L]``. The
    annuity is concave in ``L``, so by Jensen's inequality
    ``NPV(E[L]) >= E[NPV(L)]``, and substituting the expected lifespan into the
    annuity therefore overstates lifetime value in every case where ``d > 0``
    and ``0 < r < 1``. The error is one-signed and not small: at ``r = 0.8``,
    ``d = 0.05`` it is +8.2%, and at ``r = 0.9``, ``d = 0.10`` it is +22.9%.
    This function computed ``NPV(E[L])`` before version 0.7.0.

    The two modes agree where they should: at ``r = 0`` both give
    ``m / (1 + d)``, one gift discounted one year, and the ``d == 0`` branch is
    the same in both because a sum with no discounting is linear in ``L``.

    Parameters
    ----------
    average_donation : float
        The average annual donation amount.
    lifespan_years : float
        The fixed number of years the donor is expected to continue giving.
        Only used if retention_rate is None.
    discount_rate : float, default=0.05
        The discount rate used to compute the net present value of future gifts
        (e.g., 0.05 for 5%).
    retention_rate : float, default=None
        The annual retention rate of the donor (e.g., 0.80 for 80%). If
        provided, the geometric-lifetime expectation above is used and
        ``lifespan_years`` is ignored.

    Returns
    -------
    float
        The calculated Net Present Value of the expected donor lifetime value.
        ``inf`` when ``retention_rate == 1.0`` and ``discount_rate == 0``: a
        donor who never lapses, with no discounting, has unbounded value. With
        ``discount_rate > 0`` the same donor is a perpetuity worth ``m / d``.

    Raises
    ------
    ValueError
        If ``retention_rate``, ``lifespan_years``, or ``discount_rate`` is
        negative, or if ``retention_rate`` exceeds 1.

    Examples
    --------
    >>> round(donor_lifetime_value(1000.0, 5, discount_rate=0.05), 2)
    4329.48

    An 80% retention rate implies the same 5-year expected lifespan, but the
    expected NPV is lower than the 5-year annuity, not equal to it:

    >>> round(donor_lifetime_value(1000.0, 999, discount_rate=0.05,
    ...                            retention_rate=0.8), 2)
    4000.0
    """
    if discount_rate < 0:
        raise ValueError("discount_rate cannot be negative.")

    if retention_rate is not None:
        if retention_rate < 0.0:
            raise ValueError("retention_rate cannot be negative.")
        if retention_rate > 1.0:
            raise ValueError("retention_rate cannot exceed 1.")

        if discount_rate == 0:
            # Undiscounted: m * E[L] = m / (1 - r). Infinite at r == 1.
            if retention_rate == 1.0:
                return float("inf")
            return average_donation / (1.0 - retention_rate)

        # E[NPV] over a geometric lifetime. At r == 1 this is the perpetuity
        # m / d, which the expression already yields.
        return average_donation / (1.0 + discount_rate - retention_rate)

    if lifespan_years < 0:
        raise ValueError("lifespan_years cannot be negative.")

    if discount_rate == 0:
        return average_donation * lifespan_years

    return (
        average_donation
        * (1 - (1 + discount_rate) ** (-lifespan_years))
        / discount_rate
    )

demographic_parity_difference(y_pred, sensitive_features, pos_label=1)

max(selection_rate) - min(selection_rate) across protected groups.

A companion to :func:disparate_impact_ratio for when selection rates are small: two rates of 0.01 and 0.02 give a ratio of 0.5 (looks severe) but a difference of 0.01 (looks negligible), and which reading matters depends on the decision being made. Report both.

This is a diagnostic, not a fairness guarantee or legal clearance: a small difference does not certify a model as non-discriminatory, and the choice of protected groups and decision threshold materially affects the result.

Parameters:

Name Type Description Default
y_pred array-like of shape (n_samples,)

Binary decisions. Threshold continuous scores first.

 required
sensitive_features array-like of shape (n_samples,)

Protected-group label per sample.

 required
pos_label default=1

Value in y_pred that counts as "selected".

 1

Returns:

Type Description
float

Difference in [0.0, 1.0]. 0.0 is exact parity, or when only one group is present.

Raises:

Type Description
ValueError

If inputs have mismatched lengths, are empty, or contain missing values.
Source code in philanthropy/metrics/_fairness.py 
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
def demographic_parity_difference(
    y_pred: Collection,
    sensitive_features: Collection,
    pos_label: Any = 1,
) -> float:
    """``max(selection_rate) - min(selection_rate)`` across protected groups.

    A companion to :func:`disparate_impact_ratio` for when selection rates
    are small: two rates of 0.01 and 0.02 give a ratio of 0.5 (looks severe)
    but a difference of 0.01 (looks negligible), and which reading matters
    depends on the decision being made. Report both.

    This is a **diagnostic, not a fairness guarantee or legal clearance**: a
    small difference does not certify a model as non-discriminatory, and the
    choice of protected groups and decision threshold materially affects the
    result.

    Parameters
    ----------
    y_pred : array-like of shape (n_samples,)
        Binary decisions. Threshold continuous scores first.
    sensitive_features : array-like of shape (n_samples,)
        Protected-group label per sample.
    pos_label : default=1
        Value in ``y_pred`` that counts as "selected".

    Returns
    -------
    float
        Difference in ``[0.0, 1.0]``. ``0.0`` is exact parity, or when only
        one group is present.

    Raises
    ------
    ValueError
        If inputs have mismatched lengths, are empty, or contain missing values.
    """
    rates = selection_rate_by_group(y_pred, sensitive_features, pos_label=pos_label)
    values = np.array(list(rates.values()), dtype=float)
    return float(values.max() - values.min())

disparate_impact_ratio(y_pred, sensitive_features, pos_label=1)

Four-fifths-rule disparate-impact ratio across protected groups.

Computes min(selection_rate) / max(selection_rate) over the groups in sensitive_features. A value of 1.0 is exact parity; the US EEOC "four-fifths rule" flags a ratio below 0.8 as evidence of adverse impact that warrants investigation.

This is a diagnostic, not a fairness guarantee or legal clearance: a passing ratio does not certify a model as non-discriminatory, and the choice of protected groups and decision threshold materially affects the result.

Parameters:

Name Type Description Default
y_pred array-like of shape (n_samples,)

Binary decisions. Threshold continuous scores first.

 required
sensitive_features array-like of shape (n_samples,)

Protected-group label per sample.

 required
pos_label default=1

Value in y_pred that counts as "selected".

 1

Returns:

Type Description
float

Ratio in [0.0, 1.0]. Returns 1.0 when only one group is present or when no sample in any group is selected (no disparity to measure).

Raises:

Type Description
ValueError

If inputs have mismatched lengths, are empty, or contain missing values.
Source code in philanthropy/metrics/_fairness.py 
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
def disparate_impact_ratio(
    y_pred: Collection,
    sensitive_features: Collection,
    pos_label: Any = 1,
) -> float:
    """Four-fifths-rule disparate-impact ratio across protected groups.

    Computes ``min(selection_rate) / max(selection_rate)`` over the groups in
    ``sensitive_features``. A value of ``1.0`` is exact parity; the US EEOC
    "four-fifths rule" flags a ratio below ``0.8`` as evidence of adverse impact
    that warrants investigation.

    This is a **diagnostic, not a fairness guarantee or legal clearance**: a
    passing ratio does not certify a model as non-discriminatory, and the choice
    of protected groups and decision threshold materially affects the result.

    Parameters
    ----------
    y_pred : array-like of shape (n_samples,)
        Binary decisions. Threshold continuous scores first.
    sensitive_features : array-like of shape (n_samples,)
        Protected-group label per sample.
    pos_label : default=1
        Value in ``y_pred`` that counts as "selected".

    Returns
    -------
    float
        Ratio in ``[0.0, 1.0]``. Returns ``1.0`` when only one group is present
        or when no sample in any group is selected (no disparity to measure).

    Raises
    ------
    ValueError
        If inputs have mismatched lengths, are empty, or contain missing values.
    """
    rates = selection_rate_by_group(y_pred, sensitive_features, pos_label=pos_label)
    values = np.array(list(rates.values()), dtype=float)
    max_rate = values.max()
    if max_rate == 0.0:
        return 1.0
    return float(values.min() / max_rate)

metric_by_group(metric, y_true, y_pred, groups)

Any metric(y_true, y_pred) computed within each group and overall.

:func:selection_rate_by_group only answers "who gets flagged". This answers "how well does the model work for each group", for whatever metric matters to the decision, e.g. roc_auc_score on scores or recall_score on thresholded decisions (a group where the model misses most real donors is under-served even at equal selection rates). Pass extra metric arguments with :func:functools.partial.

Like the other functions here, this is a diagnostic, not a fairness guarantee. Small groups give noisy values; read count first.

Parameters:

Name Type Description Default
metric callable

metric(y_true, y_pred) -> float, e.g. any sklearn metric.

 required
y_true array-like of shape (n_samples,)

Observed outcome.

 required
y_pred array-like of shape (n_samples,)

Prediction or score, as metric expects it.

 required
groups array-like of shape (n_samples,)

Group label per sample (e.g. race, age band, region).

 required

Returns:

Type Description
DataFrame

Indexed by group, with a final "overall" row computed on all samples. Columns metric (float) and count (int). metric is NaN where it is undefined for a group, e.g. roc_auc_score on a group with only one class (a ValueError from metric is caught and reported as NaN).

Raises:

Type Description
ValueError

If inputs have mismatched lengths, are empty, contain missing group labels, or a group is itself labelled "overall".
Source code in philanthropy/metrics/_fairness.py 
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
def metric_by_group(
    metric: Callable[..., float],
    y_true: Collection,
    y_pred: Collection,
    groups: Collection,
) -> pd.DataFrame:
    """Any ``metric(y_true, y_pred)`` computed within each group and overall.

    :func:`selection_rate_by_group` only answers "who gets flagged". This
    answers "how well does the model work for each group", for whatever
    metric matters to the decision, e.g. ``roc_auc_score`` on scores or
    ``recall_score`` on thresholded decisions (a group where the model misses
    most real donors is under-served even at equal selection rates). Pass
    extra metric arguments with :func:`functools.partial`.

    Like the other functions here, this is a **diagnostic, not a fairness
    guarantee**. Small groups give noisy values; read ``count`` first.

    Parameters
    ----------
    metric : callable
        ``metric(y_true, y_pred) -> float``, e.g. any sklearn metric.
    y_true : array-like of shape (n_samples,)
        Observed outcome.
    y_pred : array-like of shape (n_samples,)
        Prediction or score, as ``metric`` expects it.
    groups : array-like of shape (n_samples,)
        Group label per sample (e.g. race, age band, region).

    Returns
    -------
    pandas.DataFrame
        Indexed by group, with a final ``"overall"`` row computed on all
        samples. Columns ``metric`` (float) and ``count`` (int). ``metric``
        is NaN where it is undefined for a group, e.g. ``roc_auc_score`` on
        a group with only one class (a ``ValueError`` from ``metric`` is
        caught and reported as NaN).

    Raises
    ------
    ValueError
        If inputs have mismatched lengths, are empty, contain missing group
        labels, or a group is itself labelled ``"overall"``.
    """
    y_true = np.asarray(y_true)
    y_pred = np.asarray(y_pred)
    groups = np.asarray(groups)
    if not (y_true.shape[0] == y_pred.shape[0] == groups.shape[0]):
        raise ValueError(
            f"y_true, y_pred and groups must be the same length, got "
            f"{y_true.shape[0]}, {y_pred.shape[0]} and {groups.shape[0]}."
        )
    if y_true.shape[0] == 0:
        raise ValueError("y_true is empty; nothing to score.")
    if pd.isna(groups).any():
        raise ValueError(
            "groups contains missing (NaN/None) labels; drop or impute those "
            "rows before computing per-group metrics."
        )
    labels = np.unique(groups)
    if "overall" in labels.tolist():
        raise ValueError(
            "A group is labelled 'overall', which collides with the overall "
            "row; rename it first."
        )

    def _score(mask: np.ndarray) -> float:
        try:
            return float(metric(y_true[mask], y_pred[mask]))
        except ValueError:
            return float("nan")

    rows = {g: (_score(groups == g), int((groups == g).sum())) for g in labels}
    everyone = np.ones(groups.shape[0], dtype=bool)
    rows["overall"] = (_score(everyone), int(groups.shape[0]))
    return pd.DataFrame.from_dict(rows, orient="index", columns=["metric", "count"])

selection_rate_by_group(y_pred, sensitive_features, pos_label=1)

Fraction selected (y_pred == pos_label) within each protected group.

Parameters:

Name Type Description Default
y_pred array-like of shape (n_samples,)

Binary decisions (e.g. "flagged for major-gift outreach"). Threshold continuous scores before calling this.

 required
sensitive_features array-like of shape (n_samples,)

Protected-group label per sample (e.g. race, age band, gender).

 required
pos_label default=1

Value in y_pred that counts as "selected".

 1

Returns:

Type Description
dict

Mapping of group value -> selection rate in [0.0, 1.0].

Raises:

Type Description
ValueError

If inputs have mismatched lengths, are empty, or contain missing values.
Source code in philanthropy/metrics/_fairness.py 
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
def selection_rate_by_group(
    y_pred: Collection,
    sensitive_features: Collection,
    pos_label: Any = 1,
) -> Dict[object, float]:
    """Fraction selected (``y_pred == pos_label``) within each protected group.

    Parameters
    ----------
    y_pred : array-like of shape (n_samples,)
        Binary decisions (e.g. "flagged for major-gift outreach"). Threshold
        continuous scores before calling this.
    sensitive_features : array-like of shape (n_samples,)
        Protected-group label per sample (e.g. race, age band, gender).
    pos_label : default=1
        Value in ``y_pred`` that counts as "selected".

    Returns
    -------
    dict
        Mapping of group value -> selection rate in ``[0.0, 1.0]``.

    Raises
    ------
    ValueError
        If inputs have mismatched lengths, are empty, or contain missing values.
    """
    y_pred = np.asarray(y_pred)
    groups = np.asarray(sensitive_features)
    if y_pred.shape[0] != groups.shape[0]:
        raise ValueError(
            f"y_pred and sensitive_features must be the same length, got "
            f"{y_pred.shape[0]} and {groups.shape[0]}."
        )
    if y_pred.shape[0] == 0:
        raise ValueError("y_pred is empty; nothing to score.")
    if pd.isna(groups).any():
        raise ValueError(
            "sensitive_features contains missing (NaN/None) group labels; a "
            "missing label would silently NaN-out the diagnostic. Drop or "
            "impute those rows before computing fairness metrics."
        )

    selected = y_pred == pos_label
    return {g: float(selected[groups == g].mean()) for g in np.unique(groups)}

qini_auc_score(y_true, uplift, treatment)

Area between the Qini curve and the random-targeting line.

The random line runs straight from (0, 0) to the curve's end point (1, gains[-1]): what targeting in random order would earn on average. The score is the trapezoidal area under :func:qini_curve minus the area under that line.

The score is not normalised (scikit-uplift's qini_auc_score divides by the perfect model's area by default; this does not). Its unit is incremental responses (or dollars, for amount outcomes) times fraction targeted, so it scales with sample size and outcome scale. Use it to compare models scored on the same holdout, not across datasets.

Positive means the ranking beats random targeting, about 0 means no better than random, negative means worse than random.

Parameters:

Name Type Description Default
y_true Collection

As for :func:qini_curve.

 required
uplift Collection

As for :func:qini_curve.

 required
treatment Collection

As for :func:qini_curve.

 required

Returns:

Type Description
float

Area between the model's Qini curve and the random line.

Raises:

Type Description
ValueError

As for :func:qini_curve.
Source code in philanthropy/metrics/_uplift.py 
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
def qini_auc_score(
    y_true: Collection,
    uplift: Collection,
    treatment: Collection,
) -> float:
    """Area between the Qini curve and the random-targeting line.

    The random line runs straight from ``(0, 0)`` to the curve's end point
    ``(1, gains[-1])``: what targeting in random order would earn on
    average. The score is the trapezoidal area under :func:`qini_curve`
    minus the area under that line.

    The score is **not normalised** (``scikit-uplift``'s ``qini_auc_score``
    divides by the perfect model's area by default; this does not). Its unit
    is incremental responses (or dollars, for amount outcomes) times
    fraction targeted, so it scales with sample size and outcome scale. Use
    it to compare models scored on the same holdout, not across datasets.

    Positive means the ranking beats random targeting, about 0 means no
    better than random, negative means worse than random.

    Parameters
    ----------
    y_true, uplift, treatment
        As for :func:`qini_curve`.

    Returns
    -------
    float
        Area between the model's Qini curve and the random line.

    Raises
    ------
    ValueError
        As for :func:`qini_curve`.
    """
    x, gains = qini_curve(y_true, uplift, treatment)
    area = float(np.sum((x[1:] - x[:-1]) * (gains[1:] + gains[:-1]) / 2.0))
    return area - float(gains[-1]) / 2.0

qini_curve(y_true, uplift, treatment)

Qini curve: cumulative incremental responses by fraction targeted.

Donors are sorted by uplift, highest first. At each cut-off holding the top k donors, the incremental gain is (Radcliffe 2007, and the definition scikit-uplift uses)::

Y_t(k) - Y_c(k) * N_t(k) / N_c(k)

where Y_t / Y_c are the summed outcomes of the treated / control donors in the top k, and N_t / N_c their counts. It reads as "extra responders (or dollars) the appeal produced among the donors targeted so far", with the control response rescaled to the treated count. While no control donor has been reached the control term is 0.

Tied scores form one cut-off, so the curve does not depend on how ties happen to be ordered.

Parameters:

Name Type Description Default
y_true array-like of shape (n_samples,)

Observed outcome: 1/0 for gave, or a gift amount.

 required
uplift array-like of shape (n_samples,)

Predicted uplift score; higher means target first.

 required
treatment array-like of shape (n_samples,)

1 if the donor was solicited, 0 if held out. Assignment must have been random.

 required

Returns:

Name Type Description
fraction_targeted ndarray

Fraction of donors targeted at each cut-off, starting at 0.0 and ending at 1.0.
gains ndarray

Cumulative incremental gain at each cut-off, starting at 0.0.

Raises:

Type Description
ValueError

If inputs are empty, of mismatched length, non-finite, if treatment is not binary 0/1, or if either arm is empty.
Source code in philanthropy/metrics/_uplift.py 
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
def qini_curve(
    y_true: Collection,
    uplift: Collection,
    treatment: Collection,
) -> Tuple[np.ndarray, np.ndarray]:
    """Qini curve: cumulative incremental responses by fraction targeted.

    Donors are sorted by ``uplift``, highest first. At each cut-off holding
    the top ``k`` donors, the incremental gain is (Radcliffe 2007, and the
    definition ``scikit-uplift`` uses)::

        Y_t(k) - Y_c(k) * N_t(k) / N_c(k)

    where ``Y_t`` / ``Y_c`` are the summed outcomes of the treated / control
    donors in the top ``k``, and ``N_t`` / ``N_c`` their counts. It reads as
    "extra responders (or dollars) the appeal produced among the donors
    targeted so far", with the control response rescaled to the treated
    count. While no control donor has been reached the control term is 0.

    Tied scores form one cut-off, so the curve does not depend on how ties
    happen to be ordered.

    Parameters
    ----------
    y_true : array-like of shape (n_samples,)
        Observed outcome: 1/0 for gave, or a gift amount.
    uplift : array-like of shape (n_samples,)
        Predicted uplift score; higher means target first.
    treatment : array-like of shape (n_samples,)
        1 if the donor was solicited, 0 if held out. Assignment must have
        been random.

    Returns
    -------
    fraction_targeted : ndarray
        Fraction of donors targeted at each cut-off, starting at 0.0 and
        ending at 1.0.
    gains : ndarray
        Cumulative incremental gain at each cut-off, starting at 0.0.

    Raises
    ------
    ValueError
        If inputs are empty, of mismatched length, non-finite, if
        ``treatment`` is not binary 0/1, or if either arm is empty.
    """
    y = np.asarray(y_true, dtype=float)
    score = np.asarray(uplift, dtype=float)
    t = np.asarray(treatment)
    if not (y.shape[0] == score.shape[0] == t.shape[0]):
        raise ValueError(
            f"y_true, uplift and treatment must be the same length, got "
            f"{y.shape[0]}, {score.shape[0]} and {t.shape[0]}."
        )
    if y.shape[0] == 0:
        raise ValueError("y_true is empty; nothing to score.")
    if not (np.isfinite(y).all() and np.isfinite(score).all()):
        raise ValueError("y_true and uplift must be finite (no NaN or inf).")
    if not np.isin(t, [0, 1]).all():
        raise ValueError("treatment must be binary 0/1.")
    t = t.astype(float)
    if t.sum() == 0 or t.sum() == t.shape[0]:
        raise ValueError(
            "Both arms must be present: treatment needs at least one 0 and "
            "one 1."
        )

    order = np.argsort(score, kind="mergesort")[::-1]
    y, score, t = y[order], score[order], t[order]

    n_t = np.cumsum(t)
    n_c = np.cumsum(1.0 - t)
    y_t = np.cumsum(y * t)
    y_c = np.cumsum(y * (1.0 - t))

    # Last index of each run of tied scores, plus the final row.
    cut = np.r_[np.flatnonzero(np.diff(score)), score.shape[0] - 1]
    ratio = np.divide(
        n_t[cut], n_c[cut], out=np.zeros(cut.shape[0]), where=n_c[cut] > 0
    )
    gains = y_t[cut] - y_c[cut] * ratio

    fraction = (cut + 1) / score.shape[0]
    return np.r_[0.0, fraction], np.r_[0.0, gains]

gift_concentration_gini(amounts)

Gini coefficient of a set of donor gift amounts.

0.0 is perfect equality (every donor gives the same); values approaching 1.0 mean revenue is concentrated in a few donors.

Parameters:

Name Type Description Default
amounts array-like of shape (n_donors,)

Per-donor total giving. NaN entries are dropped; negative or infinite values raise.

 required

Returns:

Type Description
float

Gini coefficient in [0.0, 1.0]. Returns 0.0 for an empty input or an all-zero total (no distribution to measure).

Raises:

Type Description
ValueError

If any gift amounts are negative or infinite.
Source code in philanthropy/metrics/_concentration.py 
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
def gift_concentration_gini(amounts: Collection) -> float:
    """Gini coefficient of a set of donor gift amounts.

    ``0.0`` is perfect equality (every donor gives the same); values approaching
    ``1.0`` mean revenue is concentrated in a few donors.

    Parameters
    ----------
    amounts : array-like of shape (n_donors,)
        Per-donor total giving. ``NaN`` entries are dropped; negative or
        infinite values raise.

    Returns
    -------
    float
        Gini coefficient in ``[0.0, 1.0]``. Returns ``0.0`` for an empty input
        or an all-zero total (no distribution to measure).

    Raises
    ------
    ValueError
        If any gift amounts are negative or infinite.
    """
    a = _clean_nonneg_amounts(amounts)
    total = a.sum()
    if a.size == 0 or total == 0:
        return 0.0
    a = np.sort(a)
    n = a.size
    index = np.arange(1, n + 1)
    # Mean-absolute-difference form of the Gini coefficient.
    return float((2.0 * np.sum(index * a)) / (n * total) - (n + 1.0) / n)

top_donor_share(amounts, top_fraction=0.1)

Fraction of total revenue contributed by the top top_fraction donors.

Parameters:

Name Type Description Default
amounts array-like of shape (n_donors,)

Per-donor total giving. NaN entries are dropped; negative or infinite values raise.

 required
top_fraction float

Slice of donors (ranked by giving, descending) to sum. Must be in (0.0, 1.0]. At least one donor is always counted.

 0.1

Returns:

Type Description
float

Share in [0.0, 1.0]. Returns 0.0 for an empty input or an all-zero total.

Raises:

Type Description
ValueError

If top_fraction is not in (0.0, 1.0], or if any gift amounts are negative or infinite.
Source code in philanthropy/metrics/_concentration.py 
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
def top_donor_share(amounts: Collection, top_fraction: float = 0.1) -> float:
    """Fraction of total revenue contributed by the top ``top_fraction`` donors.

    Parameters
    ----------
    amounts : array-like of shape (n_donors,)
        Per-donor total giving. ``NaN`` entries are dropped; negative or
        infinite values raise.
    top_fraction : float, default=0.1
        Slice of donors (ranked by giving, descending) to sum. Must be in
        ``(0.0, 1.0]``. At least one donor is always counted.

    Returns
    -------
    float
        Share in ``[0.0, 1.0]``. Returns ``0.0`` for an empty input or an
        all-zero total.

    Raises
    ------
    ValueError
        If ``top_fraction`` is not in ``(0.0, 1.0]``, or if any gift amounts
        are negative or infinite.
    """
    if not 0.0 < top_fraction <= 1.0:
        raise ValueError("top_fraction must be in (0.0, 1.0].")
    a = _clean_nonneg_amounts(amounts)
    total = a.sum()
    if a.size == 0 or total == 0:
        return 0.0
    a = np.sort(a)[::-1]
    k = max(1, int(np.ceil(top_fraction * a.size)))
    return float(a[:k].sum() / total)

conformal_pvalue(calibration_scores, scores)

Split-conformal p-value of each score against a calibration set.

Small p-values mean the score is high relative to the calibration donors, so conformal_pvalue(...) <= alpha selects the donors whose scores are extreme at level alpha, and the expected selection rate among exchangeable donors is at most alpha. The calibration scores must come from donors held out of training, exchangeable with the ones being scored; reusing training rows breaks the guarantee exactly the way refitting a transformer on test data does. To read alpha as a false-positive rate the calibration set must contain only donors who did not give; see the module docstring.

Parameters:

Name Type Description Default
calibration_scores array-like of shape (n_calibration,)

Scores of held-out donors, higher meaning more likely to give. Must be non-empty and finite; NaN and infinities raise rather than being dropped, because the denominator n + 1 counts them.

 required
scores array-like of shape (n_samples,)

Scores to test. May be a scalar-like sequence of any length, including empty. NaN entries yield NaN p-values.

 required

Returns:

Type Description
ndarray of shape (n_samples,)

P-values in [1 / (n_calibration + 1), 1.0]. Never 0, never above 1.

Raises:

Type Description
ValueError

If calibration_scores is empty, not one-dimensional, or contains non-finite values.

Examples:

>>> import numpy as np
>>> from philanthropy.metrics import conformal_pvalue
>>> calibration = np.arange(9, dtype=float)          # 0 .. 8, n = 9
>>> conformal_pvalue(calibration, [8.5, 4.0, -1.0])
array([0.1, 0.6, 1. ])

A score above every calibration point still gets 1 / (n + 1), not 0:

>>> float(conformal_pvalue(calibration, [1e6])[0])
0.1
Source code in philanthropy/metrics/_conformal.py 
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
def conformal_pvalue(calibration_scores: Collection, scores: Collection) -> np.ndarray:
    """Split-conformal p-value of each score against a calibration set.

    Small p-values mean the score is high relative to the calibration donors,
    so ``conformal_pvalue(...) <= alpha`` selects the donors whose scores are
    extreme at level ``alpha``, and the expected selection rate among
    exchangeable donors is at most ``alpha``. The calibration scores must come
    from donors held out of training, exchangeable with the ones being scored;
    reusing training rows breaks the guarantee exactly the way refitting a
    transformer on test data does. To read ``alpha`` as a false-positive rate
    the calibration set must contain only donors who did not give; see the
    module docstring.

    Parameters
    ----------
    calibration_scores : array-like of shape (n_calibration,)
        Scores of held-out donors, higher meaning more likely to give. Must be
        non-empty and finite; ``NaN`` and infinities raise rather than being
        dropped, because the denominator ``n + 1`` counts them.
    scores : array-like of shape (n_samples,)
        Scores to test. May be a scalar-like sequence of any length, including
        empty. ``NaN`` entries yield ``NaN`` p-values.

    Returns
    -------
    ndarray of shape (n_samples,)
        P-values in ``[1 / (n_calibration + 1), 1.0]``. Never 0, never above 1.

    Raises
    ------
    ValueError
        If ``calibration_scores`` is empty, not one-dimensional, or contains
        non-finite values.

    Examples
    --------
    >>> import numpy as np
    >>> from philanthropy.metrics import conformal_pvalue
    >>> calibration = np.arange(9, dtype=float)          # 0 .. 8, n = 9
    >>> conformal_pvalue(calibration, [8.5, 4.0, -1.0])
    array([0.1, 0.6, 1. ])

    A score above every calibration point still gets ``1 / (n + 1)``, not 0:

    >>> float(conformal_pvalue(calibration, [1e6])[0])
    0.1
    """
    cal = np.asarray(calibration_scores, dtype=float)
    if cal.ndim != 1:
        raise ValueError("calibration_scores must be one-dimensional.")
    if cal.size == 0:
        raise ValueError("calibration_scores must be non-empty.")
    if not np.all(np.isfinite(cal)):
        raise ValueError("calibration_scores must be finite (no NaN or inf).")

    s = np.asarray(scores, dtype=float)
    n = cal.size
    cal_sorted = np.sort(cal)
    # |{i : cal_i >= s}|; 'left' so calibration points equal to s are counted.
    n_ge = n - np.searchsorted(cal_sorted, s, side="left")
    p = (1.0 + n_ge) / (n + 1.0)
    return np.where(np.isnan(s), np.nan, p)

interval_report(y_true, lower, upper, alpha=0.05, trim=0.1)

Coverage, interval score and width-to-target for a set of intervals.

Coverage alone cannot tell a useful interval from [0, inf). This returns the three things that can: the proper score aggregated robustly, the width relative to the amounts being predicted, and the ratio between the bounds.

Parameters:

Name Type Description Default
y_true array-like of shape (n_samples,)

Realised amounts.

 required
lower array-like of shape (n_samples,)

The interval bounds.

 required
upper array-like of shape (n_samples,)

The interval bounds.

 required
alpha float

Miscoverage the intervals were built for; sets both the score's penalty weight and requested_level.

 0.05
trim float

Fraction of rows dropped from each end before score_trimmed_mean. Must be in [0, 0.5).

 0.1

Returns:

Type Description
IntervalReport

Raises:

Type Description
ValueError

On the same input problems as :func:interval_score, or trim outside [0, 0.5).

Examples:

>>> from philanthropy.metrics import interval_report
>>> report = interval_report([10.0, 30.0], [5.0, 5.0], [20.0, 20.0], alpha=0.5)
>>> report.coverage, report.median_width, report.median_target
(0.5, 15.0, 20.0)
>>> report.width_ratio, report.median_bound_ratio
(0.75, 4.0)

The score is heavy-tailed on gift amounts, so read all three:

>>> report.score_mean, report.score_median
(35.0, 35.0)
Source code in philanthropy/metrics/_conformal.py 
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
def interval_report(
    y_true: Collection,
    lower: Collection,
    upper: Collection,
    alpha: float = 0.05,
    trim: float = 0.1,
) -> IntervalReport:
    """Coverage, interval score and width-to-target for a set of intervals.

    Coverage alone cannot tell a useful interval from ``[0, inf)``. This returns
    the three things that can: the proper score aggregated robustly, the width
    relative to the amounts being predicted, and the ratio between the bounds.

    Parameters
    ----------
    y_true : array-like of shape (n_samples,)
        Realised amounts.
    lower, upper : array-like of shape (n_samples,)
        The interval bounds.
    alpha : float, default=0.05
        Miscoverage the intervals were built for; sets both the score's penalty
        weight and ``requested_level``.
    trim : float, default=0.1
        Fraction of rows dropped from **each** end before ``score_trimmed_mean``.
        Must be in ``[0, 0.5)``.

    Returns
    -------
    IntervalReport

    Raises
    ------
    ValueError
        On the same input problems as :func:`interval_score`, or ``trim``
        outside ``[0, 0.5)``.

    Examples
    --------
    >>> from philanthropy.metrics import interval_report
    >>> report = interval_report([10.0, 30.0], [5.0, 5.0], [20.0, 20.0], alpha=0.5)
    >>> report.coverage, report.median_width, report.median_target
    (0.5, 15.0, 20.0)
    >>> report.width_ratio, report.median_bound_ratio
    (0.75, 4.0)

    The score is heavy-tailed on gift amounts, so read all three:

    >>> report.score_mean, report.score_median
    (35.0, 35.0)
    """
    y, lo, hi, a = _interval_inputs(y_true, lower, upper, alpha)
    if not 0 <= trim < 0.5:
        raise ValueError(f"trim must satisfy 0 <= trim < 0.5, got {trim!r}.")

    per_row = _interval_score_per_row(y, lo, hi, a)
    width = hi - lo
    median_width = float(np.median(width))
    median_target = float(np.median(y))
    positive = lo > 0

    return IntervalReport(
        n=int(y.size),
        coverage=float(np.mean((y >= lo) & (y <= hi))),
        requested_level=1.0 - a,
        score_mean=float(np.mean(per_row)),
        score_median=float(np.median(per_row)),
        score_trimmed_mean=_trimmed_mean(per_row, trim),
        median_width=median_width,
        median_target=median_target,
        width_ratio=(
            median_width / median_target
            if median_target != 0
            else (np.inf if median_width > 0 else np.nan)
        ),
        median_bound_ratio=(
            float(np.median(hi[positive] / lo[positive]))
            if positive.any()
            else np.nan
        ),
    )

interval_score(y_true, lower, upper, alpha=0.05)

Mean interval score of a central 1 - alpha interval.

(u - l) + (2/alpha)(l - y)+ + (2/alpha)(y - u)+, averaged. Lower is better. Proper for a central interval, so widening to buy coverage costs more than it gains and the score cannot be gamed.

Parameters:

Name Type Description Default
y_true array-like of shape (n_samples,)

Realised amounts.

 required
lower array-like of shape (n_samples,)

The interval bounds.

 required
upper array-like of shape (n_samples,)

The interval bounds.

 required
alpha float

The miscoverage the interval was built for. This is the score's penalty weight, not a level to be attained, so the caller's requested alpha is the right value here even when the attained level differs.

 0.05

Returns:

Type Description
float

The mean score, in the target's units.

Raises:

Type Description
ValueError

If the three arrays disagree in shape, are empty, alpha is outside (0, 1), or any upper is below its lower.

Examples:

A covered row costs its width; a missed row adds 2 / alpha per dollar of miss:

>>> from philanthropy.metrics import interval_score
>>> interval_score([10.0, 30.0], [5.0, 5.0], [20.0, 20.0], alpha=0.5)
35.0
Source code in philanthropy/metrics/_conformal.py 
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
def interval_score(
    y_true: Collection,
    lower: Collection,
    upper: Collection,
    alpha: float = 0.05,
) -> float:
    """Mean interval score of a central ``1 - alpha`` interval.

    ``(u - l) + (2/alpha)(l - y)+ + (2/alpha)(y - u)+``, averaged. Lower is
    better. Proper for a central interval, so widening to buy coverage costs
    more than it gains and the score cannot be gamed.

    Parameters
    ----------
    y_true : array-like of shape (n_samples,)
        Realised amounts.
    lower, upper : array-like of shape (n_samples,)
        The interval bounds.
    alpha : float, default=0.05
        The miscoverage the interval was built for. This is the score's penalty
        weight, not a level to be attained, so the caller's requested ``alpha``
        is the right value here even when the attained level differs.

    Returns
    -------
    float
        The mean score, in the target's units.

    Raises
    ------
    ValueError
        If the three arrays disagree in shape, are empty, ``alpha`` is outside
        ``(0, 1)``, or any ``upper`` is below its ``lower``.

    Examples
    --------
    A covered row costs its width; a missed row adds ``2 / alpha`` per dollar
    of miss:

    >>> from philanthropy.metrics import interval_score
    >>> interval_score([10.0, 30.0], [5.0, 5.0], [20.0, 20.0], alpha=0.5)
    35.0
    """
    y, lo, hi, a = _interval_inputs(y_true, lower, upper, alpha)
    return float(np.mean(_interval_score_per_row(y, lo, hi, a)))