Skip to content

Datasets Reference

philanthropy.datasets

Synthetic data generators and real reference datasets for donor analytics.

load_ciob_fundraising()

Load the NYC CIOB "Official Fundraising by City Agencies" registry.

A real open-government dataset: every not-for-profit that a New York City agency reported soliciting donations for, disclosed under the Conflicts of Interest Board's (CIOB) mandate. It is an affiliation registry, one row per (year, agency, nonprofit) link, not donor-level giving data.

It carries no gift amounts, donor records, or engagement labels, so it does not support the RFM / propensity modelling in the rest of this library; use :func:generate_synthetic_donor_data for that. It is included for honest, reproducible analysis of agency ↔ nonprofit fundraising relationships (e.g. most-solicited nonprofits, agency fundraising breadth).

Returns:

Type Description
pandas.DataFrame of shape (2336, 3)

Columns:

year : int Calendar year of the reported fundraising (2019–2024). agency : string NYC agency that solicited the donation. name_of_not_for_profit : string Beneficiary not-for-profit organisation.
Notes

Source: NYC Open Data, dataset basd-2jwn (Conflicts of Interest Board), https://data.cityofnewyork.us/City-Government/Official-Fundraising-by-City-Agencies/basd-2jwn. Distributed under the NYC Open Data Terms of Use (free public use). The CSV is vendored with the package, so this loader needs no network access.

Source code in philanthropy/datasets/_ciob.py 
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
def load_ciob_fundraising() -> pd.DataFrame:
    """Load the NYC CIOB "Official Fundraising by City Agencies" registry.

    A real open-government dataset: every not-for-profit that a New York City
    agency reported soliciting donations for, disclosed under the Conflicts of
    Interest Board's (CIOB) mandate. It is an **affiliation registry**, one row
    per ``(year, agency, nonprofit)`` link, **not** donor-level giving data.

    It carries no gift amounts, donor records, or engagement labels, so it does
    **not** support the RFM / propensity modelling in the rest of this library;
    use :func:`generate_synthetic_donor_data` for that. It is included for
    honest, reproducible analysis of agency ↔ nonprofit fundraising
    relationships (e.g. most-solicited nonprofits, agency fundraising breadth).

    Returns
    -------
    pandas.DataFrame of shape (2336, 3)
        Columns:

        ``year`` : int
            Calendar year of the reported fundraising (2019–2024).
        ``agency`` : string
            NYC agency that solicited the donation.
        ``name_of_not_for_profit`` : string
            Beneficiary not-for-profit organisation.

    Notes
    -----
    Source: NYC Open Data, dataset ``basd-2jwn`` (Conflicts of Interest Board),
    https://data.cityofnewyork.us/City-Government/Official-Fundraising-by-City-Agencies/basd-2jwn.
    Distributed under the NYC Open Data Terms of Use (free public use). The CSV
    is vendored with the package, so this loader needs no network access.
    """
    csv = files("philanthropy.datasets").joinpath(
        "data/ciob_official_fundraising.csv"
    )
    with csv.open("r", encoding="utf-8") as fh:
        return pd.read_csv(
            fh,
            dtype={
                "year": "int64",
                "agency": "string",
                "name_of_not_for_profit": "string",
            },
        )

load_donorschoose(path)

Load a user-downloaded DonorsChoose Donations file (ICPSR 37898, DS0001).

DonorsChoose Open Data, United States (ICPSR 37898, doi:10.3886/ICPSR37898.v1) is distributed under its own terms of use (TermsOfUse.html in the study download): research or statistical purposes only, no investigation of specific research subjects, and no redistribution without ICPSR's written agreement. This reader never downloads, caches, or redistributes the file: point it at the delimited 37898-0001-Data.tsv (or .dta) you have already obtained yourself under your own ICPSR account. Never commit rows read by this function, or anything derived at the row level, to this repository or its docs; aggregate statistics only.

CREATED_MONTH is recorded at month resolution (YYYY-MM), not a full date; this function parses it to the first day of that month, so any date-based feature computed from the result (recency, fiscal-year boundaries, ...) inherits that one-month granularity rather than the daily resolution the synthetic and KDD98 data provide.

This reader applies no filtering: AMOUNT rows at or below zero exist in the source data and are passed through unchanged, every donor_type ("citizen donor", "organization", "teacher") is returned, and a donation's own payment-source flags are returned as booleans rather than acted on. Decide and document your own filters at the point you use this data (e.g. the benchmark script), rather than relying on this function to have applied them.

Parameters:

Name Type Description Default
path str

Path to the DS0001 Donations file, tab-separated (.tsv) or Stata (.dta, assumed to share the same column names as the .tsv).

 required

Returns:

Type Description
DataFrame

Columns donor_id (string), gift_date (datetime64, first of the donation's month), gift_amount (float64), donor_type (string), included_campaign_gift_card, included_web_purchased_gift_card, was_matched (bool): whether the donation's payment included a campaign-supplied gift card, a web-purchased gift card, or the payment was matched (codebook: "Payment matched"; matching source not specified).
Notes

Source: DonorsChoose Open Data, United States, 2002-2019 (ICPSR 37898), https://www.icpsr.umich.edu/web/ICPSR/studies/37898. Cite as doi:10.3886/ICPSR37898.v1.

Source code in philanthropy/datasets/_donorschoose.py 
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
def load_donorschoose(path: str) -> pd.DataFrame:
    """Load a user-downloaded DonorsChoose Donations file (ICPSR 37898, DS0001).

    DonorsChoose Open Data, United States (ICPSR 37898,
    doi:10.3886/ICPSR37898.v1) is distributed under its own terms of use
    (``TermsOfUse.html`` in the study download): research or statistical
    purposes only, no investigation of specific research subjects, and no
    redistribution without ICPSR's written agreement. This reader never
    downloads, caches, or redistributes the file: point it at the delimited
    ``37898-0001-Data.tsv`` (or ``.dta``) you have already obtained yourself
    under your own ICPSR account. Never commit rows read by this function,
    or anything derived at the row level, to this repository or its docs;
    aggregate statistics only.

    ``CREATED_MONTH`` is recorded at **month resolution** (``YYYY-MM``), not
    a full date; this function parses it to the first day of that month, so
    any date-based feature computed from the result (recency, fiscal-year
    boundaries, ...) inherits that one-month granularity rather than the
    daily resolution the synthetic and KDD98 data provide.

    This reader applies no filtering: ``AMOUNT`` rows at or below zero exist
    in the source data and are passed through unchanged, every ``donor_type``
    ("citizen donor", "organization", "teacher") is returned, and a
    donation's own payment-source flags are returned as booleans rather than
    acted on. Decide and document your own filters at the point you use this
    data (e.g. the benchmark script), rather than relying on this function to
    have applied them.

    Parameters
    ----------
    path : str
        Path to the DS0001 Donations file, tab-separated (``.tsv``) or Stata
        (``.dta``, assumed to share the same column names as the ``.tsv``).

    Returns
    -------
    pandas.DataFrame
        Columns ``donor_id`` (string), ``gift_date`` (datetime64, first of
        the donation's month), ``gift_amount`` (float64), ``donor_type``
        (string), ``included_campaign_gift_card``,
        ``included_web_purchased_gift_card``, ``was_matched`` (bool):
        whether the donation's payment included a campaign-supplied gift
        card, a web-purchased gift card, or the payment was matched
        (codebook: "Payment matched"; matching source not specified).

    Notes
    -----
    Source: DonorsChoose Open Data, United States, 2002-2019 (ICPSR 37898),
    https://www.icpsr.umich.edu/web/ICPSR/studies/37898. Cite as
    doi:10.3886/ICPSR37898.v1.
    """
    if path.endswith(".dta"):
        raw = pd.read_stata(path, columns=_COLUMNS)
    else:
        raw = pd.read_csv(
            path, sep="\t", usecols=_COLUMNS, dtype={"DONOR_ID": "string"}
        )

    gifts = pd.DataFrame(
        {
            "donor_id": raw["DONOR_ID"],
            "gift_date": pd.to_datetime(raw["CREATED_MONTH"].str.strip() + "-01"),
            "gift_amount": raw["AMOUNT"].astype(float),
            "donor_type": raw["DONOR_TYPE"].str.strip(),
        }
    )
    for source, name in _YES_NO_COLUMNS.items():
        gifts[name] = raw[source].str.strip() == "Yes"
    return gifts

generate_synthetic_donor_data(n_samples=1000, random_state=None)

Generate a realistic synthetic donor DataFrame for modelling and testing.

The returned dataset simulates a hospital's major-gifts prospect pool. Features are correlated in a domain-meaningful way:

  • Donors with more years_active and higher event_attendance_count have a monotonically increasing probability of being labelled as a major donor (is_major_donor = 1).
  • total_gift_amount is log-normally distributed and positively correlated with is_major_donor.
  • last_gift_date is sampled uniformly across the past five calendar years, with major donors skewed toward more recent activity.

Parameters:

Name Type Description Default
n_samples int

Number of synthetic donor records to generate.

 1000
random_state int or None

Seed for the NumPy random-number generator. Pass an integer to obtain a reproducible dataset; None draws a fresh seed on every call.

 None

Returns:

Name Type Description
df pd.DataFrame of shape (n_samples, 5)

A DataFrame with the following columns:

total_gift_amount : float Cumulative lifetime giving in USD. Drawn from a log-normal distribution (mu = 7.5, sigma = 1.4); major donors receive an additional multiplicative uplift of 3–8×. years_active : int Number of full calendar years since the donor's first recorded gift (range 1–30). Major-donor candidates are skewed toward longer tenure. last_gift_date : datetime Date of the most recent gift. Stored as datetime64[ns]. Major donors are skewed toward dates within the past two years. event_attendance_count : int Number of fundraising events attended (range 0–20). Higher values increase propensity-to-give probability. is_major_donor : int (0 or 1) Binary label indicating whether the donor is classified as a major gift prospect (gift capacity ≥ $25,000).

Examples:

>>> from philanthropy.datasets import generate_synthetic_donor_data
>>> df = generate_synthetic_donor_data(n_samples=500, random_state=42)
>>> df.shape
(500, 5)
>>> df.dtypes["is_major_donor"]
dtype('int64')
>>> bool(df["is_major_donor"].isin([0, 1]).all())
True
Notes

A latent giving capacity drives everything. It is a linear function of years_active, event_attendance_count and an unobserved wealth term, on a log-dollar scale. total_gift_amount is then drawn as a noisy realisation of that capacity, and is_major_donor as a soft threshold on it at :data:MAJOR_GIFT_CAPACITY. last_gift_date follows engagement.

The ordering matters. Capacity is a confounder that causes both the giving history and the label, so total_gift_amount is a legitimate predictor: informative, and limited by how well giving reveals capacity.

Before version 0.7.0 the label was drawn first and total_gift_amount was drawn conditional on the label. That inverted the domain's causal arrow, and it was measurable: a model given total_gift_amount beat the Bayes rate of the generator's own process by roughly 19 ROC-AUC points, which no model can legitimately do. Using cumulative lifetime giving to predict "is a major donor" is also the classic fundraising leakage this library exists to prevent, so the reference dataset was teaching the anti-pattern. last_gift_date was a second target-derived feature for the same reason.

The label remains statistically learnable and is not trivially predictable: held out on 4,000 rows, the documented feature set reaches ROC-AUC 0.814 and accuracy 0.759 against a Bayes accuracy ceiling of 0.806 given latent capacity. Sitting below that ceiling is the point.

The function never raises an error for valid inputs. Passing n_samples=0 returns an empty DataFrame with the correct column schema.

Source code in philanthropy/datasets/_generator.py 
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
def generate_synthetic_donor_data(
    n_samples: int = 1000,
    random_state: Optional[int] = None,
) -> pd.DataFrame:
    """Generate a realistic synthetic donor DataFrame for modelling and testing.

    The returned dataset simulates a hospital's major-gifts prospect pool.
    Features are correlated in a domain-meaningful way:

    * Donors with more ``years_active`` and higher ``event_attendance_count``
      have a monotonically increasing probability of being labelled as a
      major donor (``is_major_donor = 1``).
    * ``total_gift_amount`` is log-normally distributed and positively
      correlated with ``is_major_donor``.
    * ``last_gift_date`` is sampled uniformly across the past five calendar
      years, with major donors skewed toward more recent activity.

    Parameters
    ----------
    n_samples : int, default=1000
        Number of synthetic donor records to generate.
    random_state : int or None, default=None
        Seed for the NumPy random-number generator.  Pass an integer to
        obtain a reproducible dataset; ``None`` draws a fresh seed on every
        call.

    Returns
    -------
    df : pd.DataFrame of shape (n_samples, 5)
        A DataFrame with the following columns:

        ``total_gift_amount`` : float
            Cumulative lifetime giving in USD.  Drawn from a log-normal
            distribution (mu = 7.5, sigma = 1.4); major donors receive an
            additional multiplicative uplift of 3–8×.
        ``years_active`` : int
            Number of full calendar years since the donor's first recorded
            gift (range 1–30).  Major-donor candidates are skewed toward
            longer tenure.
        ``last_gift_date`` : datetime
            Date of the most recent gift.  Stored as ``datetime64[ns]``.
            Major donors are skewed toward dates within the past two years.
        ``event_attendance_count`` : int
            Number of fundraising events attended (range 0–20).  Higher
            values increase propensity-to-give probability.
        ``is_major_donor`` : int (0 or 1)
            Binary label indicating whether the donor is classified as a
            major gift prospect (gift capacity ≥ $25,000).

    Examples
    --------
    >>> from philanthropy.datasets import generate_synthetic_donor_data
    >>> df = generate_synthetic_donor_data(n_samples=500, random_state=42)
    >>> df.shape
    (500, 5)
    >>> df.dtypes["is_major_donor"]
    dtype('int64')
    >>> bool(df["is_major_donor"].isin([0, 1]).all())
    True

    Notes
    -----
    A latent **giving capacity** drives everything. It is a linear function of
    ``years_active``, ``event_attendance_count`` and an unobserved wealth term,
    on a log-dollar scale. ``total_gift_amount`` is then drawn as a noisy
    realisation of that capacity, and ``is_major_donor`` as a soft threshold on
    it at :data:`MAJOR_GIFT_CAPACITY`. ``last_gift_date`` follows engagement.

    The ordering matters. Capacity is a **confounder** that causes both the
    giving history and the label, so ``total_gift_amount`` is a legitimate
    predictor: informative, and limited by how well giving reveals capacity.

    Before version 0.7.0 the label was drawn first and ``total_gift_amount``
    was drawn *conditional on the label*. That inverted the domain's causal
    arrow, and it was measurable: a model given ``total_gift_amount`` beat the
    Bayes rate of the generator's own process by roughly 19 ROC-AUC points,
    which no model can legitimately do. Using cumulative lifetime giving to
    predict "is a major donor" is also the classic fundraising leakage this
    library exists to prevent, so the reference dataset was teaching the
    anti-pattern. ``last_gift_date`` was a second target-derived feature for
    the same reason.

    The label remains statistically learnable and is not trivially
    predictable: held out on 4,000 rows, the documented feature set reaches
    ROC-AUC 0.814 and accuracy 0.759 against a Bayes accuracy ceiling of 0.806
    given latent capacity. Sitting **below** that ceiling is the point.

    The function never raises an error for valid inputs.  Passing
    ``n_samples=0`` returns an empty DataFrame with the correct column
    schema.
    """
    rng = np.random.default_rng(random_state)

    # ------------------------------------------------------------------
    # Step 1: Generate structural features
    # ------------------------------------------------------------------
    years_active = rng.integers(1, 31, size=n_samples)          # 1–30 years
    event_attendance = rng.integers(0, 21, size=n_samples)       # 0–20 events

    # ------------------------------------------------------------------
    # Step 2: Latent giving capacity.
    #
    # This is the confounder, and the whole point of the ordering here. Capacity
    # causes BOTH the observed giving history and major-donor status. Nothing
    # below is drawn from the label, so no feature is a readout of the answer.
    #
    # Earlier versions drew the label first and then drew total_gift_amount
    # conditional on it, which inverted the domain's causal arrow: a model given
    # total_gift_amount could beat the Bayes rate of the generator's own process,
    # and using cumulative giving to predict "is a major donor" is the classic
    # fundraising leakage this library exists to prevent.
    # ------------------------------------------------------------------
    unobserved_wealth = rng.normal(0, 1, size=n_samples)
    log_capacity = (
        7.9                               # intercept, sets the base rate
        + 0.055 * years_active            # longer relationships run deeper
        + 0.075 * event_attendance        # engagement tracks affinity and means
        + 1.25 * unobserved_wealth        # wealth no column in this frame sees
    )

    # ------------------------------------------------------------------
    # Step 3: Observed giving FOLLOWS capacity.
    #
    # Donors realise some fraction of what they could give, imperfectly, so
    # total_gift_amount is a noisy proxy for capacity rather than a function of
    # the label. That makes it a legitimate predictor: informative, and bounded
    # by how well giving history reveals capacity.
    # ------------------------------------------------------------------
    total_gift_amount = np.round(
        rng.lognormal(mean=log_capacity - 1.6, sigma=0.6), 2
    )

    # ------------------------------------------------------------------
    # Step 4: The label ALSO follows capacity: a soft threshold at the
    # $25,000 gift capacity the docstring describes. Soft rather than hard
    # because a real prospect-research call is not a step function.
    # ------------------------------------------------------------------
    z = 1.6 * (log_capacity - np.log(MAJOR_GIFT_CAPACITY)) + 0.4 * rng.normal(
        0, 1, size=n_samples
    )
    propensity = 1.0 / (1.0 + np.exp(-z))
    is_major_donor = rng.binomial(1, propensity).astype(np.int64)

    # ------------------------------------------------------------------
    # Step 5: Recency follows ENGAGEMENT, not the label.
    #
    # This was also drawn from the label before (Beta for majors, uniform for
    # everyone else), which made last_gift_date a second target-derived feature.
    # More engaged donors have given more recently, which is the real mechanism.
    # ------------------------------------------------------------------
    reference_date = pd.Timestamp("2026-02-21")  # project snapshot date
    max_days = 365 * 5
    engagement = 1.0 / (1.0 + np.exp(-0.18 * (event_attendance - 8.0)))
    recency_days = np.zeros(n_samples, dtype=np.int64)
    if n_samples > 0:
        # Larger second Beta parameter skews toward 0, i.e. toward recent dates.
        recency_days = (
            rng.beta(1.0, 1.0 + 3.5 * engagement) * max_days
        ).astype(np.int64)

    last_gift_date = pd.to_datetime(
        reference_date - pd.to_timedelta(recency_days, unit="D")
    )

    # ------------------------------------------------------------------
    # Step 6: Assemble DataFrame
    # ------------------------------------------------------------------
    df = pd.DataFrame(
        {
            "total_gift_amount": total_gift_amount,
            "years_active": years_active.astype(np.int64),
            "last_gift_date": last_gift_date,
            "event_attendance_count": event_attendance.astype(np.int64),
            "is_major_donor": is_major_donor,
        }
    )

    return df

make_donor_dataset(n_donors=200, fiscal_year_start=7, start_year=2018, end_year=2024, lapse_rate=0.25, major_gift_threshold=10000.0, random_state=42)

Generate a synthetic, seeded gift-level donor dataset.

Parameters:

Name Type Description Default
n_donors int

Number of synthetic donors to generate.

 200
fiscal_year_start int

Reserved. Accepted for signature stability but not currently used when generating gift dates.

 7
start_year int

Earliest calendar year used when sampling gift dates.

 2018
end_year int

Latest calendar year used when sampling gift dates.

 2024
lapse_rate float

Reserved. Accepted for signature stability but not currently used when generating gift dates.

 0.25
major_gift_threshold float

Gift amount (inclusive) that marks a row as a major gift.

 10_000.0
random_state int or None

Seed for the NumPy random-number generator. Pass an integer to obtain a reproducible dataset; None draws a fresh seed every call.

 42

Returns:

Name Type Description
df DataFrame

A gift-level DataFrame, not a donor-level DataFrame: each donor contributes 1-5 gift rows, so len(df) > n_donors. Rows are sorted by gift_date and the index is reset. Columns:

donor_id : str Zero-padded donor id, e.g. D00001. gift_date : datetime64[ns] Sampled between start_year and end_year. gift_amount : float Log-normal gift amount, rounded to two decimal places. appeal_code : str One of ANNUAL, MAJOR, PLANNED, ONLINE, or EVENT. is_major_gift : bool True when gift_amount >= major_gift_threshold.

Examples:

>>> from philanthropy.datasets import make_donor_dataset
>>> df = make_donor_dataset(n_donors=5, random_state=0)
>>> df.shape
(15, 5)
>>> list(df.columns)
['donor_id', 'gift_date', 'gift_amount', 'appeal_code', 'is_major_gift']
>>> df['donor_id'].nunique()
5
Source code in philanthropy/datasets/_generator.py 
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
def make_donor_dataset(
    n_donors: int = 200,
    fiscal_year_start: int = 7,
    start_year: int = 2018,
    end_year: int = 2024,
    lapse_rate: float = 0.25,
    major_gift_threshold: float = 10_000.0,
    random_state: Optional[int] = 42,
) -> pd.DataFrame:
    """Generate a synthetic, seeded gift-level donor dataset.

    Parameters
    ----------
    n_donors : int, default=200
        Number of synthetic donors to generate.
    fiscal_year_start : int, default=7
        Reserved. Accepted for signature stability but not currently used
        when generating gift dates.
    start_year : int, default=2018
        Earliest calendar year used when sampling gift dates.
    end_year : int, default=2024
        Latest calendar year used when sampling gift dates.
    lapse_rate : float, default=0.25
        Reserved. Accepted for signature stability but not currently used
        when generating gift dates.
    major_gift_threshold : float, default=10_000.0
        Gift amount (inclusive) that marks a row as a major gift.
    random_state : int or None, default=42
        Seed for the NumPy random-number generator. Pass an integer to
        obtain a reproducible dataset; ``None`` draws a fresh seed every call.

    Returns
    -------
    df : pd.DataFrame
        A gift-level DataFrame, not a donor-level DataFrame: each donor
        contributes 1-5 gift rows, so ``len(df) > n_donors``. Rows are sorted
        by ``gift_date`` and the index is reset. Columns:

        ``donor_id`` : str
            Zero-padded donor id, e.g. ``D00001``.
        ``gift_date`` : datetime64[ns]
            Sampled between ``start_year`` and ``end_year``.
        ``gift_amount`` : float
            Log-normal gift amount, rounded to two decimal places.
        ``appeal_code`` : str
            One of ``ANNUAL``, ``MAJOR``, ``PLANNED``, ``ONLINE``, or ``EVENT``.
        ``is_major_gift`` : bool
            True when ``gift_amount >= major_gift_threshold``.

    Examples
    --------
    >>> from philanthropy.datasets import make_donor_dataset
    >>> df = make_donor_dataset(n_donors=5, random_state=0)
    >>> df.shape
    (15, 5)
    >>> list(df.columns)
    ['donor_id', 'gift_date', 'gift_amount', 'appeal_code', 'is_major_gift']
    >>> df['donor_id'].nunique()
    5
    """
    rng = np.random.default_rng(random_state)
    donor_ids = [f"D{str(i).zfill(5)}" for i in range(1, n_donors + 1)]
    records = []
    for donor_id in donor_ids:
        n_gifts = rng.integers(1, 6)
        for _ in range(n_gifts):
            year = rng.integers(start_year, end_year + 1)
            month = rng.integers(1, 13)
            day = rng.integers(1, 28)
            gift_date = pd.Timestamp(year=int(year), month=int(month), day=int(day))
            gift_amount = float(rng.lognormal(mean=5.5, sigma=1.2))
            gift_amount = round(gift_amount, 2)
            appeal_code = rng.choice(["ANNUAL", "MAJOR", "PLANNED", "ONLINE", "EVENT"])
            records.append(
                {
                    "donor_id": donor_id,
                    "gift_date": gift_date,
                    "gift_amount": gift_amount,
                    "appeal_code": appeal_code,
                }
            )
    df = pd.DataFrame(records).sort_values("gift_date").reset_index(drop=True)
    df["is_major_gift"] = df["gift_amount"] >= major_gift_threshold
    return df

fetch_kdd98_donors(*, data_home=None, download_if_missing=True)

Fetch the KDD Cup 1998 direct-mail donor learning set.

A real donor-level dataset: 95,412 individuals who gave at least once between June 1995 and June 1996, one row per donor, with a full per-promotion mail and response history (ADATE_2..ADATE_24, RDATE_2..RDATE_24, RAMNT_2..RAMNT_24) plus the outcome of the 1997 mailing being predicted (TARGET_B, TARGET_D). That date history is what makes as-of feature construction testable on it rather than only on synthetic data; see :func:generate_synthetic_donor_data for the synthetic equivalent used elsewhere in this library. Most date columns are encoded YYMM (year, month) rather than as a datetime dtype.

This is a read-only public research dataset, not your data. Nothing about your own donors, gifts, or environment is ever sent anywhere; the only network traffic this function makes is fetching the dataset file itself, once, to a local cache. It is never called automatically: no other function in this library imports it or calls it during fit or transform.

Under the dataset's terms of use, teaching or training material that uses it must not name the sponsoring organisation; cite it only as "KDD Cup 1998". This docstring follows that condition, and so should anything you write based on it.

Parameters:

Name Type Description Default
data_home str

Directory to cache the downloaded archive in. Defaults to the PHILANTHROPY_DATA environment variable if set, else ~/philanthropy_data.

 None
download_if_missing bool

If the archive is not already cached, download it. If False and the archive is not cached, raise OSError instead of reaching for the network.

 True

Returns:

Type Description
pandas.DataFrame of shape (95412, 481)

One row per donor, columns as documented in the data dictionary below. Column dtypes are pandas' own inference over the raw CSV; this function does not recode or impute any of them.

Raises:

Type Description
OSError

If the archive is not cached and download_if_missing is False, or if a downloaded archive fails its checksum check.
Notes

Source: the UCI KDD Archive, https://kdd.ics.uci.edu/databases/kddcup98/kddcup98.html. Field-by-field documentation: cup98dic.txt at the same location. Distributed for general research and educational use under the terms stated on that page, including the sponsor-naming restriction noted above and a request to notify the dataset's contacts of any published results.

The archive is not vendored with this package (its terms require an unmodified, individually-fetched copy); this function downloads it to a local cache on first use, the way sklearn.datasets.fetch_* functions do, and every later call reads the cached copy.

Source code in philanthropy/datasets/_kdd98.py 
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
def fetch_kdd98_donors(
    *, data_home: Optional[str] = None, download_if_missing: bool = True
) -> pd.DataFrame:
    """Fetch the KDD Cup 1998 direct-mail donor learning set.

    A real donor-level dataset: 95,412 individuals who gave at least once
    between June 1995 and June 1996, one row per donor, with a full
    per-promotion mail and response history (``ADATE_2``..``ADATE_24``,
    ``RDATE_2``..``RDATE_24``, ``RAMNT_2``..``RAMNT_24``) plus the outcome of
    the 1997 mailing being predicted (``TARGET_B``, ``TARGET_D``). That date
    history is what makes as-of feature construction testable on it rather
    than only on synthetic data; see :func:`generate_synthetic_donor_data` for
    the synthetic equivalent used elsewhere in this library. Most date columns
    are encoded ``YYMM`` (year, month) rather than as a `datetime` dtype.

    This is a **read-only public research dataset**, not your data. Nothing
    about your own donors, gifts, or environment is ever sent anywhere; the
    only network traffic this function makes is fetching the dataset file
    itself, once, to a local cache. It is never called automatically: no
    other function in this library imports it or calls it during `fit` or
    `transform`.

    Under the dataset's terms of use, teaching or training material that uses
    it must not name the sponsoring organisation; cite it only as "KDD Cup
    1998". This docstring follows that condition, and so should anything you
    write based on it.

    Parameters
    ----------
    data_home : str, default=None
        Directory to cache the downloaded archive in. Defaults to the
        ``PHILANTHROPY_DATA`` environment variable if set, else
        ``~/philanthropy_data``.

    download_if_missing : bool, default=True
        If the archive is not already cached, download it. If False and the
        archive is not cached, raise ``OSError`` instead of reaching for the
        network.

    Returns
    -------
    pandas.DataFrame of shape (95412, 481)
        One row per donor, columns as documented in the data dictionary
        below. Column dtypes are pandas' own inference over the raw CSV;
        this function does not recode or impute any of them.

    Raises
    ------
    OSError
        If the archive is not cached and `download_if_missing` is False, or
        if a downloaded archive fails its checksum check.

    Notes
    -----
    Source: the UCI KDD Archive,
    https://kdd.ics.uci.edu/databases/kddcup98/kddcup98.html. Field-by-field
    documentation: ``cup98dic.txt`` at the same location. Distributed for
    general research and educational use under the terms stated on that page,
    including the sponsor-naming restriction noted above and a request to
    notify the dataset's contacts of any published results.

    The archive is not vendored with this package (its terms require an
    unmodified, individually-fetched copy); this function downloads it to a
    local cache on first use, the way ``sklearn.datasets.fetch_*`` functions
    do, and every later call reads the cached copy.
    """
    cache_dir = _data_home(data_home)
    archive_path = os.path.join(cache_dir, "cup98lrn.zip")

    if not os.path.exists(archive_path):
        if not download_if_missing:
            raise OSError(
                f"{archive_path} is not cached and download_if_missing=False. "
                "Call with download_if_missing=True to fetch it."
            )
        _download(_LEARNING_URL, archive_path, _LEARNING_SHA256)

    with zipfile.ZipFile(archive_path) as archive:
        with archive.open(_LEARNING_MEMBER) as fh:
            return pd.read_csv(fh, low_memory=False)

fetch_kdd98_val_donors(*, data_home=None, download_if_missing=True)

Fetch KDD Cup 1998's own held-out VALIDATION set, merged with its answer key.

fetch_kdd98_donors returns the learning file, the one every model in this library fits and splits (55/15/30, see scripts/benchmark_models_vs_baselines.py). This function returns the competition's separate validation file (cup98VAL.txt): 96,367 more donors that were never part of the learning file and never touched during a learning-file split of any kind. TARGET_B/TARGET_D (whether and how much each of them gave to the 97NK mailing) were withheld from participants during the original competition and released afterwards in a separate answer key, valtargt.txt; this function downloads both and joins them on CONTROLN so the result has the same shape as :func:fetch_kdd98_donors, including TARGET_B/TARGET_D.

Use this only to score a model already fit elsewhere (e.g. on the learning file), never to fit one: a validation file used for fitting stops being held out.

This is a read-only public research dataset, not your data, under the same terms and the same sponsor-naming restriction as :func:fetch_kdd98_donors; see that function's docstring.

Parameters:

Name Type Description Default
data_home str

Directory to cache the downloaded archives in. Defaults to the PHILANTHROPY_DATA environment variable if set, else ~/philanthropy_data.

 None
download_if_missing bool

If either archive is not already cached, download it. If False and either is missing, raise OSError instead of reaching for the network.

 True

Returns:

Type Description
pandas.DataFrame of shape (96367, 481)

One row per donor, the same columns as :func:fetch_kdd98_donors.

Raises:

Type Description
OSError

If an archive is not cached and download_if_missing is False, or if a downloaded archive fails its checksum check.
Notes

Source: the UCI KDD Archive, https://kdd.ics.uci.edu/databases/kddcup98/kddcup98.html, the same page and terms as the learning file. valtargt.readme at the same location documents the answer-key file's three columns (CONTROLN, TARGET_B, TARGET_D).

Source code in philanthropy/datasets/_kdd98.py 
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
def fetch_kdd98_val_donors(
    *, data_home: Optional[str] = None, download_if_missing: bool = True
) -> pd.DataFrame:
    """Fetch KDD Cup 1998's own held-out VALIDATION set, merged with its answer key.

    ``fetch_kdd98_donors`` returns the *learning* file, the one every model in
    this library fits and splits (55/15/30, see
    ``scripts/benchmark_models_vs_baselines.py``). This function returns the
    competition's separate *validation* file (``cup98VAL.txt``): 96,367 more
    donors that were never part of the learning file and never touched
    during a learning-file split of any kind. ``TARGET_B``/``TARGET_D``
    (whether and how much each of them gave to the 97NK mailing) were
    withheld from participants during the original competition and released
    afterwards in a separate answer key, ``valtargt.txt``; this function
    downloads both and joins them on ``CONTROLN`` so the result has the same
    shape as :func:`fetch_kdd98_donors`, including ``TARGET_B``/``TARGET_D``.

    Use this only to score a model already fit elsewhere (e.g. on the
    learning file), never to fit one: a validation file used for fitting
    stops being held out.

    This is a **read-only public research dataset**, not your data, under the
    same terms and the same sponsor-naming restriction as
    :func:`fetch_kdd98_donors`; see that function's docstring.

    Parameters
    ----------
    data_home : str, default=None
        Directory to cache the downloaded archives in. Defaults to the
        ``PHILANTHROPY_DATA`` environment variable if set, else
        ``~/philanthropy_data``.

    download_if_missing : bool, default=True
        If either archive is not already cached, download it. If False and
        either is missing, raise ``OSError`` instead of reaching for the
        network.

    Returns
    -------
    pandas.DataFrame of shape (96367, 481)
        One row per donor, the same columns as :func:`fetch_kdd98_donors`.

    Raises
    ------
    OSError
        If an archive is not cached and `download_if_missing` is False, or
        if a downloaded archive fails its checksum check.

    Notes
    -----
    Source: the UCI KDD Archive,
    https://kdd.ics.uci.edu/databases/kddcup98/kddcup98.html, the same page
    and terms as the learning file. ``valtargt.readme`` at the same location
    documents the answer-key file's three columns (``CONTROLN``,
    ``TARGET_B``, ``TARGET_D``).
    """
    cache_dir = _data_home(data_home)
    val_path = os.path.join(cache_dir, "cup98val.zip")
    targt_path = os.path.join(cache_dir, "valtargt.txt")

    if not os.path.exists(val_path):
        if not download_if_missing:
            raise OSError(
                f"{val_path} is not cached and download_if_missing=False. "
                "Call with download_if_missing=True to fetch it."
            )
        _download(_VAL_URL, val_path, _VAL_SHA256)

    if not os.path.exists(targt_path):
        if not download_if_missing:
            raise OSError(
                f"{targt_path} is not cached and download_if_missing=False. "
                "Call with download_if_missing=True to fetch it."
            )
        _download(_VALTARGT_URL, targt_path, _VALTARGT_SHA256)

    with zipfile.ZipFile(val_path) as archive:
        with archive.open(_VAL_MEMBER) as fh:
            donors = pd.read_csv(fh, low_memory=False)
    targets = pd.read_csv(targt_path)
    return donors.merge(targets, on="CONTROLN", how="inner")

load_karlan_list(path)

Load the Karlan and List matching-grant experiment from a local file.

Karlan, D. and List, J. A. (2007), "Does Price Matter in Charitable Giving? Evidence from a Large-Scale Natural Field Experiment", American Economic Review 97(5): 1774-1793. About 50,000 prior donors to one US nonprofit were mailed a fundraising letter in 2005; two in three, chosen at random, were offered a matching grant, with the match ratio, the cap on the matching pool and the example ask amount each randomised.

The replication package is openICPSR 113224 (doi:10.3886/E113224V1). Its LICENSE.txt puts the data under CC BY 4.0 and the code under BSD-3, copyright American Economic Association 2007, so anything built on it must credit the authors and the AEA. This reader never downloads the file: point it at AERtables1-5.dta from your own download.

Rows with every variable missing (47 in the published file) are dropped; nothing else is filtered.

Parameters:

Name Type Description Default
path str

Path to AERtables1-5.dta.

 required

Returns:

Type Description
DataFrame

One row per donor:

  • gave (int): 1 if the donor gave in response to the letter.
  • amount (float): dollars given, 0 if not.
  • matched (int): 1 if the letter offered a matching grant.
  • match_ratio (int): 1, 2 or 3 for a 1:1, 2:1 or 3:1 match; 0 without one.
  • match_cap (int): 1, 2 or 3 for a $25,000, $50,000 or $100,000 matching pool; 4 if the letter stated no cap; 0 without a match.
  • ask_multiple (int): 1, 2 or 3 if the example amount was the donor's highest previous gift, 1.25 times it or 1.5 times it; 0 without a match.
  • prior_gifts, highest_previous_amount, months_since_last_gift, years_since_first_gift (float): giving history before the letter.
  • female, couple (float): from the donor record, NaN where unknown.
  • red_state, red_county (float): 1 if the donor's state or county voted for Bush in 2004, NaN where unknown.

Examples:

>>> from philanthropy.datasets import load_karlan_list
>>> df = load_karlan_list("AERtables1-5.dta")
>>> round(df["gave"].mean(), 3)
0.021
Source code in philanthropy/datasets/_karlan_list.py 
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
def load_karlan_list(path: str) -> pd.DataFrame:
    """Load the Karlan and List matching-grant experiment from a local file.

    Karlan, D. and List, J. A. (2007), "Does Price Matter in Charitable
    Giving? Evidence from a Large-Scale Natural Field Experiment", *American
    Economic Review* 97(5): 1774-1793. About 50,000 prior donors to one US
    nonprofit were mailed a fundraising letter in 2005; two in three, chosen
    at random, were offered a matching grant, with the match ratio, the cap
    on the matching pool and the example ask amount each randomised.

    The replication package is openICPSR 113224 (doi:10.3886/E113224V1).
    Its ``LICENSE.txt`` puts the data under CC BY 4.0 and the code under
    BSD-3, copyright American Economic Association 2007, so anything built
    on it must credit the authors and the AEA. This reader never downloads
    the file: point it at ``AERtables1-5.dta`` from your own download.

    Rows with every variable missing (47 in the published file) are dropped;
    nothing else is filtered.

    Parameters
    ----------
    path : str
        Path to ``AERtables1-5.dta``.

    Returns
    -------
    pandas.DataFrame
        One row per donor:

        - ``gave`` (int): 1 if the donor gave in response to the letter.
        - ``amount`` (float): dollars given, 0 if not.
        - ``matched`` (int): 1 if the letter offered a matching grant.
        - ``match_ratio`` (int): 1, 2 or 3 for a 1:1, 2:1 or 3:1 match; 0
          without one.
        - ``match_cap`` (int): 1, 2 or 3 for a $25,000, $50,000 or $100,000
          matching pool; 4 if the letter stated no cap; 0 without a match.
        - ``ask_multiple`` (int): 1, 2 or 3 if the example amount was the
          donor's highest previous gift, 1.25 times it or 1.5 times it; 0
          without a match.
        - ``prior_gifts``, ``highest_previous_amount``,
          ``months_since_last_gift``, ``years_since_first_gift`` (float):
          giving history before the letter.
        - ``female``, ``couple`` (float): from the donor record, ``NaN``
          where unknown.
        - ``red_state``, ``red_county`` (float): 1 if the donor's state or
          county voted for Bush in 2004, ``NaN`` where unknown.

    Examples
    --------
    >>> from philanthropy.datasets import load_karlan_list
    >>> df = load_karlan_list("AERtables1-5.dta")  # doctest: +SKIP
    >>> round(df["gave"].mean(), 3)  # doctest: +SKIP
    0.021
    """
    raw = pd.read_stata(path, columns=list(_COLUMNS), convert_categoricals=False)
    df = raw.dropna(how="all").rename(columns=_COLUMNS).reset_index(drop=True)
    df = df.astype({c: "float64" for c in df.columns})
    df[_INT_COLUMNS] = df[_INT_COLUMNS].astype("int64")
    return df

make_donor_panel(n_donors=3000, n_years=7, start_fiscal_year=2018, include_encounters=False, random_state=None)

Generate a seeded multi-year donor panel with gift-level rows.

Unlike :func:generate_synthetic_donor_data, which returns one already-aggregated row per donor, this returns the raw tables a fundraising shop actually exports: a gift log, a donor table, and optionally a clinical-encounter table. Aggregating them is left to the caller, on purpose, because when you aggregate is the whole subject of :doc:the leakage tutorial </tutorials/avoiding_temporal_data_leakage>.

Fiscal years run 1 July to 30 June and are labelled by the calendar year in which they end, so fiscal year 2019 spans 2018-07-01 to 2019-06-30. A donor gives at most once per fiscal year.

Parameters:

Name Type Description Default
n_donors int

Number of donors in the panel. Every donor appears in donors; donors who never gave contribute no rows to gifts.

 3000
n_years int

Number of consecutive fiscal years. Deriving a "gave in the following year" label costs the last one, so the default yields six usable panel years.

 7
start_fiscal_year int

Label of the first fiscal year.

 2018
include_encounters bool

Also return an "encounters" table for the grateful-patient transformers. Off by default: most callers are not an academic medical center, and an unused encounter table invites the mistake of treating synthetic clinical rows as if they meant something.

 False
random_state int or None

Seed for the NumPy generator. Pass an integer for a reproducible panel; None draws a fresh seed on every call.

 None

Returns:

Name Type Description
panel dict of str to pandas.DataFrame

"gifts" One row per gift: donor_id (int), gift_date (datetime64[ns]), gift_amount (float), fiscal_year (int), appeal (str). Sorted by gift_date. Column names match what :class:~philanthropy.preprocessing.RFMTransformer requires. "donors" One row per donor: donor_id (int), first_gift_fy (float, NaN for donors who never gave), wealth_estimate (float, ~30% NaN by design, for :class:~philanthropy.preprocessing.WealthScreeningImputer), employer (str, sometimes empty). "encounters", only when include_encounters=True Zero or more rows per donor: donor_id (int), admit_date and discharge_date (datetime64[ns]), service_line (str).
Notes

There is no label column, deliberately. A label is a claim about a point in time, and shipping one pre-computed would hand every user the exact mistake this package exists to prevent. Derive it, as of the year you are scoring:

from philanthropy.datasets import make_donor_panel panel = make_donor_panel(n_donors=200, random_state=0) gifts = panel["gifts"] gave = set(zip(gifts["donor_id"], gifts["fiscal_year"])) label = int((7, 2020) in gave) # did donor 7 give in FY2020?

Every number here is invented. The panel establishes mechanisms and shapes, never magnitudes you should quote for your own program; for magnitudes measured on a real donor file, see :doc:/explanation/real_data_replication.

Examples:

>>> from philanthropy.datasets import make_donor_panel
>>> panel = make_donor_panel(n_donors=500, random_state=42)
>>> sorted(panel)
['donors', 'gifts']
>>> list(panel["gifts"].columns)
['donor_id', 'gift_date', 'gift_amount', 'fiscal_year', 'appeal']
>>> int(panel["gifts"]["fiscal_year"].min()), int(panel["gifts"]["fiscal_year"].max())
(2018, 2024)
>>> len(panel["donors"])
500

Straight into RFM features, which the cross-sectional generator cannot do:

>>> from philanthropy.preprocessing import RFMTransformer
>>> rfm = RFMTransformer().fit_transform(panel["gifts"])
>>> list(rfm.columns)
['donor_id', 'recency', 'frequency', 'monetary']

With encounters, for the grateful-patient path:

>>> panel = make_donor_panel(
...     n_donors=500, include_encounters=True, random_state=42
... )
>>> list(panel["encounters"].columns)
['donor_id', 'admit_date', 'discharge_date', 'service_line']
Source code in philanthropy/datasets/_panel.py 
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
def make_donor_panel(
    n_donors: int = 3000,
    n_years: int = 7,
    start_fiscal_year: int = 2018,
    include_encounters: bool = False,
    random_state: Optional[int] = None,
) -> Dict[str, pd.DataFrame]:
    """Generate a seeded multi-year donor panel with gift-level rows.

    Unlike :func:`generate_synthetic_donor_data`, which returns one
    already-aggregated row per donor, this returns the *raw* tables a
    fundraising shop actually exports: a gift log, a donor table, and
    optionally a clinical-encounter table. Aggregating them is left to the
    caller, on purpose, because *when* you aggregate is the whole subject of
    :doc:`the leakage tutorial </tutorials/avoiding_temporal_data_leakage>`.

    Fiscal years run 1 July to 30 June and are labelled by the calendar year in
    which they end, so fiscal year 2019 spans 2018-07-01 to 2019-06-30. A donor
    gives at most once per fiscal year.

    Parameters
    ----------
    n_donors : int, default=3000
        Number of donors in the panel. Every donor appears in ``donors``;
        donors who never gave contribute no rows to ``gifts``.
    n_years : int, default=7
        Number of consecutive fiscal years. Deriving a "gave in the following
        year" label costs the last one, so the default yields six usable panel
        years.
    start_fiscal_year : int, default=2018
        Label of the first fiscal year.
    include_encounters : bool, default=False
        Also return an ``"encounters"`` table for the grateful-patient
        transformers. Off by default: most callers are not an academic medical
        center, and an unused encounter table invites the mistake of treating
        synthetic clinical rows as if they meant something.
    random_state : int or None, default=None
        Seed for the NumPy generator. Pass an integer for a reproducible
        panel; ``None`` draws a fresh seed on every call.

    Returns
    -------
    panel : dict of str to pandas.DataFrame
        ``"gifts"``
            One row per gift: ``donor_id`` (int), ``gift_date``
            (datetime64[ns]), ``gift_amount`` (float), ``fiscal_year`` (int),
            ``appeal`` (str). Sorted by ``gift_date``. Column names match what
            :class:`~philanthropy.preprocessing.RFMTransformer` requires.
        ``"donors"``
            One row per donor: ``donor_id`` (int), ``first_gift_fy`` (float,
            NaN for donors who never gave), ``wealth_estimate`` (float, ~30%
            NaN by design, for
            :class:`~philanthropy.preprocessing.WealthScreeningImputer`),
            ``employer`` (str, sometimes empty).
        ``"encounters"``, only when ``include_encounters=True``
            Zero or more rows per donor: ``donor_id`` (int), ``admit_date``
            and ``discharge_date`` (datetime64[ns]), ``service_line`` (str).

    Notes
    -----
    **There is no label column, deliberately.** A label is a claim about a
    point in time, and shipping one pre-computed would hand every user the
    exact mistake this package exists to prevent. Derive it, as of the year you
    are scoring:

    >>> from philanthropy.datasets import make_donor_panel
    >>> panel = make_donor_panel(n_donors=200, random_state=0)
    >>> gifts = panel["gifts"]
    >>> gave = set(zip(gifts["donor_id"], gifts["fiscal_year"]))
    >>> label = int((7, 2020) in gave)   # did donor 7 give in FY2020?

    Every number here is invented. The panel establishes mechanisms and shapes,
    never magnitudes you should quote for your own program; for magnitudes
    measured on a real donor file, see
    :doc:`/explanation/real_data_replication`.

    Examples
    --------
    >>> from philanthropy.datasets import make_donor_panel
    >>> panel = make_donor_panel(n_donors=500, random_state=42)
    >>> sorted(panel)
    ['donors', 'gifts']
    >>> list(panel["gifts"].columns)
    ['donor_id', 'gift_date', 'gift_amount', 'fiscal_year', 'appeal']
    >>> int(panel["gifts"]["fiscal_year"].min()), int(panel["gifts"]["fiscal_year"].max())
    (2018, 2024)
    >>> len(panel["donors"])
    500

    Straight into RFM features, which the cross-sectional generator cannot do:

    >>> from philanthropy.preprocessing import RFMTransformer
    >>> rfm = RFMTransformer().fit_transform(panel["gifts"])
    >>> list(rfm.columns)
    ['donor_id', 'recency', 'frequency', 'monetary']

    With encounters, for the grateful-patient path:

    >>> panel = make_donor_panel(
    ...     n_donors=500, include_encounters=True, random_state=42
    ... )
    >>> list(panel["encounters"].columns)
    ['donor_id', 'admit_date', 'discharge_date', 'service_line']
    """
    if n_donors < 1:
        raise ValueError(f"n_donors must be at least 1; got {n_donors}.")
    if n_years < 2:
        raise ValueError(
            "n_years must be at least 2, because a 'gave in the following "
            f"year' label needs a following year; got {n_years}."
        )

    rng = np.random.default_rng(random_state)

    # --- The giving process. ------------------------------------------------
    # These draws, in this order, are the ones scripts/leakage_experiment.py
    # has always made. Anything added below must stay below, or the published
    # benchmark numbers stop reproducing.
    theta = rng.normal(0, 1.2, n_donors)
    drift = np.linspace(0.3, -0.3, n_years)
    gave = np.zeros((n_donors, n_years), dtype=bool)
    amount = np.zeros((n_donors, n_years))
    for j in range(n_years):
        p = 1.0 / (1.0 + np.exp(-(theta + drift[j])))
        gave[:, j] = rng.random(n_donors) < p
        amount[:, j] = np.where(
            gave[:, j], rng.lognormal(6 + 0.35 * theta, 0.8), 0.0
        )

    # --- Everything below is presentation, and draws after the process. -----
    fiscal_years = start_fiscal_year + np.arange(n_years)
    day_offset = rng.integers(0, 365, size=(n_donors, n_years))
    appeal_idx = rng.integers(0, len(_APPEALS), size=(n_donors, n_years))

    donor_ix, year_ix = np.nonzero(gave)
    fy = fiscal_years[year_ix]
    # Fiscal year N opens on 1 July of year N-1.
    fy_start = pd.to_datetime([f"{year - 1}-07-01" for year in fy])
    gift_date = fy_start + pd.to_timedelta(day_offset[donor_ix, year_ix], unit="D")

    gifts = pd.DataFrame(
        {
            "donor_id": donor_ix,
            "gift_date": gift_date,
            # Not rounded to cents, tempting as that is. scripts/leakage_
            # experiment.py aggregates these amounts, and rounding moved the
            # published min-max ranges by 0.001 AUC. A cosmetic decimal is not
            # worth invalidating a number that is already in the docs and the
            # paper. Round at the point of display instead.
            "gift_amount": amount[donor_ix, year_ix],
            "fiscal_year": fy,
            "appeal": [_APPEALS[i] for i in appeal_idx[donor_ix, year_ix]],
        }
    ).sort_values("gift_date", ignore_index=True)

    # NaN at a realistic rate, because a wealth screen that came back for every
    # record is not a wealth screen anyone has ever received.
    wealth = np.exp(11.0 + 0.6 * theta + rng.normal(0, 0.5, n_donors))
    wealth[rng.random(n_donors) < 0.30] = np.nan

    ever_gave = gave.any(axis=1)
    first_gift_fy = np.full(n_donors, np.nan)
    first_gift_fy[ever_gave] = fiscal_years[gave[ever_gave].argmax(axis=1)]

    donors = pd.DataFrame(
        {
            "donor_id": np.arange(n_donors),
            "first_gift_fy": first_gift_fy,
            "wealth_estimate": np.round(wealth, 2),
            "employer": rng.choice(_EMPLOYERS, size=n_donors),
        }
    )

    panel = {"gifts": gifts, "donors": donors}
    if include_encounters:
        panel["encounters"] = _encounters(rng, n_donors, fiscal_years)
    return panel

load_psid_philanthropy(data_path, do_path)

Load a user-downloaded PSID individual-level cross-year extract into a long household-giving/volunteering table.

The Panel Study of Income Dynamics (PSID) is a public-use survey; its Conditions of Use (agreed to when registering at https://simba.isr.umich.edu) require: no transfer of downloaded data to a third party, no attempt to identify any individual or family, deletion of the data after the analysis it was obtained for, reporting publications that use it, and acknowledging PSID as the source. This reader never downloads, caches, or redistributes PSID data: point it at a fixed-width extract and its accompanying Stata .do file (column positions only; no Stata or any other new dependency is used to read it) that you obtained yourself from the PSID Data Center under your own registration. Never commit PSID rows, or anything derived at the row/person level, to this repository or its docs; aggregate statistics only.

This reader is built against a fixed list of PSID ER/S variables (read from PSID Data Center job 365729's own .do file and cross-checked against its codebook, covering 2001 to 2023 "Current Year Heads Individual Data": one row per person who was Head/Reference Person in at least one selected wave). ER/S variable numbers are fixed, study-wide PSID identifiers, not job-specific, so a differently-selected extract that still includes these variables reads identically. An extract with a smaller variable selection than job 365729 simply omits some columns: a wave whose Head/Reference-person variable is entirely absent is skipped outright, while a wave whose Head/Reference-person variable is present but another of its variables is missing raises a plain KeyError, since that combination means this table and the extract disagree about what was selected.

For each wave, a person counts as that household's Head only when their relation-to-head/reference-person code is 10; sequence number is not used; the household key is the person's own 1968 interview number and person number (ER30001 * 1000 + ER30002), which identifies them (and so their household, while they are its Head) across every wave.

Giving amounts, itemized contributions, and volunteering hours all use PSID's standard missing-data convention for a field of width w: a value of w nines means Not Applicable/Refused and w nines with a trailing 8 means Don't Know; both are recoded to NaN here rather than left as large sentinel integers. family_income, wealth1, and wealth2 are PSID-generated/imputed totals, not raw survey responses, and were directly checked against the real data for this convention: they carry no such sentinel values, so this reader leaves them unrecoded (a large value there, e.g. a 7-digit income, is real). total_giving sums whatever per-category amounts a wave actually asked (NaN only when every category for that wave is NaN, so a few Don't-Know categories do not blank the whole total); the categories themselves differ by wave, most notably 2001 (five categories plus one combined "checkpoint" amount for the rest) and 2021 onward (no "community" category).

Volunteering hours are three different measures that must never be combined into one series: 2001 asks hours volunteered last year, split by head and spouse; 2003 and 2005 ask hours volunteered "regularly," one battery of up to 14 slots (one per organization type) that is not split by head/spouse at all, so this reader returns it as a single household-level total; 2017, 2019, 2021 and 2023 ask hours volunteered in a typical week, again split by head and spouse. 2007 through 2015 have no hours amount in this extract (only whether-volunteered flags, which this reader does not return). The three measures are returned as separate columns, NaN in every wave they do not apply to, so a caller cannot accidentally splice them into a single trend.

Parameters:

Name Type Description Default
data_path str

Path to the fixed-width ASCII data file (PSID calls this the .txt file in a Data Center download).

 required
do_path str

Path to the accompanying Stata .do file from the same download; only its infix column-position block is read.

 required

Returns:

Type Description
DataFrame

Long format, one row per (household, wave actually observed for that household's Head). Columns: household_key, year, total_giving, one giving_<category> column per category name used anywhere in _WAVES (NaN in waves that did not ask that category), itemized_charitable_contrib_amount (PSID G102A, tax itemizers only), family_income, wealth1 (without home equity), wealth2 (with home equity), head_volunteer_hours_annual, spouse_volunteer_hours_annual, household_volunteer_hours_regular (2003 and 2005 only), head_volunteer_hours_typical_week, spouse_volunteer_hours_typical_week.
Notes

Source: Panel Study of Income Dynamics, public use dataset, produced and distributed by the Survey Research Center, Institute for Social Research, University of Michigan, Ann Arbor, MI.

Source code in philanthropy/datasets/_psid.py 
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
def load_psid_philanthropy(data_path: str, do_path: str) -> pd.DataFrame:
    """Load a user-downloaded PSID individual-level cross-year extract into
    a long household-giving/volunteering table.

    The Panel Study of Income Dynamics (PSID) is a public-use survey; its
    Conditions of Use (agreed to when registering at
    https://simba.isr.umich.edu) require: no transfer of downloaded data to
    a third party, no attempt to identify any individual or family, deletion
    of the data after the analysis it was obtained for, reporting
    publications that use it, and acknowledging PSID as the source. This
    reader never downloads, caches, or redistributes PSID data: point it at
    a fixed-width extract and its accompanying Stata ``.do`` file (column
    positions only; no Stata or any other new dependency is used to read
    it) that you obtained yourself from the PSID Data Center under your own
    registration. Never commit PSID rows, or anything derived at the
    row/person level, to this repository or its docs; aggregate statistics
    only.

    This reader is built against a fixed list of PSID ER/S variables (read
    from PSID Data Center job 365729's own ``.do`` file and cross-checked
    against its codebook, covering 2001 to 2023 "Current Year Heads
    Individual Data": one row per person who was Head/Reference Person in
    at least one selected wave). ER/S variable numbers are fixed, study-wide
    PSID identifiers, not job-specific, so a differently-selected extract
    that still includes these variables reads identically. An extract with
    a *smaller* variable selection than job 365729 simply omits some
    columns: a wave whose Head/Reference-person variable is entirely
    absent is skipped outright, while a wave whose Head/Reference-person
    variable is present but another of its variables is missing raises a
    plain ``KeyError``, since that combination means this table and the
    extract disagree about what was selected.

    For each wave, a person counts as that household's Head only when their
    relation-to-head/reference-person code is 10; sequence number is not
    used; the household key is the person's own 1968 interview number and
    person number (``ER30001 * 1000 + ER30002``), which identifies them
    (and so their household, while they are its Head) across every wave.

    Giving amounts, itemized contributions, and volunteering hours all use
    PSID's standard missing-data convention for a field of width *w*: a
    value of *w* nines means Not Applicable/Refused and *w* nines with a
    trailing 8 means Don't Know; both are recoded to ``NaN`` here rather
    than left as large sentinel integers. ``family_income``, ``wealth1``,
    and ``wealth2`` are PSID-generated/imputed totals, not raw survey
    responses, and were directly checked against the real data for this
    convention: they carry no such sentinel values, so this reader leaves
    them unrecoded (a large value there, e.g. a 7-digit income, is real).
    ``total_giving`` sums whatever per-category amounts a wave actually
    asked (``NaN`` only when every category for that wave is ``NaN``, so a
    few Don't-Know categories do not blank the whole total); the categories
    themselves differ by wave, most notably 2001 (five categories plus one
    combined "checkpoint" amount for the rest) and 2021 onward (no
    "community" category).

    Volunteering hours are **three different measures that must never be
    combined into one series**: 2001 asks hours volunteered *last year*,
    split by head and spouse; 2003 and 2005 ask hours volunteered
    "regularly," one battery of up to 14 slots (one per organization type)
    that is not split by head/spouse at all, so this reader returns it as
    a single household-level total; 2017, 2019, 2021 and 2023 ask hours
    volunteered in a *typical week*, again split by head and spouse. 2007
    through 2015 have no hours amount in this extract (only
    whether-volunteered flags, which this reader does not return). The
    three measures are returned as separate columns, ``NaN`` in every wave
    they do not apply to, so a caller cannot accidentally splice them into
    a single trend.

    Parameters
    ----------
    data_path : str
        Path to the fixed-width ASCII data file (PSID calls this the
        ``.txt`` file in a Data Center download).
    do_path : str
        Path to the accompanying Stata ``.do`` file from the same
        download; only its ``infix`` column-position block is read.

    Returns
    -------
    pandas.DataFrame
        Long format, one row per (household, wave actually observed for
        that household's Head). Columns: ``household_key``, ``year``,
        ``total_giving``, one ``giving_<category>`` column per category
        name used anywhere in ``_WAVES`` (``NaN`` in waves that did not ask
        that category), ``itemized_charitable_contrib_amount`` (PSID
        G102A, tax itemizers only), ``family_income``, ``wealth1``
        (without home equity), ``wealth2`` (with home equity),
        ``head_volunteer_hours_annual``, ``spouse_volunteer_hours_annual``,
        ``household_volunteer_hours_regular`` (2003 and 2005 only),
        ``head_volunteer_hours_typical_week``,
        ``spouse_volunteer_hours_typical_week``.

    Notes
    -----
    Source: Panel Study of Income Dynamics, public use dataset, produced
    and distributed by the Survey Research Center, Institute for Social
    Research, University of Michigan, Ann Arbor, MI.
    """
    spans = _parse_infix(do_path)

    category_names: List[str] = sorted(
        {cat for wave in _WAVES for cat in wave["giving"]}
    )

    needed_vars = set(_ID_VARS)
    for wave in _WAVES:
        needed_vars.add(wave["relation_var"])
        for key in ("income_var", "wealth1_var", "wealth2_var", "itemized_var",
                    "volunteer_head_hours_var", "volunteer_spouse_hours_var"):
            if wave[key] is not None:
                needed_vars.add(wave[key])
        needed_vars.update(wave["giving"].values())
        needed_vars.update(wave.get("regular_org_hours_vars", []))

    ordered_vars = [v for v in needed_vars if v in spans]
    colspecs = [(spans[v][0] - 1, spans[v][1]) for v in ordered_vars]
    raw = pd.read_fwf(data_path, colspecs=colspecs, names=ordered_vars)

    household_key = raw[_ID_VARS[0]].astype("int64") * 1000 + raw[_ID_VARS[1]].astype("int64")

    frames = []
    for wave in _WAVES:
        if wave["relation_var"] not in raw.columns:
            # This wave wasn't included in the caller's Data Center job
            # (a smaller variable selection than the one this reader was
            # developed against); nothing to read for it.
            continue
        is_head = raw[wave["relation_var"]] == _HEAD_RELATION_CODE
        if not is_head.any():
            continue
        frame = pd.DataFrame({
            "household_key": household_key[is_head].to_numpy(),
            "year": wave["year"],
        })

        def _amount(var: Optional[str], recode: bool = True) -> Any:
            if var is None:
                return float("nan")
            col = raw.loc[is_head, var].astype("float64")
            if not recode:
                return col.to_numpy()
            width = spans[var][1] - spans[var][0] + 1
            return _recode_missing(col, width).to_numpy()

        for cat in category_names:
            frame[f"giving_{cat}"] = _amount(wave["giving"].get(cat))
        frame["total_giving"] = frame[[f"giving_{c}" for c in category_names]].sum(
            axis=1, skipna=True, min_count=1
        )
        frame["itemized_charitable_contrib_amount"] = _amount(wave["itemized_var"])
        # family_income/wealth1/wealth2 are PSID-generated/imputed totals,
        # not raw survey responses; direct inspection of the real data
        # found zero occurrences of the 9s-sentinel convention on these
        # fields, so (unlike giving/itemized/hours) they are left unrecoded.
        frame["family_income"] = _amount(wave["income_var"], recode=False)
        frame["wealth1"] = _amount(wave["wealth1_var"], recode=False)
        frame["wealth2"] = _amount(wave["wealth2_var"], recode=False)

        for who, var_key in (("head", "volunteer_head_hours_var"), ("spouse", "volunteer_spouse_hours_var")):
            annual_col = f"{who}_volunteer_hours_annual"
            typical_col = f"{who}_volunteer_hours_typical_week"
            if wave["hours_kind"] == "annual":
                frame[annual_col] = _amount(wave[var_key])
                frame[typical_col] = float("nan")
            elif wave["hours_kind"] == "typical_week":
                frame[annual_col] = float("nan")
                frame[typical_col] = _amount(wave[var_key])
            else:
                frame[annual_col] = float("nan")
                frame[typical_col] = float("nan")

        regular_vars = wave.get("regular_org_hours_vars")
        if regular_vars:
            slots = pd.DataFrame({v: _amount(v) for v in regular_vars})
            frame["household_volunteer_hours_regular"] = slots.sum(
                axis=1, skipna=True, min_count=1
            ).to_numpy()
        else:
            frame["household_volunteer_hours_regular"] = float("nan")

        frames.append(frame)

    result = pd.concat(frames, ignore_index=True)
    return result.sort_values(["household_key", "year"]).reset_index(drop=True)