Model Selection Reference

philanthropy.model_selection

Fiscal-year–aware cross-validation for donor analytics, and the per-file model-vs-rule check.

FiscalYearGroupedSplitter

Bases: BaseCrossValidator

Walk-forward fiscal-year cross-validator for donor analytics.

This cross-validator implements a temporal expanding-window strategy that respects fiscal-year boundaries. Unlike standard :class:KFold, it never allows future data to appear in a training fold.

In each split i (0-indexed):

• Train: all rows whose fiscal year is among the i earliest distinct fiscal years present in groups.
• Test: all rows whose fiscal year is the (i+1)-th earliest fiscal year in groups.

This expands the training window by one fiscal year for each split, mirroring how a fundraising team would retrain their model at the end of each fiscal year using all prior history.

Parameters:

Name	Type	Description	Default
n_splits	int	Number of cross-validation folds. Must be >= 1 and <= n_distinct_fy - 1 (you cannot test on the first fiscal year as there is no prior training data).	5
gap_years	int	Number of fiscal years to exclude between train and test as a prophylactic leakage buffer. For example, if gap_years=1, the fiscal year immediately before the test year is withheld from training (useful when gift officers use current-year pipeline intelligence that would not have been available historically).	0
drop_repeat_donors	bool	.. versionchanged:: 0.8.0 The default changed from False to True, after 0.7.0 and 0.7.1 warned about it. Code that relied on the old default and passes one-dimensional groups now raises a ValueError asking for either donor ids or drop_repeat_donors=False. Whether to remove from each test fold any donor who already appears in that fold's training rows. Keep it True for a static per-donor label such as is_major_donor, where the same answer is attached to every one of that donor's rows, so the model can memorise it from the donor's earlier years. That is the leakage described under "What this does not prevent" below, and the reason this is the default. Set it False for a time-varying target such as "did this donor give in FY22?". There, a donor appearing in both folds is correct: the training rows precede the test rows in time, which is the point of walk-forward evaluation. When True, groups must be two-dimensional with shape (n_samples, 2): column 0 the fiscal year, column 1 the donor identifier. A :class:pandas.DataFrame with those two columns in that order works. It is not free. Donors active in both windows leave the test fold, so it shrinks and the donors remaining are systematically newer to the file. split emits a UserWarning with the number of rows removed, so the cost is visible rather than silent. A test fold emptied entirely raises rather than being skipped, because silently changing the fold count would put split and get_n_splits back out of step.	True

Raises:

Type	Description
ValueError	If n_splits < 1.
ValueError	During :meth:split if groups is None (fiscal year labels are required).
ValueError	During :meth:split if the number of distinct fiscal years is insufficient for the requested number of splits.

Examples:

>>> import numpy as np
>>> from philanthropy.model_selection import FiscalYearGroupedSplitter
>>> X = np.zeros((200, 5))
>>> fy = np.array([2018]*40 + [2019]*50 + [2020]*55 + [2021]*30 + [2022]*25)
>>> splitter = FiscalYearGroupedSplitter(n_splits=3, gap_years=0, drop_repeat_donors=False)
>>> for train_idx, test_idx in splitter.split(X, groups=fy):
...     train_fy = np.unique(fy[train_idx])
...     test_fy  = np.unique(fy[test_idx])
...     assert train_fy.max() < test_fy.min(), "No leakage"
>>> splitter.get_n_splits()
3

Integration with cross_val_score:

>>> from sklearn.model_selection import cross_val_score
>>> from sklearn.dummy import DummyClassifier
>>> y = np.random.randint(0, 2, 200)
>>> scores = cross_val_score(
...     DummyClassifier(), X, y,
...     cv=splitter,
...     groups=fy,
...     scoring="roc_auc",
... )
>>> len(scores) == 3
True

Notes

Why not TimeSeriesSplit? :class:sklearn.model_selection.TimeSeriesSplit splits on row index, not on a semantic grouping variable. Donor datasets are rarely sorted by date, and donors may have multiple rows (one per gift). FiscalYearGroupedSplitter uses groups to correctly assign all gifts from a given fiscal year to the same fold regardless of row order.

groups parameter convention: Pass groups as an integer array of fiscal years (e.g., fiscal_years = df["fiscal_year"].to_numpy()). The splitter sorts distinct values numerically and walks forward.

What this does not prevent. The grouping unit is the fiscal year, not the donor. A donor with gifts in several fiscal years therefore appears in both the training and the test fold of the same split, in different rows. That is correct and intended for a time-varying target ("did this donor give in FY22?"), because the training rows precede the test rows. It is leakage for a static per-donor label such as is_major_donor, where the same answer is attached to every one of that donor's rows and the model can memorise it from the training years.

For that case, keep the default drop_repeat_donors=True and pass groups as (n_samples, 2) with the donor identifier in column 1. Each test fold then excludes donors already present in its training rows. Aggregating to one row per donor and using a grouped holdout remains the cleaner option when the label has no time dimension at all.

See Also

sklearn.model_selection.TimeSeriesSplit : Purely index-based time series CV (does not understand fiscal years or grouping). philanthropy.preprocessing.FiscalYearTransformer : Use this first to compute the fiscal_year column from raw gift dates.

Source code in philanthropy/model_selection/_temporal_donor_splitter.py

class FiscalYearGroupedSplitter(BaseCrossValidator):
    """Walk-forward fiscal-year cross-validator for donor analytics.

    This cross-validator implements a **temporal expanding-window** strategy
    that respects fiscal-year boundaries.  Unlike standard :class:`KFold`,
    it never allows future data to appear in a training fold.

    In each split ``i`` (0-indexed):

    * **Train**: all rows whose fiscal year is among the ``i`` earliest
      distinct fiscal years present in ``groups``.
    * **Test**: all rows whose fiscal year is the ``(i+1)``-th earliest
      fiscal year in ``groups``.

    This expands the training window by one fiscal year for each split,
    mirroring how a fundraising team would retrain their model at the end of
    each fiscal year using all prior history.

    Parameters
    ----------
    n_splits : int, default=5
        Number of cross-validation folds.  Must be ``>= 1`` and ``<= n_distinct_fy - 1``
        (you cannot test on the *first* fiscal year as there is no prior training data).
    gap_years : int, default=0
        Number of fiscal years to exclude between train and test as a
        **prophylactic leakage buffer**.  For example, if ``gap_years=1``,
        the fiscal year immediately before the test year is withheld from
        training (useful when gift officers use current-year pipeline
        intelligence that would not have been available historically).
    drop_repeat_donors : bool, default=True
        .. versionchanged:: 0.8.0
            The default changed from ``False`` to ``True``, after 0.7.0 and
            0.7.1 warned about it. Code that relied on the old default and
            passes one-dimensional ``groups`` now raises a ``ValueError``
            asking for either donor ids or ``drop_repeat_donors=False``.

        Whether to remove from each test fold any donor who already appears in
        that fold's training rows.

        Keep it ``True`` for a **static per-donor** label such as
        ``is_major_donor``, where the same answer is attached to every one of
        that donor's rows, so the model can memorise it from the donor's earlier
        years. That is the leakage described under "What this does not prevent"
        below, and the reason this is the default.

        Set it ``False`` for a **time-varying** target such as "did this donor
        give in FY22?". There, a donor appearing in both folds is correct: the
        training rows precede the test rows in time, which is the point of
        walk-forward evaluation.

        When ``True``, ``groups`` must be two-dimensional with shape
        ``(n_samples, 2)``: column 0 the fiscal year, column 1 the donor
        identifier. A :class:`pandas.DataFrame` with those two columns in that
        order works.

        It is not free. Donors active in both windows leave the test fold, so it
        shrinks and the donors remaining are systematically newer to the file.
        ``split`` emits a ``UserWarning`` with the number of rows removed, so the
        cost is visible rather than silent. A test fold emptied entirely raises
        rather than being skipped, because silently changing the fold count would
        put ``split`` and ``get_n_splits`` back out of step.

    Raises
    ------
    ValueError
        If ``n_splits < 1``.
    ValueError
        During :meth:`split` if ``groups`` is ``None`` (fiscal year labels
        are required).
    ValueError
        During :meth:`split` if the number of distinct fiscal years is
        insufficient for the requested number of splits.

    Examples
    --------
    >>> import numpy as np
    >>> from philanthropy.model_selection import FiscalYearGroupedSplitter
    >>> X = np.zeros((200, 5))
    >>> fy = np.array([2018]*40 + [2019]*50 + [2020]*55 + [2021]*30 + [2022]*25)
    >>> splitter = FiscalYearGroupedSplitter(n_splits=3, gap_years=0, drop_repeat_donors=False)
    >>> for train_idx, test_idx in splitter.split(X, groups=fy):
    ...     train_fy = np.unique(fy[train_idx])
    ...     test_fy  = np.unique(fy[test_idx])
    ...     assert train_fy.max() < test_fy.min(), "No leakage"
    >>> splitter.get_n_splits()
    3

    **Integration with cross_val_score:**

    >>> from sklearn.model_selection import cross_val_score
    >>> from sklearn.dummy import DummyClassifier
    >>> y = np.random.randint(0, 2, 200)
    >>> scores = cross_val_score(
    ...     DummyClassifier(), X, y,
    ...     cv=splitter,
    ...     groups=fy,
    ...     scoring="roc_auc",
    ... )
    >>> len(scores) == 3
    True

    Notes
    -----
    **Why not TimeSeriesSplit?** :class:`sklearn.model_selection.TimeSeriesSplit`
    splits on row *index*, not on a semantic grouping variable.  Donor
    datasets are rarely sorted by date, and donors may have multiple rows
    (one per gift).  ``FiscalYearGroupedSplitter`` uses ``groups`` to correctly
    assign all gifts from a given fiscal year to the same fold regardless
    of row order.

    **groups parameter convention:** Pass ``groups`` as an integer array of
    fiscal years (e.g., ``fiscal_years = df["fiscal_year"].to_numpy()``).
    The splitter sorts distinct values numerically and walks forward.

    **What this does not prevent.** The grouping unit is the fiscal year, not
    the donor. A donor with gifts in several fiscal years therefore appears in
    both the training and the test fold of the same split, in different rows.
    That is correct and intended for a *time-varying* target ("did this donor
    give in FY22?"), because the training rows precede the test rows. It is
    **leakage** for a *static per-donor* label such as ``is_major_donor``, where
    the same answer is attached to every one of that donor's rows and the model
    can memorise it from the training years.

    For that case, keep the default ``drop_repeat_donors=True`` and pass
    ``groups`` as ``(n_samples, 2)`` with the donor identifier in column 1. Each test fold then
    excludes donors already present in its training rows. Aggregating to one row
    per donor and using a grouped holdout remains the cleaner option when the
    label has no time dimension at all.

    See Also
    --------
    sklearn.model_selection.TimeSeriesSplit :
        Purely index-based time series CV (does not understand fiscal years
        or grouping).
    philanthropy.preprocessing.FiscalYearTransformer :
        Use this first to compute the ``fiscal_year`` column from raw gift dates.
    """

    def __init__(
        self,
        n_splits: int = 5,
        gap_years: int = 0,
        drop_repeat_donors: bool = True,
    ) -> None:
        # MUST call super().__init__() for BaseCrossValidator compat.
        self.n_splits = n_splits
        self.gap_years = gap_years
        self.drop_repeat_donors = drop_repeat_donors

    # ------------------------------------------------------------------
    # Required abstract-method implementations
    # ------------------------------------------------------------------

    def _validate_params(self) -> tuple[int, int]:
        """Return ``(n_splits, gap_years)`` as validated ints.

        ``__init__`` stores raw parameters only, so both ``split`` and
        ``get_n_splits`` validate here. Without this, ``n_splits <= 0`` reached
        the ``unique_fy[-(n_splits):]`` slice, where a non-positive value flips
        the slice open-ended (``unique_fy[-(0):]`` is ``unique_fy[0:]``) and
        ``split`` silently yielded folds while ``get_n_splits`` reported zero or
        a negative count. ``cross_val_score`` sizes its result array from
        ``get_n_splits``, so the two disagreeing is a real failure, not a
        cosmetic one.
        """
        try:
            n_splits = int(self.n_splits)
            gap_years = int(self.gap_years)
        except (TypeError, ValueError):
            raise ValueError(
                "n_splits and gap_years must be integers, got "
                f"n_splits={self.n_splits!r}, gap_years={self.gap_years!r}."
            ) from None
        if n_splits < 1:
            raise ValueError(f"n_splits must be >= 1, got {n_splits}.")
        if gap_years < 0:
            raise ValueError(f"gap_years must be >= 0, got {gap_years}.")
        return n_splits, gap_years

    def split(
        self, X: Any, y: Any = None, groups: Any = None
    ) -> Iterator[Tuple[np.ndarray, np.ndarray]]:
        """Generate (train_indices, test_indices) arrays.

        Parameters
        ----------
        X : array-like of shape (n_samples, n_features)
            Training data.  Only ``X.shape[0]`` (i.e., the number of
            samples) is used; actual feature values are ignored.
        y : array-like of shape (n_samples,), optional
            Target labels.  Ignored; present for sklearn API compatibility.
        groups : array-like, **required**
            Integer fiscal year labels for each sample, shape
            ``(n_samples,)``. When ``drop_repeat_donors=True`` this must instead
            be ``(n_samples, 2)``: fiscal year in column 0, donor identifier in
            column 1.

        Yields
        ------
        train : ndarray of int
            Indices of training samples.
        test : ndarray of int
            Indices of test samples.

        Raises
        ------
        ValueError
            If ``groups`` is ``None``.
        ValueError
            If fewer than ``n_splits + 1`` distinct fiscal years are present.
        ValueError
            If ``drop_repeat_donors=True`` and ``groups`` is not
            ``(n_samples, 2)``, or its fiscal-year column is not numeric.
        ValueError
            If ``drop_repeat_donors=True`` empties a test fold entirely.
        """
        requested_splits, gap_years = self._validate_params()
        drop_repeat = bool(self.drop_repeat_donors)

        if groups is None:
            raise ValueError(
                "FiscalYearGroupedSplitter requires `groups` to be an array of "
                "fiscal year labels (integer per sample).  Pass `groups=` to "
                "`split()` or to `cross_val_score(groups=...)`."
            )

        groups_arr = np.asarray(groups)
        donor_ids = None
        if drop_repeat:
            if groups_arr.ndim != 2 or groups_arr.shape[1] != 2:
                raise ValueError(
                    "drop_repeat_donors=True (the default since 0.8.0) requires "
                    "`groups` with shape (n_samples, 2): column 0 the fiscal "
                    "year, column 1 the donor identifier. Got shape "
                    f"{groups_arr.shape}. For a time-varying target such as "
                    "'gave next year', where a donor belongs in both folds, "
                    "pass drop_repeat_donors=False with fiscal years alone."
                )
            donor_ids = groups_arr[:, 1]
            groups_arr = groups_arr[:, 0]
            # np.column_stack of integer years and string donor ids upcasts
            # everything to '<U21', which silently turns the fiscal years into
            # strings and fails much later with a bare numpy TypeError. Catch it
            # here, where the message can say what to do.
            try:
                groups_arr = groups_arr.astype(float)
            except (TypeError, ValueError):
                raise ValueError(
                    "drop_repeat_donors=True: column 0 of `groups` must be "
                    "numeric fiscal years, got dtype "
                    f"{np.asarray(groups)[:, 0].dtype!r}. If your donor ids are "
                    "strings, np.column_stack upcasts the whole array; pass a "
                    "pandas DataFrame, or factorise the ids to integers first."
                ) from None
        groups = column_or_1d(groups_arr)
        fiscal_years = groups

        n_samples = _n_samples(X)
        if len(groups) != n_samples:
            raise ValueError(
                f"`groups` length ({len(groups)}) must match the number of "
                f"samples in X ({n_samples})."
            )

        # A NaN fiscal year cannot be placed in any fold (every comparison
        # against it is False, so the row silently never trains or tests).
        # Exclude it from the distinct-year count too, or get_n_splits()
        # reports one more fold than split() actually yields.
        nan_fy = _missing_mask(fiscal_years)
        if nan_fy.any():
            warnings.warn(
                f"FiscalYearGroupedSplitter: {int(nan_fy.sum())} row(s) have "
                "a NaN fiscal year and are excluded from every split.",
                UserWarning,
            )
        unique_fy = np.sort(np.unique(fiscal_years[~nan_fy]))
        n_fy = len(unique_fy)

        if n_fy < 2:
            raise ValueError(
                f"FiscalYearGroupedSplitter requires at least 2 distinct fiscal "
                f"years in `groups`, found {n_fy}."
            )

        max_splits = n_fy - 1 - gap_years
        if max_splits < 1:
            raise ValueError(
                f"Not enough fiscal years ({n_fy}) for n_splits={requested_splits} "
                f"with gap_years={gap_years}.  Need at least "
                f"{requested_splits + 1 + gap_years} distinct fiscal years."
            )

        n_splits = min(requested_splits, max_splits)

        # Walk from the most-recent test year backward to get exactly n_splits folds.
        # Each fold tests a consecutive fiscal year, training on *all* prior years.
        indices = np.arange(n_samples)
        test_fy_sequence = unique_fy[-(n_splits):]  # Last n_splits fiscal years as test sets

        dropped_total = 0
        for test_fy in test_fy_sequence:
            test_mask = fiscal_years == test_fy

            # Training: all FYs strictly before (test_fy - gap_years)
            train_cutoff_fy = test_fy - gap_years
            train_mask = fiscal_years < train_cutoff_fy

            if not np.any(train_mask):
                # No training data before this test year, skip
                continue

            if donor_ids is not None:
                # A static per-donor label leaks through any donor present in
                # both folds. Drop those donors from the TEST side only: pulling
                # them out of training would discard history for no benefit.
                seen = np.isin(donor_ids, np.unique(donor_ids[train_mask]))
                # np.isin never matches NaN to NaN, so a row with a missing donor
                # id would be classified as unseen and kept. That is the wrong
                # default for a leakage guard: an unidentifiable donor cannot be
                # shown to be absent from training, so treat it as seen.
                unidentified = _missing_mask(donor_ids)
                seen = seen | unidentified
                n_dropped = int(np.count_nonzero(test_mask & seen))
                if n_dropped:
                    test_mask = test_mask & ~seen
                    dropped_total += n_dropped
                if n_dropped:
                    warnings.warn(
                        f"drop_repeat_donors=True removed {n_dropped} test "
                        f"row(s) from fiscal year {test_fy} whose donor already "
                        "appeared in training (or had no donor id). The test "
                        "donors that remain are systematically newer to the "
                        "file, so scores are not directly comparable to a run "
                        "without this flag.",
                        UserWarning,
                    )
                if not np.any(test_mask):
                    raise ValueError(
                        f"drop_repeat_donors=True emptied the test fold for "
                        f"fiscal year {test_fy}: every donor in it also appears "
                        "in the training years. Either the label is static per "
                        "donor and this splitter cannot help (aggregate to one "
                        "row per donor and use a grouped holdout), or the label "
                        "is time-varying and this flag is not needed."
                    )

            yield (
                indices[train_mask],
                indices[test_mask],
            )



    def get_n_splits(
        self, X: Any = None, y: Any = None, groups: Any = None
    ) -> int:
        """Return the number of splits this splitter will produce.

        Parameters
        ----------
        X, y, groups : ignored when no ``groups`` is given.
            When ``groups`` is provided, the **actual** number of splits
            (which may be less than ``self.n_splits`` if there are fewer
            than ``n_splits + 1`` distinct fiscal years) is returned.
        """
        n_splits, gap_years = self._validate_params()
        if groups is not None:
            groups = np.asarray(groups)
            drop_repeat = bool(self.drop_repeat_donors)
            if drop_repeat and groups.ndim == 2 and groups.shape[1] == 2:
                groups = groups[:, 0]
            unique_fy = np.unique(groups[~_missing_mask(groups)])
            n_fy = len(unique_fy)
            max_splits = max(0, n_fy - 1 - gap_years)
            n_splits = min(n_splits, max_splits)
            if n_splits == 0:
                return 0
            try:
                unique_fy = np.sort(unique_fy.astype(float))
            except (TypeError, ValueError):
                # Non-numeric years: split() raises the descriptive error.
                return n_splits
            # split() skips a test year with no training rows before its
            # cutoff (possible with fractional fiscal years), so don't count it.
            return int(np.count_nonzero(unique_fy[-n_splits:] - gap_years > unique_fy[0]))
        return n_splits

    # ------------------------------------------------------------------
    # sklearn clone safety: all params must be in __init__
    # ------------------------------------------------------------------

    def __repr__(self) -> str:
        return (
            f"{self.__class__.__name__}("
            f"n_splits={self.n_splits}, "
            f"gap_years={self.gap_years}, "
            f"drop_repeat_donors={self.drop_repeat_donors!r})"
        )

split(X, y=None, groups=None)

Generate (train_indices, test_indices) arrays.

Parameters:

Name	Type	Description	Default
X	array-like of shape (n_samples, n_features)	Training data. Only X.shape[0] (i.e., the number of samples) is used; actual feature values are ignored.	required
y	array-like of shape (n_samples,)	Target labels. Ignored; present for sklearn API compatibility.	None
groups	array-like, **required**	Integer fiscal year labels for each sample, shape (n_samples,). When drop_repeat_donors=True this must instead be (n_samples, 2): fiscal year in column 0, donor identifier in column 1.	None

Yields:

Name	Type	Description
train	ndarray of int	Indices of training samples.
test	ndarray of int	Indices of test samples.

Raises:

Type	Description
ValueError	If groups is None.
ValueError	If fewer than n_splits + 1 distinct fiscal years are present.
ValueError	If drop_repeat_donors=True and groups is not (n_samples, 2), or its fiscal-year column is not numeric.
ValueError	If drop_repeat_donors=True empties a test fold entirely.

Source code in philanthropy/model_selection/_temporal_donor_splitter.py

def split(
    self, X: Any, y: Any = None, groups: Any = None
) -> Iterator[Tuple[np.ndarray, np.ndarray]]:
    """Generate (train_indices, test_indices) arrays.

    Parameters
    ----------
    X : array-like of shape (n_samples, n_features)
        Training data.  Only ``X.shape[0]`` (i.e., the number of
        samples) is used; actual feature values are ignored.
    y : array-like of shape (n_samples,), optional
        Target labels.  Ignored; present for sklearn API compatibility.
    groups : array-like, **required**
        Integer fiscal year labels for each sample, shape
        ``(n_samples,)``. When ``drop_repeat_donors=True`` this must instead
        be ``(n_samples, 2)``: fiscal year in column 0, donor identifier in
        column 1.

    Yields
    ------
    train : ndarray of int
        Indices of training samples.
    test : ndarray of int
        Indices of test samples.

    Raises
    ------
    ValueError
        If ``groups`` is ``None``.
    ValueError
        If fewer than ``n_splits + 1`` distinct fiscal years are present.
    ValueError
        If ``drop_repeat_donors=True`` and ``groups`` is not
        ``(n_samples, 2)``, or its fiscal-year column is not numeric.
    ValueError
        If ``drop_repeat_donors=True`` empties a test fold entirely.
    """
    requested_splits, gap_years = self._validate_params()
    drop_repeat = bool(self.drop_repeat_donors)

    if groups is None:
        raise ValueError(
            "FiscalYearGroupedSplitter requires `groups` to be an array of "
            "fiscal year labels (integer per sample).  Pass `groups=` to "
            "`split()` or to `cross_val_score(groups=...)`."
        )

    groups_arr = np.asarray(groups)
    donor_ids = None
    if drop_repeat:
        if groups_arr.ndim != 2 or groups_arr.shape[1] != 2:
            raise ValueError(
                "drop_repeat_donors=True (the default since 0.8.0) requires "
                "`groups` with shape (n_samples, 2): column 0 the fiscal "
                "year, column 1 the donor identifier. Got shape "
                f"{groups_arr.shape}. For a time-varying target such as "
                "'gave next year', where a donor belongs in both folds, "
                "pass drop_repeat_donors=False with fiscal years alone."
            )
        donor_ids = groups_arr[:, 1]
        groups_arr = groups_arr[:, 0]
        # np.column_stack of integer years and string donor ids upcasts
        # everything to '<U21', which silently turns the fiscal years into
        # strings and fails much later with a bare numpy TypeError. Catch it
        # here, where the message can say what to do.
        try:
            groups_arr = groups_arr.astype(float)
        except (TypeError, ValueError):
            raise ValueError(
                "drop_repeat_donors=True: column 0 of `groups` must be "
                "numeric fiscal years, got dtype "
                f"{np.asarray(groups)[:, 0].dtype!r}. If your donor ids are "
                "strings, np.column_stack upcasts the whole array; pass a "
                "pandas DataFrame, or factorise the ids to integers first."
            ) from None
    groups = column_or_1d(groups_arr)
    fiscal_years = groups

    n_samples = _n_samples(X)
    if len(groups) != n_samples:
        raise ValueError(
            f"`groups` length ({len(groups)}) must match the number of "
            f"samples in X ({n_samples})."
        )

    # A NaN fiscal year cannot be placed in any fold (every comparison
    # against it is False, so the row silently never trains or tests).
    # Exclude it from the distinct-year count too, or get_n_splits()
    # reports one more fold than split() actually yields.
    nan_fy = _missing_mask(fiscal_years)
    if nan_fy.any():
        warnings.warn(
            f"FiscalYearGroupedSplitter: {int(nan_fy.sum())} row(s) have "
            "a NaN fiscal year and are excluded from every split.",
            UserWarning,
        )
    unique_fy = np.sort(np.unique(fiscal_years[~nan_fy]))
    n_fy = len(unique_fy)

    if n_fy < 2:
        raise ValueError(
            f"FiscalYearGroupedSplitter requires at least 2 distinct fiscal "
            f"years in `groups`, found {n_fy}."
        )

    max_splits = n_fy - 1 - gap_years
    if max_splits < 1:
        raise ValueError(
            f"Not enough fiscal years ({n_fy}) for n_splits={requested_splits} "
            f"with gap_years={gap_years}.  Need at least "
            f"{requested_splits + 1 + gap_years} distinct fiscal years."
        )

    n_splits = min(requested_splits, max_splits)

    # Walk from the most-recent test year backward to get exactly n_splits folds.
    # Each fold tests a consecutive fiscal year, training on *all* prior years.
    indices = np.arange(n_samples)
    test_fy_sequence = unique_fy[-(n_splits):]  # Last n_splits fiscal years as test sets

    dropped_total = 0
    for test_fy in test_fy_sequence:
        test_mask = fiscal_years == test_fy

        # Training: all FYs strictly before (test_fy - gap_years)
        train_cutoff_fy = test_fy - gap_years
        train_mask = fiscal_years < train_cutoff_fy

        if not np.any(train_mask):
            # No training data before this test year, skip
            continue

        if donor_ids is not None:
            # A static per-donor label leaks through any donor present in
            # both folds. Drop those donors from the TEST side only: pulling
            # them out of training would discard history for no benefit.
            seen = np.isin(donor_ids, np.unique(donor_ids[train_mask]))
            # np.isin never matches NaN to NaN, so a row with a missing donor
            # id would be classified as unseen and kept. That is the wrong
            # default for a leakage guard: an unidentifiable donor cannot be
            # shown to be absent from training, so treat it as seen.
            unidentified = _missing_mask(donor_ids)
            seen = seen | unidentified
            n_dropped = int(np.count_nonzero(test_mask & seen))
            if n_dropped:
                test_mask = test_mask & ~seen
                dropped_total += n_dropped
            if n_dropped:
                warnings.warn(
                    f"drop_repeat_donors=True removed {n_dropped} test "
                    f"row(s) from fiscal year {test_fy} whose donor already "
                    "appeared in training (or had no donor id). The test "
                    "donors that remain are systematically newer to the "
                    "file, so scores are not directly comparable to a run "
                    "without this flag.",
                    UserWarning,
                )
            if not np.any(test_mask):
                raise ValueError(
                    f"drop_repeat_donors=True emptied the test fold for "
                    f"fiscal year {test_fy}: every donor in it also appears "
                    "in the training years. Either the label is static per "
                    "donor and this splitter cannot help (aggregate to one "
                    "row per donor and use a grouped holdout), or the label "
                    "is time-varying and this flag is not needed."
                )

        yield (
            indices[train_mask],
            indices[test_mask],
        )

get_n_splits(X=None, y=None, groups=None)

Return the number of splits this splitter will produce.

Parameters:

Name	Type	Description	Default
X	ignored when no ``groups`` is given.	When groups is provided, the actual number of splits (which may be less than self.n_splits if there are fewer than n_splits + 1 distinct fiscal years) is returned.	None
y	ignored when no ``groups`` is given.	When groups is provided, the actual number of splits (which may be less than self.n_splits if there are fewer than n_splits + 1 distinct fiscal years) is returned.	None
groups	ignored when no ``groups`` is given.	When groups is provided, the actual number of splits (which may be less than self.n_splits if there are fewer than n_splits + 1 distinct fiscal years) is returned.	None

Source code in philanthropy/model_selection/_temporal_donor_splitter.py

def get_n_splits(
    self, X: Any = None, y: Any = None, groups: Any = None
) -> int:
    """Return the number of splits this splitter will produce.

    Parameters
    ----------
    X, y, groups : ignored when no ``groups`` is given.
        When ``groups`` is provided, the **actual** number of splits
        (which may be less than ``self.n_splits`` if there are fewer
        than ``n_splits + 1`` distinct fiscal years) is returned.
    """
    n_splits, gap_years = self._validate_params()
    if groups is not None:
        groups = np.asarray(groups)
        drop_repeat = bool(self.drop_repeat_donors)
        if drop_repeat and groups.ndim == 2 and groups.shape[1] == 2:
            groups = groups[:, 0]
        unique_fy = np.unique(groups[~_missing_mask(groups)])
        n_fy = len(unique_fy)
        max_splits = max(0, n_fy - 1 - gap_years)
        n_splits = min(n_splits, max_splits)
        if n_splits == 0:
            return 0
        try:
            unique_fy = np.sort(unique_fy.astype(float))
        except (TypeError, ValueError):
            # Non-numeric years: split() raises the descriptive error.
            return n_splits
        # split() skips a test year with no training rows before its
        # cutoff (possible with fractional fiscal years), so don't count it.
        return int(np.count_nonzero(unique_fy[-n_splits:] - gap_years > unique_fy[0]))
    return n_splits

compare_to_rule(gifts, *, task, as_of=None, fiscal_year_start=7, threshold=1000.0, band=(100.0, 999.0), n_bootstrap=1000, random_state=0)

Decide, on one gift log, whether the model or the simple rule should rank the donors for task.

The check is the one the Results pages run, applied to the caller's own file: :func:~philanthropy.ingest.build_snapshots builds one row per (donor, fiscal year T) with features from gifts dated at or before the end of T and a label read from T+1. Every labelled fiscal year but the last trains the model; the last one is the held-out test year. On that year the model is scored against the best of a short list of rules fixed in code before any run, on the top-10% hit rate (ask: mean absolute error), and the verdict is read off a 95% bootstrap interval of model - rule over the same test donors. Only a whole interval on the model's side picks the model; a tie goes to the rule, which is free and easy to explain.

Parameters:

Name	Type	Description	Default
gifts	iterable of mapping, or DataFrame	Gift-level rows with donor_id, gift_date and gift_amount.	required
task	('upgrade', 'lapse', 'ask')	Which question: will the donor reach threshold next year (from inside band), stop giving, or how much will they give next year given that they do.	"upgrade"
as_of	str or Timestamp	Gifts after this date are ignored, and only fiscal years whose following year is complete by as_of can carry a label, so a partial current year never masquerades as "gave nothing". Defaults to the latest gift date in gifts.	None
fiscal_year_start	int	Month (1-12) the fiscal year begins.	7
threshold	(float, (float, float))	The upgrade question's leadership level and candidate band; ignored by the other tasks.	1000.0
band	(float, (float, float))	The upgrade question's leadership level and candidate band; ignored by the other tasks.	1000.0
n_bootstrap	int	Resamples behind the interval.	1000
random_state	int	Seed for the model fit and the bootstrap.	0

Returns:

Type

Description

dict

task; method ("model" or "rule"); verdict, one of "Beats the rule", "About the same as the rule", "Loses to the rule", "not enough labels" or "can't test (reason)", the words the Results pages use; rule, the name of the best rule on the test year; n_rules_tried; test_fiscal_year, n_train, n_test; metric ("top10pct_hit_rate" or "mae"); model_value and rule_value on the test year (None for the model when it was not fit); difference_ci, the (low, high) interval on model - rule (None when the model was not fit); and label_floor, :func:~philanthropy.utils.check_label_floor's answer on the training labels (None for ask, which has no floor and instead needs 20 training and 5 test rows).

Raises:

Type	Description
ValueError	If task is unknown or fiscal_year_start is not 1-12.
KeyError	If gifts is missing a required column.

Examples:

>>> from philanthropy.datasets import make_donor_panel
>>> from philanthropy.model_selection import compare_to_rule
>>> gifts = make_donor_panel(n_donors=400, n_years=5, random_state=0)["gifts"]
>>> out = compare_to_rule(gifts, task="lapse", n_bootstrap=50)
>>> out["method"] in ("model", "rule")
True
>>> out["n_rules_tried"]
3

Source code in philanthropy/model_selection/_rule_check.py

def compare_to_rule(
    gifts: Union[Iterable[Mapping], pd.DataFrame],
    *,
    task: str,
    as_of: Optional[Union[str, pd.Timestamp]] = None,
    fiscal_year_start: int = 7,
    threshold: float = 1000.0,
    band: Tuple[float, float] = (100.0, 999.0),
    n_bootstrap: int = 1000,
    random_state: int = 0,
) -> Dict[str, Any]:
    """Decide, on one gift log, whether the model or the simple rule should
    rank the donors for ``task``.

    The check is the one the Results pages run, applied to the caller's own
    file: :func:`~philanthropy.ingest.build_snapshots` builds one row per
    ``(donor, fiscal year T)`` with features from gifts dated at or before
    the end of T and a label read from T+1. Every labelled fiscal year but
    the last trains the model; the last one is the held-out test year. On
    that year the model is scored against the best of a short list of rules
    fixed in code before any run, on the top-10% hit rate (ask: mean
    absolute error), and the verdict is read off a 95% bootstrap interval
    of ``model - rule`` over the same test donors. Only a whole interval on
    the model's side picks the model; a tie goes to the rule, which is free
    and easy to explain.

    Parameters
    ----------
    gifts : iterable of mapping, or DataFrame
        Gift-level rows with ``donor_id``, ``gift_date`` and ``gift_amount``.
    task : {"upgrade", "lapse", "ask"}
        Which question: will the donor reach ``threshold`` next year (from
        inside ``band``), stop giving, or how much will they give next year
        given that they do.
    as_of : str or Timestamp, optional
        Gifts after this date are ignored, and only fiscal years whose
        following year is complete by ``as_of`` can carry a label, so a
        partial current year never masquerades as "gave nothing". Defaults
        to the latest gift date in ``gifts``.
    fiscal_year_start : int, default=7
        Month (1-12) the fiscal year begins.
    threshold, band : float, (float, float)
        The upgrade question's leadership level and candidate band; ignored
        by the other tasks.
    n_bootstrap : int, default=1000
        Resamples behind the interval.
    random_state : int, default=0
        Seed for the model fit and the bootstrap.

    Returns
    -------
    dict
        ``task``; ``method`` (``"model"`` or ``"rule"``); ``verdict``, one of
        ``"Beats the rule"``, ``"About the same as the rule"``,
        ``"Loses to the rule"``, ``"not enough labels"`` or ``"can't test
        (reason)"``, the words the Results pages use; ``rule``, the name of
        the best rule on the test year; ``n_rules_tried``;
        ``test_fiscal_year``, ``n_train``, ``n_test``; ``metric``
        (``"top10pct_hit_rate"`` or ``"mae"``); ``model_value`` and
        ``rule_value`` on the test year (``None`` for the model when it was
        not fit); ``difference_ci``, the ``(low, high)`` interval on
        ``model - rule`` (``None`` when the model was not fit); and
        ``label_floor``, :func:`~philanthropy.utils.check_label_floor`'s
        answer on the training labels (``None`` for ask, which has no
        floor and instead needs 20 training and 5 test rows).

    Raises
    ------
    ValueError
        If ``task`` is unknown or ``fiscal_year_start`` is not 1-12.
    KeyError
        If ``gifts`` is missing a required column.

    Examples
    --------
    >>> from philanthropy.datasets import make_donor_panel
    >>> from philanthropy.model_selection import compare_to_rule
    >>> gifts = make_donor_panel(n_donors=400, n_years=5, random_state=0)["gifts"]
    >>> out = compare_to_rule(gifts, task="lapse", n_bootstrap=50)
    >>> out["method"] in ("model", "rule")
    True
    >>> out["n_rules_tried"]
    3
    """
    if task not in TASKS:
        raise ValueError(f"task must be one of {TASKS}; got {task!r}.")
    validate_fiscal_year_start(fiscal_year_start)

    df = _prepare_gifts(gifts, fiscal_year_start)[0]
    if as_of is None:
        as_of_ts = df["_date"].max() if not df.empty else pd.NaT
    else:
        as_of_ts = pd.Timestamp(as_of)
        df = df[df["_date"] <= as_of_ts]

    report: Dict[str, Any] = {
        "task": task, "method": "rule", "verdict": "", "rule": None, "n_rules_tried": 0,
        "test_fiscal_year": None, "n_train": 0, "n_test": 0,
        "metric": "mae" if task == "ask" else "top10pct_hit_rate",
        "model_value": None, "rule_value": None, "difference_ci": None, "label_floor": None,
    }
    if df.empty:
        report["verdict"] = "can't test (no usable gift rows)"
        return report

    # A fiscal year can only label the one before it once it is complete.
    fy_as_of = fiscal_year_for(int(as_of_ts.year), int(as_of_ts.month), fiscal_year_start)
    last_complete = fy_as_of if as_of_ts >= _fy_end(fy_as_of, fiscal_year_start) else fy_as_of - 1
    snap = build_snapshots(
        df[["donor_id", "_date", "_amount"]].rename(columns={"_date": "gift_date", "_amount": "gift_amount"}),
        kind=_KIND[task],
        fiscal_years=range(int(df["_fy"].min()), last_complete),
        threshold=threshold, band=band,
        min_years_given=_MIN_YEARS_GIVEN[task], fiscal_year_start=fiscal_year_start,
    )
    if snap.empty or snap["period"].nunique() < 2:
        report["verdict"] = "can't test (fewer than two labelled fiscal years)"
        return report

    test_fy = int(snap["period"].max())
    train, test = snap[snap["period"] < test_fy], snap[snap["period"] == test_fy]
    feature_cols = [
        c for c in snap.columns
        if c not in ("period", "target") and pd.api.types.is_numeric_dtype(snap[c])
    ]
    y_train = train["target"].to_numpy("float64")
    y_test = test["target"].to_numpy("float64")
    report.update({"test_fiscal_year": test_fy, "n_train": int(len(train)), "n_test": int(len(test))})

    metric = _mae if task == "ask" else _top_rate
    rules = _rules(task, test, y_train)
    scored = {name: metric(y_test, score) for name, score in rules.items()}
    best = (min if task == "ask" else max)(scored, key=lambda name: scored[name])
    report.update({"rule": best, "n_rules_tried": len(rules), "rule_value": scored[best]})

    if task == "ask":
        enough = len(train) >= 20 and len(test) >= 5
    else:
        report["label_floor"] = check_label_floor(y_train, _LABEL_FLOOR_TASK[task])
        enough = report["label_floor"] == "run"
    if not enough:
        report["verdict"] = "not enough labels"
        return report

    # Imported here: philanthropy.models already imports this subpackage
    # for FiscalYearGroupedSplitter, so a module-level import would cycle.
    from philanthropy.models import (
        AskAmountRecommender, LapsePredictor, MajorGiftClassifier,
    )

    X_train = train[feature_cols].fillna(0.0).to_numpy("float64")
    X_test = test[feature_cols].fillna(0.0).to_numpy("float64")
    if task == "ask":
        model_score = AskAmountRecommender(random_state=random_state).fit(X_train, y_train).predict(X_test)
    elif task == "lapse":
        model_score = LapsePredictor(random_state=random_state).fit(X_train, y_train).predict_lapse_score(X_test)
    else:
        model_score = MajorGiftClassifier(random_state=random_state).fit(X_train, y_train).predict_proba(X_test)[:, 1]
    model_score = np.asarray(model_score, dtype="float64")

    report.update(_verdict(task, y_test, model_score, rules[best], n_bootstrap, random_state))
    return report