pysare.evaluation

The following evaluation metrics are available:

pysare.evaluation.concordance_index(model: TorchSurvivalModel | NumpySurvivalModel, data: Tuple[X, T, E] | Dataset | DataLoader, censoring: Literal['KM', 'none'] | FrozenNumpySurvivalModel = 'KM', plot: bool | Axes | Iterable[Axes] = False) → DataFrame

Concordance index of a model on a dataset.

Calculates the censoring-corrected truncated time-dependent concordance index (C-index), where:

  • Time-dependent refers to that no PH assumption is made and instead the comparison is done based on the value of the predicted survival function at the time of one of the event times that are compared.

  • Censoring-corrected refers to that inverse probability weights are used to make the result less dependent on the censoring distribution

  • Truncated refers to that the result is truncated up to a specific time so that only data points with event times before this time are compared (the value at the largest truncation time corresponds to the conventional index).

See references [1] and [2] for more information.

Parameters:

  • model (SurvivalModel or NumpySurvivalModel:) – Model to be evaluated. If data is a tuple (X, T, E) of non-tensors, the model should be a NumpySurvivalModel, and otherwise it should be a SurvivalModel.

  • data (tuple(X,T,E) or Dataset or Dataloader:) – The data to be used for evaluation. If data is a tuple (X, T, E) of non-tensors, the model should be a NumpySurvivalModel, and otherwise it should be a SurvivalModel.

  • censoring ('KM' or 'none' or FrozenNumpySurvivalModel, default='KM') – Determines how the censoring distribution is calculated. If ‘KM’ it is estimated using the Kaplan-Meier estimate of the data, if ‘None’ no censoring is use, and if it is a FrozenNumpySurvivalModel this model is used.

  • plot (bool or Axis or Iterable[Axes], default=False) – If True, a plot of the result (index and censoring) will be created in a new figure; if one or two Axes are passed, the plots will be made in these.

Returns:

result – A pandas DataFrame with the result.

Return type:

DataFrame

References

[1] Antolini, L., Boracchi, P. and Biganzoli, E. (2005), A time-dependent discrimination index for survival data. Statist. https://doi.org/10.1002/sim.2427

[2] Uno, H., Cai, T., Pencina, M.J., D’Agostino, R.B. and Wei, L.J. (2011), On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Statist. Med., 30: 1105-1117. https://doi.org/10.1002/sim.4154

pysare.evaluation.negative_log_likelihood(model: TorchSurvivalModel | NumpySurvivalModel, data: Tuple[X, T, E] | Dataset | DataLoader, plot: bool | Axes | Iterable[Axes] = False) → DataFrame

Negative log-likelihood of a model on a dataset

Calculates the truncated negative log-likelihood for a model on a dataset. Truncated refers to that the mean log-likelihood of all event times up to a specific time is calculated, the last value thus corresponds to the non-truncated value.

Parameters:

  • model (SurvivalModel or NumpySurvivalModel) – Model to be evaluated. If data is a tuple (X, T, E) of non-tensors, the model should be a NumpySurvivalModel, and otherwise it should be a SurvivalModel.

  • data (tuple(X,T,E) or Dataset or Dataloader) – The data to be used for evaluation. If data is a tuple (X, T, E) of non-tensors, the model should be a NumpySurvivalModel, and otherwise it should be a SurvivalModel.

  • plot (bool or Axis, default=False) – If True, a plot of the result will be created in a new figure; if an Axes is passed, the plots will be made in this.

Returns:

result – A pandas DataFrame with the result.

Return type:

DataFrame

pysare.evaluation.brier_score(model: TorchSurvivalModel | NumpySurvivalModel, data: Tuple[X, T, E] | Dataset | DataLoader, times: T | int, integrated: bool = False, censoring: Literal['KM', 'none'] | FrozenNumpySurvivalModel = 'KM', plot: bool | Axes = False) → DataFrame

Brier and integrated Brier score

Calculates the brier score and integrated brier score. Also considers censoring, see [1] for more information.

Parameters:

  • model (SurvivalModel or NumpySurvivalModel) – Model to be evaluated. If data is a tuple (X, T, E) of non-tensors, the model should be a NumpySurvivalModel, and otherwise it should be a SurvivalModel.

  • data (tuple(X,T,E) or Dataset or Dataloader) – The data to be used for evaluation. If data is a tuple (X, T, E) of non-tensors, the model should be a NumpySurvivalModel, and otherwise it should be a SurvivalModel.

  • times (int or (n,) array_like) – Times where the score is evaluated. Should have same data type as the times (T) in the data parameter. If int, this number of linearly spaced times between zero and largest time in dataset is used. Times where the survival function of the censoring distribution is zero are removed to avoid NaN in the output.

  • integrated (bool, default=False) – Indicates if the integrated score should be calculated, False by default.

  • censoring ({'KM', 'none'} or FrozenNumpySurvivalModel, default='KM') – Determines how the censoring distribution is calculated. If ‘KM’ it is estimated using Kaplan-Meier estimator using the data, if ‘None’ no censoring is use, and if it is a FrozenNumpySurvivalModel this model is used.

  • plot (bool or Axis, default=False) – If True, a plot of the result will be created after the calculation, and if an Axis is passed the plot wil be done in this axis. If False (default), no plot is made.

Returns:

result – A pandas DataFrame with the result.

Return type:

DataFrame

References

[1] Graf, Erika, et al. “Assessment and comparison of prognostic classification schemes for survival data.” Statistics in medicine 18.17‐18 (1999): 2529-2545.

pysare.evaluation.binomial_log_likelihood(model: TorchSurvivalModel | NumpySurvivalModel, data: Tuple[X, T, E] | Dataset | DataLoader, times: T | int, integrated: bool = False, censoring: Literal['KM', 'none'] | FrozenNumpySurvivalModel = 'KM', plot: bool | Axes = False) → DataFrame

Binomial log-likelihood

Calculates the binomial log-likelihood score and its integrated counterpart. Also considers censoring, see [1] for more information.

Parameters:

  • model (SurvivalModel or NumpySurvivalModel) – Model to be evaluated. If data is a tuple (X, T, E) of non-tensors, the model should be a NumpySurvivalModel, and otherwise it should be a SurvivalModel.

  • data (tuple(X,T,E) or Dataset or Dataloader) – The data to be used for evaluation. If data is a tuple (X, T, E) of non-tensors, the model should be a NumpySurvivalModel, and otherwise it should be a SurvivalModel.

  • times (int or (n,) array_like) – Times where the score is evaluated. Should have same data type as the times (T) in the data parameter. If int, this number of linearly spaced times between zero and largest time in dataset is used. Times where the survival function of the censoring distribution is zero are removed to avoid NaN in the output.

  • integrated (bool, default=False) – Indicates if the integrated score should be calculated, False by default.

  • censoring ({'KM' , 'none'} or FrozenNumpySurvivalModel, default='KM') – Determines how the censoring distribution is calculated. If ‘KM’ it is estimated using Kaplan-Meier estimator using the data, if ‘None’ no censoring is use, and if it is a FrozenNumpySurvivalModel this model is used.

  • plot (bool or Axis, default=False) – If True, a plot of the result will be created after the calculation, and if an Axis is passed the plot wil be done in this axis. If False (default), no plot is made.

Returns:

result – A pandas DataFrame with the result.

Return type:

DataFrame

References

[1] Graf, Erika, et al. “Assessment and comparison of prognostic classification schemes for survival data.” Statistics in medicine 18.17‐18 (1999): 2529-2545.