Heart Failure Prediction#
This notebook showcases heart failure prediction on a popular Kaggle dataset using CyclOps. The task is formulated as a binary classification task, where we predict the probability that a patient will have heart failure.
Import Libraries#
[1]:
"""Heart failure prediction."""
import copy
import inspect
import shutil
from datetime import date
import numpy as np
import pandas as pd
import plotly.express as px
from datasets import Dataset
from datasets.features import ClassLabel
from kaggle.api.kaggle_api_extended import KaggleApi
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import MinMaxScaler, OneHotEncoder
from cyclops.data.df.feature import TabularFeatures
from cyclops.data.slicer import SliceSpec
from cyclops.evaluate.fairness import FairnessConfig # noqa: E402
from cyclops.evaluate.metrics import create_metric
from cyclops.evaluate.metrics.experimental.functional import (
binary_npv,
binary_ppv,
binary_roc,
)
from cyclops.evaluate.metrics.experimental.metric_dict import MetricDict
from cyclops.models.catalog import create_model
from cyclops.report import ModelCardReport
from cyclops.report.plot.classification import ClassificationPlotter
from cyclops.report.utils import flatten_results_dict
from cyclops.tasks import BinaryTabularClassificationTask
from cyclops.utils.file import join, load_dataframe
/mnt/data/actions_runners/cyclops-actions-runner-1/_work/cyclops/cyclops/.venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
CyclOps offers a package for documentation of the model through a model report. The ModelCardReport class is used to populate and generate the model report as an HTML file. The model report has the following sections:
Overview: Provides a high level overview of how the model is doing (a quick glance of important metrics), and how it is doing over time (performance over several metrics and subgroups over time).
Datasets: High level statistics of the training data, including changes in distribution over time.
Quantitative Analysis: This section contains additional detailed performance metrics of the model for different sets of the data and subpopulations.
Fairness Analysis: This section contains the fairness metrics of the model.
Model Details: This section contains descriptive metadata about the model such as the owners, version, license, etc.
Model Parameters: This section contains the technical details of the model such as the model architecture, training parameters, etc.
Considerations: This section contains descriptions of the considerations involved in developing and using the model such as the intended use, limitations, etc.
We will use this to document the model development process as we go along and generate the model report at the end.
The model report tool is a work in progress and is subject to change.
[2]:
report = ModelCardReport()
Constants#
[3]:
DATA_DIR = "./data"
RANDOM_SEED = 85
NAN_THRESHOLD = 0.75
TRAIN_SIZE = 0.8
Data Loading#
Before starting, make sure to install the Kaggle API by running pip install kaggle. To use the Kaggle API, you need to sign up for a Kaggle account at https://www.kaggle.com. Then go to the âAccountâ tab of your user profile (https://www.kaggle.com/<username>/account) and select âCreate API Tokenâ. This will trigger the download of kaggle.json, a file containing your API credentials. Place this file in the location ~/.kaggle/kaggle.json on your machine.
[4]:
api = KaggleApi()
api.authenticate()
api.dataset_download_files(
"fedesoriano/heart-failure-prediction",
path=DATA_DIR,
unzip=True,
)
[5]:
df = load_dataframe(join(DATA_DIR, "heart.csv"), file_format="csv")
df = df.reset_index()
df["ID"] = df.index
print(df)
2024-07-16 17:02:11,353 INFO cyclops.utils.file - Loading DataFrame from ./data/heart.csv
Age Sex ChestPainType RestingBP Cholesterol FastingBS RestingECG \
0 40 M ATA 140 289 0 Normal
1 49 F NAP 160 180 0 Normal
2 37 M ATA 130 283 0 ST
3 48 F ASY 138 214 0 Normal
4 54 M NAP 150 195 0 Normal
.. ... .. ... ... ... ... ...
913 45 M TA 110 264 0 Normal
914 68 M ASY 144 193 1 Normal
915 57 M ASY 130 131 0 Normal
916 57 F ATA 130 236 0 LVH
917 38 M NAP 138 175 0 Normal
MaxHR ExerciseAngina Oldpeak ST_Slope HeartDisease ID
0 172 N 0.0 Up 0 0
1 156 N 1.0 Flat 1 1
2 98 N 0.0 Up 0 2
3 108 Y 1.5 Flat 1 3
4 122 N 0.0 Up 0 4
.. ... ... ... ... ... ...
913 132 N 1.2 Flat 1 913
914 141 N 3.4 Flat 1 914
915 115 Y 1.2 Flat 1 915
916 174 N 0.0 Flat 1 916
917 173 N 0.0 Up 0 917
[918 rows x 13 columns]
Sex values#
[6]:
fig = px.pie(df, names="Sex")
fig.update_layout(
title="Sex Distribution",
)
fig.show()
Add the figure to the report
[7]:
report.log_plotly_figure(
fig=fig,
caption="Sex Distribution",
section_name="datasets",
)
Age distribution#
[8]:
fig = px.histogram(df, x="Age")
fig.update_layout(
title="Age Distribution",
xaxis_title="Age",
yaxis_title="Count",
bargap=0.2,
)
fig.show()
Add the figure to the report
[9]:
report.log_plotly_figure(
fig=fig,
caption="Age Distribution",
section_name="datasets",
)
Outcome distribution#
[10]:
df["outcome"] = df["HeartDisease"].astype("int")
df = df.drop(columns=["HeartDisease"])
[11]:
fig = px.pie(df, names="outcome")
fig.update_traces(textinfo="percent+label")
fig.update_layout(title_text="Outcome Distribution")
fig.update_traces(
hovertemplate="Outcome: %{label}<br>Count: \
%{value}<br>Percent: %{percent}",
)
fig.show()
Add the figure to the report
[12]:
report.log_plotly_figure(
fig=fig,
caption="Outcome Distribution",
section_name="datasets",
)
[13]:
class_counts = df["outcome"].value_counts()
class_ratio = class_counts[0] / class_counts[1]
print(class_ratio, class_counts)
0.8070866141732284 outcome
1 508
0 410
Name: count, dtype: int64
[14]:
print(df.columns)
Index(['Age', 'Sex', 'ChestPainType', 'RestingBP', 'Cholesterol', 'FastingBS',
'RestingECG', 'MaxHR', 'ExerciseAngina', 'Oldpeak', 'ST_Slope', 'ID',
'outcome'],
dtype='string')
From all the features in the dataset, we select 20 of them which was reported by Li et al. to be the most important features in this classification task.
[15]:
features_list = [
"Age",
"Sex",
"ChestPainType",
"RestingBP",
"Cholesterol",
"FastingBS",
"RestingECG",
"MaxHR",
"ExerciseAngina",
"Oldpeak",
"ST_Slope",
]
features_list = sorted(features_list)
Identifying feature types#
Cyclops TabularFeatures class helps to identify feature types, an essential step before preprocessing the data. Understanding feature types (numerical/categorical/binary) allows us to apply appropriate preprocessing steps for each type.
[16]:
tab_features = TabularFeatures(
data=df.reset_index(),
features=features_list,
by="ID",
targets="outcome",
)
print(tab_features.types)
{'Age': 'numeric', 'RestingECG': 'ordinal', 'Oldpeak': 'numeric', 'FastingBS': 'binary', 'Sex': 'binary', 'ChestPainType': 'ordinal', 'RestingBP': 'numeric', 'outcome': 'binary', 'MaxHR': 'numeric', 'ExerciseAngina': 'binary', 'Cholesterol': 'numeric', 'ST_Slope': 'ordinal'}
Creating data preprocessors#
We create a data preprocessor using sklearnâs ColumnTransformer. This helps in applying different preprocessing steps to different columns in the dataframe. For instance, binary features might be processed differently from numeric features.
[17]:
numeric_transformer = Pipeline(
steps=[("imputer", SimpleImputer(strategy="mean")), ("scaler", MinMaxScaler())],
)
binary_transformer = Pipeline(
steps=[("imputer", SimpleImputer(strategy="most_frequent"))],
)
[18]:
numeric_features = sorted((tab_features.features_by_type("numeric")))
numeric_indices = [
df[features_list].columns.get_loc(column) for column in numeric_features
]
print(numeric_features)
['Age', 'Cholesterol', 'MaxHR', 'Oldpeak', 'RestingBP']
[19]:
binary_features = sorted(tab_features.features_by_type("binary"))
binary_features.remove("outcome")
ordinal_features = sorted(tab_features.features_by_type("ordinal"))
binary_indices = [
df[features_list].columns.get_loc(column) for column in binary_features
]
ordinal_indices = [
df[features_list].columns.get_loc(column) for column in ordinal_features
]
print(binary_features, ordinal_features)
['ExerciseAngina', 'FastingBS', 'Sex'] ['ChestPainType', 'RestingECG', 'ST_Slope']
[20]:
preprocessor = ColumnTransformer(
transformers=[
("num", numeric_transformer, numeric_indices),
(
"onehot",
OneHotEncoder(handle_unknown="ignore"),
binary_indices + ordinal_indices,
),
],
remainder="passthrough",
)
Letâs document the dataset in the model card. This can be done using the log_dataset method, which takes the following arguments: - description: A description of the dataset. - citation: The citation for the dataset. - link: A link to a resource for the dataset. - license_id: The SPDX license identifier for the dataset. - version: The version of the dataset. - features: A list of features in the dataset. - split: The split of the dataset (train, test, validation, etc.). - sensitive_features:
A list of sensitive features used to train/evaluate the model. - sensitive_feature_justification: A justification for the sensitive features used to train/evaluate the model.
[21]:
report.log_dataset(
description="""This dataset was created by combining different datasets
already available independently but not combined before. In this dataset,
5 heart datasets are combined over 11 common features. Every dataset used
can be found under the Index of heart disease datasets from UCI
Machine Learning Repository on the following link:
https://archive.ics.uci.edu/ml/machine-learning-databases/heart-disease/.""",
citation=inspect.cleandoc(
"""
@misc{fedesoriano,
title={Heart Failure Prediction Dataset.},
author={Fedesoriano, F},
year={2021},
publisher={Kaggle}
}
""",
),
link="""
https://www.kaggle.com/datasets/fedesoriano/heart-failure-prediction
""",
license_id="CC0-1.0",
version="Version 1",
features=features_list,
sensitive_features=["Sex", "Age"],
sensitive_feature_justification="Demographic information like age and gender \
often have a strong correlation with health outcomes. For example, older \
patients are more likely to have a higher risk of heart disease.",
)
Creating Hugging Face Dataset#
We convert our processed Pandas dataframe into a Hugging Face dataset, a powerful and easy-to-use data format which is also compatible with CyclOps models and evaluator modules. The dataset is then split to train and test sets.
[22]:
dataset = Dataset.from_pandas(df)
dataset.cleanup_cache_files()
print(dataset)
Dataset({
features: ['Age', 'Sex', 'ChestPainType', 'RestingBP', 'Cholesterol', 'FastingBS', 'RestingECG', 'MaxHR', 'ExerciseAngina', 'Oldpeak', 'ST_Slope', 'ID', 'outcome'],
num_rows: 918
})
[23]:
dataset = dataset.cast_column("outcome", ClassLabel(num_classes=2))
dataset = dataset.train_test_split(
train_size=TRAIN_SIZE,
stratify_by_column="outcome",
seed=RANDOM_SEED,
)
- Casting the dataset: 0%| | 0/918 [00:00<?, ? examples/s]
</pre>
- Casting the dataset: 0%| | 0/918 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Casting the dataset: 0%| | 0/918 [00:00<?, ? examples/s]
- Casting the dataset: 100%|ââââââââââ| 918/918 [00:00<00:00, 69801.15 examples/s]
</pre>
- Casting the dataset: 100%|ââââââââââ| 918/918 [00:00<00:00, 69801.15 examples/s]
end{sphinxVerbatim}
Casting the dataset: 100%|ââââââââââ| 918/918 [00:00<00:00, 69801.15 examples/s]
Model Creation#
CyclOps model registry allows for straightforward creation and selection of models. This registry maintains a list of pre-configured models, which can be instantiated with a single line of code. Here we use a SGD classifier to fit a logisitic regression model. The model configurations can be passed to create_model based on the sklearn parameters for SGDClassifier.
[24]:
model_name = "sgd_classifier"
model = create_model(model_name, random_state=123, verbose=0, class_weight="balanced")
Task Creation#
We use Cyclops tasks to define our modelâs task (in this case, heart failure prediction), train the model, make predictions, and evaluate performance. Cyclops task classes encapsulate the entire ML pipeline into a single, cohesive structure, making the process smooth and easy to manage.
[25]:
heart_failure_prediction_task = BinaryTabularClassificationTask(
{model_name: model},
task_features=features_list,
task_target="outcome",
)
[26]:
heart_failure_prediction_task.list_models()
[26]:
['sgd_classifier']
Training#
If best_model_params is passed to the train method, the best model will be selected after the hyperparameter search. The parameters in best_model_params indicate the values to create the parameters grid.
Note that the data preprocessor needs to be passed to the tasks methods if the Hugging Face dataset is not already preprocessed.
[27]:
best_model_params = {
"alpha": [0.0001, 0.001, 0.01, 0.1, 1, 10, 100],
"learning_rate": ["constant", "optimal", "invscaling", "adaptive"],
"eta0": [0.1, 0.01, 0.001, 0.0001],
"metric": "roc_auc",
"method": "grid",
}
heart_failure_prediction_task.train(
dataset["train"],
model_name=model_name,
transforms=preprocessor,
best_model_params=best_model_params,
)
2024-07-16 17:02:34,508 INFO cyclops.models.wrappers.sk_model - Best alpha: 0.001
2024-07-16 17:02:34,509 INFO cyclops.models.wrappers.sk_model - Best eta0: 0.01
2024-07-16 17:02:34,511 INFO cyclops.models.wrappers.sk_model - Best learning_rate: adaptive
[27]:
SGDClassifier(alpha=0.001, class_weight='balanced', early_stopping=True,
eta0=0.01, learning_rate='adaptive', loss='log_loss',
random_state=123)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
SGDClassifier(alpha=0.001, class_weight='balanced', early_stopping=True,
eta0=0.01, learning_rate='adaptive', loss='log_loss',
random_state=123)[28]:
model_params = heart_failure_prediction_task.list_models_params()[model_name]
print(model_params)
{'alpha': 0.001, 'average': False, 'class_weight': 'balanced', 'early_stopping': True, 'epsilon': 0.1, 'eta0': 0.01, 'fit_intercept': True, 'l1_ratio': 0.15, 'learning_rate': 'adaptive', 'loss': 'log_loss', 'max_iter': 1000, 'n_iter_no_change': 5, 'n_jobs': None, 'penalty': 'l2', 'power_t': 0.5, 'random_state': 123, 'shuffle': True, 'tol': 0.001, 'validation_fraction': 0.1, 'verbose': 0, 'warm_start': False}
Log the model parameters to the report.
We can add model parameters to the model card using the log_model_parameters method.
[29]:
report.log_model_parameters(params=model_params)
Prediction#
The prediction output can be either the whole Hugging Face dataset with the prediction columns added to it or the single column containing the predicted values.
[30]:
y_pred = heart_failure_prediction_task.predict(
dataset["test"],
model_name=model_name,
transforms=preprocessor,
proba=True,
only_predictions=True,
)
prediction_df = pd.DataFrame(
{
"y_prob": [y_pred_i[1] for y_pred_i in y_pred],
"y_true": dataset["test"]["outcome"],
"Sex": dataset["test"]["Sex"],
}
)
- Map: 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Map: 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Map: 0%| | 0/184 [00:00<?, ? examples/s]
- Map: 100%|ââââââââââ| 184/184 [00:00<00:00, 2430.03 examples/s]
</pre>
- Map: 100%|ââââââââââ| 184/184 [00:00<00:00, 2430.03 examples/s]
end{sphinxVerbatim}
Map: 100%|ââââââââââ| 184/184 [00:00<00:00, 2430.03 examples/s]
Evaluation#
Evaluation is done using various evaluation metrics that provide different perspectives on the modelâs predictive abilities i.e. standard performance metrics and fairness metrics.
The standard performance metrics can be created using the MetricDict object.
[31]:
metric_names = [
"binary_accuracy",
"binary_precision",
"binary_recall",
"binary_f1_score",
"binary_auroc",
"binary_average_precision",
"binary_roc_curve",
"binary_precision_recall_curve",
]
metrics = [
create_metric(metric_name, experimental=True) for metric_name in metric_names
]
metric_collection = MetricDict(metrics)
In addition to overall metrics, it might be interesting to see how the model performs on certain subpopulations. We can define these subpopulations using SliceSpec objects.
[32]:
spec_list = [
{
"Age": {
"min_value": 30,
"max_value": 50,
"min_inclusive": True,
"max_inclusive": False,
},
},
{
"Age": {
"min_value": 50,
"max_value": 70,
"min_inclusive": True,
"max_inclusive": False,
},
},
{"Sex": {"value": "M"}},
]
slice_spec = SliceSpec(spec_list)
A MetricDict can also be defined for the fairness metrics.
[33]:
specificity = create_metric(metric_name="binary_specificity", experimental=True)
sensitivity = create_metric(metric_name="binary_sensitivity", experimental=True)
fpr = -specificity + 1
fnr = -sensitivity + 1
ber = (fpr + fnr) / 2
fairness_metric_collection = MetricDict(
{
"Sensitivity": sensitivity,
"Specificity": specificity,
"BER": ber,
},
)
The FairnessConfig helps in setting up and evaluating the fairness of the model predictions.
[34]:
fairness_config = FairnessConfig(
metrics=fairness_metric_collection,
dataset=None, # dataset is passed from the evaluator
target_columns=None, # target columns are passed from the evaluator
groups=["Age"],
group_bins={"Age": [40, 50]},
group_base_values={"Age": 40},
thresholds=[0.5],
)
The evaluate methods outputs the evaluation results and the Hugging Face dataset with the predictions added to it.
[35]:
results, dataset_with_preds = heart_failure_prediction_task.evaluate(
dataset=dataset["test"],
metrics=metric_collection,
model_names=model_name,
transforms=preprocessor,
prediction_column_prefix="preds",
slice_spec=slice_spec,
batch_size=-1,
fairness_config=fairness_config,
override_fairness_metrics=False,
)
- Map: 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Map: 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Map: 0%| | 0/184 [00:00<?, ? examples/s]
- Map: 100%|ââââââââââ| 184/184 [00:00<00:00, 2493.72 examples/s]
</pre>
- Map: 100%|ââââââââââ| 184/184 [00:00<00:00, 2493.72 examples/s]
end{sphinxVerbatim}
Map: 100%|ââââââââââ| 184/184 [00:00<00:00, 2493.72 examples/s]
- Flattening the indices: 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Flattening the indices: 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Flattening the indices: 0%| | 0/184 [00:00<?, ? examples/s]
- Flattening the indices: 100%|ââââââââââ| 184/184 [00:00<00:00, 3137.23 examples/s]
</pre>
- Flattening the indices: 100%|ââââââââââ| 184/184 [00:00<00:00, 3137.23 examples/s]
end{sphinxVerbatim}
Flattening the indices: 100%|ââââââââââ| 184/184 [00:00<00:00, 3137.23 examples/s]
- Flattening the indices: 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Flattening the indices: 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Flattening the indices: 0%| | 0/184 [00:00<?, ? examples/s]
- Flattening the indices: 100%|ââââââââââ| 184/184 [00:00<00:00, 26161.08 examples/s]
</pre>
- Flattening the indices: 100%|ââââââââââ| 184/184 [00:00<00:00, 26161.08 examples/s]
end{sphinxVerbatim}
Flattening the indices: 100%|ââââââââââ| 184/184 [00:00<00:00, 26161.08 examples/s]
- Map: 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Map: 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Map: 0%| | 0/184 [00:00<?, ? examples/s]
- Map: 100%|ââââââââââ| 184/184 [00:00<00:00, 9516.76 examples/s]
</pre>
- Map: 100%|ââââââââââ| 184/184 [00:00<00:00, 9516.76 examples/s]
end{sphinxVerbatim}
Map: 100%|ââââââââââ| 184/184 [00:00<00:00, 9516.76 examples/s]
- Filter -> Age:[30 - 50): 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Filter -> Age:[30 - 50): 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Filter -> Age:[30 - 50): 0%| | 0/184 [00:00<?, ? examples/s]
- Filter -> Age:[30 - 50): 100%|ââââââââââ| 184/184 [00:00<00:00, 18573.16 examples/s]
</pre>
- Filter -> Age:[30 - 50): 100%|ââââââââââ| 184/184 [00:00<00:00, 18573.16 examples/s]
end{sphinxVerbatim}
Filter -> Age:[30 - 50): 100%|ââââââââââ| 184/184 [00:00<00:00, 18573.16 examples/s]
- Filter -> Age:[50 - 70): 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Filter -> Age:[50 - 70): 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Filter -> Age:[50 - 70): 0%| | 0/184 [00:00<?, ? examples/s]
- Filter -> Age:[50 - 70): 100%|ââââââââââ| 184/184 [00:00<00:00, 10781.22 examples/s]
</pre>
- Filter -> Age:[50 - 70): 100%|ââââââââââ| 184/184 [00:00<00:00, 10781.22 examples/s]
end{sphinxVerbatim}
Filter -> Age:[50 - 70): 100%|ââââââââââ| 184/184 [00:00<00:00, 10781.22 examples/s]
- Filter -> Sex:M: 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Filter -> Sex:M: 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Filter -> Sex:M: 0%| | 0/184 [00:00<?, ? examples/s]
- Filter -> Sex:M: 100%|ââââââââââ| 184/184 [00:00<00:00, 10781.67 examples/s]
</pre>
- Filter -> Sex:M: 100%|ââââââââââ| 184/184 [00:00<00:00, 10781.67 examples/s]
end{sphinxVerbatim}
Filter -> Sex:M: 100%|ââââââââââ| 184/184 [00:00<00:00, 10781.67 examples/s]
- Filter -> overall: 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Filter -> overall: 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Filter -> overall: 0%| | 0/184 [00:00<?, ? examples/s]
- Filter -> overall: 100%|ââââââââââ| 184/184 [00:00<00:00, 11567.72 examples/s]
</pre>
- Filter -> overall: 100%|ââââââââââ| 184/184 [00:00<00:00, 11567.72 examples/s]
end{sphinxVerbatim}
Filter -> overall: 100%|ââââââââââ| 184/184 [00:00<00:00, 11567.72 examples/s]
- Filter -> Age:(-inf - 40.0]: 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Filter -> Age:(-inf - 40.0]: 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Filter -> Age:(-inf - 40.0]: 0%| | 0/184 [00:00<?, ? examples/s]
- Filter -> Age:(-inf - 40.0]: 100%|ââââââââââ| 184/184 [00:00<00:00, 19671.99 examples/s]
</pre>
- Filter -> Age:(-inf - 40.0]: 100%|ââââââââââ| 184/184 [00:00<00:00, 19671.99 examples/s]
end{sphinxVerbatim}
Filter -> Age:(-inf - 40.0]: 100%|ââââââââââ| 184/184 [00:00<00:00, 19671.99 examples/s]
- Filter -> Age:(40.0 - 50.0]: 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Filter -> Age:(40.0 - 50.0]: 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Filter -> Age:(40.0 - 50.0]: 0%| | 0/184 [00:00<?, ? examples/s]
- Filter -> Age:(40.0 - 50.0]: 100%|ââââââââââ| 184/184 [00:00<00:00, 20067.92 examples/s]
</pre>
- Filter -> Age:(40.0 - 50.0]: 100%|ââââââââââ| 184/184 [00:00<00:00, 20067.92 examples/s]
end{sphinxVerbatim}
Filter -> Age:(40.0 - 50.0]: 100%|ââââââââââ| 184/184 [00:00<00:00, 20067.92 examples/s]
- Filter -> Age:(50.0 - inf]: 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Filter -> Age:(50.0 - inf]: 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Filter -> Age:(50.0 - inf]: 0%| | 0/184 [00:00<?, ? examples/s]
- Filter -> Age:(50.0 - inf]: 100%|ââââââââââ| 184/184 [00:00<00:00, 20970.38 examples/s]
</pre>
- Filter -> Age:(50.0 - inf]: 100%|ââââââââââ| 184/184 [00:00<00:00, 20970.38 examples/s]
end{sphinxVerbatim}
Filter -> Age:(50.0 - inf]: 100%|ââââââââââ| 184/184 [00:00<00:00, 20970.38 examples/s]
[36]:
results_female, _ = heart_failure_prediction_task.evaluate(
dataset=dataset["test"],
metrics=MetricDict(
{
"BinaryAccuracy": create_metric(
metric_name="binary_accuracy",
experimental=True,
),
},
),
model_names=model_name,
transforms=preprocessor,
prediction_column_prefix="preds",
slice_spec=SliceSpec(
[{"Sex": {"value": "F"}}, {"Sex": {"value": "M"}}], include_overall=False
),
batch_size=-1,
)
- Map: 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Map: 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Map: 0%| | 0/184 [00:00<?, ? examples/s]
- Map: 100%|ââââââââââ| 184/184 [00:00<00:00, 2485.87 examples/s]
</pre>
- Map: 100%|ââââââââââ| 184/184 [00:00<00:00, 2485.87 examples/s]
end{sphinxVerbatim}
Map: 100%|ââââââââââ| 184/184 [00:00<00:00, 2485.87 examples/s]
- Flattening the indices: 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Flattening the indices: 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Flattening the indices: 0%| | 0/184 [00:00<?, ? examples/s]
- Flattening the indices: 100%|ââââââââââ| 184/184 [00:00<00:00, 3117.59 examples/s]
</pre>
- Flattening the indices: 100%|ââââââââââ| 184/184 [00:00<00:00, 3117.59 examples/s]
end{sphinxVerbatim}
Flattening the indices: 100%|ââââââââââ| 184/184 [00:00<00:00, 3117.59 examples/s]
- Flattening the indices: 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Flattening the indices: 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Flattening the indices: 0%| | 0/184 [00:00<?, ? examples/s]
- Flattening the indices: 100%|ââââââââââ| 184/184 [00:00<00:00, 26153.99 examples/s]
</pre>
- Flattening the indices: 100%|ââââââââââ| 184/184 [00:00<00:00, 26153.99 examples/s]
end{sphinxVerbatim}
Flattening the indices: 100%|ââââââââââ| 184/184 [00:00<00:00, 26153.99 examples/s]
- Map: 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Map: 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Map: 0%| | 0/184 [00:00<?, ? examples/s]
- Map: 100%|ââââââââââ| 184/184 [00:00<00:00, 9512.65 examples/s]
</pre>
- Map: 100%|ââââââââââ| 184/184 [00:00<00:00, 9512.65 examples/s]
end{sphinxVerbatim}
Map: 100%|ââââââââââ| 184/184 [00:00<00:00, 9512.65 examples/s]
- Filter -> Sex:F: 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Filter -> Sex:F: 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Filter -> Sex:F: 0%| | 0/184 [00:00<?, ? examples/s]
- Filter -> Sex:F: 100%|ââââââââââ| 184/184 [00:00<00:00, 23195.93 examples/s]
</pre>
- Filter -> Sex:F: 100%|ââââââââââ| 184/184 [00:00<00:00, 23195.93 examples/s]
end{sphinxVerbatim}
Filter -> Sex:F: 100%|ââââââââââ| 184/184 [00:00<00:00, 23195.93 examples/s]
- Filter -> Sex:M: 0%| | 0/184 [00:00<?, ? examples/s]
</pre>
- Filter -> Sex:M: 0%| | 0/184 [00:00<?, ? examples/s]
end{sphinxVerbatim}
Filter -> Sex:M: 0%| | 0/184 [00:00<?, ? examples/s]
- Filter -> Sex:M: 100%|ââââââââââ| 184/184 [00:00<00:00, 23076.64 examples/s]
</pre>
- Filter -> Sex:M: 100%|ââââââââââ| 184/184 [00:00<00:00, 23076.64 examples/s]
end{sphinxVerbatim}
Filter -> Sex:M: 100%|ââââââââââ| 184/184 [00:00<00:00, 23076.64 examples/s]
Log the performance metrics to the report.
We can add a performance metric to the model card using the log_performance_metric method, which expects a dictionary where the keys are in the following format: slice_name/metric_name. For instance, overall/accuracy.
We first need to process the evaluation results to get the metrics in the right format.
[37]:
model_name = f"model_for_preds.{model_name}"
results_flat = flatten_results_dict(
results=results,
remove_metrics=["BinaryROC", "BinaryPrecisionRecallCurve"],
model_name=model_name,
)
results_female_flat = flatten_results_dict(
results=results_female,
model_name=model_name,
)
# ruff: noqa: W505
for name, metric in results_female_flat.items():
split, name = name.split("/") # noqa: PLW2901
descriptions = {
"BinaryPrecision": "The proportion of predicted positive instances that are correctly predicted.",
"BinaryRecall": "The proportion of actual positive instances that are correctly predicted. Also known as recall or true positive rate.",
"BinaryAccuracy": "The proportion of all instances that are correctly predicted.",
"BinaryAUROC": "The area under the receiver operating characteristic curve (AUROC) is a measure of the performance of a binary classification model.",
"BinaryAveragePrecision": "The area under the precision-recall curve (AUPRC) is a measure of the performance of a binary classification model.",
"BinaryF1Score": "The harmonic mean of precision and recall.",
}
report.log_quantitative_analysis(
"performance",
name=name,
value=metric.tolist(),
description=descriptions[name],
metric_slice=split,
pass_fail_thresholds=0.7,
pass_fail_threshold_fns=lambda x, threshold: bool(x >= threshold),
)
for name, metric in results_flat.items():
split, name = name.split("/") # noqa: PLW2901
descriptions = {
"BinaryPrecision": "The proportion of predicted positive instances that are correctly predicted.",
"BinaryRecall": "The proportion of actual positive instances that are correctly predicted. Also known as recall or true positive rate.",
"BinaryAccuracy": "The proportion of all instances that are correctly predicted.",
"BinaryAUROC": "The area under the receiver operating characteristic curve (AUROC) is a measure of the performance of a binary classification model.",
"BinaryAveragePrecision": "The area under the precision-recall curve (AUPRC) is a measure of the performance of a binary classification model.",
"BinaryF1Score": "The harmonic mean of precision and recall.",
}
report.log_quantitative_analysis(
"performance",
name=name,
value=metric.tolist(),
description=descriptions[name],
metric_slice=split,
pass_fail_thresholds=0.7,
pass_fail_threshold_fns=lambda x, threshold: bool(x >= threshold),
)
We can also use the ClassificationPlotter to plot the performance metrics and the add the figure to the model card using the log_plotly_figure method.
[38]:
plotter = ClassificationPlotter(task_type="binary", class_names=["0", "1"])
plotter.set_template("plotly_white")
[39]:
# extracting the ROC curves and AUROC results for all the slices
roc_curves = {
slice_name: slice_results["BinaryROC"]
for slice_name, slice_results in results[model_name].items()
}
aurocs = {
slice_name: slice_results["BinaryAUROC"]
for slice_name, slice_results in results[model_name].items()
}
roc_curves.keys()
[39]:
dict_keys(['Age:[30 - 50)', 'Age:[50 - 70)', 'Sex:M', 'overall'])
[40]:
# plotting the ROC curves for all the slices
roc_plot = plotter.roc_curve_comparison(roc_curves, aurocs=aurocs)
report.log_plotly_figure(
fig=roc_plot,
caption="ROC Curve for Female Patients",
section_name="quantitative analysis",
)
roc_plot.show()
[41]:
# extracting the precision-recall curves and average precision results for all the slices
pr_curves = {
slice_name: slice_results["BinaryPrecisionRecallCurve"]
for slice_name, slice_results in results[model_name].items()
}
average_precisions = {
slice_name: slice_results["BinaryAveragePrecision"]
for slice_name, slice_results in results[model_name].items()
}
pr_curves.keys()
[41]:
dict_keys(['Age:[30 - 50)', 'Age:[50 - 70)', 'Sex:M', 'overall'])
[42]:
# plotting the precision-recall curves for all the slices
pr_plot = plotter.precision_recall_curve_comparison(
pr_curves,
auprcs=average_precisions,
)
report.log_plotly_figure(
fig=pr_plot,
caption="Precision-Recall Curve Comparison",
section_name="quantitative analysis",
)
pr_plot.show()
[43]:
# Extracting the overall classification metric values.
overall_performance = {
metric_name: metric_value
for metric_name, metric_value in results[model_name]["overall"].items()
if metric_name not in ["BinaryROC", "BinaryPrecisionRecallCurve"]
}
[44]:
# Plotting the overall classification metric values.
overall_performance_plot = plotter.metrics_value(
overall_performance,
title="Overall Performance",
)
report.log_plotly_figure(
fig=overall_performance_plot,
caption="Overall Performance",
section_name="quantitative analysis",
)
overall_performance_plot.show()
[45]:
# Extracting the metric values for all the slices.
slice_metrics = {
slice_name: {
metric_name: metric_value
for metric_name, metric_value in slice_results.items()
if metric_name not in ["BinaryROC", "BinaryPrecisionRecallCurve"]
}
for slice_name, slice_results in results[model_name].items()
}
[46]:
# Plotting the metric values for all the slices.
slice_metrics_plot = plotter.metrics_comparison_bar(slice_metrics)
report.log_plotly_figure(
fig=slice_metrics_plot,
caption="Slice Metric Comparison",
section_name="quantitative analysis",
)
slice_metrics_plot.show()
[47]:
# Plotting the metric values for all the slices.
# ROC curve components
pred_probs = np.array(dataset_with_preds["preds.sgd_classifier"])
true_labels = np.array(dataset_with_preds["outcome"])
roc_curve = binary_roc(true_labels, pred_probs)
ppv = np.zeros_like(roc_curve.thresholds)
npv = np.zeros_like(roc_curve.thresholds)
# Calculate PPV and NPV for each threshold
for i, threshold in enumerate(roc_curve.thresholds):
# Calculate PPV and NPV
ppv[i] = binary_ppv(true_labels, pred_probs, threshold=threshold)
npv[i] = binary_npv(true_labels, pred_probs, threshold=threshold)
runway_plot = plotter.threshperf(roc_curve, ppv, npv, pred_probs)
report.log_plotly_figure(
fig=runway_plot,
caption="Threshold-Performance plot",
section_name="quantitative analysis",
)
runway_plot.show()
We can also plot the calibration curve of the model on the test set
[48]:
calibration_plot = plotter.calibration(
prediction_df, y_true_col="y_true", y_prob_col="y_prob", group_col="Sex"
)
report.log_plotly_figure(
fig=calibration_plot,
caption="Calibration plot",
section_name="quantitative analysis",
)
calibration_plot.show()
[49]:
# Reformatting the fairness metrics
fairness_results = copy.deepcopy(results["fairness"])
fairness_metrics = {}
# remove the group size from the fairness results and add it to the slice name
for slice_name, slice_results in fairness_results.items():
group_size = slice_results.pop("Group Size")
fairness_metrics[f"{slice_name} (Size={group_size})"] = slice_results
[50]:
# Plotting the fairness metrics
fairness_plot = plotter.metrics_comparison_scatter(
fairness_metrics,
title="Fairness Metrics",
)
report.log_plotly_figure(
fig=fairness_plot,
caption="Fairness Metrics",
section_name="fairness analysis",
)
fairness_plot.show()
Report Generation#
Before generating the model card, let us document some of the details of the model and some considerations involved in developing and using the model.
Letâs start with populating the model details section, which includes the following fields by default: - description: A high-level description of the model and its usage for a general audience. - version: The version of the model. - owners: The individuals or organizations that own the model. - license: The license under which the model is made available. - citation: The citation for the model. - references: Links to resources that are relevant to the model. - path: The path to where the model is stored. - regulatory_requirements: The regulatory requirements that are relevant to the model.
We can add additional fields to the model details section by passing a dictionary to the log_from_dict method and specifying the section name as model_details. You can also use the log_descriptor method to add a new field object with a description attribute to any section of the model card.
[51]:
report.log_from_dict(
data={
"name": "Heart Failure Prediction Model",
"description": "The model was trained on the Kaggle Heart Failure \
Prediction Dataset to predict risk of heart failure.",
},
section_name="model_details",
)
report.log_version(
version_str="0.0.1",
date=str(date.today()),
description="Initial Release",
)
report.log_owner(
name="CyclOps Team",
contact="vectorinstitute.github.io/cyclops/",
email="cyclops@vectorinstitute.ai",
)
report.log_license(identifier="Apache-2.0")
report.log_reference(
link="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html", # noqa: E501
)
Next, letâs populate the considerations section, which includes the following fields by default: - users: The intended users of the model. - use_cases: The use cases for the model. These could be primary, downstream or out-of-scope use cases. - fairness_assessment: A description of the benefits and harms of the model for different groups as well as the steps taken to mitigate the harms. - ethical_considerations: The risks associated with using the model and the steps taken to mitigate them. This
can be populated using the log_risk method.
[52]:
report.log_from_dict(
data={
"users": [
{"description": "Hospitals"},
{"description": "Clinicians"},
],
},
section_name="considerations",
)
report.log_user(description="ML Engineers")
report.log_use_case(
description="Predicting risk of heart failure.",
kind="primary",
)
report.log_use_case(
description="Predicting risk of pathologies and conditions other\
than heart failure.",
kind="out-of-scope",
)
report.log_fairness_assessment(
affected_group="sex, age",
benefit="Improved health outcomes for patients.",
harm="Biased predictions for patients in certain groups (e.g. older patients) \
may lead to worse health outcomes.",
mitigation_strategy="We will monitor the performance of the model on these groups \
and retrain the model if the performance drops below a certain threshold.",
)
report.log_risk(
risk="The model may be used to make decisions that affect the health of patients.",
mitigation_strategy="The model should be continuously monitored for performance \
and retrained if the performance drops below a certain threshold.",
)
Once the model card is populated, you can generate the report using the export method. The report is generated in the form of an HTML file. A JSON file containing the model card data will also be generated along with the HTML file. By default, the files will be saved in a folder named cyclops_reports in the current working directory. You can change the path by passing a output_dir argument when instantiating the ModelCardReport class.
[53]:
np.random.seed(42)
synthetic_timestamps = pd.date_range(
start="1/1/2020", periods=10, freq="D"
).values.astype(str)
report._model_card.overview = None
report_path = report.export(
output_filename="heart_failure_report_periodic.html",
synthetic_timestamp=synthetic_timestamps[0],
last_n_evals=3,
)
shutil.copy(f"{report_path}", ".")
metric_save = None
for i in range(len(synthetic_timestamps[1:])):
if i == 3:
report._model_card.quantitative_analysis.performance_metrics.append(
metric_save,
)
report._model_card.overview = None
for metric in report._model_card.quantitative_analysis.performance_metrics:
metric.value = np.clip(
metric.value + np.random.normal(0, 0.1),
0,
1,
)
metric.tests[0].passed = bool(metric.value >= 0.7)
if i == 2:
metrics = []
for metric in report._model_card.quantitative_analysis.performance_metrics:
if metric.type == "BinaryAccuracy" and metric.slice == "Age:[30 - 50)":
metric_save = copy.deepcopy(metric)
else:
metrics.append(metric)
report._model_card.quantitative_analysis.performance_metrics = metrics
report_path = report.export(
output_filename="heart_failure_report_periodic.html",
synthetic_timestamp=synthetic_timestamps[i + 1],
)
shutil.copy(f"{report_path}", ".")
shutil.rmtree("./cyclops_report")
You can view the generated HTML report.