Skip to main content

· 4 min read
Carolyn Chong
Amrit Krishnan

We’re excited to share the release of our latest version of CyclOps, an open-source software toolkit to implement machine learning models for health care. Our new version follows the release of Vector Institute's AI Trust and Safety Principles, that are essential considerations to be taken into account when developing and deploying AI solutions.

The new version includes the following new features and improvements:

Model reports for evaluating and monitoring clinical ML models​

Tailored for clinicians, data scientists and ML engineers, CyclOps now allows users to generate model reports using a report API. ML teams building clinical ML models can use the report API to build custom model reports for their use case. The report API supports interactive plots, tooltips, trend analysis, threshold limits, and timestamps so that you can monitor the performance of your model over time. The model report also includes a sidebar that displays your model’s metadata so that your model’s information is easily visible.

We believe that increased model transparency allows clinical ML teams to develop and deploy ML models in a safe and responsible manner.

We’re excited to see radiology teams and patient safety teams at local hospitals use the model report to monitor their risk prediction and detection models so that they can make more informed decisions around patient care.

Check out an example model report here and a tutorial showcasing the use of the report API!

An experimental evaluation metrics package that demonstrates cross-framework compatibility​

We are introducing a new experimental metrics package which adopts the array API standard, and provides distributed computation support using torch.distributed and mpi4py backends. The array API standard allows array-consuming libraries like CyclOps to support functionality across array types like NumPy, CuPy and PyTorch.

Let's see how the cross-framework compatibility works with an example. In this example we compute the confusion matrix using the function provided by scikit-learn against the one provided by CyclOps. The inputs are numpy arrays.

import numpy as np
from sklearn.metrics import confusion_matrix
from cyclops.evaluate.metrics.experimental.functional.confusion_matrix import (
    binary_confusion_matrix,
)


y_true = np.random.randint(0, 2, size=10)
y_pred = np.random.rand(10)
y_pred_discrete = (y_pred > 0.5).astype(int)

print("cyclops confusion matrix\n", binary_confusion_matrix(y_true, y_pred_discrete))
print("scikit-learn confusion matrix\n", confusion_matrix(y_true, y_pred_discrete))

results in

cyclops confusion matrix
 [[2 3]
 [4 1]]
scikit-learn confusion matrix
 [[2 3]
 [4 1]]

Similarly, when the inputs are torch tensors, we compare with the function provided by the torchmetrics library.

import numpy as np
import torch
from torchmetrics.functional.classification.confusion_matrix import (
   binary_confusion_matrix as torch_binary_confusion_matrix,
)
from cyclops.evaluate.metrics.experimental.functional.confusion_matrix import (
   binary_confusion_matrix,
)


y_true = np.random.randint(0, 2, size=10)
y_pred = np.random.rand(10)
target = torch.asarray(y_true)
preds = torch.asarray(y_pred)

print("cyclops confusion matrix\n", binary_confusion_matrix(y_true, y_pred_discrete))
print("torchmetrics confusion matrix\n", torch_binary_confusion_matrix(y_true, y_pred_discrete))

results in

cyclops confusion matrix
 [[2 3]
 [4 1]]
torchmetrics confusion matrix
 [[2 3]
 [4 1]]

A simple and unified data package​

We are introducing a simple and unified python package to process and create datasets for training, inference and evaluation. We use the popular 🤗 datasets library for efficiently slicing different modalities (tabular, time-series and images) of data to create subsets for evaluation and monitoring. The package has new methods, improved generalizability, and uses Pandas 2.0.

Check out an example tutorial that showcases the use of the cyclops.data API to create slices of data for evaluation!

Next steps​

We believe that the model report will enable increased transparency around model development, and help with auditing of ML systems that are deployed in clinical settings. We are working with early adopter stakeholders on a few different use cases to pilot the model report. Stay tuned for updates and improvements as we learn more about how clinical ML teams use CyclOps!