Extending TemporAI Tutorial 04: Writing a Custom Metric Plugin¶

This tutorial shows how to extend TemporAI by wring a custom metric plugin.

Note

See also “Writing a Custom Plugin 101” section in “Writing a Custom Method Plugin” tutorial.

Inherit from the appropriate base class for the category of the metric plugin you are writing.¶

You need to find which category of metric plugin you are writing.

You can view all the different metric plugin categories as so:

[ ]:

from tempor import plugin_loader

plugin_categories = plugin_loader.list_categories(plugin_type="metric")

list(plugin_categories.keys())

['prediction.one_off.classification',
 'prediction.one_off.regression',
 'time_to_event']

Remember you can also see the existing metric plugins and how they correspond to different categories, as follows:

[ ]:

all_plugins = plugin_loader.list(plugin_type="metric")

from rich.pretty import pprint  # For prettifying the print output only.

pprint(all_plugins, indent_guides=True)

{
│   'prediction': {
│   │   'one_off': {
│   │   │   'regression': ['mse', 'mae', 'r2'],
│   │   │   'classification': [
│   │   │   │   'accuracy',
│   │   │   │   'f1_score_micro',
│   │   │   │   'f1_score_macro',
│   │   │   │   'f1_score_weighted',
│   │   │   │   'kappa',
│   │   │   │   'kappa_quadratic',
│   │   │   │   'recall_micro',
│   │   │   │   'recall_macro',
│   │   │   │   'recall_weighted',
│   │   │   │   'precision_micro',
│   │   │   │   'precision_macro',
│   │   │   │   'precision_weighted',
│   │   │   │   'mcc',
│   │   │   │   'aucprc',
│   │   │   │   'aucroc'
│   │   │   ]
│   │   }
│   },
│   'time_to_event': ['c_index', 'brier_score']
}

Let’s say you would like to write a plugin of category "prediction.one_off.regression".

You can find which base class you need to inherit from as follows.

[ ]:

plugin_categories = plugin_loader.list_categories(plugin_type="metric")

print("Base classes for all categories:")
pprint(plugin_categories, indent_guides=False)

print("Base class you need:")
print(plugin_categories["prediction.one_off.regression"])

Base classes for all categories:

{
    'prediction.one_off.classification': <class 'tempor.metrics.metric.OneOffClassificationMetric'>,
    'prediction.one_off.regression': <class 'tempor.metrics.metric.OneOffRegressionMetric'>,
    'time_to_event': <class 'tempor.metrics.metric.TimeToEventMetric'>
}

Base class you need:
<class 'tempor.metrics.metric.OneOffRegressionMetric'>

You can then find the class in the TemporAI source code, to see its method signatures etc.

Implement the methods the plugin needs.¶

Metric plugins require the following methods to be implemented: * direction property, which returns either "minimize" or "maximize", representing the “good” direction of the metric. * _evaluate() which takes in the actual and predicted values and returns the evaluated metric(s).

If you haven’t implemented some required method for the plugin, Python will notify you by raising an exception when you attempt to instantiate your plugin (see Python ``abc` <https://docs.python.org/3/library/abc.html>`__).

Register the plugin with TemporAI.¶

Registering your plugin with TemporAI is very simple, you need to use the register_plugin decorator, as shown in the example below.

You will need to specify the name of your plugin and its category in the decorator.

The plugin_type needs to be set to "datasource".

from tempor.core.plugins import register_plugin

@register_plugin(name="my_metric", category="prediction.one_off.regression", plugin_type="metric")
class MyMetric(OneOffRegressionMetric):
    ...

Example¶

Now putting this together in a minimal example.

[ ]:

from typing import Any

import numpy as np
from sklearn.metrics import d2_tweedie_score

from tempor.core import plugins
from tempor.metrics import metric, metric_typing


@plugins.register_plugin(name="my_metric", category="prediction.one_off.regression", plugin_type="metric")
class MyOneOffRegressionMetric(metric.OneOffRegressionMetric):
    """My custom metric, here we use
    D^2 regression score function, fraction of Tweedie deviance explained,
    as per `sklearn`.
    """

    @property
    def direction(self) -> metric_typing.MetricDirection:  # noqa: D102
        return "maximize"

    def _evaluate(self, actual: np.ndarray, predicted: np.ndarray, *args: Any, **kwargs: Any) -> float:
        return d2_tweedie_score(actual, predicted, power=kwargs.get("power", 0))

We now see our plugin in TemporAI:

[ ]:

from tempor import plugin_loader

all_plugins = plugin_loader.list(plugin_type="metric")

pprint(all_plugins, indent_guides=True)

my_metric_found = "my_metric" in all_plugins["prediction"]["one_off"]["regression"]
print(f"`my_metric` plugin found in the category 'prediction.one_off.regression': {my_metric_found}")
assert my_metric_found

{
│   'prediction': {
│   │   'one_off': {
│   │   │   'regression': ['mse', 'mae', 'r2', 'my_metric'],
│   │   │   'classification': [
│   │   │   │   'accuracy',
│   │   │   │   'f1_score_micro',
│   │   │   │   'f1_score_macro',
│   │   │   │   'f1_score_weighted',
│   │   │   │   'kappa',
│   │   │   │   'kappa_quadratic',
│   │   │   │   'recall_micro',
│   │   │   │   'recall_macro',
│   │   │   │   'recall_weighted',
│   │   │   │   'precision_micro',
│   │   │   │   'precision_macro',
│   │   │   │   'precision_weighted',
│   │   │   │   'mcc',
│   │   │   │   'aucprc',
│   │   │   │   'aucroc'
│   │   │   ]
│   │   }
│   },
│   'time_to_event': ['c_index', 'brier_score']
}

`my_metric` plugin found in the category 'prediction.one_off.regression': True

The plugin can be used as normal.

[ ]:

# Get the plugin.

my_metric = plugin_loader.get("prediction.one_off.regression.my_metric", plugin_type="metric")

print(my_metric)

MyOneOffRegressionMetric(
    name='my_metric',
    description='My custom metric, here we use\n    D^2 regression score function, fraction of Tweedie deviance explai...'
)

[ ]:

# Use the metric.

y_true = np.asarray([1, 2, 3])
y_pred = np.asarray([1, 3, 3])

metric_value = my_metric(y_true, y_pred, power=2)
print(metric_value)

0.7492656592741931