Extending TemporAI Tutorial 02: Testing a Custom Method Plugin¶

This tutorial gives a brief overview of how to test your custom method plugin.

For the basics of writing a custom plugin, see the “Writing Custom Method Plugin” tutorial first. This tutorial assumes you have already written a custom plugin.

⚙️ Installation for testing

You should install the [dev] TemporAI to run tests:

pip install temporai[dev]

Alternatively, the best way to install TemporAI for development is to clone the repo, checkout a branch, and install in editable mode:

git clone https://github.com/vanderschaarlab/temporai.git
cd temporai
git checkout -b my-branch-name
pip install -e .[dev]

📘 See also the contribution guide.

1. Find and adapt suitable tests¶

The simplest way to run a set of tests on your custom method plugin is to find and adapt existing tests for plugins of the same category.

Method plugin categories in TemporAI can be found as below.

[ ]:

from tempor import plugin_loader
from rich.pretty import pprint  # For prettifying the print output only.

plugin_categories = plugin_loader.list_categories(plugin_type="method")

print("Method plugin categories:")
pprint(list(plugin_categories.keys()), indent_guides=False)

print("All method plugins:")
pprint(plugin_loader.list(plugin_type="method"), indent_guides=True)

Method plugin categories:

[
    'prediction.one_off.classification',
    'prediction.one_off.regression',
    'prediction.temporal.classification',
    'prediction.temporal.regression',
    'preprocessing.encoding.static',
    'preprocessing.encoding.temporal',
    'preprocessing.imputation.static',
    'preprocessing.imputation.temporal',
    'preprocessing.nop',
    'preprocessing.scaling.static',
    'preprocessing.scaling.temporal',
    'time_to_event',
    'treatments.one_off.regression',
    'treatments.temporal.classification',
    'treatments.temporal.regression'
]

All method plugins:

{
│   'prediction': {
│   │   'one_off': {
│   │   │   'classification': ['cde_classifier', 'ode_classifier', 'nn_classifier', 'laplace_ode_classifier'],
│   │   │   'regression': ['laplace_ode_regressor', 'nn_regressor', 'ode_regressor', 'cde_regressor']
│   │   },
│   │   'temporal': {'classification': ['seq2seq_classifier'], 'regression': ['seq2seq_regressor']}
│   },
│   'preprocessing': {
│   │   'encoding': {'static': ['static_onehot_encoder'], 'temporal': ['ts_onehot_encoder']},
│   │   'imputation': {
│   │   │   'static': ['static_tabular_imputer'],
│   │   │   'temporal': ['ffill', 'ts_tabular_imputer', 'bfill']
│   │   },
│   │   'nop': ['nop_transformer'],
│   │   'scaling': {
│   │   │   'static': ['static_minmax_scaler', 'static_standard_scaler'],
│   │   │   'temporal': ['ts_minmax_scaler', 'ts_standard_scaler']
│   │   }
│   },
│   'time_to_event': ['ts_coxph', 'ts_xgb', 'dynamic_deephit'],
│   'treatments': {
│   │   'one_off': {'regression': ['synctwin_regressor']},
│   │   'temporal': {'classification': ['crn_classifier'], 'regression': ['crn_regressor']}
│   }
}

Suitable tests can be found in TemporAI source code (tests/methods/...), organized hierarchically by plugin category.

For example, tests for prediction/one_off/classification will be located under: * `tests/methods/prediction/one_off/classification <https://github.com/vanderschaarlab/temporai/tree/main/tests/methods/prediction/one_off/classification>`__.

Find an example test file in this category, and adapt the tests from it to your custom plugin. For instance, we could look at: * test file for ``nn_classifier` plugin <https://github.com/vanderschaarlab/temporai/blob/main/tests/methods/prediction/one_off/classification/test_nn_classifier.py>`__.

The test suite differs by plugin category, but in general, the following points are worth noting.

`pytest <https://docs.pytest.org/>`__ is used for testing.
Parametrization is used to run similar tests with different parameters.
Fixtures are used for reusable test elements, e.g. datasets.
Plugins are tested with two methods of importing them (PLUGIN_FROM_OPTIONS): "from_api" and "from_module", this is handled by the helper fixture get_test_plugin.
Typically, the set of test functions needed looks something like this:
- test_sanity: the very basics, check loading of your plugin works.
- test_fit: test your plugin’s fit method.
- test_predict, test_transform, ...: test the additional methods relevant to your plugin’s category.
- test_serde: test serialization and deserialization works.
The below constants are set and reused as parameters in the test suite. Adapt these as necessary. The various dataset fixtures can be found in `conftest.py <https://github.com/vanderschaarlab/temporai/blob/main/tests/conftest.py>`__. python INIT_KWARGS = {"random_state": 123, "n_iter": 5} # Input parameters for the plugin. TEST_ON_DATASETS = ["sine_data_small"] # A list of dataset fixtures to test the plugin on.

An example test file would looks something like below.

In this example, it is assumed that the plugin is of the prediction/one_off/classification category, and is named "my_classifier".

# test_my_classifier.py
# Example for illustration only - adapt to your own plugin as needed.

from typing import Callable, Dict

import pytest

from tempor.methods.prediction.one_off.classification import BaseOneOffClassifier
from tempor.methods.prediction.one_off.classification.plugin_my_classifier import MyClassifier
from tempor.utils.serialization import load, save

INIT_KWARGS = {"random_state": 123, "n_iter": 5}
PLUGIN_FROM_OPTIONS = ["from_api", pytest.param("from_module", marks=pytest.mark.extra)]
TEST_ON_DATASETS = ["sine_data_small"]


@pytest.fixture
def get_test_plugin(get_plugin: Callable):
    def func(plugin_from: str, base_kwargs: Dict):
        return get_plugin(
            plugin_from,
            fqn="prediction.one_off.classification.my_classifier",
            cls=MyClassifier,
            kwargs=base_kwargs,
        )

    return func


@pytest.mark.parametrize("plugin_from", PLUGIN_FROM_OPTIONS)
def test_sanity(get_test_plugin: Callable, plugin_from: str) -> None:
    test_plugin = get_test_plugin(plugin_from, INIT_KWARGS)
    assert test_plugin is not None
    assert test_plugin.name == "nn_classifier"
    assert test_plugin.full_name() == "prediction.one_off.classification.my_classifier"
    assert len(test_plugin.hyperparameter_space()) == 9


@pytest.mark.parametrize("plugin_from", PLUGIN_FROM_OPTIONS)
@pytest.mark.parametrize("data", TEST_ON_DATASETS)
def test_fit(plugin_from: str, data: str, get_test_plugin: Callable, get_dataset: Callable) -> None:
    test_plugin: BaseOneOffClassifier = get_test_plugin(plugin_from, INIT_KWARGS)
    dataset = get_dataset(data)
    test_plugin.fit(dataset)


@pytest.mark.parametrize("plugin_from", PLUGIN_FROM_OPTIONS)
@pytest.mark.parametrize("data", TEST_ON_DATASETS)
def test_predict(
    plugin_from: str, data: str, no_targets: bool, get_test_plugin: Callable, get_dataset: Callable
) -> None:
    test_plugin: BaseOneOffClassifier = get_test_plugin(plugin_from, INIT_KWARGS)
    dataset = get_dataset(data)
    test_plugin.fit(dataset)
    output = test_plugin.predict(dataset)
    assert output.numpy().shape == (len(dataset.time_series), 1)


# Other categories of plugins would have more / different methods to test.


@pytest.mark.parametrize("data", TEST_ON_DATASETS)
def test_serde(data: str, get_test_plugin: Callable, get_dataset: Callable) -> None:
    test_plugin: BaseOneOffClassifier = get_test_plugin("from_api", INIT_KWARGS)
    dataset = get_dataset(data)

    dump = save(test_plugin)
    reloaded1 = load(dump)

    reloaded1.fit(dataset)

    dump = save(reloaded1)
    reloaded2 = load(dump)

    reloaded2.predict(dataset)

In order to run the tests, the test file (e.g. test_my_classifier.py) should be placed into the appropriate test directory, e.g. in this example under:

`tests/methods/prediction/one_off/classification <https://github.com/vanderschaarlab/temporai/tree/main/tests/methods/prediction/one_off/classification>`__.

The tests can the be run like so:

pytest -x tests/methods/prediction/one_off/classification/test_my_classifier.py

2. Plugin loader and all method plugin tests¶

Some additional tests that apply to your plugin are found in: * `tests/test_plugin_loader.py <https://github.com/vanderschaarlab/temporai/blob/main/tests/test_plugin_loader.py>`__: The plugin registry checks. * `tests/methods/test_all_plugins.py <https://github.com/vanderschaarlab/temporai/blob/main/tests/methods/test_all_method_plugins.py>`__: The common automatic basic tests for all plugins.

In test_plugin_loader.py, your custom plugin should be added to test_methods_contents, e.g.

assert "my_classifier" in all_plugins["prediction"]["one_off"]["classification"]

test_all_method_plugins.py tests will run automatically, no changes here are needed.

To check that these two test files pass:

pytest -x tests/test_plugin_loader.py
pytest -x tests/methods/test_all_method_plugins.py

3. Finally…¶

Now is a perfect time to contribute your awesome plugin to the open sources eco-system by submitting a PR to TemporAI!

Please follow the contribution guide.