vis4d.engine.callbacks.evaluator

This module contains utilities for callbacks.

Classes

EvaluatorCallback(*args, evaluator[, ...])

Callback for model evaluation.

class EvaluatorCallback(*args, evaluator, metrics_to_eval=None, save_predictions=False, save_prefix=None, output_dir=None, **kwargs)

Callback for model evaluation.

__init__(*args, evaluator, metrics_to_eval=None, save_predictions=False, save_prefix=None, output_dir=None, **kwargs)

Init callback.

Parameters:

• evaluator (Evaluator) – Evaluator.

• metrics_to_eval (list[str], Optional) – Metrics to evaluate. If None, all metrics in the evaluator will be evaluated. Defaults to None.

• save_predictions (bool) – If the predictions should be saved. Defaults to False.

• save_prefix (str, Optional) – Output directory for saving the evaluation results. Defaults to None.

• output_dir (str, Optional) – Output directory for saving the evaluation results.

setup(trainer, pl_module, stage)

Setup callback.

Return type:

None

on_validation_batch_end(trainer, pl_module, outputs, batch, batch_idx, dataloader_idx=0)

Hook to run at the end of a validation batch.

Return type:

None

on_validation_epoch_end(trainer, pl_module)

Wait for on_validation_epoch_end PL hook to call ‘evaluate’.

Return type:

None

on_test_batch_end(trainer, pl_module, outputs, batch, batch_idx, dataloader_idx=0)

Hook to run at the end of a testing batch.

Return type:

None

on_test_epoch_end(trainer, pl_module)

Hook to run at the end of a testing epoch.

Return type:

None

run_eval()

Run evaluation for the given evaluator.

Return type:

Dict[str, Union[float, int, Tensor]]

process()

Process the evaluator.

Return type:

None

evaluate(metric)

Evaluate the performance after processing all input/output pairs.

Returns:

A dictionary containing the evaluation results. The

keys are formatted as {metric_name}/{key_name}, and the values are the corresponding evaluated values.

Return type:

MetricLogs