detectors.pipelines package

Submodules

detectors.pipelines.base module

Base abstract pipeline class.

class detectors.pipelines.base.Pipeline

Bases: object

Base Pipeline class.

load_pretrained(path)

postprocess(*args, **kwargs)

preprocess(*args, **kwargs) → Any

report(*args, **kwargs) → str

run(*args, **kwargs) → Dict[str, Any]

save_pretrained(path)

setup(*args, **kwargs)

detectors.pipelines.covariate_drift module

class detectors.pipelines.covariate_drift.CovariateDriftPipeline(dataset_name: str, dataset_splits: List[str], transform, corruptions: List[str], intensities: List[int], batch_size: int = 128, limit_fit: float = 1.0, warmup_size=2000, seed=42, **kwargs)

Bases: Pipeline

postprocess(test_scores: Tensor, test_preds: Tensor, test_targets: Tensor, test_labels: Tensor, stride=1, alpha=0.99, acc_threshold=0.9, **kwargs) → Dict[str, Any]

preprocess(method: DetectorWrapper) → DetectorWrapper

report(results: Dict[str, Any], subsample=1)

run(method, model, **kwargs)

setup()

class detectors.pipelines.covariate_drift.OneCorruptionCovariateDriftCifar100Pipeline(transform, corruption: str, intensities: List[int], batch_size: int = 128, **kwargs)

Bases: CovariateDriftPipeline

class detectors.pipelines.covariate_drift.OneCorruptionCovariateDriftCifar10Pipeline(transform, corruption: str, intensities: List[int], batch_size: int = 128, **kwargs)

Bases: CovariateDriftPipeline

class detectors.pipelines.covariate_drift.OneCorruptionCovariateDriftImagenetPipeline(transform, corruption: str, intensities: List[int], batch_size: int = 128, **kwargs)

Bases: CovariateDriftPipeline

detectors.pipelines.ood module

OOD Pipelines.

class detectors.pipelines.ood.OODBenchmarkPipeline(in_dataset_name: str, out_datasets_names_splits: Dict[str, Any], transform: Callable, batch_size: int, num_workers: int = 4, pin_memory: bool = True, prefetch_factor: int = 2, limit_fit: float = 1.0, limit_run: float = 1.0, seed: int = 42, accelerator=None)

Bases: Pipeline, ABC

OOD Benchmark pipeline.

Parameters:

• in_dataset_name (str) – Name of the in-distribution dataset.

• out_datasets_names_splits (Dict[str, Any]) – Dictionary mapping out-distribution dataset names to their splits.

• transform (Callable) – Transform to apply to the datasets.

• batch_size (int) – Batch size.

• num_workers (int, optional) – Number of workers. Defaults to 4.

• pin_memory (bool, optional) – Pin memory. Defaults to True.

• prefetch_factor (int, optional) – Prefetch factor. Defaults to 2.

• limit_fit (float, optional) – Fraction of the training set to use for fitting. Defaults to 1.0.

• limit_run (float, optional) – Fraction of the testing set to use for running. Defaults to 1.0.

• seed (int, optional) – Random seed. Defaults to 42.

• accelerator (Any, optional) – Accelerator. Defaults to None.

postprocess(test_scores: Tensor, test_labels: Tensor)

preprocess(method: DetectorWrapper) → DetectorWrapper

report(results: Dict[str, Dict[str, Any]]) → str

run(method: DetectorWrapper) → Dict[str, Any]

setup()

class detectors.pipelines.ood.OODCifar100BenchmarkPipeline(transform: Callable, limit_fit=1.0, limit_run=1.0, batch_size=128, seed=42, **kwargs)

Bases: OODBenchmarkPipeline

class detectors.pipelines.ood.OODCifar100NoiseValidationPipeline(transform: Callable, limit_fit=1, limit_run=0.1, batch_size=128, seed=42, **kwargs)

Bases: OODValidationPipeline

class detectors.pipelines.ood.OODCifar100ValidationPipeline(transform: Callable, limit_fit=1, limit_run=0.1, batch_size=128, seed=42, **kwargs)

Bases: OODCifar100BenchmarkPipeline, OODValidationPipeline

class detectors.pipelines.ood.OODCifar10BenchmarkPipeline(transform: Callable, limit_fit=1.0, limit_run=1.0, batch_size=128, seed=42, **kwargs)

Bases: OODBenchmarkPipeline

class detectors.pipelines.ood.OODCifar10NoiseValidationPipeline(transform: Callable, limit_fit=1, limit_run=0.1, batch_size=128, seed=42, **kwargs)

Bases: OODValidationPipeline

class detectors.pipelines.ood.OODCifar10ValidationPipeline(transform: Callable, limit_fit=1, limit_run=0.1, batch_size=128, seed=42, **kwargs)

Bases: OODCifar10BenchmarkPipeline, OODValidationPipeline

class detectors.pipelines.ood.OODImageNetBenchmarkPipeline(transform: Callable, limit_fit=1.0, limit_run=1.0, batch_size=64, seed=42, **kwargs)

Bases: OODBenchmarkPipeline

class detectors.pipelines.ood.OODImageNetValidationPipeline(transform: Callable, limit_fit=1, limit_run=0.1, batch_size=64, seed=42, **kwargs)

Bases: OODImageNetBenchmarkPipeline, OODValidationPipeline

class detectors.pipelines.ood.OODMNISTBenchmarkPipeline(transform: Callable, limit_fit=1, batch_size=64)

Bases: OODBenchmarkPipeline

class detectors.pipelines.ood.OODValidationPipeline(in_dataset_name: str, out_datasets_names_splits: Dict[str, Any], transform: Callable, batch_size: int, num_workers: int = 4, pin_memory: bool = True, prefetch_factor: int = 2, limit_fit: float = 1.0, limit_run: float = 1.0, seed: int = 42, accelerator=None)

Bases: OODBenchmarkPipeline, ABC

Pipeline for OOD validation.

This pipeline is used to validate the performance of a model on OOD datasets.

Parameters:

• method (DetectorWrapper) – The OOD detection method to use.

• hyperparameters (Dict[str, Union[List[Any], Tuple[Any], Dict[str, Any]]]) – The hyperparameters to use for the method.

• objective_metric (Literal["fpr_at_0.95_tpr", "auroc"], optional) – The metric to optimize. Defaults to “auroc”.

• n_trials (int, optional) – The number of trials to run. Defaults to 20.

objective(trial: Trial) → float

report(results: Dict[str, Any])

run(method: DetectorWrapper, hyperparameters: Dict[str, List[Any] | Tuple[Any] | Dict[str, Any]], objective_metric: Literal['fpr_at_0.95_tpr', 'auroc'] = 'auroc', objective_dataset: str = 'average', n_trials=20) → Dict[str, Any]

detectors.pipelines.osr module

class detectors.pipelines.osr.OSRCifar10(transform: Callable, batch_size: int, limit_fit: int | None = None, seed: int = 42, **kwargs)

Bases: OODBenchmarkPipeline

class detectors.pipelines.osr.OSRCifar100(transform: Callable, batch_size: int, limit_fit: int | None = None, seed: int = 42, **kwargs)

Bases: OODBenchmarkPipeline

class detectors.pipelines.osr.OSRImagenet(transform: Callable, batch_size: int, limit_fit: int | None = None, seed: int = 42, **kwargs)

Bases: OODBenchmarkPipeline

class detectors.pipelines.osr.SingleClassCifar10(in_dataset_name: str, in_dataset_split: str, transform: Callable, batch_size: int, num_workers: int = 4, pin_memory: bool = True, prefetch_factor: int = 2, limit_fit: float = 1.0, limit_run: float = 1.0, seed: int = 42, **kwargs)

Bases: Pipeline

setup()

detectors.pipelines.ttoodda module

Module contents

Pipeline module.

class detectors.pipelines.PipelinesRegistry(value)

Bases: Enum

An enumeration.

covariate_drift_cifar10 = 'covariate_drift_cifar10'

covariate_drift_cifar100 = 'covariate_drift_cifar100'

covariate_drift_imagenet = 'covariate_drift_imagenet'

ood_benchmark_cifar10 = 'ood_benchmark_cifar10'

ood_benchmark_cifar100 = 'ood_benchmark_cifar100'

ood_benchmark_imagenet = 'ood_benchmark_imagenet'

ood_mnist_benchmark = 'ood_mnist_benchmark'

ood_validation_cifar10 = 'ood_validation_cifar10'

ood_validation_cifar100 = 'ood_validation_cifar100'

ood_validation_imagenet = 'ood_validation_imagenet'

ood_validation_noise_cifar10 = 'ood_validation_noise_cifar10'

ood_validation_noise_cifar100 = 'ood_validation_noise_cifar100'

detectors.pipelines.create_pipeline(name: str, dataset_name: str | None = None, **kwargs) → Pipeline

Utility factory method to build a Pipeline.

Parameters:

• name (str, optional) –

  The name defining which pipeline will be returned. Currently accepted pipeline names are:

  ood_benchmark, ood_validation, ood_validation_noise, covariate_drift, osr

  Complete list:

  ood_benchmark_cifar10, ood_benchmark_cifar100, ood_benchmark_imagenet, ood_mnist_benchmark, ood_validation_cifar10, ood_validation_cifar100, ood_validation_imagenet, ood_validation_mnist, ood_validation_noise_cifar10, ood_validation_noise_cifar100, ood_validation_noise_imagenet, covariate_drift_cifar10, covariate_drift_cifar100, covariate_drift_imagenet.

• **kwargs – Additional arguments to pass to the pipeline.

Returns:

A suitable pipeline for the task.

Return type:

Pipeline

Example:

>>> import detectors
>>> pipe = detectors.create_pipeline("ood_benchmark", "cifar10")
>>> pipe.run(detector)

detectors.pipelines.list_pipeline_args(name: str) → List[Tuple[str, Any]]

List all available arguments for a given pipeline.

Parameters:

name (str) – The name of the pipeline.

Returns:

A list of available arguments and default values for the pipeline.

Return type:

list

detectors.pipelines.list_pipelines() → List[str]

List all available pipelines.

Returns:

A list of available pipelines.

Return type:

List[str]

detectors.pipelines.register_pipeline(name: str, dataset_name: str | None = None)

Decorator to register a new pipeline in the registry.

Parameters:

name (str) – The name of the pipeline to register.