Benchmark

class cl_gym.benchmarks.Benchmark(num_tasks: int, per_task_examples: Optional[int] = None, per_task_joint_examples: Optional[int] = 0, per_task_memory_examples: Optional[int] = 0, per_task_subset_examples: Optional[int] = 0, task_input_transforms: Optional[list] = None, task_target_transforms: Optional[list] = None)[source]

Bases: object

Base class for continual learning benchmarks. It implements logic for loading/serving continual learning datasets.

load(task: int, batch_size: int, shuffle: Optional[bool] = True, num_workers: Optional[int] = 0, pin_memory: Optional[bool] = True)Tuple[torch.utils.data.dataloader.DataLoader, torch.utils.data.dataloader.DataLoader][source]

Makes train/val dataloaders for a specific task. :param task: The task number. :param batch_size: The batch_size for dataloaders. :param shuffle: Should loaders be shuffled? Default: True. :param num_workers: corresponds to Pytorch’s num_workers argument. Default: 0 :param pin_memory: corresponds to Pytorch’s pin_memory argument. Default: True.

Returns

a Tuple of dataloaders, i.e., (train_loader, validation_loader).

Examples::
>>> benchmark = Benchmark(num_tasks=2)
>>> # task 1 loaders
>>> train_loader_1, val_loader_1 = benchmark.load(1, batch_size=32)
>>> # task 2 loaders
>>> train_loader_2, val_loader_2 = benchmark.load(2, batch_size=64)
load_datasets()[source]

Loading datasets from file.

load_joint(task: int, batch_size: int, shuffle: Optional[bool] = True, num_workers: Optional[int] = 0, pin_memory: Optional[bool] = True)Tuple[torch.utils.data.dataloader.DataLoader, torch.utils.data.dataloader.DataLoader][source]

Makes dataloaders for joint/multitask settings. i.e., for task t returns datasets for tasks 1, 2, …, t-1, t.

Parameters
  • task – The task number.

  • batch_size – The batch_size for dataloaders.

  • shuffle – Should loaders be shuffled? Default: True.

  • num_workers – corresponds to Pytorch’s num_workers argument. Default: 0

  • pin_memory – corresponds to Pytorch’s pin_memory argument. Default: True.

Returns

a Tuple of dataloaders, i.e., (train_loader, validation_loader).

Examples::
>>> benchmark = Benchmark(num_tasks=2, per_task_joint_examples=128)
>>> # task 1 loaders (single): returns 4 batches (i.e., 128 examples)
>>> train_loader_1, val_loader_1 = benchmark.load(1, batch_size=32)
>>> # task 1 loaders (joint): returns 4 batches (i.e., 128 examples)
>>> joint_train_loader_1, joint_val_loader_1 = benchmark.load_joint(1, batch_size=32)
>>> # task 1 loaders (single): returns 4 batches (i.e., 128 examples)
>>> train_loader_2, val_loader_2 = benchmark.load(2, batch_size=32)
>>> # task 1 loaders (single): returns 8 batches (i.e., 256 examples)
>>> joint_train_loader_2, joint_val_loader_2 = benchmark.load(2, batch_size=32)

Warning

The method will throw an error if Benchmark is instantiated without per_task_joint_examples. The reason is that, behind the scenese, we compute the indices for joint examples in precompute_joint_indices() method and this method relies on that computations.

load_memory(task: int, batch_size: int, shuffle: Optional[bool] = True, num_workers: Optional[int] = 0, pin_memory: Optional[bool] = True)Tuple[torch.utils.data.dataloader.DataLoader, torch.utils.data.dataloader.DataLoader][source]

Makes dataloaders for episodic memory/replay buffer.

Parameters
  • task – The task number.

  • batch_size – The batch_size for dataloaders.

  • shuffle – Should loaders be shuffled? Default: True.

  • num_workers – corresponds to Pytorch’s num_workers argument. Default: 0

  • pin_memory – corresponds to Pytorch’s pin_memory argument. Default: True.

Returns

a Tuple of dataloaders, i.e., (train_loader, validation_loader).

Examples::
>>> benchmark = Benchmark(num_tasks=2, per_task_memory_examples=16)
>>> # task 1 memory loaders: returns 2 batches (i.e., 16 examples)
>>> mem_train_loader_1, mem_val_loader_1 = benchmark.load_memory(1, batch_size=8)
>>> # task 2 memory loaders: returns 4 batches (i.e., 16 examples)
>>> mem_train_loader_2, mem_val_loader_2 = benchmark.load_memory(2, batch_size=4)

Note

This method uses class_uniform sampling. i.e., if each task has 10 classes, and per_task_memory_examples=20, then the returend samples have 2 examples per class.

Warning

The method will throw an error if Benchmark is instantiated without per_task_memory_examples. The reason is that, behind the scenese, we compute the indices for memory examples in precompute_memory_indices() method and this method relies on that computations.

load_memory_joint(task: int, batch_size: int, shuffle: Optional[bool] = True, num_workers: Optional[int] = 0, pin_memory: Optional[bool] = True)Tuple[torch.utils.data.dataloader.DataLoader, torch.utils.data.dataloader.DataLoader][source]
load_subset(task: int, batch_size: int, shuffle: Optional[bool] = True, num_workers: Optional[int] = 0, pin_memory: Optional[bool] = True)Tuple[torch.utils.data.dataloader.DataLoader, torch.utils.data.dataloader.DataLoader][source]
precompute_joint_indices()[source]

For each task, (randomly) computes the indices of the subset of data points in the task’s dataset. Then, once load_joint() method is called, uses these indices to return a PyTorch Subset dataset. .. note:: This method will be called only if the benchmark is initiated with per_task_joint_examples.

precompute_memory_indices()[source]

For each task, (randomly) computes the indices of the subset of data points in the task’s dataset. Then, once load_memory() method is called, uses these indices to return a PyTorch Subset dataset. .. note:: This method will be called only if the benchmark is initiated with per_task_memory_examples.

precompute_seq_indices()[source]
precompute_subset_indices()[source]
prepare_datasets()[source]

Prepares datasets: will be called after load_datasets. Responsible for computing index for various methods. E.g., selecting subset/memory indices for each task.

sample_uniform_class_indices(dataset, start_class, end_class, num_samples)List[source]

Selects a subset of size num_samples from start_class to end_class. :param dataset: The input dataset that the samples will be drawn from. :param start_class: The start_class (inclusive) :param end_class: The end_classes (inclusive) :param num_samples: Number of samples.

Returns

Indices of the selected samples.

Note

This method is specially useful for split datasets. E.g., selecting classes 0 to 4 for Split-CIFAR-100.

Warning

If num_samples > len(dataset), then the output`s shape will be equal to len(dataset).

sanity_check_inputs()[source]

A method for sanity checking arguments. E.g., checking if all provided arguments are valid.

sanity_check_transforms()[source]