Benchmark¶
-
class
cl_gym.benchmarks.
Benchmark
(num_tasks: int, per_task_examples: Optional[int] = None, per_task_joint_examples: Optional[int] = 0, per_task_memory_examples: Optional[int] = 0, per_task_subset_examples: Optional[int] = 0, task_input_transforms: Optional[list] = None, task_target_transforms: Optional[list] = None)[source]¶ Bases:
object
Base class for continual learning benchmarks. It implements logic for loading/serving continual learning datasets.
-
load
(task: int, batch_size: int, shuffle: Optional[bool] = True, num_workers: Optional[int] = 0, pin_memory: Optional[bool] = True) → Tuple[torch.utils.data.dataloader.DataLoader, torch.utils.data.dataloader.DataLoader][source]¶ Makes train/val dataloaders for a specific task. :param task: The task number. :param batch_size: The batch_size for dataloaders. :param shuffle: Should loaders be shuffled? Default: True. :param num_workers: corresponds to Pytorch’s num_workers argument. Default: 0 :param pin_memory: corresponds to Pytorch’s pin_memory argument. Default: True.
- Returns
a Tuple of dataloaders, i.e., (train_loader, validation_loader).
- Examples::
>>> benchmark = Benchmark(num_tasks=2) >>> # task 1 loaders >>> train_loader_1, val_loader_1 = benchmark.load(1, batch_size=32) >>> # task 2 loaders >>> train_loader_2, val_loader_2 = benchmark.load(2, batch_size=64)
-
load_joint
(task: int, batch_size: int, shuffle: Optional[bool] = True, num_workers: Optional[int] = 0, pin_memory: Optional[bool] = True) → Tuple[torch.utils.data.dataloader.DataLoader, torch.utils.data.dataloader.DataLoader][source]¶ Makes dataloaders for joint/multitask settings. i.e., for task t returns datasets for tasks 1, 2, …, t-1, t.
- Parameters
task – The task number.
batch_size – The batch_size for dataloaders.
shuffle – Should loaders be shuffled? Default: True.
num_workers – corresponds to Pytorch’s num_workers argument. Default: 0
pin_memory – corresponds to Pytorch’s pin_memory argument. Default: True.
- Returns
a Tuple of dataloaders, i.e., (train_loader, validation_loader).
- Examples::
>>> benchmark = Benchmark(num_tasks=2, per_task_joint_examples=128) >>> # task 1 loaders (single): returns 4 batches (i.e., 128 examples) >>> train_loader_1, val_loader_1 = benchmark.load(1, batch_size=32) >>> # task 1 loaders (joint): returns 4 batches (i.e., 128 examples) >>> joint_train_loader_1, joint_val_loader_1 = benchmark.load_joint(1, batch_size=32) >>> # task 1 loaders (single): returns 4 batches (i.e., 128 examples) >>> train_loader_2, val_loader_2 = benchmark.load(2, batch_size=32) >>> # task 1 loaders (single): returns 8 batches (i.e., 256 examples) >>> joint_train_loader_2, joint_val_loader_2 = benchmark.load(2, batch_size=32)
Warning
The method will throw an error if Benchmark is instantiated without per_task_joint_examples. The reason is that, behind the scenese, we compute the indices for joint examples in precompute_joint_indices() method and this method relies on that computations.
-
load_memory
(task: int, batch_size: int, shuffle: Optional[bool] = True, num_workers: Optional[int] = 0, pin_memory: Optional[bool] = True) → Tuple[torch.utils.data.dataloader.DataLoader, torch.utils.data.dataloader.DataLoader][source]¶ Makes dataloaders for episodic memory/replay buffer.
- Parameters
task – The task number.
batch_size – The batch_size for dataloaders.
shuffle – Should loaders be shuffled? Default: True.
num_workers – corresponds to Pytorch’s num_workers argument. Default: 0
pin_memory – corresponds to Pytorch’s pin_memory argument. Default: True.
- Returns
a Tuple of dataloaders, i.e., (train_loader, validation_loader).
- Examples::
>>> benchmark = Benchmark(num_tasks=2, per_task_memory_examples=16) >>> # task 1 memory loaders: returns 2 batches (i.e., 16 examples) >>> mem_train_loader_1, mem_val_loader_1 = benchmark.load_memory(1, batch_size=8) >>> # task 2 memory loaders: returns 4 batches (i.e., 16 examples) >>> mem_train_loader_2, mem_val_loader_2 = benchmark.load_memory(2, batch_size=4)
Note
This method uses class_uniform sampling. i.e., if each task has 10 classes, and per_task_memory_examples=20, then the returend samples have 2 examples per class.
Warning
The method will throw an error if Benchmark is instantiated without
per_task_memory_examples
. The reason is that, behind the scenese, we compute the indices for memory examples in precompute_memory_indices() method and this method relies on that computations.
-
load_memory_joint
(task: int, batch_size: int, shuffle: Optional[bool] = True, num_workers: Optional[int] = 0, pin_memory: Optional[bool] = True) → Tuple[torch.utils.data.dataloader.DataLoader, torch.utils.data.dataloader.DataLoader][source]¶
-
load_subset
(task: int, batch_size: int, shuffle: Optional[bool] = True, num_workers: Optional[int] = 0, pin_memory: Optional[bool] = True) → Tuple[torch.utils.data.dataloader.DataLoader, torch.utils.data.dataloader.DataLoader][source]¶
-
precompute_joint_indices
()[source]¶ For each task, (randomly) computes the indices of the subset of data points in the task’s dataset. Then, once load_joint() method is called, uses these indices to return a PyTorch Subset dataset. .. note:: This method will be called only if the benchmark is initiated with per_task_joint_examples.
-
precompute_memory_indices
()[source]¶ For each task, (randomly) computes the indices of the subset of data points in the task’s dataset. Then, once load_memory() method is called, uses these indices to return a PyTorch Subset dataset. .. note:: This method will be called only if the benchmark is initiated with per_task_memory_examples.
-
prepare_datasets
()[source]¶ Prepares datasets: will be called after load_datasets. Responsible for computing index for various methods. E.g., selecting subset/memory indices for each task.
-
sample_uniform_class_indices
(dataset, start_class, end_class, num_samples) → List[source]¶ Selects a subset of size num_samples from start_class to end_class. :param dataset: The input dataset that the samples will be drawn from. :param start_class: The start_class (inclusive) :param end_class: The end_classes (inclusive) :param num_samples: Number of samples.
- Returns
Indices of the selected samples.
Note
This method is specially useful for split datasets. E.g., selecting classes 0 to 4 for Split-CIFAR-100.
Warning
If num_samples > len(dataset), then the output`s shape will be equal to len(dataset).
-