Introduction

Meta-learning research and open source libraries have offered a way to get a detailed comparison between different algorithms. It offers standardized benchmarks and a wide range of datasets available allowing full control over the complexity of algorithm training and testing,However, most of the code available online suffers from the following limitations:

• The data pipeline built in traditional and conventional production systems relies on training dataset when the model is first built. The model is often tested on another dataset. Further retraining on another disjoint dataset also contributes to significant rework.

• The benchmarks in meta-learning consist of datasets, adding a layer of complexity to the data pipeline. Hence the majority of meta-learning projects implement their own specific data-loading component adapted to their method.

• Lack of a standard at the input level creates variance in the mechanisms surrounding each meta-learning algorithm, making the comparison process more challenging.

To handle this limitation Mila has introduced Torchmeta, a library built on top of PyTorch deep learning framework enables seamless and consistent evaluation of meta-learning algorithms on multiple datasets. To explain Torchmeta, leveraging meta-learning, some preliminary concepts like DataLoader and BatchLoader has been used below. This can be explained as:

“DataLoader is a generic utility to be used as part of any application data. It has an embedded fetching layer to provide a simplified and consistent API over various remote data sources. Data sources are not only limited to databases, but also includes web services via batching and caching”.

“Batching is a DataLoader’s primary feature. A batch loading function accepts a list of keys, and returns a Promise which resolves to a list of values DataLoader coalesces all individual loads which occur within a single frame of execution (executed once the wrapping promise is resolved), followed by batch function with all requested keys”.

Torchmeta comes with the following functionalities.

• Torchmeta provides DataLoaders for most of the standard benchmarks in few-shot classification and regression, with a new meta-dataset abstraction.

• The data-loaders are fully compatible with standard data components of PyTorch, such as Dataset and DataLoader.

• Torchmeta provides the “same interface” for all the available benchmarks, making the transition between different datasets as seamless as possible.

• Torchmeta also features some extensions for PyTorch to simplify the development of models compatible with meta-learning algorithms, some of which requires higher-order differentiation.

• The available benchmarks have helped to provide a reference point for the development of new meta-learning algorithms.

• Torchmeta vastly facilitates the research community by enabling a framework around which researchers can customize, extend and build their own meta-learning algorithms. It offers mush easier multiple choices over adapting the data pipeline to their methods.

• This layer of abstraction provided Torchmeta promotes code reuse, by decoupling meta-datasets from the algorithm itself.

Data-loaders for few-shot learning

Few-shot learning comes with the capability of rapidly generalizing new tasks of limited supervised experience using prior knowledge. Few-shot learning comes in three categories:

• Data uses prior knowledge to exhibit and increase the supervised experience.

• The model constrains the hypothesis space by prior knowledge.

• Algorithm uses prior learnt knowledge to alter the search space in order to select the best hypothesis.

Torchmeta has the following in its library.

• The library provides a collection of datasets corresponding to classic few-shot classification and regression problems from the meta-learning literature.

• The interface is built to support modularity between datasets, for both classification and regression, to simplify the process of evaluation on a full suite of benchmarks.

Often data recorded in few-shot learning is limited where meta-learning comes to play, meta-learning algorithms acquire some prior knowledge from a collection of datasets D-meta ={D1,…,Dn}, called the meta-training set.

In few-shot learning, each element Di contains only a few inputs/output pairs (x, y), where y depends on the nature of the problem. As these datasets can contain examples of different tasks performed in the past. Torchmeta offers a solution to automate the creation of each dataset Di, with a minimal amount of problem-specific components.

Few-shot regression

A majority of the few-shot regression problems are simple regression problems between inputs and outputs through different functions, where each function corresponds to a task. These functions are parameterised to allow variability between tasks while preserving a constant “theme” across tasks.

For example, these functions can be sine waves of the form:

fi (x) = ai sin(x + bi), with a and b varying in some range.

In Torchmeta, the meta-training set, termed as MetaDataset can be used build any dataset. It gets inherited from an object. Each dataset Di (i = 1, . . . , n, with n defined by the user) corresponds to a specific choice of parameters for the function, with all the parameters sampled once at the creation of the meta-training set.

Once the parameters of the function are known, we can create the dataset by sampling inputs in a given range, and feeding them to the function.

Few-shot classification

For few-shot classification problems, the creation of the datasets Di usually follows two steps:

• The first N classes are sampled from a large collection of candidates , termed as “N -way classification”.

• In the next step, k examples are chosen per class , termed as “k-shot learning”.

• The two-step process is automated. The consolidated object is termed as CombinationMetaDataset, and its inherited from MetaDataset. It’s capable of providing user specification of a problem-specific large collection of class candidates.

• To encourage reproducibility in meta-learning, every task is associated with a unique identifier (the N-tuple of class identifiers). Once the task has been chosen, the object returns a dataset Di with all the examples from the corresponding set of classes.

• Torchmeta also includes helpful functions to customize existing class features with variants, such as rotated images.

The above figure demonstrates the role of meta-learner. A disjoint set of tasks Tt ∼ p(T ) (with p(T ) ->the distribution of task T) is used to test the generalization ability along with the impact of variation introduced to these meta-learner.

Each Tt works on a dataset of N where dataset = {D train Tt, D test Tt }. The learner learns from training set D train Tt and tests on test set D test Tt. The averaged loss across Tt ’s is taken as the meta-learning testing error.

Training & test datasets split

• In meta-learning, each dataset Di is separated in two parts: a training set (or support set) to adapt the model to the task at hand, and a test set (or query set) for evaluation and meta-optimization.

• The two parts do not overlap while the task remains the same, no example can be in both the training and test sets.

• Torchmeta introduces a wrapper over the datasets called a Splitter that is responsible for creating the training and test datasets, as well as optionally shuffling the data.

To instantiate the meta-training set of a 5-way 1-shot classification problem based on Mini Imagenet, we use:

dataset = torchmeta.datasets.MiniImagenet(“data”, num_classes_per_task=5, meta_train=True, download=True)

dataset = torchmeta.transforms.ClassSplitter(dataset, num_train_per_class=1, num_test_per_class=15, shuffle=True)

In addition to the meta-training set, most benchmarks also provide a meta-test set for the overall evaluation of the meta-learning algorithm (and possible a meta-validation set as well). These different meta-datasets can be selected when the MetaDataset object is created, with meta_test=True (or meta_val=True) instead of meta_train=True.

Meta Data-loaders

Meta-training set objects in few shot Classification and Regression problems can be iterated over to generate PyTorch Dataset objects, included as part of any standard data pipeline (combined with DataLoader).

Meta-learning algorithms operate better on batches of tasks. Similar to how examples are batched together with DataLoader in PyTorch, Torchmeta exposes a MetaDataLoader that can produce batches of tasks when iterated over. Such a meta data-loader is able to output a large tensor containing all the examples from the different tasks in the batch, as stated below in the following:

dataset = torchmeta.datasets.helpers.miniimagenet(“data”, shots=1, ways=5, meta_train=True, download=True)

dataloader = torchmeta.utils.data.BatchMetaDataLoader(dataset, batch_size=16)

Meta-learning modules

The figure below shows the sequential steps to meta-learning using loss and error signal from a learner.

Meta learning’s sequential steps involve loss reduction by receiving error signal from a learner.

In meta-learning models in PyTorch are created from basic components called modules, which is equivalent to a layer in a neural network containing both the computational graph of that layer, as well as its parameters. The modules treat their parameters as an integral part of their computational graph, sufficient to train a model with back-propagation.

However, some meta-learning algorithms require backpropagate through an update of the parameters (like gradient update) for the meta-optimization (or the “outer-loop”), hence involving higher-order differentiation.

It is therefore critical to adapt the existing modules in PyTorch so they can handle arbitrary computational graphs as a substitute for these parameters.

Hence Torchmeta extends the existing modules, and leave an option to provide new parameters as an additional input. These new objects are called MetaModule, and their default behaviour (i.e. without any extra parameter specified) is equivalent to their PyTorch counterpart. Otherwise, if extra parameters (such as the result of one step of gradient descent) are specified, then the MetaModule treats them as part of the computational graph, and back-propagation works as expected.

• The above figure depicts how the extension of the Linear module called MetaLinear works, with and without additional parameters, and the impact on the gradients.

• The figure on the left shows the instantiation of the meta-module as a container for the parameters W & b, and the computational graph with placeholders for the weight and bias parameters.

• The figure in the middle shows the default behaviour of the MetaLinear meta-module, where the placeholders are substituted with W & b, this is equivalent to PyTorch’s Linear module.

• The figure on the right shows how these placeholders can be filled with a complete computational graph, like one step of gradient descent. In this latter case, the gradient of outer-loop with respect to W, necessary in the outer-loop update, can correctly flow all the way to the parameter W.

The code below demonstrates how can we generate training, validation and test meta datasets from existing datasets of Torchmeta (https://github.com/markdtw/meta-learning-lstm-pytorch).

from torchmeta.datasets import Omniglot, MiniImagenet, CIFARFS, FC100, TieredImagenet, TCGA
from torchmeta.transforms import Categorical, ClassSplitter, Rotation
from torchvision.transforms import Compose, Resize, ToTensor
from torchmeta.utils.data import BatchMetaDataLoader

dataset = Omniglot("data",
                   # Number of ways
                   num_classes_per_task=5,
                   # Resize the images to 28x28 and converts them to PyTorch tensors (from Torchvision)
                   transform=Compose([Resize(28), ToTensor()]),
                   # Transform the labels to integers (e.g. ("Glagolitic/character01", "Sanskrit/character14", ...) to (0, 1, ...))
                   target_transform=Categorical(num_classes=5),
                   # Creates new virtual classes with rotated versions of the images (from Santoro et al., 2016)
                   class_augmentations=[Rotation([90, 180, 270])],
                   meta_train=True,
                   download=True)
                   
dataset = ClassSplitter(dataset, shuffle=True, num_train_per_class=5, num_test_per_class=15)
dataloader = BatchMetaDataLoader(dataset, batch_size=16, num_workers=4)
for batch in dataloader:
    train_inputs, train_targets = batch["train"]
    print('Train inputs shape: {0}'.format(train_inputs.shape))    # (16, 25, 1, 28, 28)
    print('Train targets shape: {0}'.format(train_targets.shape))  # (16, 25)

    test_inputs, test_targets = batch["test"]
    print('Test inputs shape: {0}'.format(test_inputs.shape))      # (16, 75, 1, 28, 28)
    print('Test targets shape: {0}'.format(test_targets.shape))    # (16, 75)

The below image shows meta-learning datasets generated from Omniglot and MiniImagenet from Torchmeta’s datasets after download.

Here Omniglot dataset contains 50 alphabets. The dataset is split into training and test set in a range of 30 alphabets to 20 alphabets. The background set should is used to learn general knowledge about characters (e.g., feature learning, meta-learning), after being resized to 28×28 tensors.

As taken from github:

“MiniImageNet consists of 60,000, 84×84 RGB images with 600 images per class”.

With Torchmeta we can generate meta-learning datasets in HDF5 format.

Torchmeta has in-built facility of allowing download of datasets in HDF5 format that allows:

1. To use folders (including subfolders) containing HDF5 files as a data source.
2. Maintain and structure HDF5 group hierarchy in the Dataset.
3. Enable lazy data loading (i.e. upon request by the DataLoader) in order to allow working with Datasets which do not fit into memory.
4. Equipped with a data cache to speed up the data loading process, andAllow custom transforms of the source or target datasets.

TieredImagenetClassDataset used in defining the meta learning parameters of Torchmeta datasets like Omniglot, contains images from 34 categories. The meta train/validation/test splits are over 20/6/8 categories. Each category contains between 10 and 30 classes.

The splits over categories (instead of over classes) ensure that all the training classes are sufficiently distinct from the test classes (unlike Mini-Imagenet). It comes with the following set of parameters, that defines the splits of training, validation and testing datasets in addition to transformation and augmentation techniques applied to them.

Conclusion

In this blog, we understand the newly released library Torchmeta as to what features it provides and what sort of meta-learning problems it can solve. You can explore other PyTorch meta-learning libraries like https://github.com/dragen1860/MAML-Pytorch (Meta Agonistic Machine Learning) to learn a network initialisation that can quickly adapt to new tasks.

References

1. https://github.com/markdtw/meta-learning-lstm-pytorch
2. https://arxiv.org/abs/1909.06576
3. https://docs.graphene-python.org/en/latest/execution/dataloader/