Zero/Few/One-Shot Learning

Machine learning has become increasingly popular over the years, with many companies investing heavily in deep learning models to solve their more complex problems. However, collecting and labelling data can be both time-consuming and expensive, making it difficult for many companies to get started. It's a delicate balance that can leave us feeling a little like Goldilocks – too little data and it won't work, too much data and it'll overfit. It's enough to make any company think twice before diving into the world of machine learning.

This dilemma has led to the rise of alternative approaches such as zero/few/one-shot learning, which allow machines to learn from a limited amount of data or even just one example. How do they work, and are they as effective as traditional deep learning models?

‍

Zero-shot

Zero-shot learning is a type of machine learning that allows for a system to train without having any training examples from the target class. This is useful because it allows us to apply our models and techniques in new contexts where there are no labelled datasets available.

In order to achieve this, we need to build a mapping between the high level information and the input image in order to determine the class label. This can be done using a new type of neural network called a transformer, which are quickly becoming state of the art in many applications. 

This approach has been shown to work well in natural language processing tasks where there are no labelled examples available - such as when trying to transcribe an audio recording into text or translate between languages where there are no parallel corpora available.

‍

One-shot

One-shot learning is a technique that allows researchers to train a machine learning algorithm on a small amount of data, then have it perform well on a much larger dataset.

The technique uses the idea that if you train on one example from each class, the algorithm will be able to generalise to new examples from that class. This is surprising because we would expect the training set to overfit to the single example and not generalise well at all. But in fact, it turns out that this approach works surprisingly well. The algorithm is trained to compare examples and find a match on a large related dataset.

One-shot learning is being used today in many facial recognition tasks. This allows these products to work reliably while only needing to store one reference image.

‍

‍

Few-shot

Few-shot learning is a type of machine learning that can be used for solving problems with few examples.

The idea behind few-shot learning is simple: it uses a concept called the support set to find the most likely match for an input from a small set of examples. A few examples are needed because the system needs to learn from the examples.

In traditional machine learning, if you're trying to train a system to recognize dogs in photos, you need to give it enough information so that it can distinguish between different types of dogs (e.g., Labrador Retriever vs. Poodle). But how many photos should you give it? If you don't give it enough, then it will overfit and not be able to make reliable predictions.

The idea behind a support set is that we look at all possible matches for our input and then select the one that has the most support among our examples. For example, if we have an image of a dog that might be either a Labrador Retriever or Poodle (or any other type of dog), we will take all possible matches for each kind of dog and rank them by how similar they are to our original image (using some distance metric). We then choose one of these matches as the 'true' label for our input image.

‍

What are the upsides/downsides?

In the past, it was often difficult for businesses to justify the cost of collecting and labeling data for use in machine learning. But with recent advances in zero/one/few-shot learning, the rise of deep learning has been incredible. Models have gotten more accurate and are able to do things that were previously impossible without massive amounts of data. 

Deep learning models are expensive to train and can take a long time to be ready for production use. Collecting and labelling data is also expensive, which makes it hard for companies to get started with machine learning.

In the past, a lot of companies would just stick with old-fashioned decision trees or other simpler models because they don’t have the data or computational resources to train a deep learning model. Now, thanks to new research on transformer architectures for deep neural networks, zero-shot learning becomes feasible for many applications: we can train one model that can do many things! This means that we don't need separate models for every application; instead, we can use one model across multiple domains and products.

However, this comes at a cost: zero/one/few-shot models have lower accuracy than regular deep learning models on tasks with large all encompassing datasets. This is still an active area of research in many domains so there are less pre-trained tools out there for all applications/domains.

‍

Where can you use these approaches?

These approaches are often used in proof of concept projects and demos because it allows businesses to get a feel for how well their algorithms will work before they invest in collecting more data or hiring more employees. It's also used in areas where data collection/labelling is expensive or invasive (eg: medical), areas with a large amount of weakly labelled data (eg: audio, images, video), and any sort of facial recognition task. If you're working on any sort of visual recognition task, zero-shot learning will allow you to train a model without having access to any labeled images for training purposes.

Deep learning has allowed developers to find solutions to problems that were previously unsolved. Zero/one/few-shot learning has allowed computers to begin making sense of the world around them and is the next big step in machine learning. We look forward to seeing how these developments will continue to push artificial intelligence into new territories.