Avocado Chairs at the Intersection of Human Language and Neural Networks

During the last year, we’ve seen how machine learning has evolved by leaps and bounds, especially when it came to natural language processing (NLP), a branch of artificial intelligence research that explores how machines can read and decipher meaning from human languages.

One particularly important breakthrough was last year’s release of GPT-3, the massive 175-billion parameter NLP model created by artificial intelligence lab OpenAI. As the largest NLP model to date, GPT-3 performed all kinds of impressive feats, from generating texts of all kinds that were almost indistinguishable from those made by humans. Beyond the mere generation of poems, short stories, technical manuals and the like, GPT-3’s versatility was also apparent in its ability to solve simple math problems, and in even producing programming code.

Now, OpenAI has released DALL·E, a scaled-down version of GPT-3 with 12 billion parameters that’s capable of generating images when given a text description, using a dataset of text-image pairs.

“We’ve found that it has a diverse set of capabilities, including creating anthropomorphized versions of animals and objects, combining unrelated concepts in plausible ways, rendering text, and applying transformations to existing images,” explained the OpenAI team on their blog.

Awesome Avocado Chairs

The new findings are quite impressive. For example, here’s what DALL·E serves up when given the text prompt “an armchair in the shape of an avocado.”:

These samples are pretty remarkable, in the sense that a human designer could have come up with them. The OpenAI team says that in this instance, DALL·E was able to take inspiration from two unrelated ideas, while simultaneously creating something that takes into account the form and practicality of the final object. “We found that prompting DALL·E with the phrases ‘in the shape of,’ ‘in the form of,’ and ‘in the style of’ gives it the ability to do this,” they explained on their blog.

Like previous versions of the GPT models, DALL·E uses an architecture known as transformer neural networks, which are capable of handling data in a non-sequential way. This means that when applied to speech recognition, translation and text summarizing tasks, they can process the end of a sentence before parsing the beginning. This kind of ability facilitates greater parallelization than prior NLP models, which also permits transformer-based models to handle much larger data sets, while simultaneously reducing the time it takes to train such enormous models.

According to the team, the training procedure behind DALL·E involves the model receiving both the text and image as a single stream of data comprising of up to 1280 “tokens.” A token is any symbol from a distinct vocabulary; for instance, each letter from the English alphabet represents a token from an alphabet of 26 letters. In DALL·E’s case, its vocabulary has tokens from both text and image concepts.

“This training procedure allows DALL·E to not only generate an image from scratch, but also to regenerate any rectangular region of an existing image that extends to the bottom-right corner, in a way that is consistent with the text prompt,” wrote the team.

DALL·E also builds upon OpenAI’s Image GPT, which applies GPT-like transformations to object recognition and object detection tasks, allowing the model to generate coherent images using pixel sequences, rather than text-based ones, and without the need for manually labeling the data.

While the complete technical details behind DALL·E won’t be apparent until OpenAI publishes their full paper, it’s nevertheless clear that such a model will be quite powerful and have a wide variety of applications. For instance, take a look at what the model comes up with when given the prompt “a living room with two white armchairs and a painting of the colosseum. the painting is mounted above a modern fireplace”:

As one might imagine, such a tool would be helpful to architects and interior designers alike, or perhaps give a boost to game designers looking to automatically generate virtual game environments quickly.

Surprisingly, beyond being able to generate images from text captions, DALL·E is also capable of zero-shot reasoning, which means that it’s able to accomplish something that it was not initially trained to do. For instance, the team gave DALL·E the task of reproducing “the exact same cat on the top as a sketch on the bottom” and rather than creating new images, it reprised them:

Depending on how clearly the text prompt is phrased, DALL·E also seems to understand how to reproduce multiple objects with different characteristics, and in different spatial configurations in three dimensions in the same scene, in addition to visualizing various internal and external structures like cross-sections or close-up details of objects. DALL·E is also proficient in generating images that rely on geographical knowledge (“a photo of the food of china”) and temporal knowledge (“a photo of a phone from the 20s”):

Notably, DALL·E is also able to create convincing images of unrelated concepts that don’t exist in the real world, as shown when it’s given the prompt: “a professional high-quality illustration of a giraffe turtle chimera.”

Needless to say, while there are still some weaknesses in the model to improve, DALL·E appears to be quite a versatile model with plenty of potential uses, and possibly any number of broad, unforeseen socio-economic impacts in the future. The team is now working to refine the model and identify any potential biases it may have.

Read more in OpenAI’s blog.

Images: OpenAI