Microsoft today is unveiling new artificial intelligence technology that’s something of an artist – a “drawing bot.” The bot is capable of creating images from text descriptions of an object, but it also adds details to those images that weren’t included the text, indicating that the AI has a little imagination of its own, says Microsoft.
“If you go to Bing and you search for a bird, you get a bird picture. But here, the pictures are created by the computer, pixel by pixel, from scratch,” explained Xiaodong He, a principal researcher and research manager in the Deep Learning Technology Center at Microsoft’s research lab in Redmond, Washington, in Microsoft’s announcement. “These birds may not exist in the real world — they are just an aspect of our computer’s imagination of birds.”
The bot is able to generate a variety of images, researchers say, including everything from “ordinary pastoral scenes,” like those with grazing livestock, to the absurd – like “a floating double-decker bus.”
Microsoft says the bot was trained on datasets of paired images and captions, allowing it to understand how to match up words to images. It learned to draw a bird, for example, when the caption says “bird,” and it learned what a picture of a bird should look like.
“That is a fundamental reason why we believe a machine can learn,” said He.
The drawing bot’s technology consists of two machine learning models – one to generate images from text descriptions, and another that uses text descriptions to judge the authenticity of the generated images. The former tries to get fake pictures past the latter, but the latter doesn’t want to be fooled. Combined, the two work together to create higher-quality images.
In particular, it’s good at drawing images from more complex sentences. While other technology could draw a bird from a caption that said “bird,” for instance, the quality would decline if you told it to draw a bird with a green crown, yellow wings and red belly. Before, this would have led to a blurry “greenish-yellowish-reddish bird,” Microsoft explains.
What’s especially interesting is how the bot can fill in the blanks when specific details aren’t mentioned – basically, it has a little common sense and imagination of its own, thanks to its training data. In the bird example, the bot will usually draw a bird sitting on a tree branch even if that’s not stated in the text, because the images originally fed to it often showed something similar.
According to a recent research paper, this new bot represents a nearly three-fold boost in image quality compared to prior technology, Microsoft also notes.
This isn’t the first technology to combine AI with art, of course.
The intersection of the two sometimes leads to fantastic results, such as when Google’s artificial intelligence drew those trippy machine-generated images that got their own art show. Google also has a neural net that guesses what you’re drawing and an automated drawing bot, and regularly details the research it’s doing in helping machines learn how to draw.
Facebook, too, has worked on teaching neural networks to make small images of things like airplanes, cars and animals, and even creating your own Bitmoji-like avatar from a photo.
Nvidia researchers created computer-generated celebrities, using A.I.
And there have been a slew of recent advancements in this space from others, as well.
For Microsoft, teaching a bot to draw an image from text completes the circle of other work its been doing in this computer vision and natural language processing space.
This includes the development of CaptionBot which automatically writes photo captions, and a technology that can answer questions people ask about images, like the location or attributes of objects – things that can be helpful for blind people.
As to what purpose an AI artist may serve in the real world, Microsoft has some ideas.
It suggests the bot could act as a sketch assistant for painters or interior designers, or could be implemented as a tool for voice-activated photo refinement. (“Cortana, draw a bird,” perhaps?)
Further down the road, He says that, with more computing power, the technology could possibly generate animated films from screenplays to reduce the manual labor required today from animators.
But the technology isn’t there yet.
If you look closely at the images, they’ll almost always have flaws that indicate they were created by machines, not humans – like birds with blue beaks or fruit stands with odd-shaped bananas (see above).
However, with its touted 3-fold improvement, the bot represents an AI milestone of sorts, Microsoft says.
Comment