Engineering Additional Creativity

The field of artificial intelligence (AI) has long dreamed about computers that can automatically write humanlike stories. In recent years, with the advent of machine learning, AI researchers have found new opportunities for such automated text writing.

Last year, Open AI's automatic text generator GPT-3 was one of the biggest breakthroughs in AI research. GPT-3 (Generative Pre-trained Transformer 3) writes human texts in all kinds of genres, which is unique. Give a piece of text to the machine, and it generates a new piece of text which the machine considers to be the most likely continuation of the first. This can be an essay, a poem, a song, a dialogue, an interview, or any other text form that can serve as a starter. GPT-3 also can perform very different tasks from just a few examples, as long as they involve text. It can, with only a few examples, translate, generate news articles, complete a short story, correct English grammar, answer factual questions about texts, and even provide answers to simple arithmetic problems.

However, in order to produce interesting results, GPT-3 requires a lot of human intervention. There is nothing wrong with that, says AI researcher Melissa Roemmele; on the contrary, she says, it means humans can use such an AI text generator to augment their own creative writing.

Roemmele is a research scientist for the company RWS in Los Angeles, where she helps to develop interactive applications that use AI and natural language processing to facilitate content understanding and creation. In 2018, she finished her doctoral thesis on AI-augmented creative writing at the University of Southern California.

In an interview via Zoom, Roemmele spoke about the potential for computers to write text automatically.

How did you become interested in AI that writes humanlike text?

When I was a kid, I wanted to become a fiction author. I did a little bit of my own attempts in creative writing, some poetry, and some short stories, but as time went on, I started to gravitate more in the direction of science. As an undergraduate, I majored in psychology and linguistics. I was intrigued by the question of what is going on in the mind of somebody doing creative writing. That is still bit of a mystery. Later I got involved in the AI side, which became the focus of my Ph.D. work.

What is your view of AI writing text?

Media often depict automatic writing as a kind of competition between humans and computers. I try to emphasize that we don't need to look at AI this way. My view is that automatic writing provides a great opportunity for a new kind of human-computer interaction. The goal might not be to completely simulate human writing. I would like to see some people benefit from the interaction with the computer by becoming better creative writers themselves.

What are the most important results from your Ph.D. work?

In my Ph.D. work I developed an app called Creative Help, along with my advisor Andrew Gordon. I did experiments with people writing stories using that app. We were tracking what people wrote about, and when and how they interacted with the AI. The people who were getting the most out of the interaction were willing to be diverted by the suggestions of the app. Other people just hated it; they wanted to write their own story, and only sometimes to be helped by the machine. These sorts of tools are not going to be helpful for everyone. I am not trying to solve a problem for people that they don't have.

How are the people who are willing to use the AI benefitting from the interaction?

AI is really good at supporting the brainstorming process of humans. The AI generates a lot of ideas. Some are useful, some not; it is up to the human writer to judge their value. What is important is the hit rate of the ideas generated by the AI. If the ratio is 100 misses and 1 hit, that is too low, but maybe a ratio of 5 misses to 1 hit will work. That is something to explore further. Generally speaking, you can say that there is an optimum balance of suggestions that are too conventional and suggestions that are so novel that they are impossible to make sense of.

Can you give an example from your research?

Take the sentence "The child let go of the balloon." How does one continue this sentence? Many people will suggest the sentence: "The balloon flew away." That is a very predictable, and even boring, continuation. The AI might come up with a sentence like, "The wind rushed up its long straw fur to a light blue." That sounds meaningless. Between these two extremes of 'too conventional' and 'too novel' lies a sentence like, "The balloon burst into flames." This sounds surprising and intriguing, and that is what creative writers often want.

Given the current state of AI, people still tend to find that human-authored sentences overall are more meaningful than AI-generated ones. However, despite that, in my research people showed an interest in AI-generated results and in using them to become more creative themselves. In this way, I was able to demonstrate that AI can really augment human creativity.

What can we expect from AI augmentative creative writing in the near future?

I think we might see tools like GPT-3 becoming commercially viable, but there are still many challenges on the modelling and design side. We do not have a clear idea yet of what the optimal user interface should look like. Furthermore, what I found very interesting with GPT-3 is that you can instruct it on a meta-level, for example by giving the instruction "tell me a scary story." Designing good prompts for GPT-3 even became a new challenge. Finally, I also expect that automatic writing systems will learn how they can best serve an individual human writer and challenge them to go beyond their usual comfort zone.

Bennie Mols is a science and technology writer based in Amsterdam, the Netherlands.