Introduction
In my last two posts (here and here) I explored the question of whether Artificial Intelligence could write a magazine article. In particular I described a New Yorker essay on this subject and the misgivings that caused me. Parts 1 and 2 of my essay concerned mainstream applications like Google’s Smart Compose and Gboard predictive text. In this final installment I showcase my experiments with more cutting edge, experimental platforms. The website Write With Transformer gathers together a number of these, including GPT, GPT-2, and Distil-GPT-2. I tried out all three.
Distil-GPT-2
Distil-GPT-2 is the first one I tried simply because it was listed first on the website. It is supposedly an improvement over GPT-2, being “twice as fast as its OpenAI counterpart, while keeping the same generative power.” The site “lets you write a whole document directly from your browser, and you can trigger the Transformer anywhere using the Tab key.” I started writing a very simple essay about how the sentence “the quick brown fox jumps over the lazy dog” helps teach typing, as it efficiently covers all the letters in the alphabet. Here’s what I got:
(Click to zoom in, if you’re not as nearsighted as I.)
As you can see, Distil-GPT-2 didn’t do very well. I judge it based on its ability to grasp where I was trying to go with what I wrote, and how well it continued the essay so I wouldn’t have to. I don’t get the sense it had any idea what I was talking about. It did use words realistically, such that it created credible sentences (as opposed to a tossed salad of words), as long as you don’t worry about content or meaning. But it seemed to be trying to tell its own story, about some unfortunate children. And toward the end there it devolved into babble, with “sudden or unusual, sudden or unusual suddenness.” Did this A.I. learn by reading a lot of Samuel Beckett?
I accidentally tried Distil-GPT-2 a second time by (apparently) clicking the wrong link on the website. But this time, when I wasn’t getting very far with the typing theme, I tried something more concrete. Check it out:
This was kind of like trying to draw out a really small child, or somebody with a learning disability. Where things went really sideways was with “ichalba.” I still haven’t figured that out. For once, Google was at a complete loss; it could only find instances of “ich Alba” on German websites and assumed I had mistyped my query:
I tried Google Translate but it wasn’t much help either, unless it turns out this A.I. is a drunken Uzbek.
I couldn’t find Marba on a map … I’d been hoping it was in Uzbekistan. Needless to say, utter babble doesn’t make A.I.’s prose very credible.
GPT
This was the original OpenAI composition engine. Obviously I wouldn’t expect it to do as well as GPT-2, but figured it would give us a sense of the progress that has been made. I fed it the same basic prompt as the others. Let’s see how it did.
WTF?! Where did it get the whole gay thing? It’s like the A.I. hijacked my content to try to explain something about how gay men learn to type. I don’t understand this at all. I love its final conclusion … so hopeful. And yet I totally disagree that GPT has any potential whatsoever.
GPT-2
Okay, at long last, on to the cutting edge in A.I. text generation. GPT-2 is the technology described in the New Yorker article, which its creator, OpenAI, said couldn’t be released on schedule because “the machine was too good at writing” and the company had to slow down and “prepare for the potential threat posed by superintelligent machines that haven’t been taught to ‘love humanity,’ as Greg Brockman, OpenAI’s chief technology officer, put it.”
I was skeptical, curious, and a bit worried when I put this one through its paces. (I should point something out first: the website Write With Transformer notes that only three of the four GPT-2 “sizes” are publicly available. Presumably I’ve haven’t gotten my hands on the same version the New Yorker got to try out.) Here’s what this GPT-2 engine produced.
Clearly, this is way ahead of the others. The text it produced was coherent and it seemed to actually know certain things: that my cryptic sentence about jackdaws has all the letters in the alphabet; that this is useful for those using a keyboard; that the keyboard could be an old typewriter. It even figured out the most important difference between the jackdaws sentence and the old classic about the quick brown fox.
That’s the stuff it got right, anyway. Oddly, after I typed “old-fashioned” and hit tab, it almost suggested the right word, that being “typewriter.” But It only got as far as “typew.” I cannot figure that out. I hit tab again and it supplied “riter.” Just a glitch, I guess, and easily forgivable … but then it went on to suggest “cute” out of nowhere. It kind of went off the rails at that point.
Could GPT-2 help me write this blog? Not really … it could save a few keystrokes I suppose, like Smart Compose, but this is of limited value when you consider how I had to nudge it back on track a few times, and the fact that I ended up with needlessly verbose sentences (e.g., “writers who are learning to use a keyboard” instead of “budding typists”). Intrigued, I gave GPT-2 another try with the less abstract topic:
Again, GPT-2 seems pretty smart. It know that when flour is used, egg and water are probably needed. It couldn’t suggest “oil’ until I provided “olive” but at least it wasn’t suggesting “olive garden” or something. It really seemed hung up on the idea that a mixer is needed, but it got over it, and seemed to know there’s a such thing as a pasta machine. It couldn’t figure out how that machine is set up in a real kitchen, but I’m impressed that it realized the rollers would make the dough more uniform. It even had some suggestions on how to serve pasta, even if sauce and parmesan cheese are what I’d had in mind. On the basis of this performance, I’d say GPT-2 is pretty nearly ready to start writing for Real Simple magazine. But writing albertnet posts for me? I’ll wait for GPT-3.
Conclusion
Ultimately, I am really relieved at how poorly these technologies did—from Google Smart Compose to Gboard predictive text to GPT-2. I don’t mind if A.I. gives me some shortcuts around predictable words and phrases (even when it goofs), but the idea that it could create realistic prose really frightens me. I think we are entering a golden age for computer-generated writing, where everything gets so much better. The first major step in that direction is to be able to read your own code in the first place. I have no doubt that the technology is there to give us that, but until then, we are really on our own. We need an ecosystem that encourages creativity, so that our machine-generated creations can be a fun way for developers to express themselves, and also as a way to create better products for the world.
Did anything about that last paragraph bother you? Did it seem like my perspective and reason became inconsistent? What if I were to tell you that, for that paragraph, I allowed GPT-2 to generate at least one complete sentence on my behalf, based on how I started the paragraph, just like the New Yorker article did with the quotation from Steven Pinker? Well, I did. So now, go back and try to figure out where my text stopped and the GPT-2’s text began. Then scroll down to see how you did.
Ultimately, I am really relieved at how poorly these technologies
did—from Google Smart Compose to Gboard predictive text to GPT-2. I don’t mind
if A.I. gives me some shortcuts around predictable words and phrases (even when
it goofs), but the idea that it could create realistic prose really frightens
me. After all, A.I. has no concept of truth. It recommends olive oil, fresh
garlic, and fresh herbs instead of sauce, without any idea of whether that simple
fare would be tastier than a Bolognese Ragu or an Alfredo.
If the only point of A.I.-written text is to provide arbitrary text-based content to attach ads to, well … mission accomplished. But that has nothing to do with the real point of writing, which is to educate, entertain, and/or illuminate. If A.I.-written articles blithely trot out totally uninformed opinions and unverified information, with nothing guiding them but previously posted uninformed opinions and unverified information, who will lead humanity towards the light?
Other albertnet posts on A.I.
- I, Chatbot
- Almost Intelligent – Part I
- Almost Intelligent – Part II
- Smartphones & Artificial Stupidity
- A.I. Smackdown – Moto vs. Cortana vs. Siri
- Could Artificial Intelligence Replace Writers? – Part 1
- Could Artificial Intelligence Replace Writers? – Part 2
Email me here. For a complete index of albertnet posts, click here.
No comments:
Post a Comment