albertnet: GPT-2

Showing posts with label GPT-2. Show all posts

Monday, December 14, 2020

Could Artificial Intelligence Replace Writers? - Part 3

Introduction

In my last two posts (here and here) I explored the question of whether Artificial Intelligence could write a magazine article. In particular I described a New Yorker essay on this subject and the misgivings that caused me. Parts 1 and 2 of my essay concerned mainstream applications like Google’s Smart Compose and Gboard predictive text. In this final installment I showcase my experiments with more cutting edge, experimental platforms. The website Write With Transformer gathers together a number of these, including GPT, GPT-2, and Distil-GPT-2. I tried out all three.

Distil-GPT-2

Distil-GPT-2 is the first one I tried simply because it was listed first on the website. It is supposedly an improvement over GPT-2, being “twice as fast as its OpenAI counterpart, while keeping the same generative power.” The site “lets you write a whole document directly from your browser, and you can trigger the Transformer anywhere using the Tab key.” I started writing a very simple essay about how the sentence “the quick brown fox jumps over the lazy dog” helps teach typing, as it efficiently covers all the letters in the alphabet. Here’s what I got:

(Click to zoom in, if you’re not as nearsighted as I.)

As you can see, Distil-GPT-2 didn’t do very well. I judge it based on its ability to grasp where I was trying to go with what I wrote, and how well it continued the essay so I wouldn’t have to. I don’t get the sense it had any idea what I was talking about. It did use words realistically, such that it created credible sentences (as opposed to a tossed salad of words), as long as you don’t worry about content or meaning. But it seemed to be trying to tell its own story, about some unfortunate children. And toward the end there it devolved into babble, with “sudden or unusual, sudden or unusual suddenness.” Did this A.I. learn by reading a lot of Samuel Beckett?

I accidentally tried Distil-GPT-2 a second time by (apparently) clicking the wrong link on the website. But this time, when I wasn’t getting very far with the typing theme, I tried something more concrete. Check it out:

This was kind of like trying to draw out a really small child, or somebody with a learning disability. Where things went really sideways was with “ichalba.” I still haven’t figured that out. For once, Google was at a complete loss; it could only find instances of “ich Alba” on German websites and assumed I had mistyped my query:

I tried Google Translate but it wasn’t much help either, unless it turns out this A.I. is a drunken Uzbek.

I couldn’t find Marba on a map … I’d been hoping it was in Uzbekistan. Needless to say, utter babble doesn’t make A.I.’s prose very credible.

GPT

This was the original OpenAI composition engine. Obviously I wouldn’t expect it to do as well as GPT-2, but figured it would give us a sense of the progress that has been made. I fed it the same basic prompt as the others. Let’s see how it did.

WTF?! Where did it get the whole gay thing? It’s like the A.I. hijacked my content to try to explain something about how gay men learn to type. I don’t understand this at all. I love its final conclusion … so hopeful. And yet I totally disagree that GPT has any potential whatsoever.

GPT-2

Okay, at long last, on to the cutting edge in A.I. text generation. GPT-2 is the technology described in the New Yorker article, which its creator, OpenAI, said couldn’t be released on schedule because “the machine was too good at writing” and the company had to slow down and “prepare for the potential threat posed by superintelligent machines that haven’t been taught to ‘love humanity,’ as Greg Brockman, OpenAI’s chief technology officer, put it.”

I was skeptical, curious, and a bit worried when I put this one through its paces. (I should point something out first: the website Write With Transformer notes that only three of the four GPT-2 “sizes” are publicly available. Presumably I’ve haven’t gotten my hands on the same version the New Yorker got to try out.) Here’s what this GPT-2 engine produced.

Clearly, this is way ahead of the others. The text it produced was coherent and it seemed to actually know certain things: that my cryptic sentence about jackdaws has all the letters in the alphabet; that this is useful for those using a keyboard; that the keyboard could be an old typewriter. It even figured out the most important difference between the jackdaws sentence and the old classic about the quick brown fox.

That’s the stuff it got right, anyway. Oddly, after I typed “old-fashioned” and hit tab, it almost suggested the right word, that being “typewriter.” But It only got as far as “typew.” I cannot figure that out. I hit tab again and it supplied “riter.” Just a glitch, I guess, and easily forgivable … but then it went on to suggest “cute” out of nowhere. It kind of went off the rails at that point.

Could GPT-2 help me write this blog? Not really … it could save a few keystrokes I suppose, like Smart Compose, but this is of limited value when you consider how I had to nudge it back on track a few times, and the fact that I ended up with needlessly verbose sentences (e.g., “writers who are learning to use a keyboard” instead of “budding typists”). Intrigued, I gave GPT-2 another try with the less abstract topic:

Again, GPT-2 seems pretty smart. It know that when flour is used, egg and water are probably needed. It couldn’t suggest “oil’ until I provided “olive” but at least it wasn’t suggesting “olive garden” or something. It really seemed hung up on the idea that a mixer is needed, but it got over it, and seemed to know there’s a such thing as a pasta machine. It couldn’t figure out how that machine is set up in a real kitchen, but I’m impressed that it realized the rollers would make the dough more uniform. It even had some suggestions on how to serve pasta, even if sauce and parmesan cheese are what I’d had in mind. On the basis of this performance, I’d say GPT-2 is pretty nearly ready to start writing for Real Simple magazine. But writing albertnet posts for me? I’ll wait for GPT-3.

Conclusion

Ultimately, I am really relieved at how poorly these technologies did—from Google Smart Compose to Gboard predictive text to GPT-2. I don’t mind if A.I. gives me some shortcuts around predictable words and phrases (even when it goofs), but the idea that it could create realistic prose really frightens me. I think we are entering a golden age for computer-generated writing, where everything gets so much better. The first major step in that direction is to be able to read your own code in the first place. I have no doubt that the technology is there to give us that, but until then, we are really on our own. We need an ecosystem that encourages creativity, so that our machine-generated creations can be a fun way for developers to express themselves, and also as a way to create better products for the world.

Did anything about that last paragraph bother you? Did it seem like my perspective and reason became inconsistent? What if I were to tell you that, for that paragraph, I allowed GPT-2 to generate at least one complete sentence on my behalf, based on how I started the paragraph, just like the New Yorker article did with the quotation from Steven Pinker? Well, I did. So now, go back and try to figure out where my text stopped and the GPT-2’s text began. Then scroll down to see how you did.

If you were to skim my essay and not really engage with my ideas, you might be fooled … but I’m hoping you quickly grasped that the A.I. was completely contradicting me. So here’s my actual, unassisted conclusion:

If the only point of A.I.-written text is to provide arbitrary text-based content to attach ads to, well … mission accomplished. But that has nothing to do with the real point of writing, which is to educate, entertain, and/or illuminate. If A.I.-written articles blithely trot out totally uninformed opinions and unverified information, with nothing guiding them but previously posted uninformed opinions and unverified information, who will lead humanity towards the light?

Other albertnet posts on A.I.

—~—~—~—~—~—~—~—~—
Email me here. For a complete index of albertnet posts, click here.

Monday, November 30, 2020

Could Artificial Intelligence Replace Writers?

Introduction

Over a year ago, I came across this fascinating article about whether or not Artificial Intelligence could write a New Yorker article. The answer was essentially “no” or “not yet,” but it got me pretty riled up anyway. Ever since, I’ve been evaluating A.I.’s ability to correctly suggest even a word or phrase as it parses my text. In this post I wade into that history as I examine the question of A.I.’s (supposed) ascendance.

The New Yorker article

The New Yorker article, “The Next Word,” was in the October 14, 2019 issue. The writer, John Seabrook, talked with A.I. experts at Google about their “Smart Compose” feature, which predicts how your sentence ought to end and suggests the remaining words, so you can just hit tab to accept the suggestion. (If you use Gmail, you’re already familiar with this.) Seabrook also talked with the folks at another company, OpenAI, about their GPT-2 engine, which composes entire sentences and even complete paragraphs, with made-up quotations no less, in the voice of a real writer it “learns” and then mimics. GPT-2 is still under development; OpenAI claimed it has been delayed because it’s “too good at writing” and they fear society isn’t ready. (Yeah, right.) Seabrook tried it out, and in the online version of the article you can see its efforts at contributions to his story.

Seabrook included an extended quotation from Steven Pinker, a Harvard psycholinguist, that had been appended with text generated by GPT-2, and challenged the reader to figure out where the real quote ended and A.I. picked it up. (You can take the challenge in the online article.) I found this exercise really easy, but Seabrook reported that almost everybody he tried the “Pinker test” on “failed to distinguish Pinker’s prose from the machine’s gobbledygook” and concludes, “The A.I. had them Pinkered.”

Does this scare you? It sure scares me. I have no doubt that great literature will always be written by real writers, no matter how good the A.I. gets, but run-of-the-mill journalism and magazine writing, which mainly exist to serve up ads anyway, might someday be written by a clueless A.I. that has no more grasp of insight and fact than do certain famous politicians. As Seabrook puts it, “One can envision machines like GPT-2 spewing superficially sensible gibberish, like a burst water main of babble, flooding the Internet with so much writing that it would soon drown out human voices, and then training on its own meaningless prose, like a cow chewing its cud.”

Hoping against hope that this A.I. capability is overrated, I have been paying close attention to how well it has done across the devices I use, and recording its more salient failures, over the past year. In Internet time, a year is a pretty huge span—in theory I should have seen marked improvement in that period. Well, here’s what I found.

Smart Compose

I have to confess, I haven’t grabbed many snapshots of Google’s Smart Compose behavior because I only use Gmail at work, and I’m fairly religious about separating work and play. I did grab a few examples though, because they flew in the face of how the function was reputed to work. Seabrook quotes Paul Lambert, who manages this feature for Google, as saying, “If you write ‘Have a’ on a Friday, it’s much more likely to predict ‘good weekend’ than if it’s on a Tuesday.”

Weirdly, this didn’t work for me at all. Check out these samples of what Smart Compose suggested to me on a Friday morning:

Neither “great week” nor “good night” makes sense here. Meanwhile, the fact that “gr” invoked “great week” whereas “go” pointed at “good night” is illogical, since people use “great” and “good” almost interchangeably and there’s no reason to assume that we’d want somebody’s week to be great but their night to only be good. I decided to slip between the horns of the dilemma by starting the sentence over entirely. This produced:

Given that Father’s Day was more than seven months away, this suggestion struck me as totally moronic. So I added an “r” after the “F” to see how it would recover:

This would make sense if Chris had wished me a happy Friday … but he hadn’t. I decided to shoot for “weekend” to see how that would go:

Huh? Whose “week ahead” starts on Friday? Oddly, Smart Compose seemed to be utterly neglecting the context of my email.

In the year I’ve kept an eye on Smart Compose, I haven’t again seen anything as egregiously inept. Mostly what I notice is that it doesn’t suggest words or phrases all that often … perhaps my work emails are too technical or otherwise cryptic. (The A.I. that powers Smart Compose was trained on millions of real emails, but none from Google’s business customers.) The A.I. is pretty good about really basic stuff, like suggesting “you have any questions” after I type “Please let me know if,” but that’s about it. As an experiment, I composed this blog post in Gmail and it didn’t suggest anything. It’s like I overwhelmed it somehow. So much for that.

Gboard predictive text

Perhaps more useful, day-to-day, than Smart Compose is Google’s predictive text for the Gboard virtual keyboard, which is bundled with their Android operating system. Predictive text seems to come into play with every app on my phone that relies on typed input. Frankly, I don’t like typing much on the phone so I do most of my writing on the computer. The main thing I type on my phone? Text messages, which I mostly trade with my older daughter who is off at college. (Alas, texting seems to be her generation’s preferred method of communication, at least where their parents are concerned, and I’ve decided to humor my daughter in this.)

My experience? Naturally, predictive text comes in handy, usually in the context of completing words I’ve mostly typed. I hasten to point out this is a lot different from A.I. actually composing anything. If I type “has,” it’s going to suggest “has,” “was,” and “hasn’t” because those are the most likely candidates, and most of the time I’ll accept one of those. If I type “hast” it suggests “hast,” “host,” and “hash,” and if I type “haste” it suggests “haste,” “taste,” and “waste.” (After all, “haste makes waste,” we all know that.) Android is not going to suggest “hasten” because apparently not too many people use that word. This is the bulk of how predictive text behaves, and though it’s not as sophisticated as Smart Compose (much less GPT-2), it works a lot of the time. It also fails a lot.

If we’re really going to count on A.I. to create content for us at any point, I see three things it absolutely has to right. First, it needs to not make any grammar or spelling errors, obviously, since you can’t have it making the putative human author look stupid, or burdening an editor with fifty times the errors a real writer would make. Second, the A.I. can’t commit any serious gaffs that would render the text offensive or at least laughably ignorant. Finally, the A.I. will have to really understand context if it’s to reach its intended audience (well, our intended audience, since A.I. can’t really have anything like intention). An A.I.-written article for the slightly racy GQ or Men’s Health magazine better not sound like Good Housekeeping; Hunter S. Thompson shouldn’t come off like Heloise.

So here’s how the A.I. on my phone has stacked up in these areas over the last year.

Grammar and spelling

It’s kind of remarkable that anybody thinks we’re on the brink of A.I. being able to compose anything when it still doesn’t really do so well with grammar and spelling in its predictive text. I could supply countless examples of errors in this realm, but that would get dull, so I’m providing one example each of the main types of errors I see.

First, it breaks very basic rules about capitalization, failing to capitalize proper nouns or the first word of a sentence. (Predictive text’s cousin, voice recognition, screws this up quite a bit as well.) If the human fails to capitalize a word, A.I. should fix it, rather than expecting us to bother with the shift key a lot. Here’s an example:

It also screws up with predicting subject/verb agreement, so lots of its word suggestions wouldn’t work without my having to backspace and add an “s,” which is clunkier than just typing the word right to begin with. I fight with this many times a day. Here’s an example:

I mean, come on! “We’re huge fan.” That’s not very helpful. “We’re huge favor.” Look, Android, the verb is “are.” It needs a plural predicate nominative. This is not rocket science.

One of the most annoying things predictive text (and its sibling, auto-correct) does is to “fix” my errors for me on the fly, without asking. Usually I end up sending the text before I notice the problem, and then have to explain to the recipient that it’s not my error, which is way more work than for me to just type everything myself with no “help.” (Yes, I know that youngsters these days have no problem sending messages that are utterly littered with errors, but remember, we’re talking about A.I.’s ability to compose text one day ... the bar needs to be higher.) Look at this travesty:

Another category of failure is when predictive text doesn’t grasp what part of speech the next word needs to be. Consider this example where “very” has been set up, within the sentence, to be an adverb modifying another adverb. There is zero benefit in predictive text suggesting an adjective here.

It doesn’t take an English major to grasp that “Text messages don’t convey irony very groovy” simply doesn’t make sense.

Finally, suggesting anything that isn’t really a word is pretty pointless. One of the three choices typically offered up is the fragment of a word you’ve already typed. Why offer this? It’s a waste of screen real estate. And then I’ve seen suggestions that either aren’t words, or basically aren’t words. Look at this example:

Obviously “trifec” isn’t a word, so why give me the option of accepting it? If I really wanted it, I could just hit the spacebar. And “triger”? It’s basically not a word. It’s not in Google’s own spell-checker dictionary; it’s not in the American Heritage Dictionary; and it’s not in the Wiktionary. Okay, I found “triger process” in the online Merriam-Webster dictionary, so maybe Android was setting me up for that phrase, which means “a method of sinking through water-bearing ground in which a shaft is lined with tubbing and provided with an air lock so that work proceeds under air pressure.” But what are the odds this is what I was writing about? Exactly zero. Android obviously should have guessed “trifecta.” If A.I. starts to write articles, will we have to suffer through tedious asides about digging through waterlogged ground?

Some seemingly phonetic goofs

Sometimes the A.I. seems to be working phonetically and makes a suggestion that almost makes sense—but of course almost doesn’t cut it when it’s supposed to save you work while maintaining (or ideally improving) accuracy. Check this out:

Any human could have guessed “all its splendor and glory” and Android almost got it. But “all its splendor and Gloria”? Really? (Could’ve been worse, I guess … it could have changed “its” to “it’s” again.) Here’s another failure:

Many American morons have called COVID-19 a hoax, but I doubt any have called it a Hoke. (If you’re wondering where it even got “Hoke,” I can tell you I’ve used that word in eight texts, referring to a character, Hoke Mosely, who appears in four Charles Willeford novels. Among book characters he resembles a virus in no way whatsoever.)

Here’s a final example of the A.I. seeming to fail via phonetic bumbling:

It’s almost as though the predictive text software heard somebody say “thirsty” and thought it heard “Thursday.” But of course that didn’t happen … you can see where I typed “thirs.” And how could anyone be Thursday? It makes no sense. On the other hand, if you told somebody to say the first thing that popped into their head when you gave them a prompt, and the prompt was “hungry and …” I’ll bet nine out of ten would say “thirsty.” (One out of ten would be somebody on a diet who might say something like “bitter.”)

It is hard to make a case that these goofs are truly phonetic in nature. But is it feasible these errors are simply random? Well … how the hell should I know? I never said I was a brain scientist or computer technologist. But I have a couple of theories, around context and … oops, unfortunately I seem to be out of space here.

To be continued…

Tune in next week for Part 2 of this essay, where I’ll explore some more ways A.I. can go wrong, with a number of wince-worthy predictive-text FAILs.

Other albertnet posts on A.I.

—~—~—~—~—~—~—~—~—
Email me here. For a complete index of albertnet posts, click here.

Monday, December 14, 2020

Could Artificial Intelligence Replace Writers? - Part 3

Monday, November 30, 2020

Could Artificial Intelligence Replace Writers?

Followers

Blog Archive

About Me