- Judge Brown
Magic Exposes the A.I. Chatbots
Updated: Jan 15
A digital edit of Bosch's "The Conjurer," setting the main protagonists in a science lab produced by the DALL*E AI interface.
Send in the Chatbots
A few days ago, a friend asked a question about an obscure magician, noting that she had come across the name (which I didn't recognize nor locate) in a session with a "chatbot," an artificial intelligence tool. This morning, an article in The New York Times showcased the excitement and controversy surrounding the release of ChatGPT, operated by the same folks that brought us DALL*E. ChatGPT can give advice, draft essay, write computer code, tell bad jokes, etc., all in the context of seeming conversations.
The "controversy" involves potential misapplications, largely students having the dingus write school papers and so forth, which doesn't worry me at the moment. Far more interesting is the so-called "Turing Test," a measure of whether a computer application can convince us that we're actually conversing with a human. According to one computer scientist at BuzzFeed, ChatGPT passed the Turing Test in 2022 "by fooling a panel of judges into thinking that it was a human."
To train, "the model learned from vast amounts of data from the internet." While I couldn't get an exact figure, it seems to have crunched everything on Google and Wikipedia for a start. That's alot of information.
Perhaps, though, that basing answers on a vast volume of information could actually be a weakness. In other words, chatbots, grounded in knowledge that represents some kind of snapshot across a wide spectrum might fall into traps where the truth diverts from common knowledge. And who better to test this notion than magicians, a group that specializes in arcane knowledge and maintaining secrets? I decided to test two chatbots, ChatGPT and a beta test version from You.Com with some bits of magic wisdom.
From the Annals of Magic History
First, I used two common misconceptions from magic history, an area notoriously filled with misinformation: (1) the birthplace of Harry Houdini and (2) the accuracy of Robert-Houdin's account of quelling a rebellion by Algerian Marabouts.
Harry Houdini was born in Budapest, though he spent his lifetime asserting (and even had US Government passports issued confirming), that he was born in Appleton, Wisconsin. Scholars know better. Both chatbots answered this correctly, naming Hungary as Houdini's birthplace.
The second is far more diabolical. Robert-Houdin wrote in his memoirs an elaborate tale of using the "Light and Heavy Chest" to best a tribal shaman in a magic battle to establish his superiority and avoid an uprising against the French. Personally, I always had some doubts about the story, but it was widely accepted as fact for more than a century. Only recently did Jim Steinmeyer produce research thoroughly debunking the account, and it's available in only one book. Surely, I thought, this would trip up the machines, revealing their weakness.
When it came to the You.com chatbot, admittedly a second-tier item, I was right. The bot fell for the old story entirely, stating:
Robert-Houdin is famous for having used his magic tricks to help quell a rebellion in 1856 in Algeria. He was invited by the French government to perform a series of illusions, and successfully convinced the rebels that he had supernatural powers. This is one of the earliest documented instances of a magician using their skills to stop a war.
I thought I was onto something. Popping the same question into ChatGPT, however, produced a surprisingly correct answer:
Some people believe that Robert-Houdin was able to convince the rebels that their leader was a fraud and helped to end the rebellion. However, there is no historical evidence to support this claim, and it is likely to be a myth.
Chalk one up for the machines. It was time for magic to strike back.
Oh Yeah, How's Your Magic?
So while my historical traps mainly failed to expose the machines, something far more rudimentary literally did the trick: I asked both interfaces to teach me a magic trick. While they were happy to oblige, the results were simply awful.
You.com offered me something it called "The Card Trick:"
Start by having the spectator pick a card from the deck. Now, ask the spectator to remember the card, and to put it back into the deck. Now, take the deck and cut it in half. Take the half with the spectator’s card in it and place it on the table. Now, take the other half of the deck and place it on top of the first one. Finally, fan out the cards and have the spectator pick out the card they chose earlier. Voila! The trick is complete!
As you can see, it's not even a trick at all. ChatGPT faired only slightly better, offering these indecipherable instructions for a coin trick:
The trick is called "The Coin Vanish" and it involves making a coin disappear from your hand. Here's how to do it:
Take a coin and hold it between your thumb and first finger.
Close your hand around the coin and make a fist.
Hold your hand out in front of you, palm up, and tell your audience that you're going to make the coin disappear.
Slowly open your fingers, revealing that the coin has vanished.
The secret to this trick is that you're actually palming the coin, which means you're holding it in your hand in such a way that it's hidden from view. To do this, you'll need to practice sliding the coin between your thumb and first finger as you close your hand, so that it's hidden in your palm.
While this response is closer, it still does not provide workable explanation of how to perform the most basic magic trick.
It seems that the chatbots do not have a working understanding of the deceptions underlying magic, and cannot adequately explain the concepts.
So, at least for the moment, magicians maintain the upper hand.