{"id":6088,"date":"2023-04-11T10:00:00","date_gmt":"2023-04-11T10:00:00","guid":{"rendered":"https:\/\/modernsciences.org\/staging\/4414\/?p=6088"},"modified":"2023-03-31T04:27:43","modified_gmt":"2023-03-31T04:27:43","slug":"chatgpt-struggles-with-wordle-puzzles-which-says-a-lot-about-how-it-works","status":"publish","type":"post","link":"https:\/\/modernsciences.org\/staging\/4414\/chatgpt-struggles-with-wordle-puzzles-which-says-a-lot-about-how-it-works\/","title":{"rendered":"ChatGPT struggles with Wordle puzzles, which says a lot about how it works"},"content":{"rendered":"\n
\n \n
\n shutterstock.\n Shutterstock<\/a><\/span>\n <\/figcaption>\n <\/figure>\n\nMichael G. Madden<\/a>, University of Galway<\/a><\/em><\/span>\n\n

The AI chatbot known as ChatGPT, developed by the company OpenAI, has caught the public\u2019s attention and imagination. Some applications of the technology are truly impressive<\/a>, such as its ability to summarise complex topics<\/a> or to engage in long conversations<\/a>. <\/p>\n\n

It\u2019s no surprise that other AI companies have been rushing<\/a> to release their own large language models (LLMs)<\/a> \u2013 the name for the technology underlying chatbots like ChatGPT. Some of these LLMs will be incorporated into other products, such as search engines.<\/p>\n\n

With its impressive capabilities in mind, I decided to test the chatbot on Wordle<\/a> \u2013 the word game from the New York Times \u2013 which I have been playing for some time. Players have six goes at guessing a five-letter word. On each guess, the game indicates which letters, if any, are in the correct positions in the word. <\/p>\n\n

Using the latest generation, called ChatGPT-4<\/a>, I discovered that its performance on these puzzles was surprisingly poor. You might expect word games to be a piece of cake for GPT-4. LLMs are \u201ctrained\u201d on text, meaning they are exposed to information so that they can improve at what they do. ChatGPT-4 was trained on about 500 billion words: all of Wikipedia, all public-domain books, huge volumes of scientific articles, and text from many websites.<\/p>\n\n

AI chatbots could play a major role in our lives. Understanding why ChatGPT-4 struggles with Wordle provides insights into how LLMs represent and work with words \u2013 along with the limitations this brings. <\/p>\n\n

First, I tested ChatGPT-4 on a Wordle puzzle where I knew the correct locations of two letters in a word. The pattern was \u201c#E#L#\u201d, where \u201c#\u201d represented the unknown letters. The answer was the word \u201cmealy\u201d. <\/p>\n\n

Five out of ChatGPT-4\u2019s six responses failed to match the pattern. The responses were: \u201cberyl\u201d, \u201cferal\u201d, \u201cheral\u201d, \u201cmerle\u201d, \u201crevel\u201d and \u201cpearl\u201d.<\/p>\n\n

With other combinations, the chatbot sometimes found valid solutions. But, overall, it was very hit and miss. In the case of a word fitting the pattern \u201c##OS#\u201d, it found five correct options. But when the pattern was \u201c#R#F#\u201d, it proposed two words without the letter F, and a word \u2013 \u201cTraff\u201d \u2013 that isn\u2019t in dictionaries.<\/p>\n\n

\n \"Representation\n
\n California-based company OpenAI recently released its latest chatbot, known as GPT-4.<\/span>\n Shutterstock \/ Tada Images<\/a><\/span>\n <\/figcaption>\n <\/figure>\n\n

Under the bonnet<\/h2>\n\n

At the core of ChatGPT is a deep neural network<\/a>: a complex mathematical function \u2013 or rule \u2013 that maps inputs to outputs. The inputs and outputs must be numbers. Since ChatGPT-4 works with words, these must be \u201ctranslated\u201d to numbers for the neural network to work with them.<\/p>\n\n

The translation is performed by a computer program called a tokenizer<\/a>, which maintains a huge list of words and letter sequences, called \u201ctokens\u201d. These tokens are identified by numbers. A word such as \u201cfriend\u201d has a token ID of 6756, so a word such as \u201cfriendship\u201d is broken down into the tokens \u201cfriend\u201d and \u201cship\u201d. These are represented as the identifiers 6756 and 6729.<\/p>\n\n

When the user enters a question, the words are translated into numbers before ChatGPT-4 even starts processing the request. The deep neural network does not have access to the words as text, so it cannot really reason about the letters.<\/p>\n\n

Poem task<\/h2>\n\n

ChatGPT-4 is good at working with the first letters of words. I asked it to write a poem where the opening letter of each line spelled out \u201cI love robots\u201d. Its response was surprisingly good. Here are the first four lines:<\/p>\n\n

\n

I am a fan of gears and steel<\/p>\n\n

Loving their movements, so surreal,<\/p>\n\n

Over circuits, they swiftly rule<\/p>\n\n

Vying for knowledge, they\u2019re no fool,<\/p>\n<\/blockquote>\n\n

The training data for ChatGPT-4<\/a> includes huge numbers of textbooks, which often include alphabetical indices. This could have been enough for GPT-4 to have learned associations between words and their first letters. <\/p>\n\n

The tokenizer also appears to have been modified to recognise requests like this, and seems to split a phrase such as \u201cI Love Robots\u201d into individual tokens when users enter their request. However, ChatGPT-4 was not able to handle requests to work with the last letters of words.<\/p>\n\n

ChatGPT-4 is also bad at palindromes. Asked to produce a palindrome phrase about a robot, it proposed \u201ca robot\u2019s sot, orba\u201d, which does not fit the definition of a palindrome and relies on obscure words.<\/p>\n\n

However, LLMs are relatively good at generating other computer programs. This is because their training data includes many websites devoted to programming. I asked ChatGPT-4 to write a program for working out the identities of missing letters in Wordle.<\/p>\n\n

The initial program that ChatGPT-4 produced had a bug in it. It corrected this when I pointed it out. When I ran the program, it found 48 valid words matching the pattern \u201c#E#L#\u201d, including \u201ctells\u201d, \u201ccells\u201d and \u201chello\u201d. When I had previously asked GPT-4 directly to propose matches for this pattern, it had only found one.<\/p>\n\n

Future fixes<\/h2>\n\n

It might seem surprising that a large language model like ChatGPT-4 would struggle to solve simple word puzzles or formulate palindromes, since the training data includes almost every word available to it.<\/p>\n\n

However, this is because all text inputs must be encoded as numbers and the process that does this doesn\u2019t capture the structure of letters within words. Because neural networks operate purely with numbers, the requirement to encode words as numbers will not change.<\/p>\n\n

There are two ways that future LLMs can overcome this. First, ChatGPT-4 knows the first letter of every word, so its training data could be augmented to include mappings of every letter position within every word in its dictionary.<\/p>\n\n

The second is a more exciting and general solution. Future LLMs could generate code to solve problems like this, as I have shown. A recent paper demonstrated an idea called Toolformer<\/a>, where an LLM uses external tools to carry out tasks where they normally struggle, such as arithmetic calculations.<\/p>\n\n

We are in the early days of these technologies, and insights like this into current limitations can lead to even more impressive AI technologies.\"The<\/p>\n\n

Michael G. Madden<\/a>, Established Professor of Computer Science, University of Galway<\/a><\/em><\/span><\/p>\n\n

This article is republished from The Conversation<\/a> under a Creative Commons license. Read the original article<\/a>.<\/p>\n\n","protected":false},"excerpt":{"rendered":"shutterstock. Shutterstock Michael G. Madden, University of Galway The AI chatbot known as ChatGPT, developed by the company…\n","protected":false},"author":428,"featured_media":6075,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","fifu_image_url":"","fifu_image_alt":"","footnotes":""},"categories":[16],"tags":[334,693,333,497,474],"class_list":{"0":"post-6088","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-tech","8":"tag-artificial-intelligence","9":"tag-chatgpt","10":"tag-machine-learning","11":"tag-neural-network","12":"tag-the-conversation","13":"cs-entry","14":"cs-video-wrap"},"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/modernsciences.org\/staging\/4414\/wp-json\/wp\/v2\/posts\/6088","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/modernsciences.org\/staging\/4414\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/modernsciences.org\/staging\/4414\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/modernsciences.org\/staging\/4414\/wp-json\/wp\/v2\/users\/428"}],"replies":[{"embeddable":true,"href":"https:\/\/modernsciences.org\/staging\/4414\/wp-json\/wp\/v2\/comments?post=6088"}],"version-history":[{"count":1,"href":"https:\/\/modernsciences.org\/staging\/4414\/wp-json\/wp\/v2\/posts\/6088\/revisions"}],"predecessor-version":[{"id":6089,"href":"https:\/\/modernsciences.org\/staging\/4414\/wp-json\/wp\/v2\/posts\/6088\/revisions\/6089"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/modernsciences.org\/staging\/4414\/wp-json\/wp\/v2\/media\/6075"}],"wp:attachment":[{"href":"https:\/\/modernsciences.org\/staging\/4414\/wp-json\/wp\/v2\/media?parent=6088"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/modernsciences.org\/staging\/4414\/wp-json\/wp\/v2\/categories?post=6088"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/modernsciences.org\/staging\/4414\/wp-json\/wp\/v2\/tags?post=6088"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}