AN ARTIFICIAL INTELLIGENCE EXPERIMENT USING GOOGLE'S GEMINI, PART THREE: FROM NAPOLEON BONAPARTE TO MUHAMMAD ALI

 

As a recap, my artificial intelligence (AI) experiment was about finding who had the most books, whether autobiographies or biographies, written about them using Google's AI tool, Gemini, to determine the results.

Following my last two blog posts, which set out details of the experiment, this third part of the blog post, will reveal my results of the experiment. But as detailed in the earlier blog posts, the results should be treated with a lot of caution and even disregarded because of the significant limitations of Google's Gemini.

As mentioned, I searched the number of books written for nearly 11,000 people, 10,976 people to be precise, which is a very big sample. But 1,549 searches of people yielded no results, that's nearly 15 per cent of total searches that Google's Gemini drew a blank. Also because of a lack of quantification, a further 2,809 people were left out of my ranking. On top of the 1,549 people for which no search results could be found, this meant 4,358 out of 10,976 people searched for had to be excluded from my reckoning, that's just under 40 per cent of my total searches. This astonishingly high figure casts considerable doubt on the accuracy of and arguably even invalidates my findings in this experiment.

Of the 6,618 people ranked, below is Google's Gemini's top twenty people with most books written about them - in parentheses are the number of books written about them:

1. Napoleon Bonaparte ("over 60,000 to more than 300,000")

2. Abraham Lincoln - ("around 15,000")

3. Ronald Reagan ("more than 11,000 books")

4. John F Kennedy (“thousands of books dedicated to his life and presidency")

5. Rabindranath Tagore (“thousands”)

6. Dante Alighieri (“thousands of books and editions”)

7. Albert Einstein ("around 2000 books")

8. Horatio Nelson ("over a thousand so far")

9. Tokugawa Ieyasu (“at least hundreds, if not thousands”)

10. Saint George (“hundreds or thousands”)

11. Joseph Stalin (“hundreds, if not thousands”)

12. Saint Francis of Assisi (“hundreds, if not thousands”)

13. Meriwether Lewis Clark Jr (“hundreds, if not thousands, of books written”)

14. Epicurus (“hundreds or potentially thousands”)

15. Elvis Presley ("hundreds" with "Over 3,000" cited in an earlier search)

16. Franklin D Roosevelt ("at least 800 biographies”)

17. Johann Sebastian Bach (“an estimated 700 biographies” and “no known autobiographies”)

18. Skanderbeg (“over 600 books”)

19. Bob Marley ("around 500 books")

20. Muhammad Ali (“hundreds of books, with *Goodreads* listing 247 books”)

I hope you can see from that list there is a fair degree of imprecision that Google's Gemini gave as the number of books written about these people. And you'd be forgiven in not expecting nor even recognising some people appearing in Gemini's top twenty list. I had to re-familiarise myself with a few of the names on Gemini's list. My guess, bar for a handful or two, is that this list isn't what other people's top twenty would be. And there's no women on the list!

Bubbling under Google's Gemini's top twenty list include, again with number of books published presented in parentheses: Vincent van Gogh (“hundreds of biographies”), Prince Philip (“hundreds of biographies”), El Greco (“hundreds”), Caravaggio (“hundreds of biographies”), Paul Cézanne (“hundreds of biographies, academic studies, and exhibition catalogues”), George Armstrong Custer (“hundreds of biographies, studies, and personal accounts”), Mustafa Kemal Atatürk (“hundreds”), Edgar Degas (“hundreds of biographies, studies, and exhibition catalogues”), Mata Hari (“approximately 250 biographies and novels”), Lord Byron ("over 200), Alfred Hitchcock (“many biographies and books … but … there are over 200 books currently in print about Hitchcock and his films”) and Frédéric Chopin (“over one hundred biographies”). See Endnote 18 for other notable people, including the Adult FriendFinder site's owner and chief executive officer, who feature much lower down in Gemini's ranking.

Context is needed to make sense of Google's Gemini's top twenty. Please remember the following for reasons of no search results, nonsensical search results or vague search results were taken out of the game, presented in surname alphabetical order wherever possible: Akbar the Great, Alexander the Great, Jane Austen, Charles Babbage, Ludwig van Beethoven, Alexander Graham Bell, Osama bin Laden, Simón Bolívar, Usain Bolt, David Bowie, Marlon Brando, Johnny Cash, Catherine the Great, Charlemagne, Jesus Christ, Winston Churchill, Cleopatra, Bill Clinton, Kurt Cobain, Christopher Columbus, Confucius, Copernicus, Hernán Cortés, Marie Curie, Salvador Dali, Leonardo da Vinci, Charles Darwin, Princess Diana, Charles Dickens, Bob Dylan, Thomas Edison, Dwight D Eisenhower, Queen Elizabeth I, Leif Erikson, Michael Faraday, Alexander Fleming, Francisco Franco, Galileo Galilei, Mahatma Gandhi, Bill Gates, Charles de Gaulle, Che Guevara, Genghis Khan, Mikhail Gorbachev, King Henry V, King Henry VIII, Audrey Hepburn, Adolf Hitler, Robin Hood, Michael Jackson, Thomas Jefferson, Edward Jenner, Xi Jinping, Pope John XXIII, Michael Jordan, Julius Caesar, Martin Luther King, Henry Kissinger, Vladimir Lenin, John Lennon, Louis XVI, Madonna, Nelson Mandela, Karl Marx, Paul McCartney, Freddie Mercury, Lionel Messi, Moctezuma II, Narendra Modi, Moses, Wolfgang Amadeus Mozart, Robert Mugabe, Prophet Muhammad, Elon Musk, Isaac Newton, Richard Nixon, Barack Obama, Louis Pasteur, Pelé, Peter the Great, Pablo Picasso, Marco Polo, Vladimir Putin, Pierre-Auguste Renoir, Maximilien Robespierre, William Shakespeare, Socrates, Suleiman the Magnificent, Taylor Swift, Emperor Taizong of Tang, Elizabeth Taylor, Margaret Thatcher, Donald Trump, Alan Turing, Mark Twain, Queen Victoria, John Wayne, William the Conqueror, Tiger Woods and Mao Zedong. Many of these hundred people certainly would have troubled the leaderboard.

Are you surprised at Google's Gemini's findings? And are you surprised how bad Gemini is determining who has had the most books written about them?

Is there anyone in particular you'd like to know their Gemini search result?

Have I missed anyone significant out of my searches?

Though Google's Gemini managed to give me a meaningful search result for more than 6,000 people from a total of nearly 11,000 searches, its results are littered with errors. For the record, I didn't check the accuracy of each search result - that would have been a forever task. However, I can highlight five general types of errors in Gemini's search results.

The first type of error is the strangest because it relates to the lack of coherence within Google's AI tool, Gemini. For example, according to Gemini, Napoleon Bonaparte with “over 60,000 to more than 300,000" books is the person the most written about in the world. Abraham Lincoln with "around 15,000" books is the person the second most written about, but Gemini asserted that he "is the most written-about individual”. There's no logic to this claim!

Also related to a lack of Google's Gemini's coherence are these six examples, One, in searching for the number of books written about Reverend William Wood, famous for campaigning for Catholics to be allowed to hold public office in then Protestant-dominated Britain, four was the number but it excluded Charles Wellbeloved's biography of the chapel minister which was cited when I searched for the number of books written by Charles Wellbeloved, an English archaeologist. Two, Gemini told me that Chuck Colson, an American lawyer serving in the Nixon administration, had at least two autobiographies but no biographies, but when I searched for Jonathan Aitken, a disgraced British Tory cabinet minister, I found that he had written a biography of Chuck Colson. Three, Gemini claimed that the English poet, Stephen Spender, had no biographies written about him, but when I searched for John Sutherland I found that he had written an authorised biography of the poet. Four is the example of Lester Piggott, a top jockey in Britain - no results were forthcoming but when I searched for Dick Francis, a former jockey and then a best-selling author, it turns out he had written a biography of Lester Piggott. Five, when I searched for the number of books, whether autobiographies or biographies, written about Al Gore, Gemini told me it was no books, but when I searched for the number of biographies written about the former US Vice President, Gemini told me that there are several biographies. It makes no sense! And six, Gemini told me that “numerous” books had been written about Marie Curie but, when I searched for the number of books written about her husband, Pierre Curie, Gemini told me that “more than forty biographies of Marie Curie have been published worldwide”. What result should I have used for Marie Curie - the unquantifiable 'numerous' from her search or the quantifiable 'more than forty' from her husband's search?

I cannot explain how Google's Gemini failed in these six examples even though it had the information at its disposal to not make these mistakes.

The second type of error is when Google's Gemini just gets it wrong in that its search findings are factually incorrect. As I mentioned, I haven't verified the accuracy of Gemini's search results, but there are many that are just plain wrong based on my experience or from searches elsewhere. And when I say wrong, I'm not talking about spelling and grammar mistakes. I'm talking about significant errors that affect the rankings. I found over 40 significant errors in Gemini's search results. It is certain that there are far more errors as I didn't check the accuracy of all search results. See Endnote 19 for Gemini's wrong results.

Because of my interest in politics and sport, I've read or come across books of politicians (eg Diane Abbott, David Cameron, Roy Jenkins, John Major, Mo Mowlam, Alex Salmond, Nicola Sturgeon, Donald Trump, Liz Truss) and of sportspersons (eg Lee Chapman, Alex Ferguson, Wayne Gretzky, Michael Johnson, Andy Murray, Wayne Rooney, Geraint Thomas, Howard Wilkinson, Serena Williams) that weren't picked up by Google's Gemini.

David Cameron's unauthorised biography, Call Me Dave, written by Michael Ashcroft and Isabel Oakeshott was omitted by Google's Gemini - this was the book in which it was claimed that the former British Prime Minister stuck his cock in a pig's snout while in school at Eton College!

Al Gore is an interesting case. When searched for the number of autobiographies and biographies about the former US Vice President, Google's Gemini told me there was none. But when searched for the number of biographies written about him, Gemini told me there was “Several biographies” written about him. I can't explain that discrepancy.

Another strange omission by Gemini was for Edward Daly, a Catholic bishop trying to stop the British soldiers shooting unarmed Catholic civilians during a protest in Derry, Northern Ireland on 30 January 1972 - the day is known as Bloody Sunday or the Bogside Massacre depending on what side you're on. Google's Gemini yielded no search results, but Google Books revealed that he had written “two memoirs”. It seems Google's Gemini and Google Books don't talk to each other!

The funniest error that Google's Gemini made was for Wendy Richard, a British actress famous for her roles in Are You Being Served? and EastEnders. According to Gemini, Wendy Richard has three books about her, two autobiographies and one biography, with one of her autobiographies, Swing N2 My World: Insanely Different Lifestyle, about her swinging lifestyle. But Gemini confused Wendy Richard the actress with Wendy Richards the swinger!

On top of these errors, Google's Gemini generated many scarcely believable search results. Some of Gemini's results are barely credible. According to Gemini, Prince Philip has more books written about him than his wife, Queen Elizabeth II, which is just a preposterous claim. Many ancient Greeks and Romans, like Hippocrates of Kos after whom the Hippocratic Oath was named, Homer, Claudius Ptolemy and Tacitus, one of the founders of the study of history, have no books written about them. And don't get me started on Sun Tzu otherwise he may declare war on me! See Endnote 20 for Gemini's scarcely believable results.

The third type of error is the changing search results Google's Gemini gave me when searching for the same name of person at different times. This error is close to undermining one of the gold standards of scientific research. That scientific gold standard maintains that different people doing the same experiment should yield the same results; if not, it suggests the researcher affects the research findings. That's not science; the researcher shouldn't affect research findings.

Though no one else was systematically involved with this experiment, I often searched for, usually by mistake, the same person more than once. These multiple searches of the same person often and inexplicably yielded different results. See Endnote 21A and Endnote 21B for Gemini's changing search results. This is a very troubling error as I'm of the view that if someone else did this experiment, they would find very different search results than what I found. Just to prove my point, one blogger searched for the number of books written about Joe Biden. My original search came up with no results, his search came up with many books. I've just done another search for Joe Biden, and Gemini tells me he “has authored at least two major autobiographical memoirs and a children’s book, while dozens of biographies and unauthorized profiles have been written about him”. How do you explain these discrepancies? Google's Gemini is no gold standard scientist!

The problem of changing search results affected the results of Muhammad Ali, David Beckham, Ludwig van Beethoven, Beyoncé, Osama bin Laden, Simón Bolívar, Charlie Chaplin, Noam Chomsky, George Armstrong Custer, Walt Disney, Queen Elizabeth II, Mahatma Gandhi, Genghis Khan, Che Guevara, Adolf Hitler, Yusuf Islam (formerly Cat Stevens), John Lennon, Jennifer Lopez, Paul McCartney, Marilyn Monroe, Andrew Mountbatten-Windsor (formerly Prince Andrew), Elon Musk, Nebuchadnezzar II, Paracelsus, Marie Pasteur, Tadej Pogačar, Elvis Presley, Babe Ruth, Marquis de Sade, Harry Styles, Taylor Swift, Alexis de Tocqueville, Vincent van Gogh, Virgil, Mao Zedong and Mark Zuckerberg.

There are a few examples I want to highlight. I searched the author Margaret Atwood three times and each time I got three different results. Brennus, a Gallic chieftain fighting the Romans, went from no books to several books and Ron Hicklin, the American singer, went from no books to one book, both just in the space of a few minutes. Likewise a very top ice hockey player Alexander Ovechkin went from two to nine books and, conversely, Bob Geldof went from four books to one book, both in the space of a few days.

The fourth type of error concerned the search results of the same person but with different names. Quite often Google's Gemini would give me different results depending on what name I used for the search. For example, Alhazen, the Islamic polymath and so-called father of modern optometry, came up with numerous and several books; but when his full name, Ibn al-Haytham, was searched, no books was the result. See Endnote 22 for different search results for the same people with different names.

The fifth type of error is about inconsistency. Despite Google Translate seemingly at its service, Google's Gemini was useless at picking up books not written in the English language. Sometimes Gemini counted non-English language books, and sometimes it didn't. Gemini picked up non-English books for the former Prime Minister of Israel, Golda Meir, with one book written in Hebrew, two books in French and five books in English (though I suspect there are more books written in Hebrew than Gemini counted). Conversely, Gemini didn't pick up non-English books written about Muhammad Ali - Gemini told me that hundreds of books have been written about the boxer but noted that “One scholar in Pakistan has compiled a bibliography of over 10,000 titles in Urdu alone"; Gemini decided not to include these books written in Urdu as part of Muhammad Ali's count. See Endnote 23 on books not written in the English language.

Google's Gemini was very inconsistent, and such inconsistency significantly affected its rankings and even its leaderboard. My search was about any books written about a person regardless of the language the book was written in. But it seems quite clear that the Gemini search largely picked up books written in the English language and not books written in other languages. The results have significantly favoured those who live/lived and whose work is known in English-language-speaking countries like America, Australia, Britain, Canada and Ireland. My search has disadvantaged those who lived/live in or whose work isn't known in non-English-speaking countries. I realise my search hasn't picked up many people from largely non-English-speaking countries like Argentina, Brazil, China, Egypt, France, Germany, Indonesia, Iran, Italy, Japan, Mexico, Portugal, Russia, Spain, Sweden and Turkey. This is a significant shortcoming in my experiment. And it's astonishing arrogance on behalf of Google's Gemini that it 'thinks' the English language is the only language that matters.

I'm coming to an end soon. But it's no exaggeration that Google's Gemini's performance in this experiment is piss-poor to the point that its results aren't reliable and should be dismissed as rubbish (that's trash to my American friends). Just to recap, of the 10,976 people I searched for books written about them using Gemini, just under 15 per cent of the searches yielded no search results and over 25 per cent of the search results didn't yield quantifiable results, thus leaving about 60 per cent of people searched for with a quantifiable search result. But certainly hundreds, and probably thousands, of these quantifiable search results have to be seriously questioned because of errors as detailed in this blog post plus its endnotes.

When starting this experiment, I expected AI and in this case, Google's Gemini, to do much better than it did. I was only asking for a count of books written about people. Books can be easily counted - most have an ISBN (International Standard Book Number). Also AI is an aggregator tool in that it searches the internet for results, some say cannibalises and steals results. Given that many books published are listed on key sites, such as Amazon, British Library, Goodreads, Google Books, IMDb (the Amazon-owned Internet Movie Database), Library of Congress, OAPEN, Open Library, Scholar Google, The Bookseller, Waterstones and Wikipedia, I didn't think Google's Gemini would have such problems in producing a reliable ranking, but it did. I can forgive Gemini for not picking up books not written in the English language, though it would be reassuring if it accepted that limitation, but I cannot forgive Gemini for its largely inaccurate search results.

The only advantage that Google's Gemini, and for that matter probably other AI tools, has is speed. It would have taken me a very long time to do this experiment using human and not artificial intelligence. But speed is no advantage when AI results are so inaccurate.

I'm not convinced that other AI tools would have performed much better than Google's Gemini but I'm happy to be proven otherwise. I did search for the people with the most number of books written about them using ChatGPT. It told me: (1) Jesus Christ (“hundreds of thousands”), (2) Prophet Muhammad, (3) Napoleon Bonaparte, (4) William Shakespeare, and (5) Adolf Hitler. But tellingly it didn't give me numbers with its ranking. I refuse to ask Grok as I'm sure Elon Musk will be number one!

This experiment only tested one AI tool, Google's Gemini, about the number of books written about people. No more, no less. I readily accept that the use of AI has beneficial consequences in forecasting weather, diagnosing illness, synthesising court cases, writing marketing materials and even drafting blog posts.

I know I've been a sad geek doing this experiment. I've been very detailed and granular, and hopefully even forensic and scientific, to demonstrate how unintelligent AI is. My worry is that those companies who have invested heavily in AI and desperately wanting a return on their investment are foisting AI on us, without much choice, and that people consuming AI don't have the critical skills to understand AI's significant limitations. AI, at the moment, is created by humans. Its instructions, whether algorithms or codes, are created by humans, and humans aren't known to be infallible. Just because AI says it's true doesn't mean it is true. Human intelligence is needed to make sense of artificial intelligence.

Returning to Google's Gemini in this experiment, I got the impression with its vague and ever changing results that it just made things up as it went along. At best AI should stand for artificial imagination, artificial inaccuracy or, at worst, artificial ineptitude!