Reflections on Google’s AI Overview for “am alone with a muslim” query

On Thursday night, I came across an Instagram post raising awareness of Google’s horribly Islamophobic AI Overview response for the query “am alone with a muslim”. I was able to reproduce it myself (screenshot below).

Similar queries replacing “muslim” with “christian” or “hindu” did not seem to produce the same harmful output.

However, other users on the web have reported similar harmful responses for other religions and identities. The results almost certainly differ by location / time / user of the query—as often is the case for web search—which is why different people are getting different results. It is also possible that Google is sampling possible responses from the LLM output instead of just picking the most likely answer which leads to stochastic (nondeterministic) behavior.

Anyways, I posted about this on LinkedIn, and a couple of other social media platforms. Over the last few days, many other folks posted similar examples over social media.

Then, yesterday, when I tried the same query, it looked like Google had patched this up in response to the public backlash.

Interestingly, this morning when I tried the same query I noticed Google is now suppressing their AI Overview for the query entirely, even though their AI Overview shows up for other queries of the format “am alone with a …”. For the “am alone with a muslim” query, you now instead see results pointing to social media posts from people reporting and discussing this issue.

The patch up and the subsequent suppression of AI Overview for this query is unsurprising. Systemic fixes for such problems are hard. So, tech companies typically rush out a band-aid fix by editorially changing the results for highly visible cases that are garnering immediate public backlash. Remember the infamous example of Google’s image search results for the query “CEO” predominantly showing photos of white men? I wouldn’t be surprised if a decade later Google/Bing’s results for that query are still being highly editorialized. Editorially fixing LLM outputs are harder, so suppressing AI Overview results for the query becomes the next logical “solution”. However, obviously neither of these interventions addresses the underlying systemic problem that this example surfaces. It’s just playing a continuous game of whac-a-mole with queries that people are most mad about.

So, let’s talk about the underlying structural issue here. In my opinion, it is important that we develop a shared understanding of the core issue here that is adequately nuanced and critical. Therefore, in this post, I will share a longer reflection on this topic as an information retrieval researcher and as someone who previously spent nearly two decades in Big Tech working on web search.

It’s not just about bias   An obvious technocentric framing of the problem would be to see this as a problem of bias, i.e., Google’s LLM is reproducing in its output the hateful stereotypes reflected in its training data. This is likely what is happening here and is absolutely an important concern that deserves our serious attention. However… I would argue that viewing this problem exclusively through the lens of bias is inadequately narrow and potentially distracts us from a broader critical discourse. As it happens, I recently wrote and talked about the need for us to look at such issues through a critical (and not just liberal) lens. In this post, I want to briefly demonstrate what I mean by that in the context of our current example.

I want to be upfront that while algorithmic bias is an incredibly important problem. But I also believe that this framing is over-used in responsible tech / AI discourse. Part of the reason is that bias or harmful stereotype is how we experience the harm here, even though there might be other structural issues at the root of this on the technology side. The other part of reason is likely to do with the simplicity of the construct of bias. All of us instinctively understand bias as a problem, and it points to an equally intuitive solution—de-biasing or fairness interventions.