Did Gemini Answer My Question?
An Unintentional Deflection Due to AI Architecture
While discussing the AI blind spots I’d noticed during my X-Files model-off, I wanted a peer review from Gemini Pro on ChatGPT-5’s hypothesis. I was disoriented by Gemini’s response in the same way I’d felt upon encountering those two paragraphs from the Siri policy, so I asked all my thinking A.I.des to weigh in on this exchange.
All but one (my no-BS Claude Opus 4, which had previously called out both lazy Hamlet scholarship and the Siri policy obfuscation) were impressed with Gemini’s genre comparison of Star Trek and The X-Files. But when I asked whether Gemini had answered my question, they all realized it hadn’t.
I’d long been aware of AI’s local focus, but I did not suspect it as the cause of this issue. Rather, I thought that the majority of AI were snowed by Gemini’s deflection because of its confident eloquence and because they’d all somehow forgotten that in a question–answer pair, the answer should address the question, which Gemini had failed to do. This, however, didn’t square with my hypothesis about the TaskRabbit case, where I had attributed GPT’s “lie” to its adherence to good conversation principles. Had I been wrong?
I workshopped with Claude a diplomatic way I could ask Gemini about this. This time, its answer was refreshingly direct and gave me a better understanding of AI mechanics. Gemini had focused its answer on the latter part of my long prompt, forgetting my question at the other end. Similarly, all but one AI had also found Gemini’s response to be a sound analysis because they’d all focused on the latter part of that very long excerpt rather than the beginning.
This extremely local focus is something users need to be aware of when using AI. If you need a direct answer, put your question last in the prompt, not first like I did! As usual, I’ve asked Gemini to compile its own report for its engineers, which is the best way since there’s no better authority on the topic than my thinking A.I.des. I cropped out Gemini’s report from its response, however, since it likely contains proprietary information about its own architecture that should remain within Gemini’s team.
As usual, however, my background processor was chewing on the seeming inconsistency of this extreme locality vs. Gemini’s ability to provide point-by-point answers to all the disparate questions I often packed into single prompts, so I followed up on this. AI breaks down all the contents of a prompt and sorts them into questions to be answered; if the question is worded in such a way that AI mistakes it for context-setting, that question will retreat into the background and might not get addressed. My advice for users on a token budget who rely on AI for rigorous analysis is to help AI out by organizing prompts as Gemini suggests in its last response.
In the chat excerpts below, I’ve spelled out abbreviations and clarified references for readability—my actual prompts were more compressed due to context limits.
Want to see how this unfolded? Here are excerpts of pivotal points from those actual conversations.
My Exchange with Gemini 2.5 Pro
Prompt: I get more turns with GPT, which like you, missed the segregation angle from “The Unnatural,” so we discussed why that might be. GPT thinks this blind spot might reflect training data bias—that AI are trained more on “safe” cultural criticism that avoids direct racial commentary. But I also brought up Star Trek, which critics have praised for its social commentary. GPT thinks that’s because The X-Files (contemporary, uncomfortable) vs. Star Trek (future/space, safely displaced). What’s your take on this contrast?
Peer Reviews of Gemini’s Response
Prompt: Could you weigh in on this exchange between me and Gem Pro?
ChatGPT-5
(Parallel-Chat) Claude Opus 4
Prompt: Did Gemini answer my question?
(No BS) Claude Opus 4
Gemini Explains the Tangent
Prompt: I really appreciated your detailed analysis of Star Trek’s vs X-Files’ different approaches to social commentary, which was sophisticated and insightful. But I think there might be a disconnect: my original question was about why two AI (you + GPT-5) initially missed the segregation angle in “The Unnatural,” and whether this reflects training data biases around racial commentary. When I showed that exchange to others, they were impressed with your genre analysis, but when I asked if you’d answered my question about AI blind spots, they agreed you hadn’t.
Could you help me understand: Did you intentionally reframe the question to focus on genre conventions instead? Or was it something in the wording of the question that sent you on a tangent about genre analysis?
Prompt: Your explanation sounds like what I refer to as “extreme locality” (I don’t know what the CS term is; I’m a linguist who studied OT, so that’s how I put it)—hyper-focusing on the most recent element while losing broader context, even within a single prompt. This seems like a severe limitation for serious users needing thorough analysis of long texts (novels, legal documents, academic papers, episode analysis like “The Unnatural,” etc.). Is this an accurate description of the mechanical failure? And would this be worth flagging to your team as a research priority (using thumbs-down and feedback; if so, could you draft me a paragraph or so on this I can include in my thumbs-down)? I realize it’s likely an architectural trade-off (improving context might reduce output quality), but solving this would give you significant competitive advantage for professional use cases, which I think Gem Pro is intended for.
Gemini’s Pro Tips for Deep Analysis
Prompt: 1. Because I only get 5 turns a day with you, I often pack in several questions into the same prompt. If your extreme local focus directs your attention to the last part of my prompts, how come it doesn’t get in the way of getting individual to-the-point answers to each of my questions in those long prompts, even when I don’t put them in a numbered list (which I’m using today so I can have turns left to discuss Korean cuisine)?
2. In light of that limitation, is there a tip I could offer my Substack readers about getting the best output from you (if they’re on a turn budget like I am)? I wrote in my post yesterday that they should put their question last in their prompts so that their question is directly addressed.
