Eight years ago I completed a master’s in Information Management, studying how people capture and share knowledge in digital environments. For my dissertation, I interviewed and shadowed startup founders to understand their information behaviours. I wrote it up as ‘Do you even document bro?’ which I was lucky enough to present a few times. After that, I lectured for a couple of terms at the University of Technology Sydney before kids, covid and life intervened. I haven’t meaningfully engaged in any study or research since then.

But I’m on my summer holidays and my kids are (slightly) more self sufficient. I’ve been working through a pile of recent papers to understand how generative AI is changing information seeking. In 2017, the problem on my mind was getting people to externalise knowledge in a way it could be easily retrieved. Now information is being generated faster than anyone can verify it, by systems that feel authoritative but may not be. I thought I’d attempt a desktop literature review of what’s happening in the research, as a way of flexing some old muscles and attempting to stay current.

Tatsuo Miyajima, Connecting with Everything, 2017

Tatsuo Miyajima, Connecting with Everything, 2017

Like many people, I’m sure we’re in an AI bubble. Companies are laying off staff with pomp and press releases to replace them with bots, then quietly hiring the humans back. Tech giants are capitalising on AI infrastructure then sheepishly dropping sales targets for products people aren’t buying. That said, I feel that the use of generative AI for search and retrieval is impactful and likely here to stay.

To find out what’s happening, I searched for open-access empirical research published since 2023: research that measured what people do with generative AI tools. That gave me roughly a dozen studies spanning experiments, surveys, physiological measurements and observation.

My main finding is that we’re trusting AI systems in ways that don’t match their actual reliability, and our traditional credibility markers are failing us. We trust citations that don’t exist, we trust conversational interfaces more than accurate ones, and we trust unlabelled AI content more than the same content when we know it came from AI. The research shows these are predictable vulnerabilities arising from how our credibility judgments have been decoupled from actual human judgment and comprehension. Different user groups face different risks: some trust too much without enough experience to calibrate that trust, while frequent users anthropomorphise systems in ways that blur appropriate boundaries.

Individual fact-checking is not enough to solve this. We need to improve interface design, change how we check facts online, and create rules to fix how AI performance varies across different languages.

Jump to

We’re Not Searching, We’re Chatting

Search behaviour is changing. Mayerhofer et al. (2025) describe what they call ‘blending’ behaviour: users don’t just search anymore, they toggle between traditional search engines and conversational AI, often using one to verify the other. Their research shows people asking Google to fact-check ChatGPT or using ChatGPT to synthesise results from a Google search. It’s a hybrid approach that suggests we haven’t quite figured out which tool to trust for what.

Pham et al. (2024) found that in e-commerce contexts, people use longer and more conversational queries when interacting with AI chatbots compared to traditional search. Instead of typing ‘wireless headphones under $100’, users are more likely to ask ‘what are some good quality wireless headphones I can get for under $100?’. The interaction mirrors how you’d ask a shop assistant, rather than query a database.

This conversational style matters because it changes what we’re doing cognitively. Traditional search required what researchers call ‘berrypicking’ (Bates, 1989): you’d try different search terms, scan results, refine your query and gradually home in on what you needed. It was iterative and required you to stay critically engaged with whether you were finding good information. Conversation, by contrast, feels more passive. You ask, the AI answers.

We’re dealing with what philosopher Luciano Floridi (2024) calls a fundamental ‘decoupling’ between agency and intelligence. Gen AI systems act as autonomous agents. They can perform complex tasks, generate responses and cite sources, but they have no understanding of what they’re doing. For us as information seekers, this means we can’t rely on the usual signals. A citation doesn’t mean someone read and understood a source. A confident tone doesn’t mean comprehension. We’re interacting with something that acts intelligent without being intelligent. Our credibility senses weren’t built for that.

The blending behaviours documented by Mayerhofer et al. (2025) are an adaptive response to an environment where the traditional markers of human intent and accountability have been severed. We’re trying to restore verification processes by cross-referencing AI with traditional search, but even that strategy has limits when both increasingly draw from overlapping, AI-mediated sources.

Why We Trust AI

It turns out that treating AI like a conversation partner makes us less critical of the results.

Li and Aral (2025) found that when AI-generated content includes citations and reference links, people trust it more, even when those citations are incorrect or completely fabricated. The formal markers of credibility (footnotes, links, an academic appearance) override our instinct to verify the actual content.

This is another instance of Floridi’s decoupling. In traditional information environments, citations indicated that someone, a person with agency and intelligence, had consulted and evaluated those sources. That signal carried meaning. Now we have systems that can generate citation-like elements without any underlying consultation or understanding. The rule of thumb that worked before (citations = credible) fails because the relationship between form and function has broken down.

When AI Feels Human

This blind spot gets wider when you add anthropomorphism to the mix. When AI systems feel more human-like, whether through conversational tone or interface design, we trust them more (Yazan et al., 2025; Huschens et al., 2023). We’re also more willing to trade accuracy for personalisation and conversational flow. We are forgive inaccuracy if the interface feels friendly.

This creates a real design tension: creating engaging, natural interactions makes systems feel more trustworthy, but that trust may not be justified. We’re extending social trust: rapport, conversational flow, apparent understanding, to non-social entities.

What Happens When We Know It’s AI

Sun et al. (2024, 2025) explored this in health information seeking, where the stakes are high. Their research revealed that when people don’t know the source of health information, they trust AI-generated content more than human-generated content. But when information is explicitly labelled as coming from AI, people trust it less than human-attributed content.

Transparency matters. We adjust our trust based on what we think we’re dealing with, but we’re not very good at identifying AI content in the wild. Unlabelled AI content may be trusted more than it should be, while honest labelling reduces trust even in accurate information.

Trust at the Physiological Level

Sun et al. (2025) used eye-tracking, ECG, skin conductance and temperature sensors to measure physiological responses to health information. Machine learning models trained on these signals could predict whether someone trusted the information with 73% accuracy. They also found these responses could identify whether someone identified the content as AI or human-generated with 65% accuracy.

These numbers are specific to their lab setting and need validation in real-world contexts, but they suggest our bodies respond to credibility cues before our conscious minds catch up. Trust operates at multiple levels simultaneously, beyond just cognition. Interface design, credibility rules we’ve carried over from other contexts and unconscious physiological responses are all shaping whether we believe what we’re reading.

The Problem With Saying ‘I Don’t Know’

One design solution you might think would help is having AI systems signal uncertainty. If the model’s not confident, it should say so.

Li and Aral (2025) found that signalling uncertainty reduces trust and may discourage users from relying on correct information. The effect appears context-dependent, but honesty (“I’m not sure about this”) conflicts with user expectations for responsive, confident assistance.

Huang et al. (2025) propose a more sophisticated approach called ‘confidence-based response abstinence’, where AI systems decline to answer when uncertainty is high. But this requires accurate self-assessment from the model, which is not guaranteed. It also risks frustrating users who expect an answer.

There doesn’t appear to be a simple solution. Systems that admit uncertainty are more responsible but less trusted. Systems that project confidence are more persuasive but potentially misleading. It’s a direct consequence of the decoupling between agency and intelligence. We want systems that act like knowledgeable advisors but can’t ground their responses in genuine understanding.

Who Gets It Wrong

Trust dynamics also vary across different user groups, and different populations face different vulnerabilities.

Yazan et al. (2025) found that middle-aged adults show a potentially vulnerable pattern: they express higher trust in ChatGPT but use it less frequently than younger users. This means they have less opportunity to develop calibrated expectations through experience. They’re trusting, but not practised.

Meanwhile, users who employ both ChatGPT and traditional search daily show higher trust and greater anthropomorphisation of AI systems. It’s unclear whether frequent use causes this, or whether people who are already predisposed to trust AI use it more often.

The real takeaway here is that “one-size-fits-all” interface design is fundamentally broken because it assumes we all look at screens with the same level of scepticism. The data shows something much more fragmented. On one hand, you have casual users—like many middle-aged adults in the surveys—who trust the tech implicitly but don’t use it enough to spot its flaws. On the other hand, you have power users who live in these tools daily, but they’ve gotten so cosy with the conversational vibe that they start blurring the line between a software script and a human expert. You can’t fix both problems with the same warning label.

Verification at Scale

There are also broader concerns about AI as an information intermediary. When generative AI sits between us and information sources, it shapes what we’re exposed to and potentially amplifies or mitigates misinformation (Hirvonen et al., 2024; Jarrahi et al., 2025).

Kuznetsova et al. (2025) tested this directly by asking LLM-based chatbots to verify political statements. ChatGPT correctly evaluated approximately 72% of tested claims on their dataset, with substantial variability across languages and topics. High-resource languages like English performed better than low-resource languages, raising equity concerns.

If AI systems are going to serve as fact-checkers or information gatekeepers, differential performance across linguistic communities means some populations face higher misinformation vulnerability than others. The populations already disadvantaged by language and resource constraints get worse AI performance on top of that.

Training data overlap makes this worse. When multiple AI systems draw from overlapping datasets, the traditional verification strategy of cross-referencing multiple sources becomes less effective. If all the sources consulted the same underlying information (or misinformation), checking three AI-generated answers won’t help you triangulate truth. The appearance of independent verification masks shared foundational errors.

What the Research Tells Us (So Far)

I summarised the key research and findings along with their main limitations. Most of this research is less than a year old, much of it is still in preprint and the methods vary significantly. This is emerging evidence, but the field is moving quickly.

Study Study Type What They Looked At Main Finding Important Caveats
Mayerhofer et al., 2025 Observational How people use search vs chat Users blend both and toggle between them for verification Preprint; limited platform diversity; self-reported behaviour
Pham et al., 2024 Experimental E-commerce search behaviour Longer, conversational queries with chatbots Domain-specific to shopping; controlled setting
Li & Aral, 2025 Experiment Trust in AI search results Citations increase trust even when incorrect Preprint; experimental setting; needs replication
Sun et al., 2024/2025 Physiological/Survey Health information trust AI content trusted more when source is unknown; labelling matters Lab-based measures; 2025 paper is advance copy; specific to health domain
Kuznetsova et al., 2025 Evaluation study Political fact-checking by chatbots ~72% accuracy; varies by language and topic Performance sensitive to task framing; limited to political claims
Huang et al., 2025 Conceptual/Design Managing AI uncertainty Proposes systems should refuse to answer when uncertain Requires robust uncertainty estimates; theoretical proposal
Yazan et al., 2025 Survey Trust patterns across user groups Middle-aged users trust more but use less; frequent users anthropomorphise more Correlation not causation; self-reported trust measures

Verification Challenges

Across all these studies, there’s a consistent pattern: our credibility judgements are shaped as much by interface signals and situational context as by informational accuracy. We trust citations even when they’re wrong. We trust friendly-sounding AI more than cold search results. We trust unlabelled AI content more than labelled AI content. We verify less when we’re anxious or when the conversational flow feels natural.

These are predictable vulnerabilities arising from the interaction between human psychology and AI interface design. Traditional credibility markers fail in generative AI contexts.

Floridi’s framework explains this by noting these systems process information without understanding it. As more non-intelligent agents generate content, the burden of checking facts falls on the user. Currently, our tools and interfaces do not support this well.

What Should Happen Next

The research points to some practical directions. The challenges span interface design, user capacity and systemic governance.

Interface Design

We need better ways to show where information comes from and how confident the AI is. But research shows that signalling uncertainty makes people trust the system less, even when it is being honest. Designers have to balance communicating uncertainty without breaking the user experience.

Maybe the solution is to change how the system flags doubts based on what you’re asking. You probably don’t need a heavy-handed warning if you’re just looking up movie trivia, but you absolutely do if you’re asking about medical symptoms. The catch is that tailoring these warnings requires tracking user behavior pretty closely, which opens up a whole different can of worms around privacy, not to mention a UI that changes so much it becomes confusing.

Citation practices present another design challenge. If users trust citations without verifying them, systems must either ensure all citations are genuine and relevant or avoid citation-like elements entirely. The current middle ground: where systems can generate plausible but fabricated references is the worst possible outcome. But guaranteeing citation accuracy is technically demanding. Removing citations might reduce the perceived credibility of accurate information.

The anthropomorphism problem is the trickiest. Conversational, human-like interfaces increase engagement and user satisfaction but also increase misplaced trust. Do we design systems to feel less human-like, knowing this will reduce usability? Or do we accept the trust inflation as a cost of good user experience? No design choice is neutral.

For Users

Traditional information literacy emphasises source evaluation and cross-referencing multiple independent sources. These strategies need adaptation for AI-mediated information environments.

Users need frameworks for:

  • Recognising AI-generated content even when unlabelled. That means watching out for weird stylistic flatlines, algorithmic formatting patterns, and references that look plausible but lead nowhere.
  • Treating citations as claims rather than proof. If the stakes are high, you simply can’t assume a cited source is real or that it actually says what the chatbot claims. You have to trace it back to the primary material.
  • The illusion of consensus. Don’t fall into the trap of cross-referencing three different models to verify a fact. If they are all trained on the same overlapping data bucket, an identical answer isn’t independent confirmation, it’s most likely an echoed error.
  • Maintaining critical distance despite conversational design. Actively remind yourself that conversational flow doesn’t imply understanding, and that a friendly tone isn’t a proxy for accuracy.

Educational interventions should cover technical understanding of AI systems, but also how conversational design shapes trust. We need training that helps people recognise when they’re being swayed by interface features rather than information quality.

But individual verification strategies only scale so far. Not everyone has the time, skills or motivation to fact-check AI responses rigorously, and it’s not reasonable to expect them to.

For Institutions

If AI systems increasingly mediate access to information, performance disparities across languages, topics and cultural contexts create equity concerns that can’t be solved by better design or user education alone.

Governance frameworks need to address:

Transparency and attribution: require AI-generated content to be labelled and training data sources to be disclosed. People can’t calibrate their trust if they don’t know what they’re reading. This needs enforcement and international coordination.

Performance requirements across languages and topics: Kuznetsova et al. (2025) showed that AI gets things right far more often in English than in other languages. Any rules we set need to account for that. Otherwise we’re building systems that work well for some people and fail everyone else.

Audit and accountability: when AI provides misinformation, who is responsible? Current liability rules weren’t designed for systems that generate content without explicit programming. We need ways to identify harm, attribute responsibility and provide remedy.

Citation standards: if AI systems provide citations, those must be real and verifiable. If they can’t reliably do that, perhaps they shouldn’t provide citations at all.

The methodological gaps are real too. Most research reviewed here is either experimental (artificial lab settings) or survey-based (what people say they do, not what they actually do). We need longitudinal studies that follow people over time, cross-cultural work that doesn’t assume everyone behaves like Western educated populations, and studies that actually measure verification behaviour in natural contexts.

Bringing It Back

When I was researching startup founders eight years ago, the challenge was getting people to document what they knew. Knowledge management felt like a problem of capture and retrieval: making tacit knowledge explicit, building systems where people could find what others had learned. The cognitive work was in getting information into systems where it could be found.

Now we’re drowning in confident-sounding information from systems that don’t know anything. The research suggests we haven’t figured out how to navigate this environment yet. We’re trusting the wrong signals: citations that mean nothing, conversational warmth that implies understanding that isn’t there, formal presentation that masks fabrication. We’re verifying less than we should, doing so unevenly across different populations. We face systemic vulnerabilities that individual scepticism can’t fully address. The cognitive work has flipped: instead of focusing on capture, we now need to verify and sense-make in an environment where information is abundant but accountability is absent.

Instead of focusing on capturing knowledge, we now have to verify it in an environment with high volume and low accountability. Moving forward, this requires improved interface design, updated literacy programs for AI information, and governance that addresses transparency.

My information management training emphasised context, provenance and the socio-technical systems that shape how knowledge flows through organisations. Those frameworks feel newly relevant. Understanding where information comes from, who generated it and for what purpose matters more than it used to.

Information can now be generated by agents that act without understanding. Credibility markers can be fabricated. Verification strategies that worked for human-generated content fail.

We are probably in a bubble. Companies will continue quietly walking back their AI promises. But whether the bubble bursts or not, a change in how we seek and trust information is here to stay.

AI assistance disclosure: While the analysis and perspectives here are mine, I did use some generative AI tools during the process. I used Google’s Notebook LM to help with coding some of the papers I’d collated. I used Anthropic’s Claude Opus 4.5 to fact check my interpretations and format the references. All the sources below are real, non-hallucinated, open-access* research papers.

(* With the exception of Bates’ berrypicking, which is a classic.)

References

Bates, M. J. (1989). The design of browsing and berrypicking techniques for the online search interface. Online Review, 13(5), 407–424. https://doi.org/10.1108/eb024320

Floridi, L. (2024). On the future of content in the age of artificial intelligence: Some implications and directions. Philosophy & Technology, 37(3), 112. https://doi.org/10.1007/s13347-024-00806-z

Hirvonen, N., Jylhä, V., Lao, Y., & Larsson, S. (2024). Artificial intelligence in the information ecosystem: Affordances for everyday information seeking. Journal of the Association for Information Science and Technology, 75(10), 1152–1165. https://doi.org/10.1002/asi.24860

Huang, Y., et al. (2025). Confidence-based response abstinence: Improving LLM trustworthiness via activation-based uncertainty estimation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. ACL. https://arxiv.org/abs/2510.13750

Huschens, M., et al. (2023). Do you trust ChatGPT? Perceived credibility of human and AI-generated content. arXiv preprint arXiv:2309.02524. https://arxiv.org/abs/2309.02524

Huynh, M.-T., & Aichner, T. (2025). In generative artificial intelligence we trust: Unpacking determinants and outcomes for cognitive trust. AI & Society, 40, 5849–5869. https://doi.org/10.1007/s00146-025-02378-8

Jarrahi, M. H., Li, L., Robinson, A. P., & Meng, S. (2025). Generative AI and the augmentation of information practices in knowledge work. Behaviour & Information Technology. Advance online publication. https://doi.org/10.1080/0144929X.2025.2551570

Kuznetsova, E., Makhortykh, M., Vziatysheva, V., Stolze, M., Baghumyan, A., & Urman, A. (2025). In generative AI we trust: Can chatbots effectively verify political information? Journal of Computational Social Science, 8, 15. https://doi.org/10.1007/s42001-024-00338-8

Leschanowsky, A., Rech, S., Popp, B., & Bäckström, T. (2024). Evaluating privacy, security, and trust perceptions in conversational AI: A systematic review. Computers in Human Behavior, 161, 108344. https://doi.org/10.1016/j.chb.2024.108344

Li, H., & Aral, S. (2025). Human trust in AI search: A large-scale experiment. arXiv preprint arXiv:2504.06435. https://arxiv.org/abs/2504.06435

Mayerhofer, A., et al. (2025). Blending queries and conversations: Understanding tactics, trust, verification, and system choice in web search and chat interactions. arXiv preprint arXiv:2504.05156. https://arxiv.org/abs/2504.05156

Pham, V. K., Pham Thi, T. D., & Duong, N. T. (2024). A study on information search behaviour using AI-powered engines: Evidence from chatbots on online shopping platforms. SAGE Open, 14(4). https://doi.org/10.1177/21582440241300007

Sun, X., Ma, R., Wei, S., Cesar, P., Bosch, J. A., & El Ali, A. (2025). Understanding trust toward human versus AI-generated health information through behavioural and physiological sensing. International Journal of Human-Computer Studies. Advance online publication. https://arxiv.org/abs/2512.12348

Sun, X., Ma, R., Zhao, X., Li, Z., Lindqvist, J., El Ali, A., & Bosch, J. A. (2024). Trusting the search: Unraveling human trust in health information from Google and ChatGPT. arXiv preprint arXiv:2403.09987. https://doi.org/10.48550/arXiv.2403.09987

Yazan, M., Situmeang, F. B. I., & Verberne, S. (2025). Personality over precision: Exploring the influence of human-likeness on ChatGPT use for search. In Proceedings of NIP@IR 2025. https://doi.org/10.48550/arXiv.2511.06447