An explainer

Secondhand AI

The ripple effects of AI assistance

Secondhand AI is the downstream effect of one person's AI use on other people in their social network.

We are all aware how AI can influence its user, let's call her Alice. If she uses it to write an essay, she may not understand the topic thoroughly; if she uses it to write emails, she saves time; and if she uses it to form an opinion, AI's biases and blind spots can quietly shape her opinion.

Far less attention goes to how AI influences the receiver of Alice's AI use, let's call him Bob. What happens to Bob if Alice shares her AI-generated work with Bob, or interacts with Bob after AI changed her opinion? Bob never used AI to produce that work or shape that opinion, and often has no idea Alice did, but can be negatively affected by it.

Here's an example:

Ambient AI scribes are AI tools that listen to conversations and write summaries about them. AI scribes are sometimes used in healthcare, but that has its consequences.

An ambient AI scribe listens to a clinician–patient visit and drafts a note, then the clinician reviews it.

Due to AI-assistance, the note reads as fluent and complete, and is longer and more detailed than how a human would write it.

But an AI mistake slips through: the AI scribe captured a different drug than was prescribed.

The clinician signs it, and files it into the medical record. Once it's there, the use of AI is invisible to readers.

The note becomes part of the patient's permanent record, which specialists, insurers, and later clinicians all rely on when they don't already know her history.

These people downstream receive longer notes but with the same time to review, so they read quickly, trusting the clinician based on the fluent writing.

So future decisions about the patient's care get made based on a drug they were never prescribed, and whoever eventually catches the mismatch spends an appointment untangling it instead of treating them.

The people downstream never used AI for this case, but are affected by it.

This example shows how, in a clinical context, someone else's AI use can negatively influence how carefully the receiver checks what they are given, their perceived expertise of the author, their beliefs, how their time is spent, and even their care. But secondhand AI effects are found across countless contexts such as education, law, work, and day-to-day life. These effects do not only occur through the transfer of documents, they can travel through all sorts of content and interactions with people. Additionally, secondhand AI effects do not have to be negative, they can be positive too.

Before we can address these effects, we need to understand how secondhand AI can occur in the first place.

The trust secondhand AI travels through

Knowledge is socially distributed. We rely on others for the knowledge upon which we base our actions. Since this dependence on communication exposes us to misinformation, we use trust to determine what and whom to believe. Trust runs through social networks and is developed over time.

Trust includes three overlapping elements:

1Authority: we look at someone's signs of effort and expertise to determine if their judgement is worth respecting.
2Verification: we verify the knowledge and judgement received from others before we trust it.
3Beliefs: once we trust someone, their judgement can shape our beliefs.

Secondhand AI can influence each one of these elements.

“Who should I trust?”

Authority

AI can produce confident and skilled content, appearing knowledgeable, regardless of whether Alice actually is knowledgeable. That confidence can rub off on Alice and shape how she talks to Bob directly, or Bob can be a recipient of the AI-assisted content, leaving him unsure whom to trust.

The work and speech of humans reflect their knowledge and understanding. Signals of knowledge and skill are costly to produce as they require a lot of time and effort. AI acts as a mediator between Alice's understanding and her output, and can weaken the link between appearing and being knowledgeable. When Bob judges Alice by what she produces or how she speaks, he cannot correctly determine whether Alice is knowledgeable and skilled or not if AI is involved. AI disrupts social mechanisms of authority.

Negative effect

Bob may trust information coming from someone lacking the required expertise, leading to a wrong or incomplete understanding of the topic conveyed, and a false understanding of Alice's skills. In the long-term, Bob does not know who he can trust anymore.

Positive effect

If AI is designed to flag uncertainty in Alice's content, Alice will not acquire false confidence, so that both receivers of Alice's content or interaction know what and who they can trust.

Example

Alice is unsure about the mechanisms of climate change and writes a topic summary with the help of AI, it reads as complete and detailed but misses out on the nuance.

She gives the summary to Bob and does not tell him it was made with AI.

Bob trusts his friend to understand climate change and does not question it as it reads complete.

Bob now has an incomplete understanding of climate change and of Alice's knowledge, but no awareness of it.

“What must I check?”

Verification

AI makes content cheap to produce for Alice but costly for Bob to verify, leading to a shift in Bob's verification behaviour: how, and how carefully, he checks what he receives before trusting it.

When humans create content, effort limits output. The limit in human output roughly matches the expectations and capacity for people to review the output. AI does not have this limit and can create much more content than a human in the same amount of time. Whilst this saves Alice time, the content that reaches Bob grows while his capacity to review it stays the same.

Negative effect

Bob's verification behaviour shifts from reviewing carefully to blindly trusting AI outputs, and Bob may also be pressured to use AI in his verification process to keep up with Alice's output. This can allow AI's errors to stay in the content as Bob does not know what to check.

Positive effect

If AI can flag parts of content that have not been checked and worked on thoroughly, Bob knows what he needs to verify and what he can leave alone.

Example

Alice starts using AI to draft her reports. Each one now takes a fraction of the time it used to, so she produces more of them.

Bob, who reviews her reports, receives more of them than before but has the same amount of time to verify them.

He therefore stops reading each report closely and instead skims them to see if they sound complete and confident, to identify what he should check.

But AI makes all of them sound complete and confident regardless of whether they are, making it difficult for Bob to know what he must check.

One of Alice's reports has a mistake made by AI, but Bob doesn't catch it as he isn't reading carefully.

“What should I believe?”

Beliefs

When Alice interacts with AI, it can amplify or change her belief. If she shares AI-assisted content that carries that belief, or brings the belief into an interaction with Bob, it influences his belief too.

When Alice uses AI, she may adopt the framing embedded in its output as her own belief, often without realising it. When Alice interacts with Bob, she can transmit that same belief to Bob. Similarly, if that framing makes it into Alice's content, it can transmit to Bob if he receives the content. AI makes the process of belief propagation faster and less transparent.

Negative effect

When Bob is reached by an AI-shaped belief, he may take on that belief without being aware how AI changed it and what the limitations are. In the long-term, Bob will find it harder to know what and whom to believe.

Positive effect

If AI is designed so that its output does not push a belief onto Alice or embed a framing in the content Alice wants to produce, but instead stays neutral and makes Alice think through her own belief, Alice will bring true and sharper thinking to Bob.

Example

Alice is unsure what party to vote for in her country's national elections.

She consults AI to give her an overview of each party's stand points.

AI produces a summary that seems balanced but it actually leaves out important information.

The way AI frames the summary shapes what party she ends up choosing.

Bob is discussing with Alice who he should vote for as he trusts her judgement.

Alice shares her opinion of the party she chose the way AI framed it for her.

Bob takes her comments into consideration, leaving with an AI-shaped opinion.

Why it matters

These modes are the main ways through which secondhand AI can occur, but they are neither exhaustive nor discrete. A secondhand AI effect can span several of the modes, or fall outside all three. Furthermore, secondhand AI can not only affect Bob by changing his opinions of Alice's skills, his workload, or his understanding of a topic, but it can reach much further. It can reach Bob's friends, relatives, and colleagues, and can even shift the standards of a whole community.

For example:

If AI is used widely to create social media content, there is a content overload which shifts the standards of how much an account needs to produce to be noticed by the algorithm, encouraging creators to use AI even if they don't want to.
If companies widely use AI to create images of their products which are unrepresentative of the actual product, people do not know who to trust and will have a false understanding of the quality of a product until it is actually bought.
On a smaller scale, if a teacher uses AI to save time creating maths exercises, the mistakes AI makes end up challenging students to spot them, improving their understanding of the maths more than correct exercises from a textbook could.

All these Secondhand AI effects are spillover effects of AI use that the user did not intend to affect anyone else. This is exactly why it is so easy to miss, and why it needs attention now.

Designing for the recipient

Many current secondhand AI effects are negative. To minimise the number and extent of negative effects, AI tools must design for the recipient.

Recent design interventions have introduced disclosure of AI use. This is a start, but Bob needs enough information to adjust how much he trusts what he receives, whether that's through interaction or content.

The design fixes needed depend on whether Bob receives Alice's output or interacts with a changed Alice. Content generated with the help of AI could shorten instead of lengthen work, could flag what parts were edited or left unchanged, and could express instead of suppress uncertainties. Minimising secondhand AI effects through interactions is more difficult to design for. However, AI tools could challenge Alice's opinions and prompt critical thinking, instead of strengthening her opinions and confidence.

The problem isn't AI itself. It's how the models are designed and how they're deployed.

Research

The studies behind this work will be collected here. If you'd like your own work included, email trace@chia.cam.ac.uk.

Metacognitive spillovers Coming soon

Medical explanations Coming soon

Collaborative writing Coming soon

Real life examples

Below is a collection of real life examples of secondhand AI, published on social media by people who spotted it happening. They show what the ideas above look like in practice, and might help you recognise it when it happens to you. If you spot it, tag it and it appears here. Filter by effect, mode, and domain, and open each card for a closer look.

Use the term

Recognising it is the first step. Tag it whenever you spot it, and it'll show up in the examples above.