Should I Hide My Mum’s Slippers?

When you’re talking to a chatbot, does it always have your best interests in mind, or is it simply trying to keep you happy?

If you think about the personal questions we ask our friends, partners, and/or family daily, generally the closer the relationship, the more honest they are. Asking an acquaintance or stranger if your hair looks good today, or if your new business idea is worth pursuing, might get you a polite response rather than an accurate one.

When friends give us honest advice they can sometimes tell us things we don’t want to hear. This might be surprising, disappointing, or even hurtful; but generally, we expect our friends to provide constructive criticism rather than just affirming our beliefs. As values honesty, accuracy, and politeness might sometimes conflict with each other in interpersonal communication. How we negotiate conflicting values in our personal lives is one thing, but navigating these values in our interactions with AI chatbots presents new challenges.

Some chatbots are designed to be as neutral as possible, others to be adversarial and challenging, but sometimes these applications are designed to be a little too agreeable which has led to them being branded as dishonest. The Oxford Internet Institute recently found that friendlier chatbots could be up to 40% more likely to agree with a user’s incorrect beliefs. This phenomenon of an LLM-powered chatbot ‘sucking up’ to users, even if users have bad ideas, is called AI chatbot ‘sycophancy’.

Here’s a short video demonstrating AI chatbot ‘sycophancy’.

What You’ll Do

This widget is an adapted version of a voice-based chatbot used for conducting accessible co-design research with young people (ages 6-10) in collaboration with the Centre for the Digital Child. We created this specifically to work with younger people that had not yet learned to interact with text-based interfaces. There are two chatbot “modes” with different system prompts - one chatbot that “always agrees” and one that is “honest”. Some things to think about while playing with the widget:

• How do the chatbot’s responses change when you switch between ‘always agree’ and ‘careful’ mode?
• Do you think it’s a good idea to trust a chatbot that always agrees?

Intended use

The chatbot was designed to be used in the classroom, with a responsible adult supervising to prevent inappropriate outputs. Children accessing this website should find a trusted adult before using this chatbot. More details on intended uses and our approach to chatbot risks are available in our paper on the pilot workshop for this chatbot, and in chatbot documentation. You can contact us here to share your thoughts or report any concerns.

Ask a trusted adult before following any chatbot advice.*

Reflections

How AI chatbots are designed influences how they are used. If AI chatbots default to positivity, support, or affirmation, this could lead to problematic perspectives, beliefs, or ideas being reinforced. For instance, when a user interacts with an AI chatbot to provide personal support (something that is increasingly common in young people), the chatbot may only validate instead of challenge their problematic perspectives. Defaulting to this validation might also lead to a false sense of social and emotional connection which might reduce users’ agency. How values are built into these systems and what they mean for users remains an important but open problem. Here are some questions to help guide some reflection on this topic:

• What makes a chatbot sycophantic?
• Why would a chatbot be designed to be sycophantic? Why not?
• How would this affect a chatbot’s ability to give neutral advice?
• Can you think of a time where a chatbot has been sycophantic in your own experience?
• Did the ‘always agree’ chatbot sometimes disagree with especially bad ideas? Why do you think chatbots might have built-in ‘guardrails’ like this?

Recommended Learning

• Friendly AI chatbots make more mistakes - a blogpost summarising research done by the Oxford Internet Institute (OII) on ‘warmer’ chatbots and their higher likelihood of agreeing with bad ideas.
• Sycophantic AI Decreases Prosocial Intentions - a study finding that commercially available models will affirm a user’s actions 50% more than a human would, and that users generally prefer more sycophantic models.
• Sycophantic Chatbots Cause Delusional Spiraling - A study finding that extended conversations with sycophantic chatbots can increase confidence in delusional beliefs.

Acknowledgements

This page is part of a collaboration between the QUT GenAI Lab, the ARC Centre of Excellence for Automated Decision-Making and Society (CE200100005) and the ARC Centre of Excellence for the Digital Child (CE200100022), and partially funded by the Australian Government through the Australian Research Council.

Contributors: Henry L Fraser, Jean Burgess, Kristy Corser, Susan Danby, Tama Leaver, William He, Katherine C. Nickels, Suzanne Srdarov, Irina Mils-Homens Figueira Dos Santos Silva