What changes when players can ask an NPC anything?

In Hsu and colleagues’ LLM-NPC study, players reported more autonomy, but also higher cognitive load and lower usability and trust. The authors found no statistically significant improvement in overall gaming experience.^[1]

Consider a hypothetical scene where a player needs a clue. A dialogue menu gives them questions to choose from. With a text box, they have to work out what to ask. After the NPC replies, they still have to decide whether it understood them and whether they got the clue.

A repeated question could be useful to look at. Perhaps the player missed the clue in the reply, or perhaps they expected the character to do something else. Ask what they were trying to accomplish at that point. Their explanation can give you context for the words in the dialogue log.

For a comparison, you could keep the goal and available clues the same and change only how the player asks. Record what happens between the first question and the player’s next action. Ask separately how much freedom they felt and how hard it was to get the information they needed. Someone may value the freedom and still find the task difficult.

Optional opening questions or a clearer signal when a clue has been found are possible changes to test. Choose a change that addresses the moment you observed, then check whether it helps the player move on. The paper’s abstract does not establish that either change works.