Log In

LLMs’ Ways of Seeing User Personas

Research Questions

  1. How do LLMs perceive and interpret user personas?
  2. How does cultural context—particularly user profiles from India—shape LLM interpretations?
  3. Can LLMs reconstruct demographic profiles based on persona descriptions?

Results

  • GPT-3.5 and GPT-4 showed strong alignment across the three India-centered personas analyzed.
  • The highest scores were assigned to Consistency (GPT-3.5: 6.34, GPT-4: 7).
  • The lowest scores were given to Credibility (GPT-3.5: 5.67, GPT-4: 6.33), reflecting limitations in perceived realism.
  • GPT-4 consistently reconstructed demographic traits such as age, income, tech proficiency, and occupation.
  • High agreement was observed between the two models’ outputs.

Findings

  • Consistency:

    • LLMs achieved the strongest performance in capturing internal coherence among persona attributes.

  • Credibility Challenges:

    • The lower realism scores (“Does this persona feel like a real person?”) highlight a known limitation in persona generation across LLMs.

  • Demographic Reconstruction:

  • Models were able to infer demographic profiles from persona descriptions:

    • Persona B (Dependent Family Talker): estimated as 50+, low-to-middle income, low tech proficiency
    • Persona C: predicted as a small business owner/entrepreneur with medium-to-high tech proficiency

  • Model Agreement:

    • GPT-3.5 and GPT-4 produced highly similar ratings, with minimal divergence across evaluated dimensions.

  • LLM Models: 5

  • Synthetic Data: 2

  • Method: 4

  • Speed: 1

  • Ethics: 2

  • Accuracy: 4

  • Demographics: 5

If you would like to access more detailed information about this article, click here to view the supplementary material.

5 min read

Related Articles

Analyzing Brand Perception in LLMs

Evaluation of bias and consistency levels in LLMs toward brands.

Read more

Beyond Demographics: Aligning Role-playing LLM-based Agents Using Human Belief Networks

Alignment of LLM agents with human behavior: the impact of demographic information and belief networks on opinion imitation accuracy.

Read more

Can LLMs Capture Human Preferences?

The ability of LLMs to imitate human preferences and the effects of language and chain-of-thought methods.

Read more

Comparing Traditional and LLM-based Search for Consumer Choice: A Randomized Experiment

Examining the effects of LLM-based search systems on efficiency, accuracy, user perception, and error handling compared to traditional search, and the role of confidence cues.

Read more

Do LLM Agents Exhibit Social Behavior?

LLM behavior in social interactions: the role of group identity, reciprocity mechanisms, and chain-of-thought processes in predicting social behavior.

Read more

Enhancing LLM Decision-Making with Factor Profiles and Analogical Reasoning (DEFINE)

Rational decision-making by LLMs in financial scenarios under uncertainty: improving decision accuracy and transparency through factor profiles and historical analogy.

Read more

Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?

The capacity of LLMs to imitate human agents in behavioral economics experiments: model diversity, endowment effects, and experimental reproducibility.

Read more