Research Questions
- How do LLMs perceive and interpret user personas?
- How does cultural context—particularly user profiles from India—shape LLM interpretations?
- Can LLMs reconstruct demographic profiles based on persona descriptions?
Results
- GPT-3.5 and GPT-4 showed strong alignment across the three India-centered personas analyzed.
- The highest scores were assigned to Consistency (GPT-3.5: 6.34, GPT-4: 7).
- The lowest scores were given to Credibility (GPT-3.5: 5.67, GPT-4: 6.33), reflecting limitations in perceived realism.
- GPT-4 consistently reconstructed demographic traits such as age, income, tech proficiency, and occupation.
- High agreement was observed between the two models’ outputs.
Findings
-
Consistency:
- LLMs achieved the strongest performance in capturing internal coherence among persona attributes.
-
Credibility Challenges:
- The lower realism scores (“Does this persona feel like a real person?”) highlight a known limitation in persona generation across LLMs.
-
Demographic Reconstruction:
-
Models were able to infer demographic profiles from persona descriptions:
- Persona B (Dependent Family Talker): estimated as 50+, low-to-middle income, low tech proficiency
- Persona C: predicted as a small business owner/entrepreneur with medium-to-high tech proficiency
-
Model Agreement:
- GPT-3.5 and GPT-4 produced highly similar ratings, with minimal divergence across evaluated dimensions.
-
LLM Models: 5
-
Synthetic Data: 2
-
Method: 4
-
Speed: 1
-
Ethics: 2
-
Accuracy: 4
-
Demographics: 5
If you would like to explore this research in more detail, click here to read the full paper.