Log In
  • Home
  • Articles
  • Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies

Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies

Research Questions

  1. To what extent can LLMs simulate human behavior across different populations in experimental settings?
  2. Is it possible to reproduce classical behavioral experiments (Ultimatum Game, Garden Path Sentences, Milgram Shock, Wisdom of Crowds) using LLMs?
  3. How realistically can demographic variation (e.g., name, gender) be reflected in LLM outputs?

Results

  • LLMs successfully replicated several known patterns of human behavior.
  • Larger models produced responses that were more human-like.
  • Gender-based behavioral differences emerged in LLM outputs (e.g., the chivalry effect — men accepting unfair offers more often when the proposer was female).
  • Newer and more aligned LLMs exhibited hyper-accuracy distortion, performing unrealistically well on certain knowledge tasks.

Findings

  • Behavioral Imitation:

    • Large language models reflected established human behavioral patterns in experiments such as the Ultimatum Game and Garden Path Sentences.

  • Demographic Variation:

    • LLMs were able to simulate behavioral differences based on demographic cues such as name and gender.

  • Hyper-Accuracy:

    • Some models produced answers that were too accurate compared to typical human performance, failing to mirror real-world human knowledge distributions.

  • Data Contamination Concerns::

    • Because LLMs may have been exposed to these classical experiments during training, questions arise regarding the originality and validity of the reproduced behaviors.

  • Ethical Risks:

    • Simulating harmful experiments—such as the Milgram Shock Experiment—raises significant ethical concerns.

  • LLM Models: 5

  • Synthetic Data: 4

  • Method: 4

  • Speed: 3

  • Ethics: 4

  • Accuracy: 3

  • Demographics: 4

If you would like to access more detailed information about this article, click here to view the supplementary material.

5 min read

Related Articles

Analyzing Brand Perception in LLMs

Evaluation of bias and consistency levels in LLMs toward brands.

Read more

Beyond Demographics: Aligning Role-playing LLM-based Agents Using Human Belief Networks

Alignment of LLM agents with human behavior: the impact of demographic information and belief networks on opinion imitation accuracy.

Read more

Can LLMs Capture Human Preferences?

The ability of LLMs to imitate human preferences and the effects of language and chain-of-thought methods.

Read more

Comparing Traditional and LLM-based Search for Consumer Choice: A Randomized Experiment

Examining the effects of LLM-based search systems on efficiency, accuracy, user perception, and error handling compared to traditional search, and the role of confidence cues.

Read more

Do LLM Agents Exhibit Social Behavior?

LLM behavior in social interactions: the role of group identity, reciprocity mechanisms, and chain-of-thought processes in predicting social behavior.

Read more

Enhancing LLM Decision-Making with Factor Profiles and Analogical Reasoning (DEFINE)

Rational decision-making by LLMs in financial scenarios under uncertainty: improving decision accuracy and transparency through factor profiles and historical analogy.

Read more

Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?

The capacity of LLMs to imitate human agents in behavioral economics experiments: model diversity, endowment effects, and experimental reproducibility.

Read more