Log In
  • Home
  • Articles
  • Reduce Disparity Between LLMs and Humans: Optimal LLM Sample Calibration

Reduce Disparity Between LLMs and Humans: Optimal LLM Sample Calibration

Research Questions

  1. How can LLM outputs be calibrated across demographic groups to better approximate human responses?
  2. Can this approach be made model-agnostic (applicable to any LLM)?
  3. To what extent can calibration be transferred across domains or geographic regions?

Results

  • Human Mimicry Calibration (HMC) significantly improved alignment between LLM responses and human data.
  • HMC outperformed traditional population-density–based weighting methods.
  • Spatial transfer (Texas → New York, California, Florida) was successful, while topic transfer—especially in politics and sensitive domains—was limited.
  • The highest weights were assigned to young (18–34) and low-income demographics, which produced outputs most similar to human behavior.

Findings

  • Calibration Technique:

    • HMC improves LLM-human alignment by reweighting demographic persona outputs toward groups that better mimic human data.

  • Performance:

    • HMC achieved 20–30% higher accuracy compared to uniform and population-weighted baselines.

  • Transferability:

    • Geographic transfer worked effectively.
    • Cross-domain transfer, particularly in sensitive or political topics, showed reduced accuracy.

  • Evaluation Metrics:

    • Differences between human and LLM responses were quantified using Kendall’s Tau and Wasserstein Distance, both showing substantial reductions after calibration.

  • Key Insight:

    • Younger and lower-income groups emerged as the most representative demographic segments in terms of producing human-like LLM outputs.

  • LLM Models: 5

  • Synthetic Data: 4

  • Method: 5

  • Speed: 3

  • Ethics: 3

  • Accuracy: 5

  • Demographics: 5

If you would like to access more detailed information about this article, click here to view the supplementary material.

5 min read

Related Articles

Analyzing Brand Perception in LLMs

Evaluation of bias and consistency levels in LLMs toward brands.

Read more

Beyond Demographics: Aligning Role-playing LLM-based Agents Using Human Belief Networks

Alignment of LLM agents with human behavior: the impact of demographic information and belief networks on opinion imitation accuracy.

Read more

Can LLMs Capture Human Preferences?

The ability of LLMs to imitate human preferences and the effects of language and chain-of-thought methods.

Read more

Comparing Traditional and LLM-based Search for Consumer Choice: A Randomized Experiment

Examining the effects of LLM-based search systems on efficiency, accuracy, user perception, and error handling compared to traditional search, and the role of confidence cues.

Read more

Do LLM Agents Exhibit Social Behavior?

LLM behavior in social interactions: the role of group identity, reciprocity mechanisms, and chain-of-thought processes in predicting social behavior.

Read more

Enhancing LLM Decision-Making with Factor Profiles and Analogical Reasoning (DEFINE)

Rational decision-making by LLMs in financial scenarios under uncertainty: improving decision accuracy and transparency through factor profiles and historical analogy.

Read more

Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?

The capacity of LLMs to imitate human agents in behavioral economics experiments: model diversity, endowment effects, and experimental reproducibility.

Read more