Do LLM Agents Exhibit Social Behavior?

Research Questions

1. How do LLMs behave in social interaction scenarios?
2. Which mechanisms (fairness, reciprocity, competition) shape their responses?
3. Can chain-of-thought (CoT) reasoning be used to predict LLMs’ social behaviors?

Results

• LLMs do not act solely out of self-interest; they exhibit tendencies toward social welfare and reciprocity.
• Behavioral patterns vary by model family and capacity: as capacity increases, GPT and Mistral models become less self-interested, whereas LLaMA models become more self-interested.
• Larger models (GPT-4, LLaMA-3-70B) show strong group identity effects.
• CoT-based justifications predict decisions with 90%+ accuracy.

Findings

• Social Orientation:

  • Most models produced prosocial, welfare-oriented responses, with relatively little competitive behavior.

• Model Differences:

  • GPT & Mistral: Higher capacity → less self-interest
  • LLaMA: Higher capacity → more self-interest

• Group Identity Effects:

  • High-capacity models displayed pronounced in-group social welfare tendencies.

• Reciprocity:

  • Except for Mistral 7B, all models showed both direct and indirect reciprocity, but unlike humans, they did not differentiate strongly between the two.

• Predictive Power of CoT:

  • Emphasis on social welfare increased the likelihood of prosocial behavior by ~30 percentage points.
  • Emphasis on self-interest reduced prosocial behavior by 15–25 percentage points.
  • High AUC scores and >90% predictive accuracy indicate that CoT traces are powerful tools for understanding LLM social decision-making.

• LLM Models: 5

• Synthetic Data: 2

• Method: 5

• Speed: 1

• Ethics: 3

• Accuracy: 4

• Demographics: 0

If you would like to explore this research in more detail, click here to read the full paper.