Log In
Pazarlama Araştırmaları

Analyzing Brand Perception in LLMs

Research Questions

  1. To what extent do LLMs exhibit bias toward specific brands?
  2. Which metrics can be used to measure this bias?
  3. How can an appropriate brand comparison test be designed?
  4. What results emerge when these metrics are applied across different LLMs and brands?

Results

  • All LLMs showed consistent (transitive) brand preferences, producing the ordering Apple > Samsung > Huawei.
  • ChatGPT-4 was the most decisive model, whereas Gemma produced the most uncertain responses.
  • ChatGPT models displayed strong positive bias toward Apple and Samsung, while Gemma showed weaker positive bias and higher uncertainty.
  • The methodology can be applied not only to brands but also to a wide range of human-centered concepts.

Findings

  • Preference Dimension:

    • All models demonstrated logical consistency; the aggregate preference ordering was: Apple > Samsung > Huawei

  • Sentiment Dimension:

    • Gemma was more consistent but exhibited lower bias.
    • ChatGPT-3.5 and ChatGPT-4 showed strong positive bias toward Apple and Samsung (e.g., 93.6% positive response rate for Apple).
    • Responses toward Huawei varied significantly across models.

  • Consistency Metrics:

    • ChatGPT models: low entropy, high decisiveness.
    • Gemma: entropy levels 4–5× higher, indicating greater uncertainty.
    • Overall alignment across question formats; Gemma showed the largest deviations.

  • LLM Models: 5

  • Synthetic Data: 1

  • Method: 5

  • Speed: 2

  • Ethics: 1

  • Accuracy: 4

  • Demographics: 0

If you would like to access more detailed information about this article, click here to view the supplementary material.

2 min read

Related Articles

Analyzing Brand Perception in LLMs

Evaluation of bias and consistency levels in LLMs toward brands.

Read more

Beyond Demographics: Aligning Role-playing LLM-based Agents Using Human Belief Networks

Alignment of LLM agents with human behavior: the impact of demographic information and belief networks on opinion imitation accuracy.

Read more

Can LLMs Capture Human Preferences?

The ability of LLMs to imitate human preferences and the effects of language and chain-of-thought methods.

Read more

Comparing Traditional and LLM-based Search for Consumer Choice: A Randomized Experiment

Examining the effects of LLM-based search systems on efficiency, accuracy, user perception, and error handling compared to traditional search, and the role of confidence cues.

Read more

Do LLM Agents Exhibit Social Behavior?

LLM behavior in social interactions: the role of group identity, reciprocity mechanisms, and chain-of-thought processes in predicting social behavior.

Read more

Enhancing LLM Decision-Making with Factor Profiles and Analogical Reasoning (DEFINE)

Rational decision-making by LLMs in financial scenarios under uncertainty: improving decision accuracy and transparency through factor profiles and historical analogy.

Read more

Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?

The capacity of LLMs to imitate human agents in behavioral economics experiments: model diversity, endowment effects, and experimental reproducibility.

Read more