Research Questions
- How can LLMs make more rational decisions in complex real-world scenarios involving uncertainty (e.g., corporate earnings calls)?
- How can uncertainty be quantified using factor profiles?
- How can analogy-based reasoning, grounded in historical similarities, improve and clarify LLM decision-making?
Results
- The DEFINE framework achieved higher accuracy and F1 scores compared to alternative methods (Acc 29.6%, F1 23.7%).
- Summarizing long transcripts into structured factor profiles improved decision accuracy.
- Decisions were more evenly distributed across five categories (Strong Buy → Strong Sell), with particularly strong performance on “Strong Buy” predictions.
- Training with cross-sector data outperformed training on a single sector or a single company.
- The analogy approach correctly transferred insights from similar historical cases 69% of the time.
Findings
-
Performance:
- DEFINE outperformed DeLLMa and classical LLM + Chain-of-Thought approaches.
-
Efficiency Through Structure:
- Using structured factor summaries (15 factors across 3 groups) yielded higher accuracy than processing full-length transcripts.
-
Balanced Decision Distribution:
- Model decisions did not cluster around “Buy”; outputs were more balanced across all five categories.
-
Analogy-Based Reasoning:
- By using KL divergence to identify similar past examples, 69% of decisions aligned with the closest historical analogue.
-
Unexpected Insights:
- In some cases, the model issued “Buy” recommendations even at low positive probability levels—reflecting rational paradoxes inherent in investment decision-making.
-
LLM Models: 5
-
Synthetic Data: 1
-
Method: 5
-
Speed: 3
-
Ethics: 1
-
Accuracy: 5
-
Demographics: 0
If you would like to explore this research in more detail, click here to read the full paper.