Your LLM Ran Out of Knowledge — Now What?
Feb 22, 2025 · 12:51
Speaker 1 presents a technique for applying LLMs to low-knowledge domains like corporate negotiations and geopolitics, where structured training data is scarce. The method uses a parsing engine to identify the problem type and match domain-specific heuristics—binary rules like 'must prioritize agreements with highest combined value' for negotiations or 'must have three independent paths for critical resources' for geopolitics. After reformatting the input for consistency, the system sends it to a reasoning model (e.g., Anthropic's Claude with a world sim) that applies the rule set alongside powerful reasoning capabilities. The demo shows an intelligence estimate scenario where the model identifies geopolitics, reformats the query, and evaluates options against rules—eliminating those that breach heuristics (e.g., scenario five violating a rule on control thresholds). The approach enables generating dozens of scenarios in a fraction of the time a human expert would take, allowing practitioners to explore more options and find novel solutions while keeping a human in the loop for subject-matter oversight.
