82 labs
← Field notes

Stanford’s STORM method, adapted for business research

How the method works, and what we changed to use it for business decisions.

STORM is a research method from Stanford. A model researches a question from several points of view, then writes from an outline in which every section cites its sources. We adapted it for business questions, and this post describes our version.

The name stands for Synthesis of Topic Outlines through Retrieval and Multi-perspective Question Asking. Stanford’s OVAL Lab published it at NAACL 2024, and built it to write Wikipedia-style articles from scratch. The original works in three steps. It finds the points of view that matter for the topic. For each one, it runs a simulated interview in which a writer asks questions and the answers come from web sources. Then it builds an outline from the interviews and writes the article from that outline, with citations.

Our version starts from a fixed set of five points of view and adds one step: a list of the places where they disagree.

Practitioner Academic Skeptic Economist Historian Disagreements Where the views differ Briefing Each claim cited
Five points of view, one list of disagreements, one briefing with sources.

The five points of view

Each one asks different questions of the same sources.

1

Practitioner

Has done the work. Asks what holds up in the field and what needs adjusting.

2

Academic

Asks what the published evidence supports and how strong the data is.

3

Skeptic

Looks for the assumptions behind a claim, the conditions it depends on, and the strongest case against it.

4

Economist

Looks at cost and incentives: who benefits, what it costs, and what happens once everyone does the same.

5

Historian

Asks whether this happened before and how it turned out.

The set changes with the question. A pricing decision mostly needs the economist and the practitioner. A compliance question mostly needs the skeptic and the academic.

The six steps

The steps run in order, and the outline is fixed before any of the final text is written.

  1. 1

    Points of view

    Chosen to fit the decision. The five above are the usual start.

  2. 2

    Interviews

    Each point of view asks its own questions. Each answer cites a source that anyone can open and check.

  3. 3

    Disagreements

    Each place where two points of view conflict goes on a list.

  4. 4

    Outline

    Built once the interviews stop turning up new material. Every section maps to its sources.

  5. 5

    Draft

    Written from the outline only, so every line traces to a source.

  6. 6

    Gaps

    A last pass looks for what is missing: a point of view to add, a question still open. Co-STORM, Stanford’s later system, adds a moderator that raises questions the discussion missed.

The list of disagreements is the part we use most. It shows which questions the sources leave open, and those are the ones a person still has to decide.

What Stanford measured

Stanford asked experienced Wikipedia editors to rate the articles. The baseline was a system that also retrieves sources and writes from an outline. Against it, the editors rated more of the STORM articles as organized (an absolute increase of 25 percent) and as broad in coverage (10 percent). Those results are for encyclopedia articles, the job the method was built for.

When it is worth the effort

The method takes more time and more tokens than a single prompt. It suits decisions that are expensive to get wrong: due diligence on an acquisition, entry into a new market, a platform a company will rely on for five years, a competitor analysis that goes to the board.