Better Ideas, Narrower Search: Designing LLM Brainstorming as a Portfolio

TL;DR for operators

LLM brainstorming should not be evaluated only by how good its individual ideas look. In a product-ideation experiment, GPT-4 generated ideas with higher average purchase intent than human participants and was seven times as likely to place an idea in the top decile. Yet its idea pools were also much more concentrated around similar regions of the solution space.

For an innovation team, that combination is attractive but incomplete. A single strong model can be an efficient source of promising candidates while still leaving parts of the opportunity landscape underexplored. The paper shows that this narrowing is not fixed: mixing models from different vendors, varying task-relevant starting points, preserving human problem selection, and simply generating many more ideas can all expand search in different ways.

The managerial choice is therefore not “human or AI brainstorming.” It is how to construct an idea portfolio that keeps AI’s high candidate yield without mistaking repeated strength in a narrow region for comprehensive exploration.

The quality advantage is real enough to preserve

A product team typically wants two things from early ideation: many plausible candidates and a reasonable chance that a few are unusually strong. Terwiesch and colleagues test both in AI and Its Impact on Creativity and Diversity: An Empirical Study of LLM-Generated Product Ideas.¹

Their focal comparison uses 200 human-generated product ideas, 100 GPT-4 zero-shot ideas, and 100 GPT-4 few-shot ideas for physical products aimed at college students and priced at $50 or less. College-age U.S. respondents evaluated the ideas without seeing their source.

Average purchase intent was 40.4% for the human ideas, 46.4% for GPT-4 zero-shot ideas, and 49.3% for GPT-4 few-shot ideas. The upper tail was more striking. Among the 40 ideas in the top decile, five came from humans, 15 from zero-shot GPT-4, and 20 from few-shot GPT-4. Because the human and combined AI pools each contained 200 ideas, an AI-generated idea was seven times as likely to appear in that top group.

The paper also tests an obvious confound. Perhaps the AI concepts merely sounded better because GPT-4 presented them more persuasively. In a separate experiment designed specifically to test that explanation, the authors had GPT-4 rewrite the human ideas in an AI-like style. Purchase intent rose by only 0.009 and the difference was not statistically significant.

That makes the practical result harder to dismiss as presentation polish. Within this task, AI generation produced a genuinely stronger candidate distribution as judged by the paper’s purchase-intent measure.

Better candidates did not mean broader exploration

The same experiment produces a different answer when the unit of analysis changes from an individual idea to the whole portfolio.

The authors represent idea descriptions as embeddings and measure how close the ideas are to one another in meaning. Lower average pairwise cosine similarity means the pool covers more distinct semantic territory. Human ideas had average similarity of 0.221, compared with 0.414 for GPT-4 zero-shot and 0.428 for GPT-4 few-shot.

So GPT-4 was producing stronger-rated ideas from a narrower region.

This is not just an artifact of one embedding calculation. The paper rephrases the focal ideas as a style robustness check, uses alternative diversity measures, and then reanalyzes data from earlier creativity studies. Across four studies and five datasets, all 32 comparisons in the alternative diversity analyses pointed toward lower AI diversity. For pairwise similarity, reported effect sizes ranged from Cohen’s $d=0.380$ to $d=5.538$.

That broader analysis is best read as robustness and generalization evidence for the diversity finding, not as a second claim that AI ideas are uniformly inferior. The quality result and the diversity result coexist.

The search interpretation is straightforward: a model can be very productive inside regions it explores well while repeatedly returning to similar associations. For innovation teams, high average quality is closer to effective exploitation; portfolio diversity determines how much exploration is occurring around it.

Diversity can be designed into the generation system

The paper’s later experiments are especially useful because they treat homogenization as something to modify rather than merely diagnose.

Model choice alone matters. Using diversity defined as one minus average pairwise cosine similarity, the human benchmark was 0.773. GPT-4 Turbo reached 0.756, Gemini Pro 2.0 and DeepSeek R1 each reached 0.752, while other tested models were materially lower.

More revealingly, a balanced composite pool assembled from GPT-4 Turbo, Gemini Pro 2.0, DeepSeek R1, and Claude Sonnet 3.5 reached 0.763 and was not statistically distinguishable from the human benchmark. No individual tested model reached the human score.

The implication for a firm running serious innovation search is not that four vendors are always required. It is that model standardization has a search cost. If different models concentrate probability mass in somewhat different regions, pooling them can diversify the candidate portfolio without changing model weights.

Prompt design creates a second lever. Against GPT-4o zero-shot diversity of 0.679, random personas reached 0.757, human-picked problems reached 0.739, and random functional constraints reached 0.735.

These interventions have something in common: they alter where the model begins searching in ways that remain connected to the task. Random Shakespeare sonnets produced a smaller increase, to 0.686. More randomness by itself was not equivalent to more useful exploration.

Some seemingly creative interventions actually moved in the wrong direction. Iteratively asking the model to make ideas “bolder” reduced diversity to 0.575. Generating one idea per independent session produced diversity of only 0.452.

For operators, prompt variation should therefore be treated as search-space design, not as decorative prompt engineering.

Humans may be more valuable upstream of solution generation

One of the paper’s more practical workflow tests gives humans a specific role: identify the unmet problem, then let the model generate alternative solutions.

That human-picked-problem agent reached diversity of 0.739, the second-highest AI strategy in the corresponding experiment.

The evidence does not show that humans are universally better at problem discovery, nor does it test a live organizational team iterating with AI over time. But it supports a useful design hypothesis: human input may create more value by directing the search toward heterogeneous problems than by competing with AI to produce every solution.

Cognaptus inference: innovation workflows should consider separating problem-space authority from solution-generation throughput. Humans can nominate neglected needs, customer contexts, or constraints; models can then search each assigned region at high volume.

That preserves a role for human judgment without making human-versus-AI idea quality the organizing question.

Scale improves coverage even when equal-sized pools remain narrow

Diversity measures describe how spread out a fixed pool is. A company with inexpensive generation has another option: make the pool much larger.

The paper therefore measures coverage by asking how close each of 200 human ideas is to its nearest GPT-4o-generated counterpart. As the AI pool grows from 100 to 1,000 ideas, mean nearest-neighbor distance falls from 0.546 to 0.478; the maximum distance falls from 0.762 to 0.711. The authors report no plateau in mean coverage through 1,000 ideas.

This does not mean the AI pool becomes intrinsically as diverse as human ideation. Some human regions remain difficult to reach. It shows something operationally different: brute-force generation can compensate for part of the exploration deficit because the marginal cost of producing another candidate is low.

That makes downstream screening a necessary part of the architecture. Generating 10 times more ideas only helps if the organization can rank, cluster, eliminate duplicates, and surface genuinely distinct candidates without moving the bottleneck from generation to evaluation.

What the evidence does not establish

The main comparison is a college-product ideation task, not a randomized field experiment of firms adopting AI. The human ideas were produced in 2021, while the initial GPT-4 ideas were generated later. The paper also studies text descriptions rather than sketches, prototypes, or implementation work.

It does not test whether AI can reliably select and develop the best ideas after generation, and perceived novelty is not a direct audit of whether an idea appeared in model training data. Nor does higher semantic diversity automatically imply higher commercial value: the paper itself observes that pushing temperature too high can produce unstable or nonsensical output.

The safer business conclusion is narrower. Within the tested ideation settings, AI can improve the yield of highly rated candidates while concentrating search. That concentration can be reduced through deliberate architecture and workflow choices.

Build an idea portfolio, not a single-model suggestion box

The paper changes how LLM brainstorming should be evaluated.

If the objective is merely to obtain several credible concepts quickly, a strong model may already perform extremely well. If the objective is innovation search across a wide opportunity space, candidate quality is only one portfolio metric.

An operating design that follows the evidence would use AI heavily but avoid giving one model a monopoly over exploration: mix generators where diversity matters, vary meaningful starting conditions, let humans identify neglected problems, generate at a scale justified by cheap inference, and evaluate quality and breadth separately.

The result is less a contest between human and machine creativity than an allocation problem: which parts of search should be diversified, which should be scaled, and where should human judgment still determine the direction of exploration?

Cognaptus: Automate the Present, Incubate the Future.