Divide and Conquer: How LLMs Learn to Teach

TL;DR for operators

The useful finding is not “LLMs can write lessons.” They can, in the same way a junior analyst can write a memo: quickly, plausibly, and with enough confidence to become dangerous if nobody reads it.

The paper tests GPT-4o with retrieval-augmented generation (RAG) for creating interactive, scenario-based lessons used to train novice human tutors in online middle-school mathematics.¹ The lesson topics are practical rather than ornamental: encouraging student independence, encouraging help-seeking behaviour, and persuading students to turn cameras on during online tutoring.

The strongest result is operational. When the researchers asked GPT-4o to generate an entire lesson in one prompt, the average quality score was 10.67 out of 17. When they decomposed the task into three linked segments, the score rose to 14.67. But when they decomposed it further into five segments, performance slipped to 13.33. So, yes, divide and conquer. But do not atomise and pray.

For edtech teams, the lesson is simple: use LLMs to accelerate lesson drafting, scenario variation, and multiple-choice question creation. Do not use them as unsupervised curriculum factories. The model produced useful structures and realistic tutoring scenarios, but still struggled with generic feedback, unclear instructional logic, inconsistent terminology, and fabricated or inaccurate references even under RAG. Apparently “retrieval” is not the same thing as “bibliographic honesty.” A tragic discovery for anyone who has met a language model.

The business case is workflow compression, not quality abdication. The review layer is not a nice-to-have. It is the product.

The result worth starting with: three segments beat one prompt, but five segments did not

The paper’s main evidence comes from a comparison of five prompting strategies. The researchers generated the same kind of tutor-training lesson using different levels of task decomposition: one segment, two segments, three segments, four segments, and five segments.

The quality scores were assigned by two trained human coders using a 17-point rubric. Their inter-rater reliability, Cohen’s $\kappa = 0.72$, indicates substantial agreement by the standard interpretation used in the paper. Disagreements were resolved through discussion with a third reviewer.

Here is the core result:

This is the article’s centre of gravity. The researchers did not merely show that decomposition helps. They showed that decomposition has a shape.

One-shot prompting performed worst. That is unsurprising: a full scenario-based tutor-training lesson has multiple moving parts, including learning objectives, two tutoring scenarios, open-ended questions, multiple-choice questions, research-backed instruction, feedback, conclusions, and references. Asking a model to do all of that at once is not “efficient.” It is delegation by dumping.

The three-segment approach performed best. In this setup, the model first generated the opening material and Scenario I. It then used that output to generate the instructional content. Finally, it used the earlier sections to generate Scenario II and the conclusion. That sequence matters because the model is not just filling slots. It is carrying forward a pedagogical thread.

The five-segment approach, despite being the most granular, did not win. That matters because many AI workflow designs quietly assume that finer decomposition is always better. Split every task. Assign every subtask. Chain every output. Add an agent, preferably with a dramatic name. Then wonder why the final product reads like five interns sharing one trench coat.

This paper suggests a more disciplined view: decomposition helps when it reduces cognitive burden while preserving continuity. Push it too far, and the workflow may fragment the lesson’s logic.

What the paper actually built

The system was designed for a specific educational setting: training novice human tutors who teach middle-school mathematics online. The goal was not to create lessons for students directly, but to create lessons that help tutors learn how to behave in live tutoring situations.

The lesson format came from an existing tutor-training platform and followed five main sections:

This structure is important because it makes the task harder than generic educational content generation. A blog-style explanation can be fluent and still be useful. A tutor-training lesson must maintain alignment across objectives, scenarios, questions, feedback, instruction, and references. That is a coordination problem, not merely a writing problem.

The researchers used GPT-4o, specifically gpt-4o-2024-05-13, with RAG. A lesson designer first retrieved articles related to the tutoring practice. Those articles were then used as supporting material in the prompts. The paper’s purpose was not to test whether RAG exists, which would be mercifully unnecessary by now. It was to test whether RAG plus decomposition could generate structured, scenario-based tutor-training lessons of usable quality.

The generated lessons covered three topics:

The human-crafted versions of these three lessons had already gone through review and iteration for publication on the tutor-training platform. The LLM-generated lessons were compared against this context, not against a vague fantasy of educational quality.

The mechanism: decomposition buys attention, but coherence still has to survive

The section-level rubric explains why the three-segment result is interesting.

The model did well at reproducing the basic structure of scenarios. Across segment strategies, “problem structure” scored consistently well for both Scenario I and Scenario II. Plain language was also strong in the scenarios. In other words, GPT-4o was good at imitating the visible form of the lesson: scenario, question, answer, explanation.

The improvements showed up where the task required more instructional control. Feedback improved when the model had more segmented attention. Scenario I feedback rose from 1/3 under one- and two-segment prompting to 3/3 under three-segment prompting. Instructional plain language also improved from 1/3 in one-segment prompting to 3/3 in the three-, four-, and five-segment approaches.

But the deepest signal is “pedagogy grounded.” This code measured whether the instruction bridged theory and practice. It scored 0/3 for one-segment prompting, 1/3 for two segments, 3/3 for three segments, then dropped back to 1/3 for both four and five segments.

That is the decomposition curve in miniature.

A plausible interpretation is that the three-segment design gave the model enough context to maintain continuity while narrowing each generation step enough to improve focus. One segment asked too much at once. Five segments may have asked too little at a time, forcing later sections to stitch together content that had lost a shared centre.

The paper itself is careful here. It does not claim to have proven the exact causal mechanism behind the five-segment drop. The authors note that the lower clarity and weaker pedagogical grounding in the five-segment approach could stem from excessive decomposition disrupting logical flow, or from rating bias or manual rating inconsistencies. That caution is warranted. The sample is small: three lesson topics, one model family, one evaluation context.

Still, for product design, the implication is useful. The unit of decomposition should match the unit of meaning. Splitting by interface component is not always the same as splitting by pedagogical function.

RAG made the content relevant, not citation-safe

The most commercially important failure in the paper is not the lowest score. It is the reference problem.

The generated lessons included conclusion sections with academic references. The system used RAG, meaning the model was prompted with retrieved research articles related to the topic. Yet the “authenticity of references” code performed badly across the board: 0/3 for one, two, three, and five segments, and only 1/3 for four segments.

The appendix gives a concrete example from the “Encouraging Students’ Independence” lesson. The model generated in-text claims about politeness in tutoring and then produced references with hallucination issues: incorrect authors, incorrect year, incorrect title and venue, incomplete title, and incorrect venue. This was not a tiny formatting blemish. It was the familiar LLM problem dressed in a mortarboard.

This matters because RAG is often sold as a grounding mechanism. That is partly fair and partly slippery. RAG can provide relevant source material. It can improve topical specificity. It can reduce the chance that the model free-associates from its training data. But RAG does not automatically enforce citation integrity. A model can retrieve a real paper, read useful content, and still invent a bibliographic record that looks academically plausible.

For operators, the distinction is non-negotiable:

The safe workflow is not “RAG, therefore publish.” It is “RAG, therefore draft from a controlled source set, then validate every cited claim and reference.” Less glamorous, more useful. Product teams may recover from a dull workflow. They do not recover as easily from fabricated references in professional learning content.

The human comparison says “draft accelerator,” not “designer replacement”

The second research question compared LLM-generated lessons with human-crafted lessons through structured written feedback from two lesson designers. This is qualitative evidence, not a controlled learning-outcome test. It is still useful because the designers were evaluating the practical work of lesson creation.

Their positive feedback clustered around three areas.

First, the LLM saved time in scenario drafting. Creating paired scenarios is tedious because the two situations must be similar enough to assess the same skill but different enough to avoid repetition. The designers reported that GPT could quickly generate variations, allowing humans to spend more time refining rather than starting from a blank page.

Second, the model generated diverse and plausible tutoring scenarios. In the independence lesson, for example, one scenario involved a student who waits for direct answers because of low confidence, while another involved a student who repeatedly seeks reassurance before making progress. Those are not identical situations; they represent two different ways student dependence can appear in tutoring.

Third, the designers did not observe biased or offensive content in the generated lessons they reviewed. That should not be inflated into “the system is unbiased.” It means no such content was identified in this small review. A bias audit would require a different design.

The weaknesses are more revealing.

The generated feedback was often generic. Human-crafted lessons tend to explain why each answer option is correct or incorrect. The LLM-generated content tended to explain the correct answer and then wave vaguely at the others. One appendix example shows feedback for a multiple-choice question where the model says the correct answer provides scaffolding, then says the other options “do not align with the research recommendations.” That is not targeted feedback. That is a shrug wearing a lab coat.

The lessons also had clarity problems. Designers noted inconsistent use of “learners,” “teachers,” “tutors,” and “students.” In ordinary prose, that might be mildly annoying. In tutor training, it is structurally damaging. The whole point is to help novice tutors understand how they should act toward students. If the lesson blurs who is doing what, the training object slips.

The instruction sections sometimes became too long, over-cited, or logically disconnected. One cited example introduced the Science Writing Heuristic without explaining what it was or why it mattered for the tutoring strategy. This is a classic LLM failure mode in educational writing: the output sounds research-rich while leaving the learner to assemble the bridge between theory and action. That is not instruction. That is a scavenger hunt.

What each part of the evidence is doing

The paper includes several forms of evidence, and they should not be treated as interchangeable.

This distinction matters because AI papers often get translated into business claims too quickly. “The model scored higher under three-segment prompting” is evidence. “We can now automate tutor training content at scale” is an inference. It may be a reasonable inference if the workflow includes review, measurement, and source verification. Without those, it is just ambition with a subscription plan.

The operating model: generate in modules, review as a lesson

The best practical takeaway is a workflow pattern.

Do not ask an LLM to “create a complete training lesson.” That prompt is cheap, which is exactly the problem. It hides the complexity from the user and forces the model to guess the structure, sequencing, and quality standards in one pass.

A better process has three layers.

First, define the lesson skeleton. The team should specify the target trainee, tutoring context, learning objectives, scenario requirements, question types, feedback standards, and acceptable source set. This is not where the model should be creative. This is where the organisation should be stubborn.

Second, generate in meaningful instructional chunks. The three-segment pattern from the paper is useful because each step carries forward context:

1. Generate the title page and first scenario.
2. Generate the instruction using the first segment as context.
3. Generate the second scenario and conclusion using the earlier sections.

This is not merely chunking for token management. It is sequencing by pedagogical dependency. The second scenario depends on the first and the instruction. The conclusion depends on all of them. Revolutionary concept: the end should know what happened before it.

Third, review against a rubric. The paper’s 17-point rubric is useful not because every company should copy it exactly, but because it turns vague quality control into observable criteria: clarity, alignment, problem structure, feedback quality, plain language, use of examples, pedagogical grounding, and reference authenticity.

A production version should add stricter gates:

The business value appears when this review process is faster than full manual drafting while still catching the model’s predictable errors. That is where LLMs make sense: not as autonomous instructional designers, but as draft engines inside a controlled production line.

What Cognaptus infers for edtech and training teams

The paper directly shows that, in this setting, a three-segment GPT-4o + RAG workflow outperformed one-shot generation and more granular alternatives on human rubric scores. It also shows that expert designers saw practical value in scenario generation and drafting speed, while identifying serious quality gaps.

Cognaptus infers three business lessons.

First, lesson generation should be treated as a workflow design problem, not a model selection problem. GPT-4o was not simply “good” or “bad.” Its output quality depended on how the task was structured. That is awkward for teams hoping that model upgrades will rescue weak process design. It is also good news for teams that can build better workflows before spending more on frontier model access.

Second, the highest ROI may be in scenario variation, not full lesson automation. Scenario drafting is repetitive, cognitively demanding, and highly reusable across training topics. If AI reduces that burden, instructional designers can spend more time on feedback, clarity, sequencing, and evaluation. Those are the parts where humans still have a clear comparative advantage.

Third, citation verification should be separated from text generation. If references matter, they should be managed by a source-control layer, not trusted to the generator. The model can draft claims based on retrieved articles. A separate process should verify that the claim maps to a real source and that the final reference is exact. This can be partly automated, but it cannot be wished into existence by saying “RAG” loudly enough in a roadmap meeting.

The boundaries: useful evidence, small sandbox

The study is valuable because it is concrete. It is also bounded.

It used three lesson topics. That is enough to observe a pattern, not enough to establish a universal law of decomposition. A three-segment strategy worked best here, but other educational formats may require different segmentation. A compliance training module, a clinical simulation, a coding exercise, and a sales enablement lesson do not share the same internal dependencies.

It used GPT-4o. The results may not transfer cleanly to smaller models, newer models, or specialised education models. The workflow lesson probably generalises more than the exact score table, but that remains an inference.

It evaluated lesson quality through human ratings and designer feedback. It did not test whether tutors trained on the LLM-generated lessons performed better with students. That is a decisive boundary. A lesson can be well-structured and still fail to improve behaviour. Education has a long and distinguished history of materials that look excellent until learners encounter them.

It did not measure actual production time saved in a controlled way. Designers reported time-saving potential, especially in scenario and MCQ creation, but the paper does not provide a cost model or time-motion analysis.

It did not conduct a full bias audit. Designers reported that they did not observe biased or offensive content, which is reassuring but narrow. Organisations deploying such systems still need scenario diversity testing and review standards.

Finally, the reference failures are not a side note. They directly limit publication readiness. A lesson with invented references can still be a useful internal draft. It should not become a published training resource without verification. This is not pedantry. It is the difference between assisted design and institutional embarrassment.

The strategic lesson: the model learns to teach only when the workflow teaches the model

The paper’s title is about automatic LLM creation of interactive learning lessons. The more interesting story is semi-automatic creation. Human designers retrieve sources, define lesson formats, evaluate outputs, compare weaknesses, and polish the final material. The LLM accelerates the middle, where blank-page labour and scenario variation are expensive.

That is the realistic future for much of educational AI: not a single machine writing perfect curriculum, but a production system where humans define the learning architecture and models generate candidate content inside it.

The three-segment result gives that future a useful design principle. Good AI workflows are not just decomposed. They are decomposed at the right level of meaning.

Too little decomposition overwhelms the model. Too much decomposition fragments the pedagogy. RAG improves relevance but does not guarantee truth. Fluent scenarios are useful but not sufficient. Feedback must diagnose, not decorate. References must be verified, not hallucinated in APA cosplay.

So, yes: divide and conquer.

Just remember what is being conquered. It is not teaching. It is the drafting bottleneck.

Cognaptus: Automate the Present, Incubate the Future.