Split Only What Hurts: A Constraint-First Guide to LLM Parallelism

TL;DR for operators

When a model must fit across a fixed accelerator cluster without sacrificing throughput, splitting the workload more aggressively is not automatically better. Amer et al.¹ show that on their eight-NPU setup, pure data parallelism performs best for both tested 1B models. At 7B, the best configurations add only limited model parallelism: pipeline parallelism for LLaMA and tensor parallelism for Mamba.

The operating rule is constraint-first: keep as much data parallelism as memory permits, then add only enough model or activation partitioning to relieve the constraint preventing the workload from fitting or running efficiently. Sharding can reduce memory pressure, but it can also shrink efficient matrix operations and add communication, synchronization, or pipeline overhead, lowering the share of accelerator compute actually used.

These rankings are not universal recipes. They come from one host with eight Ascend 910B NPUs, 4096-token sequences, and LLaMA and Mamba models at roughly 1B and 7B parameters.

Dividing the workload can make eight accelerators behave like fewer

Suppose a team has eight accelerators and a model that already fits comfortably on one of them. The cluster still needs to be divided somehow, but every additional split changes more than memory placement. It can change matrix dimensions, introduce collective communication, create synchronization points, or leave stages waiting for one another.

The paper measures this effect through MFU, the share of nominal accelerator compute actually realized by the model:

Its main case-study results make the cost visible.

Table 15 is the paper’s main empirical comparison, not an ablation or sensitivity exercise. For both 1B models, where the workload fits without model partitioning, pure data parallelism wins. LLaMA 1B falls from 43.3% MFU in the best configuration to 6.2% in the heavily partitioned worst case. Mamba 1B falls from 20.4% to 3.1%.

The devices have not disappeared. Their useful execution has.

Each parallelism dimension buys a different kind of capacity

The paper’s more durable contribution is the framework behind those rankings. It treats parallelism as allocation of a fixed accelerator budget across dimensions rather than as a ladder where a higher degree means greater scalability.

Data parallelism distributes examples while keeping a model replica on each device. It is attractive when the model fits because it preserves comparatively large local computations, although gradients still require synchronization.

Tensor parallelism splits individual layer weights and computations. Its strongest direct benefit is reducing per-device weight memory, but the split creates frequent communication and can shrink matrix multiplications into less efficient local kernels.

Pipeline parallelism assigns different groups of layers to different devices. That reduces the model footprint per device but introduces dependencies between stages, with bubbles or stage imbalance potentially leaving hardware idle.

Context parallelism divides sequence-dependent work across devices. This can reduce activation pressure and becomes especially relevant to long contexts, but weights remain replicated unless another technique shards them, and sequence/state communication must reconnect the computation.

The paper’s analytical work is useful because it shows that these choices redistribute costs rather than reducing total computational work. Under its fixed-device accounting, the overall FLOP requirement is unchanged by the TP/CP/DP split. What changes materially is where memory sits, how large local operations remain, and how much data devices must exchange.

Communication formulas do not determine runtime by themselves

Some of the paper’s theoretical comparisons show why architecture matters.

For grouped-query attention using the analyzed pass-KV context-parallel scheme, communication scales as

so when the number of KV heads $k$ is below the number of attention heads $a$, the modeled communication volume for context parallelism is lower than for tensor parallelism.

For Mamba-2 prefill, the paper similarly derives

When $CP\ll\mathrm{seqlen}$, the latter has the better asymptotic communication scaling.

These are analytical comparisons, not evidence that context parallelism should always run faster. In the actual 4096-token benchmarks, aggressive CP performs poorly. The operation-level profiling supplies the mechanism: partitioning can reduce the share of runtime spent in high-throughput matrix operations while communication, synchronization, and fixed vector work occupy more of the step.

This distinction matters for system design. A favorable communication formula identifies one cost advantage. It does not settle end-to-end performance when kernel efficiency and synchronization remain different.

Architecture changes which compromise is least expensive

At 7B, the model can no longer be treated as merely a larger version of the 1B allocation problem.

For LLaMA 7B, the best tested setup is DP=4 and PP=2, reaching 63.7% MFU, 101.8 seconds per step, and 41.2 thousand tokens per second. Limited pipeline splitting relieves the relevant memory pressure without the stronger penalties observed under heavy tensor/context partitioning.

Mamba 7B cannot fit under pure data parallelism in the tested environment. Its best configuration instead uses DP=4 and TP=2, producing 20.2% MFU. The preferred model-parallel dimension therefore differs even though both models use the same eight devices.

The paper links that difference to execution structure. LLaMA spends a larger share of runtime in high-throughput Cube/GEMM operations on the evaluated hardware, while Mamba contains a larger proportion of vector-oriented work. Partitioning changes those operation mixes differently.

The replacement for an architecture-agnostic recipe is straightforward: determine what prevents the workload from fitting or meeting its target, then choose the split whose execution costs are least damaging for that architecture.

For infrastructure teams, parallelism is a capacity-allocation decision

Cognaptus inference: an infrastructure team choosing a distributed configuration should treat unused MFU as an economic signal, not merely a benchmark statistic. If two configurations occupy the same accelerator count but one spends substantially more time communicating, synchronizing, or executing inefficiently small kernels, its accelerator cost per useful token can be higher even when its memory footprint looks attractive.

A practical search order follows from the paper:

1. Determine whether the model and workload fit with maximal feasible data parallelism.
2. Identify the binding constraint—weights, activations/context, model depth, communication, or a service-level requirement.
3. Introduce the parallelism dimension that directly relieves that constraint.
4. Profile realized kernel mix, communication, memory, throughput, and latency rather than stopping at theoretical capacity.
5. Increase the partitioning degree only while the additional constraint relief exceeds its execution cost.

This is also why automatic parallelization remains commercially relevant. Searching hybrid configurations manually becomes expensive as architecture, hardware topology, sequence length, training stage, and service objective vary. The paper reviews cost-model and learning-based approaches, but does not introduce a new auto-parallelization algorithm, and notes that cost-model accuracy and transfer across hardware remain open problems.

The configuration rankings stop at the tested regime

The empirical result is strong enough to reject one assumption within the tested environment: adding more TP, PP, or CP does not automatically improve accelerator efficiency.

It is not strong enough to prescribe (4,2,1,1) for LLaMA or (4,1,2,1) for Mamba generally. The study uses one host with eight Ascend 910B NPUs, 60 GB memory per device, sequence length 4096, micro-batch size 1, and roughly four million tokens per training step. It tests only LLaMA and Mamba at approximately 1B and 7B parameters.

Ultra-long-context workloads may change the value of context parallelism; the paper’s own analysis gives reasons to expect that, but the benchmark does not test it. Multi-node clusters, very large models, Mixture-of-Experts architectures, expert parallelism, detailed memory optimizations, and energy consumption are also outside the empirical comparison.

The durable result is therefore not a winning tuple of parallelism degrees. It is a selection discipline: partition because a specific constraint requires it, and stop when additional partitioning costs more execution efficiency than the constraint relief is worth.

Cognaptus: Automate the Present, Incubate the Future.