Share the Trunk, Spare the Averaging: Federated Actor-Critic Gets Personal

A fleet looks unified on a dashboard. It is rarely unified in the world.

The warehouse robots share a navigation objective, but one floor has glossy tiles, another has uneven concrete, and a third has humans who treat marked lanes as casual decoration. The delivery drones may use the same controller family, but wind, payload, battery ageing, and local regulation quietly rewrite the operating problem. Industrial arms may repeat the same task, until a supplier swaps a component and the “same” movement is no longer quite the same.

This is where a common instinct in federated learning becomes dangerous: average everything and call it collaboration. In supervised learning, that instinct is already imperfect. In reinforcement learning, where the policy changes the data it sees, it can become a very expensive way of manufacturing compromise.

The paper Collaborative Yet Personalized Policy Training: Single-Timescale Federated Actor-Critic proposes a cleaner split: share the representation, personalise the policy components.¹ Its algorithm, pFedAC, lets multiple agents collaborate on a common latent subspace while keeping local actor-critic heads adapted to each agent’s own Markov decision process. In plain English: learn the common trunk together; do not force every branch to bend in the same direction.

That distinction is the useful business lesson. The paper is mathematically dense, because apparently one must suffer for subspaces. But the central design idea is simple enough to survive translation: federated reinforcement learning should not always mean one global policy. It can mean shared structure with local control.

The mechanism: share what transfers, personalise what collides

The misconception worth removing first is that federated reinforcement learning naturally means training one shared policy across all clients. That is convenient for diagrams. It is less convenient for heterogeneous control problems.

In a standard federated setup, each client collects experience locally. A server periodically aggregates model updates. The obvious FedAvg-style approach averages parameters across clients, broadcasting the result back. This can work when clients are variations on the same problem. It becomes brittle when their environments differ in ways that make a single policy internally conflicted.

pFedAC attacks the problem by decomposing the value approximation into two pieces:

$$ z^{k,}_{\theta} = B^ \omega^{k,*}_{\theta} $$

Here, $B^$ is the shared subspace representation and $\omega^{k,}_{\theta}$ is the local head for agent $k$. The assumption is not that every agent has the same value function or the same optimal policy. It is that their value functions can be represented through a common low-dimensional subspace, with agent-specific coefficients.

That is the whole architecture in miniature:

This is not a sentimental compromise between centralisation and local autonomy. It is a specific answer to negative transfer. If two agents interpret the same action differently, or face different transition dynamics, averaging their full policies can erase useful local adaptation. Sharing only the representation gives the system a common language without forcing everyone to speak the same sentence.

pFedAC updates three moving targets at once

The paper’s formal algorithm operates in a federated actor-critic setting. Each agent is an MDP with the same state and action spaces but potentially different transition kernels and rewards. The objective is personalised: each agent wants to maximise its own cumulative reward, not an averaged objective in a fictional blended environment. One cannot pay invoices with rewards from an imaginary average robot. Tedious, but true.

Each round of pFedAC does three things.

First, every agent updates its local critic head using temporal-difference information. The critic estimates how valuable states are under the current local policy, but it does so through the shared subspace and the local head.

Second, each agent forms a local update to the shared subspace. These local subspace updates are sent to the server, averaged, orthonormalised through QR decomposition, and then broadcast back.

Third, each agent updates its actor, the personalised policy parameters, using critic feedback. The policy is not averaged across agents. This is the point, not a missing feature.

The authors also use auxiliary sampling for policy updates to handle the mismatch between the limiting state distribution used by the critic and the discounted visitation distribution used by policy gradients. In practical terms, the analysis leans on simulator-style access for the actor-side sampling. That matters later when we talk about deployment boundaries.

The important phrase in the title is “single-timescale”. In much actor-critic theory, the critic and actor are analysed on separate clocks: one learns quickly, the other slowly, so the proof can pretend the fast piece is almost settled before the slow piece moves. pFedAC instead updates the local heads, shared subspace, and actors on the same order of step size. This is more realistic for many implementations, and substantially less polite to the proof.

The theorem says collaboration buys speed, but only inside a carefully fenced garden

The main theoretical result is a finite-time convergence guarantee under Markovian sampling and heterogeneous transition kernels. The paper bounds two central quantities: the critic approximation error and a policy-gradient stationarity measure. Under its assumptions, it obtains rates that, after choosing the rollout length $L = K^2$ and assuming higher-order terms are dominated, scale as:

$$ \bar{X}_T \leq \tilde{O}\left(\frac{1}{(1-\gamma)^4\sqrt{TK}}\right) $$

and

$$ \bar{G}_T \leq \tilde{O}\left(\frac{1}{(1-\gamma)^6\sqrt{TK}}\right) $$

where $T$ is the number of rounds, $K$ is the number of agents, and $\gamma$ is the discount factor.

The business translation is not “add more agents and magic happens”. It is more specific: under shared-representation structure, collaborative sampling can produce a linear speedup in the number of agents. The dominant term behaves like $1/\sqrt{TK}$, which is the familiar signal of parallelism helping nonconvex stochastic optimisation.

The catch is that the theorem earns this result under serious conditions. The analysis assumes linear value approximation, bounded features, sufficient exploration, uniform ergodicity and contraction of the Markov chains, smooth policy classes, small stepsizes, and a well-covered shared subspace. The agents’ local truths must span the shared subspace well enough. If all clients are nearly identical, there may not be enough variation to identify the common structure. If they are wildly unrelated, the shared subspace assumption may be fiction with notation.

This is not a criticism. It is the price of a theorem in reinforcement learning. But it means the result should be read as an architectural guarantee under disciplined conditions, not as a blanket production certificate for every swarm, factory, or fleet.

The proof difficulty is the product, not a decorative appendix

The paper’s proof architecture is revealing because it shows why this problem is harder than supervised personalised federated learning.

In supervised PFL, clients have data. The data may be non-IID, but it does not usually change because the model blinked. In reinforcement learning, the policy influences the trajectory, which influences the critic, which influences the next policy update. Add federated sharing on top, and the server is aggregating signals from clients whose data-generating processes are evolving locally.

pFedAC therefore has to control three coupled error dynamics:

The paper introduces a perturbation analysis for the projected subspace updates and QR decomposition steps. It also uses conditional mixing arguments for heterogeneous Markovian noise, plus auxiliary “temporally frozen” chains to compare what the algorithm samples with what the analysis needs.

This is where a simple summary of the paper would be actively unhelpful. The mechanism matters because the proof is not merely proving FedAvg with longer equations. It is proving that representation sharing can remain stable when local critics, shared geometry, and evolving policies are all moving at once. That is exactly the kind of failure mode practitioners meet when a federated control system appears to train, then quietly teaches every client the wrong lesson.

The PPO experiment is evidence for the design pattern, not the theorem

The empirical section instantiates the shared-trunk, personalised-head idea inside PPO and calls the resulting method FedPer PPO. This is not the same object as the linear pFedAC theorem. It is a deep-network implementation inspired by the same architecture. That distinction matters. The theory supports the mechanism under linear approximation; the experiment tests whether the mechanism still looks useful in a more recognisable deep RL setting.

The benchmark is federated Hopper-v5 with action-map heterogeneity. Each client uses the same underlying MuJoCo Hopper environment, but actions are permuted and scaled before being passed to the simulator. From the policy’s perspective, the same emitted action can create different physical torque effects across clients. The paper treats this as a way to induce heterogeneous transition behaviour while keeping the benchmark controlled.

The comparison is between:

• Single PPO: each client trains independently;
• FedAvg PPO: all parameters are averaged across clients;
• FedPer PPO: only the shared trunk is averaged; actor heads, critic heads, and log-standard-deviation parameters remain personalised.

The main empirical result is that FedPer PPO performs best when heterogeneity is fully distinct. In the grouped setup, where clients share two action maps, FedPer beats Single PPO by 2.04× and FedAvg PPO by 1.40× at the fair reporting budget. In the 6-UNIQUE setup, where every client has a distinct action map, FedPer beats Single PPO by 2.63× and FedAvg PPO by 3.78×.

That pattern is the interesting part. FedAvg does less badly when there are only two shared action maps because an averaged model can still partly specialise. When every client has a different action interface, full averaging has to reconcile conflicting policies. FedPer, by contrast, is pushed towards a trunk that is invariant to the action interface, leaving local heads to specialise. The experiment is therefore not merely “our method got higher reward”. It tests the central mechanism: partial sharing becomes more useful precisely when full sharing becomes less reliable.

The appendix checks a possible confound, not a second thesis

The paper’s experimental appendix is short but useful. It clarifies the environment construction, algorithm settings, and fairness controls.

The downstream transfer test is especially easy to overread. The authors load a trunk from a FedPer 6-UNIQUE checkpoint, randomly re-initialise actor and critic output heads, and compare three settings: training from scratch, fine-tuning the full warm-started network, and freezing the trunk while training only the new heads. At 0.4 million environment steps, the frozen-trunk setting reaches 1328 ± 74 on vanilla Hopper versus 245 ± 45 from scratch, a 5.42× ratio. On an out-of-distribution variant with an action scale below the pretraining range, frozen trunk reaches 701 ± 101 versus 200 ± 23 from scratch, a 3.50× ratio.

That is good evidence that the trunk learned something reusable across these controlled action interfaces. It is not evidence that we now possess a universal Hopper soul. Let us remain adults.

Curiously, the frozen trunk outperforms full fine-tuning in both transfer settings. The authors suggest that on-policy fine-tuning may disrupt the pretrained representation before the randomly initialised heads recover. For practitioners, the lesson is familiar: fine-tuning everything is not always sophistication. Sometimes it is just a larger blast radius.

The business interpretation is architectural, not magical

The paper’s business relevance sits in systems where multiple agents have related but non-identical control problems. Robotics is the obvious case, but the pattern also applies to embodied AI, industrial automation, fleet optimisation, smart infrastructure, and any environment where local dynamics matter.

The operational design principle is:

Federate the representation layer that captures reusable structure; personalise the policy layer that touches local dynamics.

This maps cleanly onto several deployment patterns.

For a robotics fleet, the shared trunk might encode reusable perception and control abstractions, while local heads adapt to hardware wear, floor material, payload, or site-specific constraints. For industrial process control, the trunk could capture common production dynamics, while local heads adapt to individual machines or plants. For logistics optimisation, shared representations may encode route, demand, and constraint patterns, while local policies adapt to city-specific traffic, labour rules, or depot layouts.

The result also suggests a governance point. A central team should not evaluate federated RL only by whether global training converges. It should ask which parameters are shared, which remain local, and whether the shared layer is actually learning reusable structure rather than averaging away the client differences that matter most.

A practical evaluation checklist would look like this:

The likely ROI is not simply cheaper training. It is cheaper adaptation. If a fleet can reuse a shared trunk and retrain only small local heads for new sites or variants, deployment cycles shorten. If local policies remain personalised, performance is less likely to degrade when one site’s dynamics diverge. If full averaging is avoided, the system may also reduce the hidden cost of negative transfer: the kind that looks like collaboration in the meeting and regression in the field.

The boundaries are where the procurement slide should stop

The paper is not a production validation study. It is a theoretical contribution with a compact empirical demonstration.

The theory assumes linear value approximation. The empirical PPO implementation uses a neural shared trunk, which is more relevant to practice but outside the exact theorem. The theorem also depends on technical mixing, exploration, smoothness, boundedness, and subspace coverage assumptions. These are not decorative. They define the kind of world in which the convergence guarantee lives.

The discount-factor dependence is another boundary. The authors explicitly note that the dependence on $(1-\gamma)^{-1}$ may not be optimal. Since long-horizon control often operates near the undiscounted regime where $\gamma$ approaches one, this matters. A rate with strong dependence on $(1-\gamma)$ can look elegant on paper and then become very large in the environments businesses actually care about. Mathematics has a talent for putting the knife in the exponent.

The empirical evidence is also deliberately narrow. The Hopper-v5 experiments use six clients, controlled action-map heterogeneity, three seeds, and CPU-based runs. The appendix improves confidence that the effect is not merely a longer-episode confound, but it does not convert the result into cross-domain validation. There are no real robots, no multi-site industrial deployments, no safety-critical constraints, and no communication-cost analysis deep enough for an enterprise architecture review.

So the right inference is bounded:

• The paper directly shows a finite-time convergence result for pFedAC under stated assumptions.
• It directly shows that a FedPer PPO instantiation beats Single PPO and FedAvg PPO on the paper’s controlled federated Hopper-v5 benchmark.
• It provides evidence that the learned trunk transfers to related downstream Hopper variants when the heads are reinitialised.
• Cognaptus infers that shared-representation/personalised-head designs are a promising architecture for heterogeneous control systems.
• It remains uncertain how far the mechanism generalises across real-world robotics, richer simulators, safety constraints, communication budgets, nonstationary hardware, and large-scale multi-client deployments.

That is already enough. Not every useful paper needs to arrive wearing a hard hat and carrying a purchase order.

The useful lesson: averaging is not collaboration

pFedAC’s contribution is not that federated reinforcement learning should become more complicated for sport. It is that heterogeneity forces a choice. Either the system pretends local differences are noise and averages them away, or it learns a structure that lets commonality and local adaptation coexist.

The paper’s mechanism-first lesson is durable: in federated control, share the parts that encode transferable structure and keep local the parts that must answer to local reality. The theorem gives that idea a convergence story under strict assumptions. The PPO experiment gives it a controlled behavioural demonstration. The business implication is a design principle, not a deployment guarantee.

Averaging one global policy is administratively neat. So is putting every vehicle in a fleet on the same tyre pressure, regardless of load and terrain. Neatness is not intelligence.

The better architecture is collaborative where collaboration helps, personalised where reality refuses to standardise. pFedAC is a serious step in that direction.

Cognaptus: Automate the Present, Incubate the Future.