Verified Maneuvers, Unmeasured Reliability: Lessons from CARLA Imitation Learning

TL;DR for operators

An engineering team can train a driving controller to reproduce recorded steering, throttle, and braking with very little prediction error. That does not establish how reliably the controller will drive when its own decisions determine the next situation it encounters.

Jordy Kieto’s Imitation Learning for Autonomous Driving in CARLA¹ investigates this gap through a compact driving policy trained on systematically collected demonstrations. Its data-generation procedure verifies completed maneuvers, yet 89.47% of the resulting training windows represent straight driving. The released model achieves strong offline prediction accuracy, while the author also reports extended autonomous driving, operation in an unfamiliar town, and recovery from substantial deviations.

The collection procedure and offline measurements are documented quantitatively. The driving outcomes remain qualitative observations, without repeated-route reliability statistics.

For autonomous-driving and robotics developers, the immediate value is a more auditable demonstration pipeline and a clearer separation between accurate imitation, observed autonomous behavior, and demonstrated operational reliability.

Verified maneuvers do not guarantee balanced training examples

A recurring difficulty in collecting driving demonstrations is controlling what the expert actually does. An autopilot may be instructed to approach an intersection, but its eventual maneuver can depend on route geometry and navigation decisions. Recording a requested turn without checking the executed action can produce misleading labels.

Kieto addresses this problem through systematic demonstration generation in CARLA, a simulated urban-driving environment.

The procedure enumerates vehicle spawn points, examines the lane graph for feasible intersection maneuvers, introduces small randomized perturbations to initial vehicle poses, and records autopilot driving. A capture is accepted only after verifying the maneuver actually performed, subject to maneuver-specific recording quotas.

This makes the collection process more reproducible and its maneuver labels more defensible. Its contribution concerns control over the construction of expert demonstrations, not a novel neural-network architecture.

The released dataset comprises 448 recordings from CARLA Town01 and Town02, producing 236,882 overlapping five-frame windows—approximately 3.3 hours of simulated driving at 20 Hz.

However, the effective training distribution is considerably less balanced than the capture procedure might suggest.

Source: Section 5.3, Table 2. Percentages are reported at the training-window level.

A recording classified as a turning maneuver can still contain substantial straight-driving time before and after the intersection. Consequently, balancing maneuver captures does not balance the individual temporal windows consumed by the model. A direction-consistency filter also removes some windows around maneuver transitions.

Only 2.0% of training windows contain positive braking targets.

For teams developing imitation-learning datasets, this creates two distinct quality-control tasks. The first is confirming that recorded demonstrations have the intended labels. The second is measuring the actual distribution of model-training examples, including maneuver transitions and relatively infrequent controls such as braking.

The study establishes a reproducible approach to the first task and supplies useful measurements for the second. It does not experimentally establish that its coverage strategy causes better closed-loop driving than alternative collection strategies.

Low prediction error does not measure autonomous-driving reliability

The released policy contains approximately 1.36 million parameters and combines five-frame histories of four observation modalities: RGB camera images, LiDAR bird’s-eye-view occupancy histograms, vehicle telemetry, and lane waypoints.

Separate encoders process these inputs before a two-layer temporal Transformer integrates their representations. The model predicts continuous steering, throttle, and brake commands at 20 Hz.

Training uses behavioral cloning: the model learns to reproduce the actions of the demonstration expert rather than optimizing directly against the consequences of its own driving.

The released checkpoint, saved after epoch 10, achieved the following offline validation results:

Source: Section 6.3, Table 3. Results use withheld temporal observations from the recorded demonstrations.

These measurements demonstrate accurate action imitation on observations drawn from the expert-recording distribution. They do not measure the consequences of repeatedly executing the predicted controls.

The validation design reinforces this distinction. Training and validation contain different temporal portions of the same recordings, rather than independently collected routes or driving conditions. It therefore evaluates prediction on familiar trajectory distributions more directly than generalization to unfamiliar roads.

During autonomous driving, even a small steering mistake changes the vehicle’s subsequent position. The next camera image, LiDAR observation, and vehicle state can then differ from the situations represented in the demonstrations. Further mistakes may accumulate.

This feedback-driven mismatch is known as covariate shift. It explains why low offline prediction error cannot substitute for closed-loop evaluation, in which the learned controller actually operates the simulated vehicle.

The training history also illustrates why aggregate loss requires careful interpretation. Across epochs 50–53, average validation loss fell to 0.00070, while turn-detection F1 increased only from 0.78 to 0.82. Left-turn validation mean squared error remained approximately 2.6 times the straight-driving error.

Those later results belong to the training history, not the released epoch-10 checkpoint. They suggest that improving aggregate action regression does not necessarily produce comparable improvements across individual driving maneuvers.

Neither metric establishes how often the vehicle completes an autonomous route successfully.

Hours of driving are observations, not a measured reliability rate

The report goes beyond offline evaluation by describing autonomous deployment of the released policy.

The author reports hours of intervention-free driving without collisions on training and held-out CARLA routes. The policy also reportedly operated on curved, multi-lane geometry in an unseen CARLA town and sometimes returned to drivable road surface after substantial trajectory deviations.

These observations are meaningful because the controller operated autoregressively: its own predictions influenced subsequent observations, without expert intervention, an independent safety controller, or additional control filtering.

But the closed-loop assessment was conducted without other traffic, and the report does not provide standardized route-completion rates, collision frequencies, evaluation distances, or repeated multi-seed rollout statistics.

The unseen-town observation is consequently an example of possible transfer, not an estimate of cross-town robustness.

The recovery observations require particularly careful interpretation.

The released checkpoint did not train on dedicated recovery recordings collected on May 17, 2026, or on online DAgger data. This rules out those later recordings as an explanation for its observed recovery behavior.

It does not demonstrate that recovery emerged independently of all recovery-relevant training exposure. Small initial-pose perturbations were already part of the demonstration-generation procedure, and their contribution was not isolated through an ablation.

Furthermore, the controller receives privileged lane-waypoint information derived from the simulator’s HD-map lane graph and ground-truth vehicle pose. Debug waypoints rendered into camera observations introduce additional route information.

These inputs can help explain successful lane following and unfamiliar-road operation without establishing that the model independently learned the necessary geometry from ordinary sensor observations.

The waypoint construction also follows the first lane-graph successor at junctions, which may differ from the expert’s intended navigation maneuver.

Thus, the reported recovery and transfer remain promising behavioral observations whose frequency, causal explanation, and dependence on privileged information have not been established.

Checkpoint provenance determines which claims survive inspection

One of the report’s stronger contributions is its reconstruction of the released model’s training history.

Appendix A connects the Keras checkpoint to the May 10 training run, identifies its epoch-10 origin through timestamps, reconstructs the associated demonstration recordings, and confirms that later dedicated recovery recordings were excluded.

The ONNX export also matches the Keras policy outputs within a maximum absolute deviation below $4\times10^{-7}$.

This evidence helps establish which model was released and which data were available during its training.

The reconstruction identifies implementation details relevant to interpretation. The temporal positional embedding was fixed and random rather than learned, and two telemetry inputs encode mathematically identical speed information.

Such details complicate claims about which architectural components explain the observed performance. No input or architecture ablations establish their individual contributions.

The audit also reveals a small reporting inconsistency: the stated training and validation window counts sum to 236,885, three more than the reported total of 236,882.

This discrepancy does not materially change the dataset-composition findings, but it illustrates the value of checking numerical claims against underlying artifacts.

For engineering organizations, checkpoint provenance provides a concrete governance function. It helps reviewers determine whether a claimed capability belongs to the released artifact, a later experiment, or an unverified observation.

What a controlled release evaluation should measure

The report proposes a quantitative closed-loop evaluation protocol in Section 7.2, Table 4. That protocol was not executed as part of the reported results.

It specifies fixed routes and spawn points, multiple random seeds, and evaluation without human intervention. Measurements would include route completion, collisions per kilometer, off-road events per kilometer, and intervention counts, covering both training towns and previously unseen towns.

For recovery, the proposed tests introduce controlled lateral offsets of up to three meters and heading offsets of up to 30 degrees. Recovery success would require returning to a specified lane-position and heading tolerance within five seconds, without collision, and maintaining that tolerance for one second.

These proposed measurements would address questions that neither offline control errors nor selected successful rollouts can answer.

A development team could use this distinction to establish three promotion requirements.

First, audit the demonstration distribution at the unit actually consumed by the model, including rare actions and maneuver transitions.

Second, measure the released checkpoint through repeated closed-loop rollouts under specified conditions. Report failures alongside successes, and keep familiar-route performance separate from unseen-town performance.

Third, isolate plausible explanations through controlled tests: remove privileged waypoints, vary initial-pose perturbations, compare data-coverage strategies, and evaluate difficult junction maneuvers separately.

The last category is especially relevant before committing resources to additional demonstration collection or a larger architecture. Without ablations, the observed results cannot identify whether performance improvements should be attributed to better coverage, geometric guidance, temporal modeling, or their combination.

These are proposed development practices inferred from the study, not benefits experimentally established by its reported comparisons.

The boundaries remain substantial. No quantitative traffic-interaction, pedestrian, adverse-weather, or real-world driving results are presented. Even successful execution of the proposed traffic-free benchmark would not establish real-world safety.

The stronger contribution is control over the evidence

Kieto’s study shows how a relatively small multimodal controller can accurately imitate expert actions and exhibit sustained autonomous behavior in a controlled simulated environment. Its verified demonstration-generation procedure and checkpoint audit make the experiment more inspectable than a report consisting only of training losses and successful driving examples.

The evidence is strongest for data construction, offline fitting, and model provenance. It is weaker for the reliability, generalization, and mechanisms of the observed autonomous behavior.

For teams deciding whether to advance an imitation-trained controller, the study supports a disciplined sequence: verify what was collected, measure what the model learned, and test what happens when its own actions control the environment.

Reliable deployment remains a separate empirical question.

Cognaptus: Automate the Present, Incubate the Future.