TL;DR for operators
A commit may be labeled as a refactor and still change behavior that users can observe. For release, backport, rollback, and dataset decisions, the label matters less than whether the parent and modified versions behave differently.
The existing repository may not provide a practical way to reach the changed code: in the benchmark, original callers were absent in 50% of cases. SemaDiff addresses this by creating a small caller—called a generated dependent class—that invokes stable public APIs in both versions. It then runs the same generated tests against each version and treats a divergent outcome as evidence of a semantic-changing commit, a process known as differential execution.
This generated route covered more than 98% of the semantic-changing cases SemaDiff detected, and approximately 85% were exposed only through the generated code. A positive result is therefore strong evidence for escalation or relabeling, but an unflagged commit is not cleared because the generated tests may still miss a behavioral difference.
A refactor label does not settle release risk
Suppose a maintenance commit renames methods, moves code, extracts classes, or reorganizes control flow. A static detector can often identify those transformations accurately. The operational decision is different: can the commit be merged, backported, rolled back, or recorded as a pure refactor without changing observable behavior?
That distinction matters because structural and functional edits can be tangled in the same commit. A benchmark or training corpus labeled from refactoring metadata alone may therefore mix behavior-preserving maintenance with functionality changes. A release process can make the same mistake, only with production consequences.
Tests seem like the natural arbiter, but they depend on reachability. The repository may have no unchanged caller for the modified code, or the existing call path may be so indirect that generated tests barely exercise the relevant branch. In the paper’s benchmark, original callers were absent for the changed code in 50% of cases. The bottleneck was not simply writing more tests. It was creating a usable test target.
The missing artifact is a caller that survives both versions
SemaDiff’s central design choice is to generate a small dependent class before generating tests. The class calls stable public APIs and is intended to compile against both the parent and modified versions of the commit.
This generated class is not a replacement implementation. It is a controlled bridge into the changed code. By reducing the distance between a test and the modified behavior, it gives the test generator a simpler surface to exercise than a large production caller with unrelated state, dependencies, and setup logic.
The workflow is:
- Analyze the commit, identify modified Java code, and locate unchanged dependent classes.
- Generate an additional dependent class that compiles against both versions.
- Generate tests over the original and generated dependent code, repairing compilation errors through limited retries.
- Run the same tests on the parent and modified versions.
- Mark the commit as semantic-changing when at least one test outcome differs.
This is differential execution: the signal is not whether a test passes in isolation, but whether the same test produces a different observable result across versions. Identical outcomes provide evidence relative to the generated suite. They do not prove full semantic equivalence.
The generated caller drives the result
The strongest mechanism evidence is the comparison between generated and original dependent code. This functions as an ablation: it asks whether the additional caller is doing real work or merely adding another prompt step.
Generated dependent code covered more than 98% of the semantic-changing cases SemaDiff identified. Approximately 85% of those cases were exposed only through generated code. Tests aimed at generated dependent classes exercised about 74% of those classes on average, compared with about 30% for tests aimed at original repository dependents. The generated route also produced roughly 42 percentage points more modified-class coverage and 48 points more modified-line coverage.
Those figures explain why the system works when it works. Test generation quality matters, but tests cannot reveal behavior they do not reach. SemaDiff improves the geometry of the problem first: it constructs a short, cross-version path into the changed public behavior, then asks the model to produce discriminating tests.
For teams building similar systems, this shifts the implementation priority. Measuring prompt quality without measuring compilability and changed-code coverage would miss the main operational mechanism.
Precision changes how the system should be used
On the 183-commit benchmark, the ChatGPT-based configuration correctly identified 53 of 95 semantic-changing commits and produced no false semantic-change flags among 88 semantic-preserving commits. That yields 75.95% accuracy, 100% precision, and 58.89% recall, with a 3.82% execution-failure rate.
The precision-recall profile is more informative than the headline accuracy. A positive result was highly reliable in this sample. A negative result was not clearance: more than two-fifths of the semantic-changing commits were not detected.
That makes SemaDiff a conservative triage tool. It is suited to decisions where false alarms are expensive, such as interrupting a backport, escalating a rollback review, or excluding a supposedly pure refactor from a curated dataset. It is not suited to silently approving every unflagged commit.
The comparison results separate screening from evidence
The paper’s additional experiments serve different purposes, and they should not be read as one undifferentiated leaderboard.
| Comparison | Likely purpose | Result | Operational interpretation |
|---|---|---|---|
| ChatGPT vs. DeepSeek vs. CodeLlama inside SemaDiff | Robustness and implementation reliability | Accuracy: 75.95%, 63.93%, 36.61%; all at 100% precision | Model choice affects whether code and tests compile and execute, not just classification quality |
| Generated vs. original dependent code | Ablation of the core design | Generated code exposed nearly all detected changes and reached much more modified code | The caller is a primary mechanism, not preprocessing decoration |
| LLM tests vs. Randoop | Test-generation comparison | Randoop variant: 34.9% accuracy, 37.9% recall, 45.35% execution failures | General random generation struggled with commit-specific reachability and execution |
| Direct LLM classification | Lightweight baseline | ChatGPT classifier: 86.89% accuracy, 84.47% precision, 91.58% recall | Better overall screening, but weaker assurance that a positive flag is behavior-backed |
| PurityChecker | Static-analysis baseline | 20.77% accuracy; no result for 65.03% of commits | Restricted pattern coverage limits use as a general semantic-change detector |
The direct LLM result is the most important operational contrast. A classifier looking at the commit achieved higher accuracy and recall, but its precision was materially lower than SemaDiff’s. The two approaches answer different workflow needs.
Paper evidence: execution-backed comparison produced fewer false semantic-change alarms in this benchmark.
Cognaptus inference: a practical pipeline can use direct LLM classification as a low-cost screening layer, then reserve SemaDiff-style execution for commits where a false alarm would trigger costly review or block release activity.
Unresolved uncertainty: the paper does not provide production cost, latency, or controlled economic comparisons, so the optimal routing threshold remains an engineering decision rather than an empirical finding.
A two-stage workflow is the practical deployment pattern
The paper supports a layered process rather than replacement of existing refactoring tools.
First, use static refactoring detection to identify candidate commits. This stage is broad and structurally grounded. Second, run behavior-backed differential execution on candidates that affect release, backport, rollback, or dataset decisions. A divergent outcome should move the commit into manual review with the generated caller and failing test retained as diagnostic evidence.
For dataset curation, the same pattern can reduce contamination. Commits mined as “refactoring” are often used in empirical studies or model training. A positive behavioral-change flag gives curators a defensible reason to exclude or relabel the record. An unflagged commit should remain provisional unless other evidence supports preservation.
For library code, generated callers may be especially valuable when the changed public API has no reachable in-repository consumer. The design effectively creates a temporary client for the purpose of semantic comparison.
Operational monitoring should include more than classification metrics. Teams should track dependent-code compilation success, test-repair frequency, execution-failure rate, and coverage of modified classes and lines. The model comparison shows why: execution failures rose from 3.82% with ChatGPT to 14.20% with DeepSeek and 36.61% with CodeLlama, while precision remained 100% among completed positive detections.
The evidence is strong enough for triage, not certification
The benchmark is manually adjudicated and spans seven open-source Java repositories, 183 commits, 262 modified Java files, and 10,399 refactored lines. That provides useful comparative evidence, but it is not broad enough to establish language-independent or organization-wide performance.
Three boundaries should govern adoption.
First, the method has incomplete recall. Missed changes can result from failed generation, failed compilation, insufficient coverage, or tests that simply do not expose the behavioral difference.
Second, the system depends on external tools and evolving models, including Spoon, RefactoringMiner, test generators, and LLMs. A model update can alter compilability and execution reliability even if the high-level pipeline remains unchanged.
Third, the arXiv version did not provide a public replication-package link, and the paper contains several numerical inconsistencies in prose. The analysis here follows the tabulated values: 88 preserving and 95 changing commits, 34.9% accuracy for the Randoop variant, and 75.95% accuracy for the main SemaDiff configuration.
These limits do not erase the mechanism result. They define its proper role: SemaDiff supplies high-confidence behavioral evidence for some commits, not a formal proof that all remaining commits preserve semantics.
Build the test path before judging the test
The paper’s most transferable idea is not that an LLM can classify a refactoring commit. It is that semantic comparison may fail because the system lacks a stable route into the changed behavior.
Generating that route changes the testing problem. A focused caller that survives both versions can reach code that production callers miss, make differential execution feasible, and turn a positive result into concrete review evidence.
For operators, the deployment rule is straightforward: use structural detection to find the candidates, execution to escalate the strongest risks, and additional review to handle everything the generated suite cannot establish.
Cognaptus: Automate the Present, Incubate the Future.