Refactor, Then Run It: SemaDiff Tests Whether Behavior Actually Changed
TL;DR for operators A commit may be labeled as a refactor and still change behavior that users can observe. For release, backport, rollback, and dataset decisions, the label matters less than whether the parent and modified versions behave differently. The existing repository may not provide a practical way to reach the changed code: in the benchmark, original callers were absent in 50% of cases. SemaDiff addresses this by creating a small caller—called a generated dependent class—that invokes stable public APIs in both versions. It then runs the same generated tests against each version and treats a divergent outcome as evidence of a semantic-changing commit, a process known as differential execution. ...