Cleaning the Book: How Structural Filtering Sharpens High-Frequency Signals

TL;DR for operators

Structural filtering is useful, but not in the simple “delete noisy orders, get cleaner alpha” way.

The paper tests three order-level filters — order lifetime, modification count, and modification timing — on BANKNIFTY January 2023 futures, then asks whether filtered order book imbalance better aligns with short-horizon returns.¹ The diagnostic ladder is sensible: start with Pearson correlation, move to discretised OBI-return regimes, then use Hawkes excitation norms to examine whether imbalance regimes are followed by return regimes of the same sign.

The important result is a contrast. For imbalance computed from the standing limit order book, filtration barely moves the needle. The average residualised Pearson correlation rises from 0.01018 under the unfiltered book to 0.01133 under the modification-time filter, while the lifetime filter is essentially unchanged and the modification-count filter performs worse. Hawkes excitation for standing-book OBI also changes only slightly.

For trade-based imbalance, the story changes. When the same filters are applied to the parent orders behind executed trades, Hawkes excitation from imbalance regimes to return regimes rises much more clearly. On 23 January 2023, for example, the unfiltered trade-based excitation score is 11.59; under the modification-time filter it rises to 24.74. That is not a backtest. It is not a licence to print money. It is a diagnostic sign that executed trades linked to certain order histories carry a cleaner event-time imprint than the raw visible book.

For exchanges and regulators, the paper’s practical value is a market-quality audit: do order-level surveillance thresholds change the directional information carried by the order flow, not just spreads, depth, and volatility? For trading teams, the lesson is even less romantic: do not assume quote filtration improves a feature just because the filter sounds microstructurally sophisticated. Test where the signal survives.

The easy story is wrong in a useful way

A familiar market-data story goes like this: the order book is full of junk. Fleeting quotes appear, modify, and vanish. Some reflect legitimate liquidity management; some are strategic noise; some are just machines endlessly nudging prices because apparently the machines were bored. Remove the junk, and the signal should improve.

That story is plausible. It is also too clean.

The paper studies a setting where the intuition should have a chance to work. Order book imbalance, or OBI, is a standard short-horizon directional primitive: if buy-side pressure exceeds sell-side pressure over a short interval, prices may be more likely to move up, and vice versa. The authors apply structural filters to tick-by-tick BANKNIFTY futures data from the National Stock Exchange of India and compare unfiltered and filtered imbalance against realised short-horizon returns.

The obvious expectation is that filtration should make standing-book imbalance more predictive. After all, if short-lived or rapidly modified orders pollute the visible book, removing them should leave something closer to genuine trading intent.

The paper’s actual evidence is more interesting: filtration of the standing book produces only small and unstable improvements, but filtration of the parent orders behind executed trades produces a much stronger Hawkes excitation pattern. The signal is not simply “in the cleaner book.” It is more visible at the point where quoted intent becomes executed commitment.

That distinction is the article. Everything else is plumbing.

The paper compares two imbalance worlds

The study has two related but different objects.

The first is ordinary order-based OBI, computed from the reconstructed standing order book after applying each structural filter. This is the version most people imagine when they hear “clean the book.” You remove orders with undesirable lifecycle patterns, rebuild the book, compute imbalance, and compare it with returns.

The second is trade-based imbalance, denoted in the paper as $\text{OBI}^{(T)}$. This version ignores unexecuted quotes and computes imbalance from signed trades. The twist is that the same structural filters are then mapped back to the parent orders that produced those trades. A trade is retained only if its parent order survives the relevant lifecycle filter.

That gives the paper a clean comparison:

This is why the earlier, tempting headline — “structural filtering sharpens high-frequency signals” — needs discipline. It is true only if “sharpens” means “reveals clearer event-time excitation in parent-filtered trade imbalance.” It is much less true if it means “makes the standing book obviously more predictive.”

Financial microstructure has many such traps. The thing that sounds like the signal is often just the thing with the nicest dashboard.

The diagnostic ladder is the real contribution

The paper’s first contribution is not one particular filter. The filters are simple by design:

1. Lifetime filter: remove orders whose total survival time is below a threshold.
2. Modification-count filter: remove orders with too many modifications.
3. Modification-time filter: remove orders with tightly clustered modification behaviour.

The more transferable contribution is the diagnostic ladder used to evaluate whether those filters improve directional association.

This ladder matters because high-frequency signals can look good at one layer and unimpressive at another. A correlation can rise because both series are noisy in compatible ways. A regime table can show alignment without offering useful event-time dynamics. A Hawkes excitation norm can reveal clustering and directional follow-through, but it is still a model-based diagnostic, not a courtroom confession from the order book.

The authors are explicit that the exercise is diagnostic. That restraint should survive into any business interpretation. A metric that says “this filtered signal has stronger directional event-time association” is valuable. It does not say “deploy this into production and expect lunch money from the market.”

Standing-book filtration mostly changes decimals

The standing-book result is the part that should disappoint anyone hoping for a clean filtration fairy tale.

Across the three sample days and threshold settings, the paper reports only small and heterogeneous changes for OBI computed from the standing book. The averaged residualised scores make the point:

The modification-time filter produces the highest average Pearson correlation and Hawkes excitation among the standing-book variants. But the scale matters. Moving from 0.01018 to 0.01133 is not a dramatic discovery; it is a small relative gain on a tiny absolute correlation. The Hawkes score rises from 8.9292 to 9.1745, also modest.

The appendix reinforces the point. At different horizons and on different days, some filter-lag combinations improve and others deteriorate. The pattern is not “MTF wins.” It is “MTF often looks best on average, but the evidence is too mixed to call standing-book filtration a robust signal upgrade.”

This corrects the most likely reader misconception. Removing fleeting or heavily modified orders sounds like it should clean OBI. In the paper’s standing-book tests, it mostly does not. The raw book is noisy, but the aggregate imbalance measure may already average out much of that noise. Or, less flatteringly, simple lifecycle filters may be too blunt to isolate the part of the book that matters.

Either way, the business takeaway is the same: do not sell filtration to a trading desk or surveillance committee with a before-and-after chart unless the “after” survives multiple diagnostics. Microstructure theatre is still theatre, even when performed at nanosecond resolution.

The trade-linked result is where the signal moves

The stronger evidence appears when the authors shift from the standing book to trade-based imbalance. Here, they compute imbalance from signed trades and then filter those trades according to the lifecycle properties of their parent orders.

The Hawkes excitation scores become much more responsive:

This table should be read carefully.

On 2 January, all three filters improve the trade-based Hawkes excitation score relative to the unfiltered benchmark, with the lifetime filter strongest. On 13 January, the lifetime filter nearly doubles the unfiltered score, while modification-time and modification-count filtration also improve it. On 23 January, the modification-time filter dominates, raising the score from 11.59 to 24.74, while modification-count is slightly below unfiltered.

So the result is not “one filter always wins.” The result is that filtering parent orders of executed trades reveals a much clearer directional excitation structure than filtering the standing book.

That is a subtler and more useful claim. It suggests that not all executions carry the same informational content. Trades arising from parent orders with certain structural histories — longer-lived, less suspiciously modified, or not embedded in frantic modification bursts — appear to carry cleaner event-time association with subsequent return regimes.

Notice what this does not say. It does not say that trades from filtered-out parent orders are fake, manipulative, or irrelevant in all contexts. It does not say a firm should ignore them in execution logic. It says that, in this dataset and under these diagnostics, the directional Hawkes structure is clearer among trades whose parent orders survive certain lifecycle filters.

That is enough to be useful. It is not enough to be smug.

Why the book can stay noisy while trades become informative

The asymmetry between standing-book OBI and trade-based $\text{OBI}^{(T)}$ is the paper’s most important interpretive problem.

One possible mechanism is aggregation. Standing-book imbalance compresses many messages into a windowed net pressure measure. Fleeting orders, rapid cancellations, and repeated modifications may be abundant, but their effect can wash out when the signal is aggregated. If both sides of the book are constantly churning, deleting some churn may change the event stream without materially changing the imbalance-return relationship.

The paper gives a useful scale clue. On 23 January 2023, the unfiltered book contains 83,547,534 final ticks. The lifetime filter removes 5,928,152 order IDs and leaves 51,023,701 ticks. The modification-count filter removes 5,416,670 order IDs and leaves 72,228,161 ticks. The modification-time filter removes only 168,058 order IDs but leaves 47,531,611 ticks.

That last number is the eyebrow-raiser. A relatively small set of rapidly modifying orders accounts for a very large amount of message traffic. Yet removing that traffic still produces only modest changes in standing-book OBI diagnostics. Apparently, deleting a lot of messages is not the same thing as deleting a lot of directional information. Shocking. Somewhere, a data vendor is pretending not to hear this.

At the execution boundary, however, the filter has a sharper meaning. A trade linked to a parent order is not just visible intent; it is realised commitment. Filtering the parent orders behind trades asks a more precise question: among trades that actually happened, which order histories carry cleaner directional event-time structure?

That is why $\text{OBI}^{(T)}$ moves while standing-book OBI barely does. The filter is no longer trying to purify the entire visible book. It is segmenting executed flow by the structural history of the orders that produced it.

This is a more operationally meaningful distinction than “quotes versus trades.” The point is not that quotes are useless. The point is that quote-level filtration and execution-linked filtration answer different questions. Confusing them is how one ends up optimising a beautiful feature that does not survive contact with fills.

The regulatory reading is diagnostic, not punitive

The paper is partly motivated by market surveillance. Exchanges and regulators often worry about high order-to-trade ratios, excessive cancellations, repeated modifications, and persistent noisy order flow. Those concerns are legitimate, but translating them into evidence is difficult.

Traditional market-quality studies tend to look at spreads, depth, volatility, and related measures. This paper adds another lens: after applying a candidate structural filter, does the remaining order flow exhibit stronger directional association with returns?

That is valuable because it links surveillance thresholds to informational content. A rule may reduce message traffic without improving the price-relevant clarity of the book. Conversely, a rule may reveal that certain executed-flow segments carry cleaner directional structure even if the standing book remains noisy.

For market operators, the paper suggests a practical audit loop:

This is where the paper is more useful to an exchange or regulator than to a retail alpha-hunter. It gives a way to ask whether order-flow rules change the informational structure of the market. It does not give a universal threshold for “bad” orders.

That distinction matters. A market maker modifying quotes rapidly during volatility is not automatically a manipulator. A short-lived order is not automatically fake. A high message count is not automatically toxic. Structural filters are blunt instruments. Their value is in comparative diagnosis, not moral classification.

For trading teams, the feature lesson is uncomfortable

Trading teams often like feature engineering because it feels controllable. Add a filter. Remove noise. Improve signal. Repeat until the backtest becomes emotionally supportive.

The paper is a useful antidote. It shows that a plausible filter can remove millions of events and still barely improve standing-book OBI. That should make feature teams more humble about microstructure intuitions.

The more actionable workflow is not “use the modification-time filter.” It is:

1. Build the raw imbalance feature.
2. Apply structurally interpretable filters independently.
3. Compare unfiltered and filtered variants across multiple diagnostic layers.
4. Separate standing-book features from trade-linked features.
5. Treat Hawkes excitation as a signal-audit metric, not as proof of deployable alpha.
6. Validate across more contracts, dates, regimes, and market conditions before caring too much.

This is less glamorous than announcing a new high-frequency signal. It is also less likely to embarrass the firm.

For execution desks, the result may be particularly relevant. Parent-order histories could help distinguish trades that carry cleaner directional information from trades embedded in noisy quote-revision dynamics. That could inform short-horizon risk controls, fill interpretation, or post-trade analytics.

For surveillance teams, the same idea becomes a monitoring problem. If a small number of rapidly modifying parent orders generates a disproportionate share of book activity, the next question is whether those orders are associated with degraded price formation, reduced informational clarity, or simply benign automated liquidity management. The paper does not answer that fully. It gives a better diagnostic path.

Where the evidence stops

The limitations are not decorative here. They materially affect how the result should be used.

First, the empirical sample is narrow: three BANKNIFTY January 2023 futures trading days — 2 January, 13 January, and 23 January — chosen to span different parts of the expiry cycle and different activity levels. That is enough for a methodological illustration, not enough for a universal claim about futures markets, Indian markets, or high-frequency trading generally.

Second, the paper does not observe trader identities or individual latencies. That means it cannot distinguish benign liquidity-provider revisions from strategic noise or manipulative behaviour. The filters classify order histories, not intentions.

Third, Hawkes excitation is not structural causality. The paper uses Hawkes kernel norms to diagnose event-time excitation between imbalance regimes and return regimes. This is a richer object than correlation, but it remains model-based evidence of dependency. It does not prove that filtered imbalance causes returns in an economic or policy sense.

Fourth, the paper does not present an execution strategy or P&L backtest. For a trading firm, that matters. A cleaner event-time diagnostic may still be too weak, too unstable, too costly, or too crowded to monetise.

Finally, the filters are intentionally simple. That makes them interpretable and implementable, but also blunt. More adaptive thresholding, cross-asset validation, volatility-conditioned filters, or client-level surveillance data could change the picture. The authors themselves point toward future calibration frameworks where thresholds become design parameters optimised against out-of-sample diagnostic objectives.

So the safest reading is not conservative for the sake of being dull. It is precise: the paper demonstrates a diagnostic asymmetry between quote-level and trade-linked filtration. It does not settle optimal market design or deliver a ready-made alpha engine.

The better headline is not “filter harder”

The old instinct says: the book is dirty, so clean it.

The paper says: clean where the evidence tells you cleaning matters.

For standing-book OBI, simple structural filters barely and inconsistently improve directional association with short-horizon returns. The visible book may be noisy, but the aggregate imbalance signal is not transformed by deleting short-lived or heavily modified orders.

For trade-based imbalance, filtering by the lifecycle of parent orders produces a clearer Hawkes excitation pattern. That is the useful contrast. The market’s directional structure appears less in the filtered display of intent and more in the filtered record of commitment.

For operators, that changes the question. Do not ask whether a filter sounds microstructurally sensible. Ask whether it changes the diagnostic layer that matters for your decision: correlation, regime alignment, event-time excitation, surveillance calibration, execution risk, or actual P&L.

The book can be cleaned. The harder part is knowing whether cleanliness is information, cosmetics, or just another expensive way to organise noise.

Cognaptus: Automate the Present, Incubate the Future.