Partial fills of resting orders are never applied to position/balance (PartialFillExchange + prob queue models)

[readerwei]

Summary

When using partial_fill_exchange() with a probability queue model (e.g. power_prob_queue_model), a resting limit order that gets partially filled via the queue model never has that execution applied to the strategy's position/balance/fee. The local order copy shows status=PARTIALLY_FILLED with the correct exec_qty, but hbt.position() and the Recorder output report no fill at all.

The root cause is that the local processor only applies fills for Status::Filled responses:

// hftbacktest/src/backtest/proc/local.rs (line ~102, present on master and in all 2.x releases)
// Processes receiving order response.
if order.status == Status::Filled {
    self.state.apply_fill(&order);
}

PartiallyFilled responses fall through and are only used to update the local order copy. Additionally, since State::apply_fill consumes the per-event order.exec_qty (each fill overwrites it), even when the order later fills completely, only the last chunk is credited — so position ends up undercounting by every partially-filled chunk:

// hftbacktest/src/backtest/state.rs
pub fn apply_fill(&mut self, order: &Order) {
    let amount = self.asset_type.amount(order.exec_price(), order.exec_qty);
    self.state_values.position += order.exec_qty * AsRef::<f64>::as_ref(&order.side);
    ...
}

Severity / impact

Any backtest where a resting order's first queue-model execution is partial (i.e. order qty exceeds the executable volume at that moment) silently reports a smaller fill than the model actually produced:

• On 2.3.0 (current PyPI release) the reported fill is zero, because of a second, since-fixed problem: check_if_buy_filled/check_if_sell_filled pushed the order id to filled_orders unconditionally on any queue-model fill, so a partially filled order was also removed from the exchange book and could never fill again — even when price subsequently traded through it.
• On master / py-v2.4.4 the exchange side is fixed (the order survives and the remainder fills correctly later), but due to local.rs above, only the final Filled chunk reaches the accounting. E.g. an order that fills 455 (partial) + 25 (final) reports position 25.

Minimal reproduction (hftbacktest 2.3.0 from PyPI)

Synthetic market: bid 100.0 x 10, ask 100.5 x 10. A resting buy is placed at 100.0 (queue ahead = 10), then a sell trade of 15 @ 100.0 arrives, so 5 contracts of the resting order are executable. In one variant the ask afterwards drops to 100.0, crossing the order, which should fill the remainder.

import numpy as np
from numba import njit
from numba.types import uint64

from hftbacktest import (
    GTC, LIMIT,
    BacktestAsset, HashMapMarketDepthBacktest,
    DEPTH_EVENT, TRADE_EVENT, event_dtype,
)

EXCH_LOCAL_BUY = 0xE0000000
EXCH_LOCAL_SELL = 0xD0000000
S = 1_000_000_000


def ev(ts, kind, px, qty, ival):
    e = np.zeros(1, dtype=event_dtype)[0]
    e["ev"], e["exch_ts"], e["local_ts"] = kind, ts, ts
    e["px"], e["qty"], e["ival"] = px, qty, ival
    return e


def build_events(with_trade, with_cross):
    rows = [
        # t=1s: establish book  bid 100.0 x 10, ask 100.5 x 10
        ev(1 * S, EXCH_LOCAL_BUY | DEPTH_EVENT, 100.0, 10, -1),
        ev(1 * S, EXCH_LOCAL_SELL | DEPTH_EVENT, 100.5, 10, 1),
    ]
    if with_trade:
        rows += [
            # t=3s: sell trade 15 @ 100.0 (sweeps queue of 10, then 5 of ours)
            ev(3 * S, EXCH_LOCAL_SELL | TRADE_EVENT, 100.0, 15, -1),
            ev(3 * S + 1, EXCH_LOCAL_BUY | DEPTH_EVENT, 100.0, 0, -1),
            ev(3 * S + 2, EXCH_LOCAL_BUY | DEPTH_EVENT, 99.5, 8, -1),
        ]
    if with_cross:
        # t=5s: ask reprices down to 100.0 -> crosses the resting buy
        rows += [ev(5 * S, EXCH_LOCAL_SELL | DEPTH_EVENT, 100.0, 7, 1)]
    rows += [ev(8 * S, EXCH_LOCAL_BUY | DEPTH_EVENT, 99.5, 9, -1)]
    out = np.zeros(len(rows), dtype=event_dtype)
    for i, r in enumerate(rows):
        out[i] = r
    return out


@njit
def sim(hbt, order_qty):
    placed = False
    while hbt.elapse(100_000_000) == 0:
        if not placed and hbt.current_timestamp >= 2 * 1_000_000_000:
            hbt.submit_buy_order(0, uint64(1), 100.0, order_qty, GTC, LIMIT, False)
            placed = True
    pos = hbt.position(0)
    st, ex, lv = -1.0, -1.0, -1.0
    vals = hbt.orders(0).values()
    while vals.has_next():
        o = vals.get()
        st, ex, lv = float(o.status), o.exec_qty, o.leaves_qty
    hbt.close()
    return pos, st, ex, lv


def run(order_qty, with_trade, with_cross):
    asset = (
        BacktestAsset()
        .add_data(build_events(with_trade, with_cross))
        .linear_asset(1.0)
        .constant_latency(1_000_000, 1_000_000)
        .power_prob_queue_model(2)
        .partial_fill_exchange()
        .trading_value_fee_model(0.0, 0.0)
        .tick_size(0.5)
        .lot_size(1)
    )
    hbt = HashMapMarketDepthBacktest([asset])
    pos, st, ex, lv = sim(hbt, order_qty)
    print(f"qty={order_qty:4.0f} trade={with_trade!s:5} cross={with_cross!s:5}  "
          f"position={pos:5.0f}  order(status={st:.0f} exec={ex:.0f} leaves={lv:.0f})")


print("status codes: 1=NEW 3=FILLED 5=PARTIALLY_FILLED")
run(5.0, True, False)    # full fill by trade
run(20.0, True, False)   # partial fill 5, no cross
run(20.0, True, True)    # partial fill 5, then ask crosses the order
run(20.0, False, True)   # control: no partial fill, cross only

Actual output (2.3.0)

status codes: 1=NEW 3=FILLED 5=PARTIALLY_FILLED
qty=   5 trade=True  cross=False  position=    5  order(status=3 exec=5 leaves=0)
qty=  20 trade=True  cross=False  position=    0  order(status=5 exec=5 leaves=15)
qty=  20 trade=True  cross=True   position=    0  order(status=5 exec=5 leaves=15)
qty=  20 trade=False cross=True   position=   20  order(status=3 exec=20 leaves=0)

Expected

qty=   5 ...  position=    5   (ok)
qty=  20 trade=True  cross=False  position=    5   (partial fill of 5 applied)
qty=  20 trade=True  cross=True   position=   20   (5 by trade + 15 when crossed)
qty=  20 trade=False cross=True   position=   20   (ok)

Rows 2–3 show both problems: the executed 5 never reaches position, and (2.3.0 only) the crossing that fills a fresh order (row 4) does nothing for a partially filled one, because the order was already removed from the exchange book.

Suggested fix

In local.rs (and presumably l3_local.rs), apply fills for partial executions as well:

if order.status == Status::Filled || order.status == Status::PartiallyFilled {
    self.state.apply_fill(&order);
}

Since fill() sets order.exec_qty to the per-event executed quantity, this accumulates correctly across chunks.

Environment

• hftbacktest 2.3.0 (PyPI wheel), Python 3.10, Linux x86_64
• HashMapMarketDepthBacktest, partial_fill_exchange(), power_prob_queue_model(2), constant_latency
• Behavior is deterministic; also reproduced with real CME treasury futures L2 data, where a 480-lot resting order at a swept level reported zero fill while the order object showed exec_qty=455.