Monitoring a live book

01The daily loop

We run the same loop after every session. The strategy stamps its targets when it produces them. When the market closes, the monitor takes fresh snapshots from the broker (positions, fills, the statement of P&L) and from the price feed. Each check reads those snapshots, recomputes what it needs, and reports one of three results with the evidence it used. The report lists all the checks, each day, and only the results that need a person become alerts. The monitor also sends a heartbeat on each pass to a watcher that runs elsewhere, so a monitor that has stopped is noticed.

02What to measure each session

Each check compares two things that should agree and were produced independently. A position is only wrong relative to something, and the choice of that something decides what the check can catch.

Holdings are read from the broker, never from our own records, because our records are what a bug corrupts. The last column matters as much as the third. Holdings can equal targets even when the target itself is wrong: in the synthetic session, a strategy that asks for forty times its intended position passes the holdings check and is caught only by the exposure check.

03Pass, fail and unavailable

A check has three possible results. It passed: it looked, and what it saw was right. It failed: it looked, and something was wrong. Or it was unavailable: it could not look, because the broker did not answer, returned an empty list, or the check itself crashed. The report’s headline is fail if anything failed, else unavailable if anything could not look, and pass only when every check looked and passed.

def run(d: Day) -> list:
    """Every check, each isolated: a check that raises is UNAVAILABLE, never a crash of the report.
    On a day the exchange calendar has no session there is nothing to check, and each check says so."""
    if d.close_utc is None:
        return [Result(name, PASS, "no session today by the exchange calendar: nothing to check", "calendar: exchange closed")
                for name in CHECKS]
    out = []
    for name, fn in CHECKS.items():
        try:
            out.append(Result(name, *fn(d)))
        except Exception as e:                                   # noqa: BLE001
            out.append(Result(name, UNAVAILABLE, f"the check itself failed ({type(e).__name__})", "no result"))
    return out


def overall(results: list, unavailable_is_pass: bool = False) -> str:
    """The report's headline. Correctly: any FAIL, else any UNAVAILABLE, else PASS. The naive
    version, which many dashboards amount to, reads UNAVAILABLE as PASS."""
    s = {r.status for r in results}
    if FAIL in s:
        return FAIL
    if UNAVAILABLE in s and not unavailable_is_pass:
        return UNAVAILABLE
    return PASS

The rule matters on the evening the broker’s interface is down. On many dashboards an empty panel, a metric that stopped updating and a green light look the same at a glance.

04Witnesses and planted faults

Each result carries a witness: the evidence the check examined in this pass, with its date. “6 broker positions read at 20:05, 6 targets from 19:40” is a witness; “OK” is not. Witnesses make silent failures visible. A check that read nothing (an empty snapshot, a missing file) is unavailable, not passed, and says so. A price from the day before is named as such: “stale: B stamped the day before at 20:00”. We never store “the check passed” and read it back; each pass recomputes from snapshots taken in that pass.

A check is trusted only after it has been seen to fail. For each check we plant the fault it exists for and confirm that it fires, a practice borrowed from mutation testing in software, where a test suite is judged by the deliberate defects it catches.¹ Run over all the planted faults, the synthetic suite gives the matrix below. Each planted fault is caught by at least one check, and 5 of the 11 rest on a single check: those are the places where a second, independent check would earn its keep.

def session_close_utc(day: date):
    """The session's close in minutes from midnight UTC of `day`, from the exchange calendar,
    or None when the exchange does not open that day."""
    if day.weekday() >= 5 or day in HOLIDAYS:
        return None
    utc = datetime.combine(day, EARLY_CLOSES.get(day, REGULAR_CLOSE), EXCHANGE_TZ).astimezone(ZoneInfo("UTC"))
    return (utc.date() - day).days * 1440 + utc.hour * 60 + utc.minute


def check_freshness(d: Day) -> tuple:
    if not d.price_utc:
        return UNAVAILABLE, "no prices were read", "empty price snapshot"
    age = {s: d.close_utc - t for s, t in d.price_utc.items()}
    stale = {s: a for s, a in age.items() if abs(a) > FRESH_MIN}
    w = f"{len(age)} prices, the oldest stamped {when(d.close_utc - max(age.values()))}"
    if not stale:
        return PASS, "every price within 15 minutes of the close", w
    offsets = set(stale.values())
    off = next(iter(offsets))
    if len(stale) == len(age) and len(offsets) == 1 and off % 30 == 0 and abs(off) <= 14 * 60:
        return FAIL, (f"every price is off by exactly {abs(off) / 60:g} h: check the time zone before "
                      "blaming the feed"), w
    return FAIL, "stale: " + ", ".join(f"{s} stamped {when(d.close_utc - a)}" for s, a in sorted(stale.items())), w

The close the check measures against is the session’s own, from the exchange calendar, never a fixed clock time. The regular close in New York falls at a different hour in UTC in summer and in winter, and earlier on a half-day, so a fixed time would report every price off by a whole hour for half the year and blame the time zone for what is really the calendar. On a holiday there is no session, and nothing to be fresh: the report says so for every check (run, in the first listing).

05Explaining the day’s P&L

The day’s P&L should be explainable from three things we already have: yesterday’s holdings times the day’s price moves, each fill marked to the close, and fees. Whatever the broker reports beyond that is unexplained, and its size and sign say where to look. A gap equal to a position times a price difference points at a price; a round number points at a fee; a gap that recurs every day points at something we do not model, such as financing or borrow.

When an order is doubled, this check still passes: the broker’s P&L follows the broker’s fills, and those fills explain it exactly. The P&L is right; the book is wrong, and the holdings and orders checks are the ones that say so. A P&L check run against the intended book instead would also catch it. The checks are designed as a set. Live versus backtest reconciliation compares the same holdings with the backtest’s, where a doubled order shows up at once.

06Dashboards and checks

A dashboard answers the questions a person thinks to ask; a check answers one question every day whether anyone asks or not. Google’s site reliability engineers list dashboards and ad hoc analysis among the uses of monitoring, and keep alerts for symptoms that need a person now.² Few’s rules for dashboard design, one screen with the exceptions made obvious, help the person looking;³ they cannot help on an evening when nobody looks.

The SEC’s order on Knight Capital describes the cost of relying on the first kind. Knight’s position monitoring tool relied entirely on people watching it, generated no automated alerts, and did not display the limits it was meant to police, and it could not stop the entry of orders, so the orders kept flowing while positions grew.⁴ End-of-day checks find problems after the fact; preventing them takes pre-trade controls in the order path, which the SEC’s market access rule requires of brokers and which Going live describes.⁵

Which of these results should wake someone is the subject of Alerting that people act on.