Data for systematic trading

01From delivery to strategy

We treat market data as a pipeline with a record of what arrived, when, and in what form. Raw deliveries are kept exactly as received and never edited. Everything a strategy reads is derived from them by code: adjusted prices, continuous futures, calendars, and the checks that decide whether a day’s data may be used at all. When a number looks wrong, the raw record shows whether the vendor sent it or we made it.

A raw record is useful only if it can answer, months later, what the system knew and when. Ours holds, for every delivery:

The same reader serves research, the backtest and the live system. When they read different copies of the data, the backtest and the live book disagree for reasons that have nothing to do with the strategy; QuantConnect and LEAN describes how one engine keeps that path shared.

02Point-in-time data

A fact about a period has at least two timestamps: the period it describes and the moment it became known. Quarterly earnings for a quarter that ends in March are published weeks later and are sometimes restated months after that. Macroeconomic series are routinely re-estimated after publication. Croushore and Stark built a real-time data set for US macroeconomic variables because results computed from the latest figures can differ materially from what could have been computed at the time;¹ the Federal Reserve Bank of St. Louis publishes every vintage of its series free of charge in ALFRED.² Index membership, sector classifications and credit ratings change too, and a database that stores only the latest value rewrites history each time.

A point-in-time record keeps every vintage with its publication time, and a backtest asks it one question: what was known when the decision was made? Fama and French handled this for accounting data by matching fiscal years ending in calendar year t − 1 to returns from July of year t, a gap of at least six months that makes sure the figures were public.³ A lag rule of that kind is a safe default when publication dates are missing. Where they exist, use them, with a time of day: a figure released after the close cannot inform that day’s closing trade. Our listing assumes a date-only publication arrives after the close.

def known_on(facts: pd.DataFrame, day: pd.Timestamp) -> pd.DataFrame:
    """The record a strategy could use at the close of `day`: each period in its latest vintage.
    Publication dates carry no time of day, so a fact is assumed to arrive after the close and
    is first usable at the next session's close (strictly earlier date). With timestamps,
    compare them with the decision time instead."""
    seen = facts[facts["published"] < day].sort_values("published")
    return seen.groupby("period_end", as_index=False).last()


def point_in_time(days: pd.DatetimeIndex, facts: pd.DataFrame, period: str = None) -> pd.Series:
    """What a backtest may use at each day's close: the latest period's value as known then,
    or, with `period`, that period's value in the vintage known then."""
    out = []
    for d in days:
        k = known_on(facts, d).sort_values("period_end")
        if period is not None:
            k = k[k["period_end"] == pd.Timestamp(period)]
        out.append(k["value"].iloc[-1] if len(k) else float("nan"))
    return pd.Series(out, index=days)


def naive(days: pd.DatetimeIndex, facts: pd.DataFrame) -> pd.Series:
    """The look-ahead mistake: today's final vintage, joined on the period it describes."""
    final = facts.sort_values("published").groupby("period_end", as_index=False).last()
    left = pd.DataFrame({"day": days.astype(final["period_end"].dtype)})
    out = pd.merge_asof(left, final.sort_values("period_end"), left_on="day", right_on="period_end")
    return out.set_index("day")["value"]

Banz and Breen showed how much this matters: treating accounting figures as known at the fiscal year end, and using only the firms still on the current file, gave significantly different answers to standard questions from those of a point-in-time record.⁴ Kothari, Shanken and Sloan argued that part of the book-to-market effect in early studies came from the way firms entered the accounting database with their histories already attached;⁵ Chan, Jegadeesh and Lakonishok found that bias small in their sample.⁶ Either way, the question has to be asked of every dataset.

03Survivorship and delisting returns

A list of stocks that trade today is a list of winners. Companies that went bankrupt, were delisted for low prices or were taken over have left it, and a backtest that starts from today’s list and looks backwards never holds them. Brown, Goetzmann, Ibbotson and Ross showed that truncating a sample on survival can make performance look persistent and predictable when it is neither,⁷ and Elton, Gruber and Blake measured the effect on mutual fund returns directly.⁸

Keeping dead companies is not the end of it. When a stock is delisted, its holders receive something: a takeover price, or the last trade on an over-the-counter market after a fall. Shumway found that the Center for Research in Security Prices (CRSP) file was missing most delisting returns for stocks delisted for poor performance, that for NYSE and AMEX stocks the missing returns averaged about −30%, and suggested using that figure when the true one is unknown;⁹ for Nasdaq stocks, Shumway and Warther put it nearer −55%.¹⁰ CRSP’s files carry delisting returns, with codes for why one is missing.¹¹

If the database drops delisting returns instead of recording them at −30%, the point-in-time universe compounds at 6.87%: 0.71 percentage points a year better than the truth.

The bias is worst for strategies that buy weakness. Buying each month the fifth of stocks with the worst past year earns 12.8% a year on the survivors and 3.0% on the point-in-time universe. Among survivors, no stock fell far enough to be delisted, so buying the fallen ones never meant buying a stock on its way to zero.

def listed(r: np.ndarray, floor: float, delist_return: float = None):
    """Returns as a point-in-time database records them, and who was listed when.

    A stock whose price ends a month below `floor` is delisted at that month's end. Following
    CRSP's convention, that month's recorded return compounds the month's trading return with
    the delisting return (what the shares fetch after leaving the exchange). None means the
    database has no delisting return: the month shows only the trading return, and the loss on
    exit is missing (Shumway's problem with early CRSP data).

    Returns (rec, alive, survivor): `alive[t, i]` is True if stock i is listed at the start of
    month t; `rec[t, i]` is its return that month (NaN once it is gone); `survivor[i]` is True
    if the stock is still listed at the end.
    """
    months, n = r.shape
    price = np.ones(n)
    alive = np.zeros((months, n), dtype=bool)
    rec = np.full((months, n), np.nan)
    live = np.ones(n, dtype=bool)
    for t in range(months):
        alive[t] = live
        rec[t, live] = r[t, live]
        price[live] = price[live] * (1.0 + r[t, live])
        out = live & (price < floor)
        if delist_return is not None:
            rec[t, out] = (1.0 + r[t, out]) * (1.0 + delist_return) - 1.0
        live = live & ~out
    return rec, alive, live

Identifiers

A related error is quieter. Tickers change when companies rename or move exchange, and a ticker freed by a delisting is reassigned to another company. A history joined on ticker can splice two companies into one series or lose a company halfway. Key every series by a permanent identifier (CRSP’s PERMNO, or a vendor’s own), map tickers to it by date, and take the universe as it stood on each rebalancing date from a source that keeps dead securities and their delisting returns. The same logic applies to futures markets that were discontinued, funds that closed and indices whose rules changed.

04Corporate actions and adjusted prices

Prices on the tape are quoted in the shares of the day. When a company splits its stock four for one, the price falls to a quarter and the holder owns four times as many shares. When it pays a dividend, the price drops by about the dividend on the ex-date, which fixes who is entitled; the cash arrives on the payment date, often weeks later. A raw price series treats both as losses. Adjusted series correct them by scaling every earlier price, and data vendors publish them in two common forms: split-adjusted, and total-return adjusted, which adjusts for dividends as well.

For a split of ratio k on day e, prices before e are divided by k. For a cash dividend D going ex on day e, prices before e are multiplied by

where P_{e − 1} is the raw close before the ex-date. The cumulative factor for any day is the product over every event after it.

An adjusted history is therefore a function of the date it is read. Each new dividend or split lowers every price before it. In the synthetic stock below, the first close printed at 180.00; the total-return series read a year later shows 176.99, and read at the end shows 41.17. Returns computed from the series do not change; price levels do. A rule written in price levels (“buy below 100”, “stocks above 5 dollars”) tested on today’s adjusted history uses information from the future.

The adjusted return on an ex-date is P_e / (P_{e − 1} − D) − 1, while the holder’s return is (P_e + D) / P_{e − 1} − 1. They differ by a second-order term, about the dividend yield times the day’s return: in the synthetic stock by 0.6 bp on an average ex-date and at most 2.1 bp. Over five years the total-return series compounds to 57.90% against an exact 57.94%. For precise work, compute returns from raw prices and the cash flows, as CRSP does for its holding-period return. Both forms assume the gross dividend is reinvested; an investor who suffers withholding tax, or holds the stock in another currency, earns something else, and the backtest should say which.

def factors(raw, div, split, asof: int):
    """The cumulative adjustment factors as a vendor computes them on day `asof`: for each day,
    the product over events AFTER it (and no later than `asof`) of 1/ratio for a split and
    1 - D / previous close for a dividend. Events after `asof` have not happened yet."""
    n = len(raw)
    fs = np.ones(n)            # splits only
    fd = np.ones(n)            # dividends only
    for e in range(asof, 0, -1):          # walk back from the reading date
        fs[e - 1] = fs[e] / split[e]
        fd[e - 1] = fd[e] * (1.0 - div[e] / raw[e - 1]) if div[e] > 0 else fd[e]
    return fs, fd

On raw prices the split shows up as a one-day return of −74.4%; the holder actually made 2.3%. A 50-day moving-average rule run on raw and on adjusted prices disagrees on 64 sessions, 50 of them in the 60 sessions after the split, when the raw series sits far below its own average. Over the whole sample the raw price “returned” −63.9%, the split-adjusted price 44.5% and the holder 57.9%.

Backtesting engines make this choice for you unless you make it yourself. LEAN, for example, feeds split- and dividend-adjusted prices by default (its Adjusted mode) and lets each subscription choose another, including Raw and SplitAdjusted; a history request can also use ScaledRaw, which adjusts prices only for the events known at the algorithm’s current time. Its TotalReturn mode applies splits and then adds the accumulated dividends to the price instead of scaling earlier prices for them, so it is a different series from the one called total-return adjusted here.^12,13 Know which one your engine uses before you read a fill price.

05Continuous futures and the roll

A futures contract expires, so a history of “the” futures price has to be assembled from a chain of contracts. A trader holding the front contract sells it some days before expiry and buys the next: the roll. On the roll date the two contracts trade at different prices, and the difference is carry: financing and storage, less dividends or convenience yield, depending on the market. When later contracts are dearer the curve is in contango; when they are cheaper, in backwardation. Three constructions are common, and they disagree:

Nothing is lost at the roll itself: one contract is exchanged for another at market prices. At the laboratory’s default carry of +5% a year, the loss in the readout above accrues day by day as each contract, priced above spot, drifts down to it at expiry. When the carry is the financing rate, as here, interest on the collateral earns back roughly what the roll-down costs: the collateralised position tracks spot plus interest minus carry, which here is about spot, while the continuous contract on its own, an excess return, tracks spot minus carry. The ratio-adjusted series gets the excess return right to the last digit. The difference-adjusted series, read as a percentage, says −35.5%, because in contango each gap is added to every earlier price, the early prices are inflated, and every percentage computed from them is too small. In backwardation the same construction goes the other way: at −5% the difference-adjusted history bottoms at −13.9, a negative price for an asset that never traded below 45.

def continuous(carry: float = 0.05, offset: int = 5, years: int = YEARS) -> dict:
    """The unadjusted, difference-adjusted and ratio-adjusted series, and the truth they are
    judged against: a one-contract position's P&L, and the growth of a fully collateralised
    position (its excess return: no interest on the collateral).

    On a roll day the series still shows the old contract; the new one takes over the next day.
    Each roll's gap (new minus old, or new over old, at the roll's close) is applied to that
    day and every day before it."""
    spot, today, prev, rolls = chain(carry, offset, years)
    n = len(spot)
    gap_add, gap_mul = np.zeros(n), np.ones(n)
    for d, old, new in rolls:
        gap_add[d], gap_mul[d] = new - old, new / old
    add = np.cumsum(gap_add[::-1])[::-1]          # the sum of the gaps on or after each day
    mul = np.cumprod(gap_mul[::-1])[::-1]
    pnl = np.r_[0.0, np.cumsum(today[1:] - prev[1:])]
    growth = np.r_[1.0, np.cumprod(today[1:] / prev[1:])]
    return {"spot": spot, "unadjusted": today, "difference": today + add, "ratio": today * mul,
            "pnl": pnl, "growth": growth, "roll_days": [d for d, _, _ in rolls]}

We use ratio-adjusted series for signals and returns, difference-adjusted series for P&L in points, and the raw contract’s price to trade and size. The roll rule belongs in the backtest: the date (a fixed number of sessions before expiry, before first notice day for physically delivered contracts, or when open interest moves to the next contract), the price at which both legs trade, and the cost of the spread trade. The daily “close” of a future is usually the exchange’s settlement price, set by its own procedure and published after the session, which can differ from the last trade. A continuous series built with one roll rule and a strategy traded with another will not reconcile.

06Time zones and session calendars

Exchanges set their hours in local time, and local time moves against UTC when daylight saving starts and ends. The United States moves its clocks on the second Sunday of March and the first Sunday of November;¹⁴ the European Union, and the United Kingdom under its own law, on the last Sundays of March and October.^15,16 For a few weeks a year New York is four hours behind London instead of five. In 2025, the London and New York sessions overlapped for two hours on most weekdays and for three hours from 10 to 28 March and from 27 to 31 October.

Store every timestamp in UTC with its source’s time zone recorded, and convert to local time only to apply an exchange’s rules. Use the IANA database through the language’s own library, never a fixed offset.¹⁷ Key daily data by the session it belongs to, not the calendar date of its timestamp: many futures sessions open on the evening before their trade date, and a daily bar stamped at midnight UTC can belong to either day. Keep an exchange calendar with holidays and shortened sessions, and treat a bar on a day the exchange was closed as an error.

def utc_hours(day: date, zone: str, local: time) -> float:
    """The UTC clock time, in hours, of a local time on `day`."""
    t = datetime.combine(day, local, tzinfo=ZoneInfo(zone)).astimezone(UTC)
    return t.hour + t.minute / 60

Signals that combine markets need the same care. A close in Tokyo is known hours before the close in New York; a signal that uses both “closes of day t” is fine for a New York trade and a look-ahead for a Tokyo one. Write down, for every input, the UTC time at which it becomes available, and join on that.

07Quality checks

Every delivery goes through the same checks before anything reads it. Each check is a small function, tested against the fault it is meant to catch. Two standard references on tick data, by Brownlees and Gallo and by Barndorff-Nielsen and colleagues, describe cleaning rules that follow the same logic at a finer scale: remove what is impossible, then compare each print with its neighbours.^18,19

The spike check is the one that needs thought. A 20% fall is suspect in a quiet stock on a quiet day and unremarkable on a day the whole market fell 20%. So the check compares each return with a reference, here the market with a beta of one, and scales the residual by the median absolute deviation of recent residuals, which a single bad print cannot inflate. A flagged move that reverses the next session is a bad print. One that stays, at a split ratio relative to the market, is an unadjusted split if the corporate-actions file has one that day and a move for review if it does not. On the day a bar arrives its reversal cannot yet be seen, so a live check quarantines the bar and classifies it a session later; an earnings calendar removes most of the false alarms on single stocks.

def check_spikes(bars: pd.DataFrame, ref: pd.Series, actions: dict = None, z: float = 8.0, window: int = 63) -> pd.DataFrame:
    """Returns that the market does not explain.

    The residual is the bar's return minus the reference's over the same interval (a beta of
    one; use a rolling beta or a sector reference for stocks far from it). Its scale is robust:
    1.4826 times the median absolute deviation of recent residuals from their rolling median,
    so one bad print cannot widen it. A residual beyond `z` scales is flagged and classified:
      - it reverses the next session (in log terms, by more than half): a bad print;
      - it stays, and the move relative to the reference is a split ratio: an unadjusted split
        if the corporate-actions file (`actions`, date -> ratio) has one that day, otherwise a
        real move at a suspicious size, for review;
      - anything else: a jump for a person to look at.
    On the latest bar the reversal cannot be seen yet, so the flag says so and the bar stays in
    quarantine until the next session. The first `window` sessions or so have no scale and are
    not checked: a new listing needs a cross-sectional scale instead."""
    actions = actions or {}
    close = bars["close"].where(bars["close"] > 0)        # a non-positive price is ohlc's to report
    r = close.pct_change(fill_method=None)
    rr = ref.reindex(close.index).pct_change(fill_method=None)
    e = r - rr
    m = window // 2
    dev = (e - e.rolling(window, min_periods=m).median()).abs()
    scale = 1.4826 * dev.rolling(window, min_periods=m).median().shift(1)
    lr = np.log1p(r)
    rows, skip = [], -1
    for i in np.flatnonzero((e.abs() > z * scale).to_numpy()):
        if i == skip:                                      # the day a bad print reverted
            continue
        d = close.index[i]
        rel = (1 + r.iloc[i]) / (1 + rr.iloc[i])           # the move the market does not explain
        at_ratio = min(abs(np.log(rel / q)) for q in SPLIT_RATIOS) < 3 * scale.iloc[i]
        if i + 1 >= len(e) or not np.isfinite(lr.iloc[i + 1]):
            kind = "unconfirmed until the next session: quarantine"
        elif np.sign(lr.iloc[i + 1]) == -np.sign(lr.iloc[i]) and abs(lr.iloc[i + 1]) > 0.5 * abs(lr.iloc[i]):
            kind, skip = "spike that reverses: a bad print", i + 1
        elif at_ratio and d in actions:
            kind = "unadjusted split: one is on the corporate-actions file"
        elif at_ratio:
            kind = "move at a split ratio, none on file: review"
        else:
            kind = "unexplained jump: review"
        rows.append((d, "spike", kind))
    return pd.DataFrame(rows, columns=["date", "check", "detail"])


def run_checks(bars: pd.DataFrame, ref: pd.Series, sessions: pd.DatetimeIndex, actions: dict = None) -> pd.DataFrame:
    """The suite. Duplicates are reported by the calendar check and then dropped, keeping the
    later delivery (a resent bar is usually a correction), so the price checks see one bar per
    timestamp."""
    once = bars[~bars.index.duplicated(keep="last")].sort_index()
    out = [check_calendar(bars, sessions), check_ohlc(once), check_stale(once), check_spikes(once, ref, actions)]
    return pd.concat(out, ignore_index=True).sort_values(["date", "check"], ignore_index=True)

Missing data in pandas

08When vendors disagree

Two vendors rarely agree on every number, and the differences have causes that can be listed. Ince and Porter compared Datastream’s US equity returns with CRSP’s and found enough discrepancies, in coverage, in rounding and in the treatment of dead and foreign securities, to change the results of standard studies unless the data were screened first.²⁰

Historical bars change after the fact as well: late prints are added, cancelled trades removed, closes corrected. The history a backtest reads next month is not what the live system saw today. Keep what the live feed actually delivered in the raw store, and treat the vendor’s later history as a second source to reconcile against.

Before switching vendors, or adding one, reconcile the two on the overlapping history: returns day by day, corporate actions event by event, and the universe on each date. Explain every difference larger than rounding. The same exercise, run between a backtest and a live account, is the subject of Live versus backtest reconciliation.

With the data in order, the next step is to turn an idea into a rule and test it: Indicators and the 200-day moving average.