The Historical Surface

Concept reference

This page renders the bounded-termination historical contract for Kairos site readers. Implementation details live with the engine and are not part of the public client contract.

Context

The same analytics need to run on three ingest sources behind one analytic pipeline:

1. Live — the upstream live feed via Client::connect.
2. Historical — internally downloaded from the upstream historical service for a given date range. Institutional users replay last week's option flow before opening their book on Monday; quants regression-test model changes against a year of intraday Greeks; risk runs end-of-day reports against a synthetic intraday volatility surface.
3. User-supplied datasets — pre-fetched trade + quote ticks staged on the engine link. Required for back-tests where the user already holds the data, for replay against vendor-recorded fills, and for offline correctness regressions.

The analytic outputs stay bit-identical across modes. An analytic that fires on a live upstream quote and the same analytic fired on the same (date, ms_of_day, bid, ask, ...) replayed from the upstream historical service or from a recorded dataset MUST produce identical output.

Surface

1. Client shape — mode namespaces on Client

The thin client exposes three mode namespaces. Each returns the SAME per-analytic builder accessors; the underlying engine call differs by mode:

impl Client {
    pub fn live(&self) -> Scope<'_>;

    #[cfg(feature = "historical-mode")]
    pub fn historical(&self, start: i32, end: i32) -> Scope<'_>;

    #[cfg(feature = "replay-mode")]
    pub fn replay(&self) -> Scope<'_>;
}

impl<'a> Scope<'a> {
    pub fn volatility(&self) -> AnalyticBuilder<'a, VolatilityRequest>;
    pub fn greeks(&self)     -> AnalyticBuilder<'a, GreeksRequest>;
    pub fn gex(&self)        -> AnalyticBuilder<'a, GexRequest>;
    // … one per-analytic accessor …
}

The historical and replay namespaces are gated behind the matching Cargo features (historical-mode / replay-mode). They remain preview while the C-ABI staging entries land.

2. Terminal verb — .on_event in every mode

use kairos::GreeksRow;

// Live — unbounded; the callback fires until you unsubscribe.
client.live().greeks(["QQQ"])
    .on_event(|row: &GreeksRow| { /* … */ })?;

// Historical — bounded over the `YYYYMMDD` date range; the callback
// fires once per row, then the stream ends.
client
    .historical(20251103, 20251107)
    .greeks(["QQQ"])
    .on_event(|row: &GreeksRow| { /* … */ })?;

// Replay — bounded over the engine-staged dataset.
client.replay().greeks(["QQQ"])
    .on_event(|row: &GreeksRow| { /* … */ })?;

start / end are YYYYMMDD-encoded i32s per the engine's integer-time convention. .on_event is the terminal verb in every mode; it returns an EventSubscription handle. The only difference is lifetime — on the bounded historical / replay namespaces the stream ends after the last row and the handle goes inactive (sub.is_active() returns false); a live subscription runs until you unsubscribe().

3. Synthetic watermark — derived from tick event-time

Live mode advances watermarks on a fixed interval. Historical mode replaces that interval with a tick-coupled advance:

• The historical orchestrator downloads one (contract, date) batch at a time.
• It walks the tick stream in source order and feeds each quote / trade event into the engine.
• Every N ticks (default 256) and at every source-data minute boundary the engine advances the watermark to the most recent tick's (date, ms_of_day).
• At the end of each unit the engine emits a final watermark at (date, end_of_session_ms) to flush any window still open at the close.

The minute-boundary advance is the load-bearing rule: a worst-case 256-tick gap inside a minute cannot delay an OHLCVC bar flush past the boundary.

4. Dataset replay surface

The replay dataset is staged on the engine link. The thin client does not marshal a dataset over the wire; the FFI staging entry arms the dataset on the client handle, and the in-tree test link owns it directly. From the caller's point of view, the dataset is provisioned out of band; the public client call is just client.replay().<analytic>([...]).on_event(|row| …).

The same synthetic-watermark contract applies: the engine walks the dataset in event-time order and synthesises watermarks exactly the same way. Replay is byte-equivalent to historical except the source is the user's staged dataset instead of an upstream historical download.

5. Progress reporting

The thin client surfaces progress through the callback itself — each emission is one row delivered to .on_event. A consumer can drive any progress UI (CLI bar, web-UI WebSocket push, Prometheus gauge, log lines) from inside the callback without engine-side hooks, and watch sub.is_active() to detect completion on a bounded run.

6. Upstream endpoint selection and concurrency

The engine resolves the contract filter against its symbol classifier to produce the concrete contract list and fans out the historical download per (contract, date). The per-contract concurrency cap is an engine config knob; the public client surfaces the bounded stream uniformly.

7. Output shape — same typed rows

The .on_event callback and the typed <Camel>Row it receives are the same in every mode. The unbounded live case runs until you unsubscribe(); the bounded historical / replay cases end after the last row and mark the handle inactive.

8. Backpressure differs by mode

• Live — drop-oldest if the consumer falls behind. The live feed cannot be slowed without losing the connection's reconnect-replay window. If a callback consumer falls far enough behind that the buffer overflows, the subscription ends rather than skipping frames silently, and its handle goes inactive.
• Historical / Replay — block the producer. The bounded stream suspends until the callback keeps up.

9. Tier checks — per-asset historical entitlement

The upstream account model carries separate streaming and historical entitlements per asset class. The historical() namespace verifies the per-asset historical tier before issuing the first upstream call. A user with streaming Pro / historical Standard sees Free-tier rejections from historical(...).<analytic>(...).on_event(…) while the live builder (client.live().<analytic>([...]).on_event(…)) continues to work.

The replay path (client.replay()) does not check entitlement: the dataset came from the user, the engine is not the gatekeeper.

10. Memory — one (contract, date) batch in flight per concurrent unit

The historical orchestrator fans out one (contract, date) unit at a time per concurrent unit; the per-unit watermark publisher streams ticks into the engine as they arrive rather than buffering the whole unit. The bounded peak memory tracks the engine's per-unit concurrency cap and the typical per-day tick count for the contract class — the public client does not own a buffer that grows with the unit count.

Surface trade-offs

Strengths

• One client, one per-analytic builder shape, three ingest sources.
• The same source-order processing holds across all three sources — every event reaches the analytic in source order.
• The mode is opt-in via Cargo feature; production FFI builds that do not enable the preview features cannot reach the unwired paths.
• Replay-from-staged-dataset gives correctness regressions a determinism harness: stage the same dataset twice, expect bit-identical output.

Trade-offs

• The synthetic-watermark contract is enforced engine-side, not by the public type system. A future replay surface that exposed a custom iterator constructor would need to enforce ordering separately.
• Multi-day historical queries fan out per (contract, date). The upstream historical endpoint set drives this shape.
• Consumption is uniform across modes: .on_event fires per row in every namespace. A bounded caller that wants the rows collected gathers them inside the callback (push onto a Vec) and watches sub.is_active() for completion.
• The replay() namespace bypasses tier checks. A user with a Free account cannot drive the live builder (client.live().<analytic>()...) or the historical namespace but can still drive replay() with their own staged data. This is correct behaviour.

Industry comparables

• databento-rs — split client design with separate live::Client and historical::Client. The namespace design here preserves the same downstream call-site uniformity without splitting client identity.
• Bloomberg BLPAPI — single Session exposing both subscribe(...) (real-time) and sendRequest(...) (request / reply). The namespace design mirrors the single-session model.
• Refinitiv RTSDK — same data types across modes. The mode-agnostic data type contract is what this surface commits to: the .on_event callback and its typed rows are the same regardless of which namespace produced them.