Appearance
The Tick Parquet Schema
Context
The engine's replay surface accepts on-disk parquet files matching the schema defined here. The schema is closed at the shipping release; readers ignore unknown columns so future extensions land without breaking earlier readers.
On the public Rust thin client the replay namespace (client.replay(), gated behind the replay-mode Cargo feature) runs the engine's parquet reader against a dataset staged on the engine link. Callers do not construct a dataset value through the public API; the dataset is provisioned out of band.
Decision
1. Library choice — arrow-rs + parquet (Apache official)
The Apache Foundation publishes arrow-rs + parquet on crates.io under the apache-arrow GitHub organisation. The crates are the canonical Rust implementation of the Arrow + Parquet specs and the only ones the upstream Arrow community supports.
The alternative arrow2 / parquet2 ecosystem (an unaffiliated fork, archived 2025) is rejected: archived upstream, no security advisory channel, no further releases. Pinning to an archived dependency on a financial-engine production codebase is unacceptable.
2. Schema — row-typed with tick_kind discriminator
One columnar RecordBatch carries both TickEvent::Quote and TickEvent::Trade rows. A tick_kind: u8 discriminator column distinguishes the two:
tick_kind == 0→ quote row; quote-side columns populated, trade-side columns nullable and unused.tick_kind == 1→ trade row; trade-side columns populated, quote-side columns nullable and unused.
The full column list (Arrow types in parentheses):
| Column | Type | Description |
|---|---|---|
tick_kind | UInt8 | 0 quote / 1 trade |
contract_id | Int32 | Upstream wire id (0 on replay) |
symbol | Utf8 | Underlying symbol |
sec_type | UInt8 | SecType discriminator (0 Stock, 1 Option, 2 Index, 3 Rate, 255 Unknown) |
expiration | Int32 (nullable) | YYYYMMDD option expiration; null for non-options |
strike | Int32 (nullable) | dollars × 1000 option strike; null for non-options |
is_call | Boolean (nullable) | true call / false put; null for non-options |
date | Int32 | Trading-session date YYYYMMDD |
ms_of_day | Int32 | Milliseconds since midnight ET |
bid / bid_size / bid_exchange / bid_condition | Float64 / Int32 / Int32 / Int32 (nullable) | Quote bid side |
ask / ask_size / ask_exchange / ask_condition | Float64 / Int32 / Int32 / Int32 (nullable) | Quote ask side |
received_at_ns | UInt64 | Engine-side ingest timestamp (0 on replay) |
price / size | Float64 / Int32 (nullable) | Trade price + size |
exchange | Int32 (nullable) | Trade exchange |
condition | Int32 (nullable) | Primary OPRA condition |
ext_cond_1 / ext_cond_2 / ext_cond_3 / ext_cond_4 | Int32 (nullable) | OPRA extended conditions |
sequence | Int32 (nullable) | Per-exchange wire sequence |
volume_type | Int32 (nullable) | Upstream volume type |
records_back | Int32 (nullable) | Out-of-sequence correction depth |
price_flags / condition_flags | Int32 / Int32 (nullable) | Trade flags |
Adding columns is a non-breaking extension (readers ignore unknown columns); removing columns is a breaking change that requires a follow-up schema decision.
3. File-naming convention (informational)
One parquet file per (symbol, date) unit, named <symbol>_<yyyymmdd>.parquet (e.g. AAPL_20260506.parquet). The convention is informational — from_parquet(path) accepts any parquet file matching the schema regardless of naming.
4. Ordering — sort-merged on (date, ms_of_day)
from_parquet decodes every row independently and sort-merges the resulting (QuoteTick, TradeTick) lists by (date, ms_of_day) per the synthetic-watermark cadence (anchored on source-data event-time and requiring non-decreasing tick timestamps).
5. Cycle metadata is reset
QuoteTick::cycle_label, cycle_provisional, cycle_emit_seq are populated by the live cycle tracker on first emission and are reset on parquet load (PreTrade / false / 0). Replay analytics that consume cycle-aware fields run their own cycle tracker on top of the loaded ticks; the parquet file does not preserve the original cycle stamps.
Alternatives considered
arrow2/parquet2(unaffiliated fork). Rejected — archived 2025.- Row-typed: separate quote / trade
RecordBatches in one file. Rejected. Each batch carries its own schema and one parquet column subset; loading requires multiplexing across batches in event-time order — extra complexity for no win over a single batch with a discriminator column. - Two separate files per unit (
*_quotes.parquet+*_trades.parquet). Rejected. Doubles per-unit IO; the sort-merge would have to seek between two files for every emit, breaking parquet's strength (sequential column scan). - Strict columnar with
sec_type-specific schemas. Rejected. A schema-per-asset-class table would force callers to pre-bucket their data and load each bucket separately.
Consequences
Positive
- One call site. Users with parquet bind their file on the replay chain and drive it through
client.replay().<analytic>([...]).from_parquet(path).on_event(cb)directly. - Closed schema. The column list is fixed at the shipping release; readers ignore unknown columns so future extensions land without breaking earlier readers.
- Industry vocabulary. Apache Arrow / Parquet is the canonical institutional file format; databento, BLPAPI, and Refinitiv RTSDK all accept parquet as a first-class playback source.
Negative
- Dependency closure grows by ~2–3 MB compiled. The two crates carry a substantial transitive set (
flatbuffers,chrono,zstd/snap/lz4codecs). Mitigated by selecting only the parquet features the schema needs. - Schema is closed at the shipping release. Adding new tick variants (
OpenInterest,VendorOhlcvc) requires either an extension column or a major-version schema bump. - Cycle metadata is reset on load. Replay analytics that depend on the cycle tracker's full output must run their own cycle tracker on top of the loaded ticks.
Industry comparables
- databento-rs —
zstd-compressed parquet playback files. - Bloomberg BLPAPI —
bpiperecording format. - Refinitiv RTSDK —
rkfxrecording format.