TokSeq documentation/Data

Data acquisition

What is collected, at what granularity, how much of it there is, and the quality controls that decide whether a row is allowed to exist at all.

Everything the platform claims rests on data it collected itself. No third-party dumps are ever merged into the source of truth. The provenance rule is clean-room: every row originates from the platform's own poll of a documented public API, performed by the platform's own code, with the response timestamped on arrival.

What is collected

Forty-five distinct streams are running. They fall into seven families.

• Microstructure – trade-by-trade tape, order book depth aggregates, liquidation events and derived order-flow aggregates for the three primary perpetual-futures assets.
• Price history – OHLCV for perpetual and spot instruments across eight timeframes, from one minute to one day, plus open interest across six timeframes and the funding series.
• Volatility and options – implied volatility indices, at-the-money implied volatility, risk reversals, skew and term-structure slope, and full option chains by strike with mark implied volatility, open interest, volume and greeks.
• Cross-asset and macro – equity index and dollar-index proxies, a gold proxy, and a set of macro series covering policy rates, the yield curve, money supply and central bank balance sheet.
• On-chain – active addresses, transaction count and volume, hashrate, difficulty, fees and mempool state, plus stablecoin supply and exchange net flow computed from a registry of exchange addresses built from the exchanges' own disclosures.
• Positioning and cross-venue derivatives – open interest and funding compared across several venues, long-short ratios, taker volume balance, insurance fund levels and public large-holder positions.
• Narrative and events – fourteen news feeds across crypto, policy and macro, a global event database, and a curated economic calendar of scheduled releases.

Two barrier-ladder prediction-market sources are also collected, as an implied distribution of market expectations rather than as a price series.

Granularity

The microstructure layer samples once per second, offset from the second boundary, and aggregates into one, five, fifteen and sixty-second tiers. The trade tape and liquidation events are stored raw, one row per event. Book state is stored as a forty-bucket depth heatmap around the mid, with a per-asset band width, in two time tiers.

The polling layer runs at cadences declared per source: ninety seconds for the fast market cycle, roughly fifteen minutes for the full timeframe set, five minutes for volatility and positioning snapshots, ten minutes for news, an hour for the slower on-chain and supply series.

One rule is worth stating because it is unusual: a second-level row is written only if that second was actually observed. A second during which the socket said nothing produces no row at all, rather than a row of zeros. Gaps are never interpolated, anywhere in the platform, at any layer.

What is deliberately not collected

• Raw level-by-level order book with queue positions. The exchange cap makes the marginal coverage gain small, and taking it would cost a five-fold degradation in sampling cadence. The trade-off was measured and declined.
• Full differential depth streams, whose bandwidth cost was measured at over three gigabytes per day per venue for a use case that has no confirmed consumer.
• Any source requiring the scraping of a third party's rendered page, and any source whose terms forbid derived public use.
• Paid data feeds. When a keyless public endpoint degrades, the stream stops and is documented as stopped. It is not replaced by a purchase.

Volume and depth

Coverage is asymmetric by design. Second-level microstructure is collected for three primary assets, because a socket per instrument at a one-second cadence is the most expensive stream in the system and the research it feeds needs depth rather than breadth. Price history, derivatives context, macro and on-chain series cover a wider set. That base is now being extended: the instrument screen is being widened after a pre-registered transfer probe, and a second market with its own option series is being added for the volatility program. Storage was sized for that expansion in advance – about ten additional gigabytes a year against 86 free.

Storage is partitioned by day in a columnar format with compression, with per-stream retention declared in configuration: two years for order-flow aggregates and liquidations, four hundred days for the raw trade tape with an emergency reduction under disk pressure, and keep-all for news and snapshot series.

Quality control

Data quality is not a review step, it is a set of refusals built into the write path.

• Schema is verified on every write. A mismatch refuses the write rather than coercing the row.
• Each row carries two timestamps: the event time reported by the source, and the receive time stamped when the response arrived. Partitioning uses event time; joins and replays use receive time. This is what makes point-in-time reconstruction honest, and a clock-skew measurement confirmed there is no host clock offset, while the real hazard – publication lag – reaches a 95th percentile of several minutes on some slow endpoints.
• Deduplication uses a declared natural key per source, not a single timestamp. This rule exists because a timestamp-only deduplication once reduced a nine-row cross-venue update to one row and a seven-strike ladder to one strike, while reporting success.
• Four independent detectors for data that is silently frozen: consecutive successful-but-empty polls, a minimum coverage floor for composite sources, wallet-rotation detection for address-based streams, and a freshness bound per dataset derived from its own cadence.
• Writes are atomic through a temporary file, an fsync and a rename. An unreadable batch is quarantined rather than deleted, and raises an error with a count in the status file.
• Unformed bars are dropped before they reach the lake; on a timestamp collision the fresher version wins.

Known limitations

The platform documents its own data defects publicly to itself rather than fixing history retroactively, because a retroactive fix is indistinguishable from a fabrication:

• The liquidation side field carries the side of the liquidated position, which is the inverse of the naive reading. This was established empirically, not assumed, and the semantics differ between venues – two major venues use opposite conventions, and the price column means different things in each.
• Before a fix in late July 2026, roughly eighteen per cent of seconds were lost, and the cumulative volume delta series contained thirteen resets over 459 550 rows.
• A host clock jump of forty-nine seconds means cross-stream rows before that date require an embargo to be usable.
• Several composite sources needed key normalization before their history became comparable, and the part of the history preceding that normalization is not recoverable.

Each of these is recorded with its date, its magnitude and its effect on downstream use.