TokSeq documentation/Methodology

Research methodology

Pre-registration, purged walk-forward validation, multiple-comparison control, and the acceptance gate a finding has to clear before it is called an edge.

The research loop is deliberately hostile to its own findings. The platform generates tens of thousands of candidate configurations; at that scale the best result of a pure noise process is a large number, and any method that does not price the search itself will produce a confident, reproducible, worthless answer. Most of the engineering below exists to price the search.

The loop

Hypotheses are declared, either by hand or by an algorithmic generator, then fingerprinted and deduplicated. A budget gate prices each one in compute. The whole population is pre-generated, filtered for feasibility, and only then executed by a parallel executor. Planning does not run concurrently with execution, so the result set is independent of the order in which it was produced.

Execution builds features, labels regimes, runs purged walk-forward validation, scores the result, and applies the significance gate. Results are written whether they are positive or negative. A validation pass then attacks the survivors with ablation, robustness, leakage detection, feature importance and combinatorial cross-validation. Surviving signals seed the next generation of hypotheses.

Between one and two per cent of the pre-generated population are placebo configurations whose labels are known to carry no information. They are not a formality: they are the empirical monitor of the pipeline's own false-positive rate. If placebos start passing, the pipeline is broken, regardless of what the real hypotheses say.

Pre-registration

Before a run starts, the platform writes down what would count as a finding: the effective sample size computed over non-overlapping blocks, the multiplicity correction for the declared number of trials, and the threshold the metric must clear. That record cannot be overwritten. A finding produced without a matching pre-registration record is annulled – not argued about.

Every pre-registration carries two controls alongside the hypothesis: a directional control whose expected discrimination is confined to a narrow band around chance, and a placebo control built by shuffling features in day-long blocks.

Validation

• Purged walk-forward. Splits are chronological. A label spanning a horizon contaminates the end of the training window, so that tail is purged, and an embargo scaled to the horizon separates train from test. A fold with too few surviving rows is skipped; if no valid fold remains, the run fails rather than reporting on a degenerate split.
• Overlapping labels. Nominal sample count is never used for inference. Effective sample size divides by the horizon, because a five-minute cadence with a four-hour label produces rows that are anything but independent.
• Combinatorial cross-validation and the probability of backtest overfitting. Performance is evaluated across all balanced train-test partitions of the block matrix; the fraction of partitions where the selected configuration ranks below median out of sample is the overfitting probability. Pure noise gives about one half; a real effect drives it toward zero.
• Costs. Every result is net of costs. Taker fees and slippage are charged against turnover, and the position construction includes a dead zone and smoothing so that only an edge which survives its own turnover survives at all.
• Leakage. A non-physical Sharpe, a shuffle that does not destroy the result, or an embargo stress test that does not degrade it, all raise a leakage flag on the run.

Multiple-comparison control

Multiplicity is counted inside a declared family – volatility, regime, direction, conditional direction, positioning, seasonality, inversion, volatility premium – rather than across the whole population, and the correction uses the number of hypotheses rather than the number of variants. Assets are pooled by default, so that the same hypothesis measured on three correlated assets counts once.

The search stage and the confirmation stage use different machinery on purpose:

• Search uses false-discovery-rate control with a block bootstrap across assets, blocked by day, and an effective count of independent trials derived from a correlation clustering of the candidate space. Here the deflated Sharpe ratio is a ranking component, not a gate.
• Confirmation uses pre-registration, a correction against the number of candidates that entered, forward evaluation with confidence sequences, and e-value evidence accumulation. Here the deflated Sharpe ratio is a hard gate.

The choice of correction was itself measured rather than assumed. A synthetic benchmark of 720 trials over 250 daily blocks, with realistic correlation between variants and sixty repetitions per scenario, was run across eight regimes of effect size, calibrated to the effect sizes the platform actually finds. The variant originally planned – false-discovery control with an estimated null proportion – failed to hold the error rate in five of eight scenarios, with a worst-case false discovery proportion of 0.333. The plain variant held control in all eight scenarios at a worst case of 0.117, at the cost of power, and was adopted on that evidence.

The acceptance gate

A result is called significant only when all three of the following hold simultaneously:

1. A permutation test breaking the link between forecast and forward return rejects at the declared level.
2. The lower bound of a block-bootstrap confidence interval on the Sharpe ratio is above zero.
3. The deflated Sharpe ratio clears its threshold, where the deflation prices the number of trials, the sample length, and the skew and kurtosis of the return distribution, using effective sample size for overlapping labels.

Hyperparameter search is not free: the number of tuning trials inflates the trial count that deflation is computed against.

Verdicts are discrete – edge, fragile edge, needs validation, no edge, regime detector – and the default is not "promising". It is "insufficient evidence".

Sample size is declared in advance, not discovered

Minimum sample size is treated as a rate per unit of time rather than an absolute count, because an absolute floor transplanted into a shorter window silently permits empty cells in every fold. Where a forward window is required, its power is stated before it starts: for a desk whose daily result has a standard deviation in a stated range, a thirty-day window can distinguish a mean daily effect of a stated size and nothing smaller. That sentence is written down before the window opens, so that a null result at the end is a result and not a disappointment.

Reproducibility

Every run records the random seed and the hashes of the dataset, the feature set, the configuration and the model. Research is offline: experiments never pull live data, they read a frozen store. Profit and loss is always net of cost. And nothing is asserted without the sample it rests on.