Does a bigger model trade better?
The finding: across 95 machine-learning experiments, a higher accuracy score did not reliably mean better simulated trading once costs were charged. Simpler models held up.
One recipe, one score, two open questions
The club brief was to rebuild one published model — DeepLOB — and report how accurate it is. A fair brief. But it leaves open two questions a trader would ask first:
- Is all of that machinery needed, or would something smaller do the same job?
- Does a good accuracy score turn into good trading once every trade costs money?
The brief did not ask for answers. I made them the project.
The old way: one published recipe, rebuilt as-is, judged by one accuracy number. Dashed = the questions the brief left open.
A test bench, not a bigger model
Instead of one model, a bench that trains and scores many. Every model gets the same data and the same question: over the next 10 to 100 order-book updates, does the price go up, down or stay flat? Each run is a settings file, queued on the Amarel cluster and saved with its results; a second scorecard then charges every simulated trade its real cost.
Average accuracy across the five horizons
documented averages from the experiment log · higher is better
Test accuracy, % · sky = deep models · grey = classical baselines · dashed = the rebuilt recipe · one seed, one dataset · the rebuild sits below the paper’s own numbers (gap documented in the report).
Same models, horizon by horizon
does more complexity help? · redrawn from the experiment figures
Test accuracy, % per horizon (k = order-book updates ahead) · same values as the project’s figure, redrawn · the simplest model is never far behind.
A better score did not reliably mean better trading
The second scorecard turns each model’s predictions into a simple trading rule and charges half the bid–ask spread every time the position changes — the price of changing your mind. With that cost charged, the accuracy lead did not carry over consistently. In the exploratory runs, what moved the simulated result was trading less: holding longer, acting only when several horizons agreed, sitting out volatile moments.
Simulated on one dataset with one random seed: a research finding.
schematic · not to scale
What a person still decides
The bench produces numbers; it does not say which ones matter. A person still decides whether a small gain in accuracy is worth a bigger, slower model, whether a result that only looks good in simulation deserves belief, and which negative results go into the report. Here that person was me: I kept the simpler models on the table, charged the costs, and wrote up what did not flatter the biggest model.
Removed: guesswork
Whether extra complexity pays off was measured, not assumed — same data, same question, seven models side by side.
Removed: hand-run experiments
Every run is a settings file, queued on the cluster and saved with its history — 95 runs in one format, none retyped by hand.
Removed: trusting one score
A second scorecard charges real trading costs, so a model is judged on what it would do, not only on what it gets right.
Five steps, in detail
Get the data ready
A public order-book benchmark: five Finnish stocks, ten trading days, ten price levels each side. Each example is the last 100 updates; the label is whether the price then went up, down or stayed flat.
Rebuild the published model
DeepLOB in PyTorch: a pattern-spotting layer, a layer that looks at several time scales at once, and a memory layer. Three classical baselines beside it: straight-line statistics, a small neural net and decision trees.
Try simpler and different builds
Remove the memory layer. Swap memory for attention. Drop the multi-scale block. Seven models across five horizons: 35 comparisons, all from the same settings files.
Train for trading, not only for accuracy
Sixty more runs with four training objectives that punish costly behaviour: weight rare moves, focus on hard cases, penalise churn, reward a smoother return. All queued on the Amarel cluster.
Charge the costs, then write it up
A backtest that charges half the bid–ask spread on each position change, plus filters that trade less. Then a research report with the limits stated plainly: one seed, one dataset, simulated.
- Data
- FI-2010 · 40 features · lookback 100 events · horizons 10 / 20 / 30 / 50 / 100 · days 1–7 train, 8–10 test
- Models
- DeepLOB = CNN + Inception + LSTM · variants: CNN-only (global average pooling), CNN + Transformer, CNN + Inception + Transformer · baselines: logistic regression, MLP, XGBoost · early stopping on validation
- Training
- YAML configs · 35 model × horizon runs + 60 cost-aware runs (weighted CE, focal, turnover-penalised CE, differentiable Sharpe; 3 settings × 5 horizons each) · SLURM jobs on GPU
- Backtest
- half the bid–ask spread charged per position change · filters: holding period, ensemble agreement, volatility · one seed, one dataset · report + experiment log
My part, not my part
| Mine | Shared with AI | Not mine |
|---|---|---|
Mine
|
Shared with AI
AI-assisted throughout: I directed, questioned and checked. I can explain everything on this page. |
Not mine
The report was submitted to the Rutgers Quant Finance Club as a showcase piece. |