Northwind Commodities · 2025
Replacing Northwind's seven-hour valuation batch with an incremental dependency graph
A seven-hour overnight batch left Northwind's traders working from yesterday's P&L whenever a curve was corrected. We replaced it with an incremental graph over versioned inputs, cutting the full run to 34 minutes and a curve correction to 90 seconds, with 12 months of valuations replayable to the input.
- Energy & Utilities
- 11 months
- 8 engineers, 1 quant analyst, 1 delivery lead
- 34 minutes
- Full end-of-day run, down from 7 hours
- 90 seconds
- Revaluation after a corrected forward curve
- 12 months
- Valuations replayable to their exact inputs
The challenge
What was wrong
Northwind's end-of-day valuation ran as a single overnight batch across roughly 40,000 gas and power positions. It took seven hours, failed about twice a month, and any corrected forward curve meant re-running everything, so traders regularly started the day on stale P&L. Auditors also wanted a published valuation reproduced from its exact inputs, which the batch could not do because it read curves live from the market data store as it went.
The approach
What we did
We modelled valuation as a dependency graph over versioned curve and position inputs, so a corrected curve revalues only the 1 to 3 per cent of positions that touch it rather than the whole book. Inputs are written immutably and referenced by version, which turns audit replay into a lookup instead of a reconstruction. The cost is storage: 90 days of inputs stay hot in ClickHouse and anything older replays from object storage in about 20 minutes. We took that over keeping five years hot.
“The number I care about is 90 seconds. That is how long a curve correction now takes to reach the desk P&L. It used to take the rest of the trading day.”
Stack
What it runs on
- Python
- Polars
- Ray
- ClickHouse
- Apache Iceberg
- Airflow
- dbt
- Kubernetes
Services
Practices involved
AI & Data
Data Engineering
Pipelines with contracts, tests and an on-call rota, not a folder of scheduled scripts.
Engineering
Enterprise Software
Platforms for organisations with multiple entities, real auditors and a ten-year horizon.
AI & Data
Business Intelligence
One definition per metric, and a report someone actually opens on Monday morning.
Cloud & Platform
Platform Engineering
An internal platform with golden paths, run as a product with users and a roadmap.
More work
Other engagements
Meridian Exchange · 2024
Bidding engine rebuild
Meridian was losing bids to lock contention in the closing seconds of every auction, and disputes over bid order had to be settled manually. We rebuilt the engine around per-lot single-writer partitions and an append-only log, trading cross-lot transactions for provable ordering at 1,800 bids per second.
Kestrel Industrial · 2023
Shop-floor system modernisation
Four plants, 300 machines and a VB6 application nobody could test safely. We replaced it one production cell at a time behind a routing gateway, keeping Oracle as the book of record until the final cell was live. Total unplanned stoppage across the entire programme came to 11 minutes.
Corvus Logistics · 2024
Dispatch planning and ETA models
Corvus planned 220 daily routes by hand and quoted ETAs that were often an hour wrong. We paired a solver seeded with the previous day's plan with a gradient-boosted ETA model, and gave drivers an app that queues work offline. Planning fell from four hours to 22 minutes a day.