AI & Data
Pipelines with tests,
contracts and an owner
Most data platforms fail slowly: a column changes upstream, three dashboards go wrong, nobody notices for a fortnight. We build ingestion, transformation and storage with schema contracts, freshness checks and lineage, so breakage is loud and traceable.
Overview
Pipelines with contracts, tests and an on-call rota, not a folder of scheduled scripts.
The deliverable is a platform your analysts trust enough to stop keeping their own spreadsheet. That requires three unglamorous things: agreed contracts with the systems that produce data, tests that run on every load, and lineage that answers where a number came from without a two-hour investigation. All three are organisational as much as technical.
We build on the warehouse or lakehouse you already have — Snowflake, BigQuery, Databricks, Redshift or Postgres — and we are direct about when your current one is fine. Migrating platforms is rarely the fix; roughly two-thirds of the performance complaints we investigate turn out to be modelling and clustering problems, not engine problems.
Cost gets designed in. Partitioning, clustering keys, incremental models, sensible warehouse sizing and query cost monitoring by team are all decisions made during the build. Retrofitting them after a platform bill has tripled is possible but takes months and involves rewriting the models everyone now depends on.
- 22 min
- Nightly load window at Northwind Commodities, down from 4h 40m
- £41k/yr
- Warehouse spend removed by incremental models and clustering keys
- 1,340
- Automated quality tests running per load across 62 models
Capabilities
What this covers
Six areas we staff properly. If your problem sits outside them, the honest note at the foot of this page says so.
Ingestion and change data capture
Batch and streaming ingestion from operational databases, SaaS APIs, files and event streams, with log-based CDC where the source supports it and a documented replay procedure for when a load fails.
Data contracts and schema evolution
Producing systems commit to a versioned schema with a deprecation window. Breaking changes fail in the producer's CI rather than in a dashboard three days later, which moves the cost of a rename onto its author.
Transformation and modelling
Dimensional or wide-table models built in dbt with tests, documentation and lineage generated from the code itself. Business definitions live in one place, so revenue means the same thing in every report.
Data quality and observability
Freshness, volume, distribution and referential checks running on each load, with alerts routed to the team that owns the source. Silent nulls and duplicated rows get caught before anyone builds on them.
Streaming and near-real-time
Kafka or Kinesis pipelines with exactly-once sinks and a stated end-to-end latency target, usually 3 to 60 seconds. We will ask whether you need it, because streaming costs several times batch to run and to operate.
Governance, access and cost control
Row and column-level access tied to your identity provider, PII tagging carried through lineage, and per-team query cost dashboards with budgets. Governance people can see tends to be governance people follow.
Deliverables
What you get
- Ingestion pipelines with replay procedures and documented source contracts
- Modelled warehouse layer in dbt with tests, docs and generated lineage
- Data quality monitors with ownership routing and alert thresholds
- Orchestration in Airflow or Dagster with retry, backfill and SLA policies
- Access control model mapped to your identity provider, with PII tagging
- Cost dashboard by team and query, with budget alerts and tuning notes
Stack
What we build it with
- dbt
- Apache Airflow
- Dagster
- Snowflake
- Databricks
- Apache Kafka
- Debezium
- Apache Iceberg
- Great Expectations
- DuckDB
- BigQuery
- Terraform
Process
How the engagement runs
Two-week increments against a written definition of done. You can stop at any increment boundary and keep everything built so far.
Source and consumer mapping
We inventory every producing system and every downstream consumer, including the spreadsheets. The shadow reporting layer shows exactly where the platform currently fails.
Contract negotiation
Schemas, freshness expectations and ownership agreed with the teams that produce the data. This is a people exercise and it takes longer than the engineering does.
Build one domain end to end
One business domain taken from source to modelled tables to a live report, with tests and monitoring, so the pattern is proven before it gets repeated nine times.
Migrate and parallel-run
Remaining domains ported with old and new outputs reconciled row by row. Nothing is decommissioned until the numbers match and someone signs that they match.
Handover and cost review
Runbooks, backfill procedures and an on-call rota your team owns, plus a cost review at 30 and 90 days with specific tuning recommendations.
When this is the wrong engagement
If you have one operational database, under 50 GB of data and three analysts, a well-indexed read replica with scheduled views will serve you better than a warehouse and an orchestration layer.
FAQ
Questions we get asked
- Do we need a lakehouse, or is a warehouse enough?
A warehouse is enough for most organisations under a few hundred terabytes with mainly structured data. Lakehouse formats earn their place when you have large semi-structured volumes, multiple compute engines, or a genuine need to avoid lock-in at the storage layer.
- How long until the first useful thing exists?
One domain from source to a working report in six to ten weeks, depending on how many producing systems it touches and how quickly their owners engage. Access approvals are consistently the slowest item, so we ask for them in the first week.
- Who runs this after you leave?
Your team, and we design for that from the start: standard tools, code in your repositories, no bespoke framework of ours. Handover includes an on-call rota, backfill runbooks and two rehearsed incident drills. If you have no data engineer, we will say so before starting.
- What do you do about data that is simply wrong at source?
We surface it rather than silently correcting it. Quality checks fail loudly, affected rows are quarantined and routed to the owning team, and the report shows a gap instead of a plausible wrong number. Correcting bad data in the pipeline hides the problem until it resurfaces.
Related