Why Data Pipelines Need Vertical Slicing in Agile Sprints

Sprint six demo. The team shows dozens of ingested tables, a warehouse model nobody has queried, and a dashboard mockup. A stakeholder asks which shipments were late last week, and the honest answer is 'sprint seven'.

That is layer-by-layer planning: ingest everything, model everything, then build the dashboard. Every layer is technically done and nothing is usable. The fix is vertical slicing: build data pipelines one thin, end-to-end slice at a time, so each slice takes one source to one consumer's question and is usable when the sprint ends.

Why layered data work ships nothing for six sprints

A story like 'ingest all tables from the warehouse management system' has no user, so nobody can say whether it is right. The Agile Manifesto says working software is the primary measure of progress, and a table nobody reads is not working software.

Layering also pushes every risk to the end:

• Definitions. 'Late' might mean past the carrier's date or past the date promised to the customer. You only find out when someone looks at a number, so make that happen early.
• Access. The security review, the PII question and the source team's rate limits show up after the model is built.
• Quality. Duplicate rows and broken keys surface in the dashboard, in front of the stakeholder.

Vertical slicing data pipelines: one source, one question

A slice is one source, one consumer question, one named person who can say yes or no, and every layer in between: ingestion, transformation, a quality check, access control and a visible output. Write it down before the sprint starts:

slice: late-shipments-v1
consumer: warehouse ops lead
question: Which shipments missed their promised date last week, by carrier?
source: shipments table from the WMS (one table, last 30 days)
done_when:
  - the ops lead opens the chart unaided and the numbers match their own spreadsheet
  - key and sanity checks run on every load and fail the job when they break
  - only the ops_analytics role can read the output table
out_of_scope: returns, other warehouses, history older than 30 days, customer joins

The out_of_scope line matters as much as the rest. It is what keeps the slice thin.

A sequence of slices that each ship

1. Slice 1: one table, last 30 days, late shipments by carrier, one chart, one check, one access rule.
2. Slice 2: backfill older history and add a freshness alert, so the ops lead knows when the chart is stale.
3. Slice 3: add the orders source, so late shipments can be ranked by order value.
4. Slice 4: open it to a second consumer, finance, with its own access rule.

Each slice ends with a demo to the person who asked. Each one also changes the plan, because the demo produces real feedback.

Put the quality check and access control in slice one

These are the first things cut when they come last. Add them on day one, even tiny, so the plumbing exists and later slices only add rules. With dbt data tests, the first check is a few lines of YAML (data_tests: is the current key, tests: is the older name):

models:
  - name: late_shipments
    columns:
      - name: shipment_id
        data_tests:
          - unique
          - not_null
      - name: carrier
        data_tests:
          - not_null

Add one business-rule check as a singular test. dbt treats any rows the query returns as failures:

-- tests/assert_delivery_after_shipping.sql
select shipment_id, shipped_at, delivered_at
from {{ ref('late_shipments') }}
where delivered_at < shipped_at

Then give the consumer, and only the consumer, access:

-- Snowflake syntax; other warehouses use GRANT SELECT ... TO <role>
grant select on table analytics.late_shipments to role ops_analytics;

None of this is sophisticated. The point is that the first user meets a pipeline that already checks itself and already knows who may read it.

The 'we need the schema designed up front' objection

Some up-front design is real, and it is smaller than it sounds. Decisions that are expensive to reverse deserve a conversation before slice one: the grain of the table (one row per shipment or per parcel?), the key, how customers are identified across systems, and what counts as PII. Settle those for the data in the slice, on one page, not for the whole company.

What you should not design up front is every entity nobody has asked about yet. That model is a guess, and it gets reworked the first time a real question disagrees with it. With vertical slices the rework is cheap, because slice one has one consumer: renaming a column breaks one chart, not forty.

The honest exception is a consumer with a fixed contract, such as a regulatory report with a prescribed layout. There the output shape is a requirement, so it goes into slice one's definition, and you still slice by source or by report section.

Pick the thinnest first slice

1. Find a person who answers a question by hand today, with a spreadsheet or a weekly export. They already know what a right answer looks like.
2. Pick the single table that holds most of the answer. If it needs three systems, narrow the question until it needs one.
3. Cut the time range and the scope until it fits in less than a sprint. Last 30 days beats all history.
4. Write done_when as something that person can check without you in the room.
5. Add one quality check and one access rule, however small.

Next sprint, take the next slice, not the next layer.

AI features sit on top of whatever data you give them, so a pipeline that reaches a real user with a check and an access rule in its first sprint is the only kind you can safely build on in production.