Everything else in this course assumed the data was sitting in a tidy file you could load into memory. Reality is messier and bigger: data arrives as a firehose of clicks and sensor readings, in a dozen incompatible formats, faster than any one machine can store — let alone analyse. Data engineering is the unglamorous, indispensable discipline of building the plumbing that turns that chaos into the clean table the analyst takes for granted. A famous rule of thumb says data scientists spend 80% of their time just getting data into shape; this page is about the machinery that automates it.
"Big" is really the point where any one V breaks the single-machine, single-format assumptions of ordinary analysis — which is why the answer is nearly always to distribute the work across many machines and to build pipelines that handle the flow automatically.
A data pipeline is a series of stages that data flows through automatically. The canonical shape:
Ingest pulls data from its sources (apps, databases, sensors, third-party APIs). Store lands it somewhere durable. Transform cleans, joins and reshapes it into analysis-ready tables. Serve delivers the result to dashboards, models and reports. Each stage is automated and monitored so the whole thing runs on a schedule without a human babysitting it.
Two design axes shape every pipeline. First, batch vs streaming. A batch pipeline processes a big chunk on a schedule — "recompute yesterday's sales every night at 2 a.m." A streaming pipeline processes each event as it arrives, for things that can't wait: fraud detection, live dashboards, recommendations. Batch is simpler and cheaper; streaming is harder but gives fresh answers.
Second, ETL vs ELT — where the "transform" happens.
| Approach | Order | Idea |
|---|---|---|
| ETL | Extract → Transform → Load | Clean the data before storing it — the classic warehouse approach. |
| ELT | Extract → Load → Transform | Dump raw data in first, transform later on demand — cheap storage makes this the modern default. |
Where does all this data live? Two archetypes. A data warehouse stores clean,
structured tables with a fixed schema, optimised for fast
When data is too big for one machine, the winning idea is embarrassingly simple: split the data across many machines, have each machine apply the same computation to its own shard in parallel, then combine the partial results. That is exactly the MapReduce pattern (the "map" step applies, the "reduce" step combines), and it is the conceptual heart of Spark, the dominant modern engine. To count words across a petabyte of text, you don't build a bigger computer — you give each of a thousand computers a slice, let them count locally, and sum the counts. The framework handles the hard parts: distributing the data, recovering from a machine that dies mid-job, and shuffling intermediate results between stages.
This was Google's radical bet in the early 2000s. Instead of buying ever-more-expensive specialised hardware (scaling up), they wired together vast numbers of ordinary, unreliable commodity machines (scaling out) and wrote software — MapReduce and the Google File System — that assumed machines would constantly fail and simply routed around the failures. The economics were unbeatable: commodity hardware is far cheaper per unit of compute, and you can keep adding machines almost without limit. The catch is that not every problem splits neatly into independent pieces — anything requiring lots of communication between shards (many graph algorithms, for instance) fights the model. But for the "do the same thing to every row" workloads that dominate data processing, scaling out won so decisively it reshaped the entire industry.
The seductive failure of data engineering is the pipeline that works but that nobody can rerun, explain or trust. If a result came from a hand-run script, an undocumented manual fix, and a data source that has since changed, you cannot reproduce it — and an unreproducible number is not evidence, it's a rumour. Mature pipelines are treated like software: version-controlled code, orchestration tools (Airflow, Dagster and the like) that schedule stages and track their dependencies as a DAG, automated data-quality checks that fail loudly when an input goes weird, and clear lineage so you can trace any number back to its raw source. "It ran on my laptop last Tuesday" is not a data platform.
Kaggle Learn's Advanced SQL course teaches the joins, aggregations and large-scale querying (over BigQuery) that sit at the transform-and-serve heart of a data pipeline — the hands-on next step from the architecture ideas on this page.