A newcomer imagines a data-science project as a straight line: someone hands you a clean table, you train a clever model, and out pops the answer. Every practitioner knows the truth is messier and far more interesting. Real projects loop. You frame a question, wrestle with grubby data, discover the question was slightly wrong, reframe it, and go round again. The most quoted number in the field is that data scientists spend roughly 80% of their time just finding, cleaning and shaping data — and only the remaining fifth on the modelling everyone thinks the job is about.
This lesson gives you the map of the whole journey — the data-science lifecycle — so that every later module has a home. The map is old and battle-tested: its most famous form is CRISP-DM (the Cross-Industry Standard Process for Data Mining, 1999), still the most widely used template two decades on.
Read the diagram clockwise, but keep one finger on the arrows that jump backwards — you will use them constantly. Nothing here is a one-way street.
The single most important arrow is the one that closes the loop, from communicate back to frame. Shipping a result is not the end; it is what teaches you the next question worth asking.
| Stage | The real work | The trap |
|---|---|---|
| 1. Frame the question | Turn a vague business wish ("reduce churn") into a measurable target and a decision it will inform. | Solving a well-posed problem nobody actually asked. |
| 2. Get & wrangle data | Locate sources, join them, fix types, handle missing values and outliers. The 80%. | Assuming the data means what its column name says. |
| 3. Explore (EDA) | Plot distributions and relationships; find skew, gaps and surprises before modelling. | Skipping straight to a model and being fooled by the data's shape. |
| 4. Engineer features | Build the variables the model actually needs — ratios, aggregates, encodings, transforms. | Leaking information about the answer into a feature. |
| 5. Model | Fit candidate models; tune them; keep it as simple as the problem allows. | Reaching for a neural net when a line would do. |
| 6. Evaluate | Score honestly on held-out data against the business metric, not just the loss. | Grading on the data you trained on. |
| 7. Communicate / deploy | Turn the result into a decision — a dashboard, a report, a shipped feature — and monitor it. | A brilliant model that never leaves your laptop. |
Notice how many "traps" are failures of an earlier stage that only surface later. That is exactly why the lifecycle is a loop: evaluation sends you back to feature engineering, a strange plot sends you back to wrangling, and a stakeholder's reaction sends you all the way back to the question.
The discipline that separates a data scientist from a person who owns a laptop and a copy of the data is starting at stage 1, not stage 5. A crisp question fixes three things at once: the unit of analysis (a customer? a transaction? a day?), the target you will predict or estimate, and the decision the answer changes. If you cannot name the decision, you are not ready to model.
A rough budget of effort, which surprises every newcomer, looks like this:
The glamorous stage is the smallest slice. The Primer's data-science course spends its time where the real hours go: on the data itself.
CRISP-DM was published in 1999 by a consortium including SPSS and Daimler-Benz, in the days when "data mining" was the fashionable phrase. Its six phases — business understanding, data understanding, data preparation, modelling, evaluation, deployment — map almost one-to-one onto the lifecycle above. What is remarkable is its staying power: survey after survey, decades later, still finds it the most widely used process model in industry, out-living the tools and even the vocabulary of its era. Rivals exist (Microsoft's TDSP, the "OSEMN" mnemonic — Obtain, Scrub, Explore, Model, iNterpret), but they are all recognisably the same loop wearing different names. Learn one and you have learned them all.
The classic beginner mistake is treating these stages as a checklist you tick once, top to bottom, like a software "waterfall." In reality you revisit every stage many times. A model that scores badly (stage 6) usually sends you back to features (stage 4) or even to wrangling (stage 2), not forward to a fancier algorithm. A surprising plot in EDA (stage 3) can reveal that your data is corrupt (stage 2) or that your question was wrong (stage 1). If your project only ever moves forwards, you are not doing data science — you are doing data theatre. Budget for iteration from the start.
Every remaining lesson in this course is one stage of this loop seen up close. Next we pin down the
shape data must take before any of it works —
The lifecycle lives or dies on your fluency with data tables. Kaggle Learn's free, in-browser Pandas course walks you through the get-and-wrangle stages hands-on, so the loop you just met stops being a diagram and becomes muscle memory.