← back to the section

A developer writes: "fixed it, works on my machine." You open the staging environment — it doesn't work. Half an hour later it turns out staging runs the build from two days ago, and the fix lives only on that laptop. The next day the same story with another task: someone forgot to push, someone built the wrong branch.

Discipline doesn't cure this — a machine does. A separate server takes every change that lands in the shared branch and builds the product from it on a clean machine: the same way for everyone, with no "well, on my box". That automatic build on every change is called continuous integration (CI), and together with the next step — delivering that build automatically all the way to release — CI/CD.

change a3f1e2 landed in the shared branch — the pipeline started itself build build✓ unit tests unittests✓ staging staging✓ smoke set smokeset✗ release six minutes from change to report goes no further run report · build 4.19 · a3f1e2place an order ✓pay with points ✓pay by card ✗ expected 201, got 500service log, request and response — right there, in the run artifacts only what passed the machine reaches people

The steps run in turn, and each one decides whether the build travels further. The smoke set found a red check — the release is crossed out, and the report already says which check failed and what the service answered instead of the expected result.

The pipeline: five steps and the gates between them

Building a project is not the same as checking it, so a pipeline is almost never a single action. The list of steps sits as a file in the same repository as the code and is changed the same way the code is.

stages:
  - build          # turn sources into something runnable
  - unit           # developers' unit tests, minutes
  - deploy-staging # put the built thing on the test environment
  - smoke          # automated checks of the main path
  - report         # what passed, what failed, where to look

A step here is also a gate. Unit tests failed — there will be no deployment: putting something knowingly broken on staging is pointless. Staging didn't come up — the smoke set won't start, otherwise you get twelve red marks for one reason. This is the same entry criterion as in a test plan, only the machine enforces it.

That is exactly where the benefit is: breakage is visible five minutes after the change, while the author still remembers what they did — not a week later, when the investigation eats a day.

Where your autotests live in the pipeline

Autotests that people run by hand whenever they remember stop being run by the third week. They start paying off when the run is tied to an event and the sets are split by how long they take. The fast and reliable part — a smoke set, a dozen API checks of the main path — is attached to every change: five minutes the team will wait. Nobody waits forty minutes for a full regression, so it runs at night on a schedule and the report is reviewed in the morning. Interface tests go into a separate set and run least often: they are the slowest and the most capricious.

Manual checks belong in the same row: you start them on a green pipeline, not when "the build seems to have arrived".

A test must not depend on its neighbour

Locally the set is green, in the pipeline it is red every other time, and always different tests. The cause is almost always the same: the tests made an arrangement between themselves, and the machine never knew about it — it runs them in its own order and often in several threads at once. A test "log into the account" that counts on the customer qa@shop.com having been created by the test "sign up" turns red from a simple reordering. Worse: two threads start signing up the same customer, the second one gets "address already taken" — and a test without a single defect in it fails.

Hence three rules, and they are not about CI but about the tests themselves: every test prepares its own data and cleans up after itself; values that must be unique are made unique (qa+1726212000@shop.com); and you wait for a state, not for time — sleep 5 on a busy environment doesn't save you, it only slows the run down.

The artifact: what is left of the build

"Which version is on which environment?" is the question half of all investigations start with. The answer comes from the artifact: the file or image produced by the build step that can be run as is. It carries a build number and the change it was built from — the very pair that goes into a bug report.

There is one rule here: the artifact is built once and after that only moved around. Rebuild it for every environment and production gets something other than what you checked, and the difference shows up exactly where nobody expects it.

The second meaning of the word is run artifacts: the report, service logs, screenshots, video, requests and responses. They are stored next to the run, and that matters: staging will be reset overnight, while a week-old report stays the only trace of the failure.

How a red run is investigated

The order of investigation saves hours. First the name of the failed check and its message — what was expected and what came back. Then the step it failed at: unit tests and the smoke set break for different reasons. Then the artifacts — the service log, the screenshot, the response body. And only then your hands: repeat the same step on the same build.

There are three outcomes, and mixing them up is expensive. A defect in the product — file a report with a link to the run. The test is outdated: the requirement changed and the answer is legitimately different now — fix the test, not the product. The environment: staging didn't start, the disk ran out of space, a neighbouring system went away — the product is fine, but red still gets investigated, otherwise it turns into background noise.

A flaky test: why you can't simply rerun it

A check failed, you pressed "retry" — green. Same code, same environment, different result: such a test is called flaky. You recognise it by run history: it fails on unchanged code, say seven times out of a hundred.

Rerunning does two bad things. It hides a real defect: a race condition in the product looks exactly the same — "usually works, sometimes doesn't" — and a user will hit it the same way. And it ruins the team's relationship with red: people get used to hitting "retry" without looking and one day rerun a genuine failure.

So a flaky test gets treated, not rerun: mark it, open a task with the number of failures, move it out of the blocking set into quarantine — so it doesn't hold the release yet doesn't disappear from view — and fix the cause: waiting on time, shared data, dependence on order. Silently switching a check off is the same as deleting it, only without a record in history.

Delivery versus deployment

The pipeline is green — and now the question is who presses the release button. The two letters CD are expanded in two ways, and the difference is exactly that button: continuous delivery — the pipeline brings the artifact to a "ready to release" state and a human releases it; continuous deployment — there is no button, a green pipeline goes to production on its own. For testing this is a question about the last point where you can still say "we are not releasing": in the second case that role is played by the autotests.

In short

  • CI is the automatic build and check of every change on a clean machine: the proof that something works is a run, not somebody's laptop.
  • A pipeline is a chain of steps with gates: build, unit tests, staging, smoke set, report; a red step lets nothing through.
  • Fast checks are attached to every change, the long regression runs at night; manual checks start on a green pipeline.
  • A test must prepare its own data and not depend on order: the pipeline runs checks however it likes, often in parallel.
  • An artifact is built once and moved between environments; run artifacts — report, logs, screenshots — outlive the reset of the environment.
  • A flaky test is not rerun: mark it, move it into quarantine and fix the cause, otherwise the team stops believing red.