CI/CD Pipelines
CI vs CD vs Continuous Deployment, pipeline stages and gates, deploy strategies and rollback, runners, artifacts and caches, pipeline secrets and supply-chain security.
15 questions
JuniorTheoryVery commonWhat is the difference between CI, Continuous Delivery, and Continuous Deployment?
What is the difference between CI, Continuous Delivery, and Continuous Deployment?
CI (Continuous Integration) auto-builds and tests each merged change, catching integration breaks early. Continuous Delivery keeps every passing build releasable behind a human approval. Continuous Deployment drops that gate — passing changes ship to production automatically.
Common mistakes
- ✗Treating Continuous Delivery and Continuous Deployment as the same thing
- ✗Thinking CI includes deploying to production rather than build-and-test
- ✗Believing Continuous Deployment still needs a manual approval gate
Follow-up questions
- →Why does Continuous Deployment demand a strong automated test and monitoring suite?
- →What organizational changes let a team move from Delivery to Deployment?
MiddleDesignVery commonYour team runs a stateless web API behind a load balancer on a fixed fleet. Deploys currently cause a brief outage, and rollbacks are slow and manual. You have budget to roughly double capacity during a release window. Design a blue-green deployment: how you stand up the new version, cut traffic over, verify it, and roll back instantly if it misbehaves. Explain the capacity cost, and one limitation blue-green has around database schema changes.
Your team runs a stateless web API behind a load balancer on a fixed fleet. Deploys currently cause a brief outage, and rollbacks are slow and manual. You have budget to roughly double capacity during a release window. Design a blue-green deployment: how you stand up the new version, cut traffic over, verify it, and roll back instantly if it misbehaves. Explain the capacity cost, and one limitation blue-green has around database schema changes.
Two identical environments — blue (live), green (idle). Deploy the new version to green and smoke-test it while blue serves users. Cut over by repointing the load balancer to green — an instant switch; keep blue warm to roll back. Needs ~2x capacity, and schema changes stay backward-compatible.
Common mistakes
- ✗Cutting traffic to green before smoke-testing it against blue
- ✗Forgetting blue-green needs roughly double capacity during the window
- ✗Shipping a breaking schema change that only the new version tolerates
Follow-up questions
- →How would you drain in-flight requests from blue during the cutover?
- →How do you split a breaking schema change into backward-compatible steps?
SeniorDesignVery commonYou run a high-traffic user-facing service and want to shrink the blast radius of bad releases — a full rollout occasionally ships a regression that pages the on-call before anyone notices. Design a canary release with automated, metric-based promotion: how you route a small slice of traffic to the new version, which signals gate promotion, how the rollout progresses, and what triggers an automatic rollback. Contrast it with blue-green, and note when canary is the wrong choice.
You run a high-traffic user-facing service and want to shrink the blast radius of bad releases — a full rollout occasionally ships a regression that pages the on-call before anyone notices. Design a canary release with automated, metric-based promotion: how you route a small slice of traffic to the new version, which signals gate promotion, how the rollout progresses, and what triggers an automatic rollback. Contrast it with blue-green, and note when canary is the wrong choice.
Deploy the new version beside the old and route a small traffic slice (say 5%). Watch error rate, latency, and saturation vs a healthy baseline. Hold through a bake period, raise the share (5 to 25 to 50 to 100%); breach a threshold, roll back to 0%. Unlike blue-green's instant switch, it limits exposure gradually.
Common mistakes
- ✗Sending full traffic at once instead of a small initial slice
- ✗Gating promotion on elapsed time rather than health signals
- ✗Assuming canary always beats blue-green regardless of traffic or metrics
Follow-up questions
- →How do you pick thresholds that catch regressions without false rollbacks?
- →How does a canary compare a new version fairly against the baseline?
MiddleDebuggingCommonAn integration-test job passes locally but flakes in CI on about 1 run in 5 with no code change. Find and fix it.
An integration-test job passes locally but flakes in CI on about 1 run in 5 with no code change. Find and fix it.
The sleep 3 is a race — under CI load the app or DB isn't always up in three seconds, so some runs test early and fail with connection-refused. Replace it with a readiness poll of the health endpoint with a timeout before pytest. Wait on the condition, not the clock.
Common mistakes
- ✗Fixing a race by lengthening the sleep instead of waiting on readiness
- ✗Masking flakiness with a blanket job retry rather than the root cause
- ✗Blaming CI hardware instead of the timing assumption in the config
Follow-up questions
- →How would a readiness poll with a timeout look in the script?
- →When is a bounded job retry acceptable, and when does it hide real bugs?
MiddleTheoryCommonHow do feature flags decouple deploying code from releasing a feature?
How do feature flags decouple deploying code from releasing a feature?
A feature flag wraps new code in a runtime condition, so it ships disabled. This splits deploy (binary onto production) from release (turning it on for users). You merge unfinished code safely, enable it gradually, and kill it instantly by flipping the flag, no redeploy.
Common mistakes
- ✗Thinking a flag is compile-time rather than a runtime toggle
- ✗Conflating deploy (binary on prod) with release (feature on for users)
- ✗Missing that flags enable gradual rollout and an instant kill switch
Follow-up questions
- →What operational risk do stale, never-removed feature flags create?
- →How do flags support A/B testing or a percentage-based rollout?
MiddleDesignCommonA CI pipeline needs a registry password to push the image, a cloud key to deploy, and a database URL for integration tests — today they sit in plaintext in the repo's YAML. Design a safe secrets flow: where the secrets live, how jobs receive them, how you keep them out of build logs, and how you limit the blast radius if a runner or a fork is compromised.
A CI pipeline needs a registry password to push the image, a cloud key to deploy, and a database URL for integration tests — today they sit in plaintext in the repo's YAML. Design a safe secrets flow: where the secrets live, how jobs receive them, how you keep them out of build logs, and how you limit the blast radius if a runner or a fork is compromised.
Keep secrets out of the repo — in a secrets manager or protected CI variables, injected as env vars only into jobs that need them. Scope each to the right environment and branches, mask it in logs, never echo it. Prefer short-lived, rotatable credentials (OIDC tokens) over long-lived keys.
Common mistakes
- ✗Committing secrets to the repo, even base64-encoded, thinking it is encryption
- ✗Exposing every secret to every job and to untrusted fork pipelines
- ✗Echoing secrets to logs or relying only on long-lived keys
Follow-up questions
- →How does OIDC let a job get a cloud token without any stored key?
- →Why are fork pipelines especially dangerous for protected secrets?
MiddlePerformanceCommonA CI build takes 40 minutes and blocks every merge. Where do you look first for the highest-value speedup?
A CI build takes 40 minutes and blocks every merge. Where do you look first for the highest-value speedup?
Profile first to find the slowest stages — don't guess. The biggest wins are usually caching dependencies, parallelizing independent jobs (shard tests, run lint and build together), reusing Docker layer caching via Dockerfile ordering, and running only affected work. Fail fast so a broken build stops early.
Common mistakes
- ✗Optimizing by guesswork instead of profiling the slowest stages first
- ✗Skipping dependency and Docker layer caching as unsafe
- ✗Running everything serially when independent jobs could be parallel
Follow-up questions
- →How does Dockerfile instruction order change which layers stay cached?
- →What lets you run only the tests affected by a given change?
JuniorTheoryOccasionalWhat are the typical stages of a CI/CD pipeline, and why should it fail fast?
What are the typical stages of a CI/CD pipeline, and why should it fail fast?
A pipeline runs ordered stages — build, unit tests, static analysis and security scans, package the artifact, then deploy. Cheap fast checks run first, so a failure halts the run immediately (fail-fast): the developer gets quick feedback, and later slow stages waste no minutes once an early one has already broken.
Common mistakes
- ✗Running slow end-to-end tests before the cheap fast checks
- ✗Continuing the pipeline after a failure instead of stopping early
- ✗Skipping static analysis or security scans as optional
Follow-up questions
- →Where do you place an integration or end-to-end stage relative to unit tests?
- →How does fail-fast interact with jobs you want to run in parallel?
JuniorTheoryOccasionalHow do shared and self-hosted CI runners differ, and when do you need self-hosted?
How do shared and self-hosted CI runners differ, and when do you need self-hosted?
Shared runners are pooled cloud machines the provider manages — zero setup, generic environment. Self-hosted runners are ones you own and register. Choose self-hosted for a GPU or other special hardware, private-network or on-prem access, big caches, or tighter control.
Common mistakes
- ✗Swapping the definitions of shared and self-hosted runners
- ✗Thinking a shared runner can reach on-prem or private-network resources
- ✗Missing that special hardware or big caches are why you self-host
Follow-up questions
- →What new security duties come with running your own self-hosted runners?
- →Why do you often make self-hosted runners ephemeral rather than long-lived?
JuniorTheoryOccasionalHow do you verify that a deployment actually succeeded before trusting it?
How do you verify that a deployment actually succeeded before trusting it?
Don't trust the deploy tool's zero exit — verify the running app. Confirm health and readiness endpoints are healthy, run smoke tests over critical paths, watch error rate and latency, and check the new version is served. A gate blocks or rolls back on any failure.
Common mistakes
- ✗Treating a zero exit code as proof the deploy is live and healthy
- ✗Skipping smoke tests and health checks against the running app
- ✗Not confirming which version is actually being served
Follow-up questions
- →What is the difference between a health check and a smoke test here?
- →How would you wire an automatic rollback into that promotion gate?
MiddleTheoryOccasionalHow does a CI runner's execution model work — how does a job get picked up and run?
How does a CI runner's execution model work — how does a job get picked up and run?
A runner is an agent that polls the CI server for jobs matching its tags. On claiming one it spins up a clean executor — a fresh container, VM, or shell — clones the repo at that commit, restores caches, runs the script, uploads artifacts, then tears it down.
Common mistakes
- ✗Assuming state persists between jobs instead of a clean executor per job
- ✗Thinking the server pushes jobs rather than the runner polling for them
- ✗Forgetting the runner clones the repo at the job's specific commit
Follow-up questions
- →Why is a fresh executor per job important for reproducible builds?
- →How do runner tags decide which runner claims a given job?
MiddleTheoryOccasionalWhat is the difference between a pipeline artifact and a cache, and when do you use each?
What is the difference between a pipeline artifact and a cache, and when do you use each?
An artifact is a build output you deliberately keep and pass on — a jar, image, or report — versioned and immutable; correctness depends on it. A cache is a speed optimization: reusable inputs like dependencies, keyed by a lockfile hash. The pipeline must still work with an empty cache.
Common mistakes
- ✗Depending on the cache for correctness instead of treating it as optional
- ✗Storing large immutable outputs as caches rather than artifacts
- ✗Not keying the cache to a lockfile hash so it never invalidates
Follow-up questions
- →What makes a good cache key, and what happens when it never invalidates?
- →Why should artifacts be immutable and uniquely versioned across a release?
MiddleDebuggingOccasionalThe deploy job succeeds and exits zero, but production keeps serving the previous build. Where did the pipeline break?
The deploy job succeeds and exits zero, but production keeps serving the previous build. Where did the pipeline break?
The mutable latest tag never changes the Deployment spec between releases, so kubectl set image sees no diff and triggers no rollout, nothing pulled. Fix it with immutable tags: push app:$CI_COMMIT_SHA and point the Deployment at it. Build once, promote that artifact — never rebuild per env.
Common mistakes
- ✗Using a mutable
latesttag so the Deployment spec never changes - ✗Expecting a rollout when
kubectl set imagesees no diff - ✗Rebuilding a separate image per environment instead of promoting one
Follow-up questions
- →Why is build-once-promote-many safer than rebuilding for each environment?
- →How does
imagePullPolicyinteract with a reused mutable tag?
SeniorDesignOccasionalAfter a dependency-poisoning incident in the industry, your org wants to harden the software supply chain of its CI/CD. Design the controls: how you make builds trustworthy and reproducible, how you know exactly what went into a released artifact, how you stop a tampered or malicious dependency or build step from reaching production, and how a downstream consumer can verify an artifact really came from your pipeline. Name the key artifacts and mechanisms.
After a dependency-poisoning incident in the industry, your org wants to harden the software supply chain of its CI/CD. Design the controls: how you make builds trustworthy and reproducible, how you know exactly what went into a released artifact, how you stop a tampered or malicious dependency or build step from reaching production, and how a downstream consumer can verify an artifact really came from your pipeline. Name the key artifacts and mechanisms.
Pin dependencies by version and hash in a lockfile from a vetted internal proxy, not the open internet. Generate an SBOM (software bill of materials) and scan it for known vulnerabilities. Build in an isolated least-privilege runner and record signed provenance (SLSA). Sign it, and admit only signed, policy-passing artifacts.
Common mistakes
- ✗Pulling dependencies unpinned and un-hashed straight from public indexes
- ✗Treating an SBOM, provenance, and signing as paperwork with no value
- ✗Building with broad privileges instead of an isolated least-privilege runner
Follow-up questions
- →How does signed provenance stop a compromised build step from hiding?
- →What does admitting only signed, policy-passing artifacts enforce at deploy time?
JuniorTheoryRareIn GitLab CI, how do stages and jobs in .gitlab-ci.yml structure a pipeline?
In GitLab CI, how do stages and jobs in .gitlab-ci.yml structure a pipeline?
.gitlab-ci.yml defines a pipeline as jobs, each in a stage. Jobs in one stage run in parallel; stages run sequentially in the listed order, and the next starts only after every job in the current one passes. Each job has its own script.
Common mistakes
- ✗Thinking stages run in parallel and jobs run serially (it is the reverse)
- ✗Expecting the next stage to start before the current one fully passes
- ✗Believing a job cannot declare its own artifacts, cache, or rules
Follow-up questions
- →How do
needslet a job start before its whole prior stage finishes? - →What does
allow_failurechange about a job's effect on the pipeline?
