Understanding Stages, Jobs, and Steps in Azure Pipelines
Azure Pipelines are a powerful CI/CD orchestration tool. Understanding the hierarchical structure of pipelines—stages, jobs, and steps—is fundamental to building efficient and maintainable build and release processes.
1. Stages
Stages represent the highest level of division in a pipeline. They typically correspond to different phases of your software delivery process, such as:
- Build: Compiling code, running unit tests, and creating artifacts.
- Test: Running integration tests, performance tests, or security scans.
- Deploy: Deploying the application to various environments (e.g., Dev, QA, Staging, Production).
- Approval: Manual gates or automated checks before proceeding to the next stage.
Stages run sequentially by default, meaning the next stage only begins after the previous one has successfully completed. However, you can configure stages to run in parallel to speed up your pipeline.
Key Characteristics of Stages:
- Isolation: Each stage typically runs in a clean environment.
- Dependencies: Stages can depend on the successful completion of prior stages.
- Approvals/Gates: Can include manual approvals or automated checks.
- Artifacts: Stages can consume artifacts produced by previous stages.
2. Jobs
Jobs are containers within a stage that execute a set of tasks. A stage can contain one or more jobs.
- Parallel Execution: Jobs within the same stage can be configured to run in parallel on different agents, significantly reducing execution time.
- Agent Pool: Each job is executed on an agent (a machine or VM that runs the pipeline tasks). You can specify which agent pool a job should use.
- Dependencies: Jobs can also have dependencies on other jobs, although this is less common than stage dependencies.
Job Execution:
By default, jobs within a stage run sequentially. To enable parallel execution, you define a matrix or explicitly specify dependencies between jobs.
Example: Parallel Jobs for Different OS
You can run the same set of steps on multiple operating systems concurrently:
stages:
- stage: Build
jobs:
- job: BuildLinux
pool:
vmImage: 'ubuntu-latest'
steps:
- script: echo Building on Linux...
- job: BuildWindows
pool:
vmImage: 'windows-latest'
steps:
- script: echo Building on Windows...
3. Steps
Steps are the smallest units of work in a pipeline. They are individual tasks that perform a specific action, such as running a script, compiling code, publishing an artifact, or deploying a service.
- Tasks: Pre-defined building blocks provided by Azure DevOps or third-party extensions.
- Scripts: Custom scripts written in Bash, PowerShell, Python, etc.
- Execution Order: Steps within a job execute sequentially in the order they are defined.
Types of Steps:
- Script: Executes a script.
- Task: Runs a specific task (e.g.,
DotNetCoreCLI@2,AzureRmWebAppDeployment@4). - Checkout: Checks out code from a repository.
- DownloadBuildArtifacts: Downloads artifacts from a previous pipeline run.
Example: A Job with Multiple Steps
A job demonstrating build, test, and publish steps:
jobs:
- job: BuildAndTest
pool:
vmImage: 'ubuntu-latest'
steps:
- checkout: self
- task: DotNetCoreCLI@2
displayName: 'Restore Dependencies'
inputs:
command: 'restore'
projects: '**/*.csproj'
- task: DotNetCoreCLI@2
displayName: 'Build Project'
inputs:
command: 'build'
projects: '**/*.csproj'
arguments: '--configuration Release'
- task: DotNetCoreCLI@2
displayName: 'Run Unit Tests'
inputs:
command: 'test'
projects: '**/*Tests.csproj'
arguments: '--configuration Release'
- task: PublishBuildArtifacts@1
displayName: 'Publish Artifact'
inputs:
pathToPublish: '$(Build.ArtifactStagingDirectory)'
Relationship Between Stages, Jobs, and Steps
The hierarchy is straightforward:
- A Pipeline contains one or more Stages.
- A Stage contains one or more Jobs.
- A Job contains one or more Steps.
This structure provides a clear and organized way to define your CI/CD workflows, allowing for parallel execution, conditional logic, and dependency management at different levels.
| Level | Purpose | Execution | Parallelism |
|---|---|---|---|
| Stage | Major phase (Build, Deploy) | Sequential (by default) | Across stages (optional), within stages (jobs can run parallel) |
| Job | Container for tasks on an agent | Sequential within stage (by default) | Within a stage (can run parallel on different agents) |
| Step | Single unit of work (task/script) | Sequential within job | No (steps are sequential within a job) |
Best Practices
- Organize logically: Use stages for distinct deployment environments or major process phases.
- Leverage parallel jobs: Configure jobs within a stage to run in parallel to save time.
- Keep steps focused: Each step should perform a single, well-defined action.
- Use templates: For complex pipelines, consider using YAML templates to promote reusability and maintainability.
By mastering the concepts of stages, jobs, and steps, you can build robust, efficient, and scalable CI/CD pipelines in Azure DevOps.