pypeline - Define Your CI/CD Pipeline Once, Run It Anywhere

pypeline - Define Your CI/CD Pipeline Once, Run It Anywhere#

CI Status Documentation Status Test coverage percentage

Poetry ruff pre-commit

PyPI Version Supported Python versions License

Pypeline lets you define your build, test, and deployment pipeline in a single YAML file and run it consistently across your local development environment and any CI/CD platform (GitHub Actions, Jenkins, etc.). No more platform-specific configurations – write once, run anywhere.

Key Features

  • Unified Pipeline Definition: Users can define their entire pipeline in a single YAML file, eliminating the need to switch between different syntaxes and configurations for different CI/CD tools.

  • Extensibility: Pypeline supports execution steps defined not only through installed Python packages but also from local scripts.

  • Execution Context: Allow sharing information and state between steps. Each step in the pipeline receives an execution context that can be updated during step execution.

  • Dependency Handling: Every step can register its dependencies and will only be scheduled if anything has changed.

  • Use as a Library: PipelineConfig and PipelineLoader[T] are generic — define your own step base class and use pypeline’s infrastructure (config loading, step resolution, shared context) to build custom pipelines for any domain, not just CI/CD. See Use Pypeline as a Library.

Installation#

Use pipx (or your favorite package manager) to install and run it in an isolated environment:

pipx install pypeline-runner

This will install the pypeline command globally, which you can use to run your pipelines.

[!NOTE] The Python package is called pypeline-runner because the name pypeline was already taken on PyPI. The command-line interface is pypeline.

Documentation: pypeline-runner.readthedocs.io

Quick Start#

pipx install pypeline-runner
pypeline init --project-dir my-pipeline
cd my-pipeline
pypeline run

Walkthrough: Getting Started with Pypeline#

To get started run the init command to create a sample project:

pypeline init --project-dir my-pipeline

The example project pipeline is defined in the pipeline.yaml file.

pipeline:
  - step: CreateVEnv
    module: pypeline.steps.create_venv
    config:
      bootstrap_script: .bootstrap/bootstrap.py
  - step: WestInstall
    module: pypeline.steps.west_install
    description: Download external modules
  - step: MyStep
    file: steps/my_step.py
    description: Run a custom script

This pipeline consists of three steps:

  • CreateVEnv: This is a built-in step that creates a Python virtual environment.

  • WestInstall: This is a built-in step that downloads external modules using the west tool.

  • MyStep: This is a custom step that runs a script defined in the steps/my_step.py file.

You can run the pipeline using the run command:

pypeline run --project-dir my-pipeline

Contributing#

The project uses Poetry for dependencies management and packaging. You can set up the development environment using one of the following methods:

Option 1: Using Poetry directly (minimal setup)

poetry install

Option 2: Using pypeline-runner (runs the full pipeline including tests and checks)

pipx install pypeline-runner
pypeline run

To execute the test suite, call pytest inside Poetry’s virtual environment via poetry run:

poetry run pytest

For those using VS Code there are tasks defined for the most common commands:

  • run tests

  • run all checks configured for pre-commit

  • generate documentation

See the .vscode/tasks.json for more details.

This repository uses commitlint for checking if the commit message meets the conventional commit format.

Credits#

This package was created with Copier and the browniebroke/pypackage-template project template.

Contents