Theoretical foundations and architecture
Laboratory session: building a carrier-grade NetDevOps CI/CD pipeline using Git version control, GitHub Actions / GitLab CI, and ephemeral test testbeds (Containerlab / EVE-NG). Fragile manual CLI modifications are deprecated in favor of an automated pipeline: 1. The engineer declares service intent inside a structured Git repository (e.g., editing customers.yml) and opens a Merge Request. 2. Linting & Schema Validation: CI runners validate the data model against strict JSON schemas, ensuring RFC compliance, unique RD/RT allocation, and non-overlapping IP assignments. 3. Ephemeral Staging & Automated Testing: CI spins up an isolated digital twin inside Containerlab, renders target vendor configurations via Jinja2, deploys them to virtual nodes, and runs automated testing frameworks (PyATS / Batfish / Robot Framework) verifying BGP peering, L3VPN ping reachability, and zero route leaking. 4. Canary Deployment: upon pipeline pass, changes are deployed to production PE nodes with automated rollback triggers monitoring real-time telemetry.
The pipeline as a change contract
PE configuration is changed not by a person on the device but by a Git commit that passes a set of checks. The pipeline solves three problems: it does not let a clearly wrong intent through (data, policy), it shows exactly what will change (render and diff), and it delivers the change so that it can be stopped and rolled back (canary, confirmation, rollback). Each stage must fail loudly and before the next device is touched.
| Stage | What it checks | Tools | What it blocks |
|---|---|---|---|
| lint | YAML syntax, playbook style, render stability | yamllint, ansible-lint, flake8, ansible-playbook --syntax-check | broken YAML and wrong modules |
| policy | the customer RT is in the allowlist, RD uniqueness, the ASN:NN format | Pydantic/JSON Schema, pytest | route leaks between customers |
| render | deterministic rendering of the four vendor templates | Jinja2 with StrictUndefined | empty values in the config |
| diff | what changes relative to the reference in Git and on the device | git diff, --check --diff, compare_config | unexpected deletions |
| static | analysis of rendered configurations | Batfish (Cisco, Junos), own tests for Huawei and RouterOS | loops and policy overlaps |
| twin | a live test in a virtual topology | containerlab, vrnetlab (if images are available) | what is not visible statically |
| canary | one PE with a rollback timer and a post-check | Ansible serial, commit confirmed | mass damage |
| rollout | the remaining PEs after manual confirmation | when: manual, resource_group | races between pipelines |
Data and policy matter more than syntax
Syntax errors are rejected by the device itself: % Invalid input is visible at once. What is dangerous are correct but semantically wrong changes: customer A's RT equal to customer B's RT, overlapping RDs, a VRF bound to someone else's interface. The device accepts such a config without objection, and the consequences show up as a route leak. So the main gate in CI is policy over the data: an RT allowlist per customer, RD uniqueness on a PE, a ban on importing a foreign RT. A negative test is mandatory: a commit with RT 65000:200 for A must fail the pipeline. If there is no such test, you do not know whether the gate works.
Render in Git and a diff for review
An effective practice is to commit not only the data but also the rendered configurations to rendered/<pe>.cfg. Then the reviewer sees a real diff for each device in the merge request, not an abstract YAML change. The idempotency check comes down to this: a repeated render on unchanged data does not change any file, and --check --diff on the device returns changed=0.
# idempotency gate: second run must not change anything
python render.py --out build/
git diff --exit-code build/ rendered/
ansible-playbook pe_vrf.yml --check --diff | grep -q "changed=0"Canary, confirmation and rollback
- Junos:
commit check, thencommit confirmed N: if the finalcommithas not arrived in N minutes, the configuration rolls back by itself. This protects against loss of management. - IOS-XE: before the change a reference is saved (
copy running-config flash:pre-change.cfg); rollback goes throughconfigure replace flash:pre-change.cfg. The modeconfigure replace ... time Nstarts a timer that must be confirmed with theconfigure confirmcommand. - Huawei VRP: in VRPv8 the configuration is applied with the
commitcommand, and the history is available throughdisplay configuration commit list; rollback is done to a saved state. The exact commands depend on the version and are verified in the lab. - RouterOS: no transactions. Before the change,
/system/backup/saveand/exportare made; restoring from a backup (/system/backup/load) reboots the router, so on RouterOS the strategy is built on small idempotent changes and manual confirmation.
A canary is not only one PE but also a waiting time: the post-check reads the state (the RT is in place, the customer prefixes are visible on the second PE, the session with the RR is Established), then the confirmation fires, and on error the rollback starts. Rolling back the configuration and restoring the state of the network are different things: after a rollback you need a soft BGP clear, otherwise the tables will not be recalculated.
Concurrency and drift
Two pipelines changing the same PE at the same time create a race: the last one wins, even if its branch is stale. Protection: resource_group in GitLab (or concurrency in GitHub Actions), rollout only from the protected main branch, a requirement for an up-to-date merge request base, and a rendered from <sha> label in the configuration. A scheduled pipeline runs check mode and looks for drift: a manual edit on a device that nobody committed. Any non-zero diff is an incident or a task to fix the data.
Secrets, permissions and audit
Secrets do not live in the repository: masked and protected CI variables, Vault, short-lived tokens. The automation account has a minimal role, and the source is restricted to the runner address. The change record consists of the merge request, pipeline logs, diff artifacts and entries on the device (show system commit on Junos, %SYS-5-CONFIG_I on IOS-XE). Two approvers are required for RT policy changes and a mandatory manual step before rollout. Job logs must not print secret values (no_log: true, set -x disabled).
A virtual lab as the last filter
Static analysis sees only configurations. Containerlab with virtual router images tests real behavior: it brings up PEs and an RR, applies the generated configurations, and checks vpnv4 sessions and a ping between customer sites. Vendor images are licensed separately, so not everyone has this stage; without it the canary remains the real protection on live equipment.
