Build a GenAIOps pipeline with GitHub Actions

Design around the release unit

A GenAIOps pipeline should make a proposed AI change reviewable and repeatable, not simply run a model call on every commit. The release unit may include application code, prompt templates, tool definitions, retrieval configuration, evaluation cases, infrastructure, and a model or deployment identifier. Decide which of those assets must move together and give the combination a version or commit reference.

• Keep prompts, tool schemas, and retrieval settings in reviewed source control
• Version evaluation cases and expected outcomes separately from production data
• Record the model deployment name or immutable model version when available
• Keep secrets, customer content, and live conversation data out of test fixtures
• Make rollback target a known-good release bundle rather than an undocumented prompt edit

Start with credential-free pull-request checks

Pull requests from forks and untrusted contributors should be treated as hostile input. Their workflow should not receive Azure credentials, production secrets, or write-capable tokens. Run deterministic checks first; add bounded offline evaluation only when the job can use safe fixtures and no privileged credentials.

name: infrastructure-check

on:
  pull_request:
    paths:
      - "infra/**"
      - ".github/workflows/infra-check.yml"
  push:
    branches: [main]
    paths:
      - "infra/**"
      - ".github/workflows/infra-check.yml"

permissions:
  contents: read

jobs:
  validate:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: infra
    steps:
      - name: Check out source
        uses: actions/checkout@v4
      - name: Install Terraform
        uses: hashicorp/setup-terraform@v3
      - name: Check formatting
        run: terraform fmt -check -recursive
      - name: Initialize without a backend
        run: terraform init -backend=false -input=false
      - name: Validate configuration
        run: terraform validate -no-color

• Lint and unit-test the application
• Validate schemas, prompt templates, tool contracts, and retrieval configuration
• Check Terraform formatting and configuration without contacting a cloud backend
• Run a small, versioned evaluation set with explicit pass/fail criteria
• Publish a concise summary that links the tested commit to its evaluation results

Evaluate behavior, not just successful execution

A model returning HTTP 200 is not evidence that an AI feature is useful or safe. Build a small evaluation set from representative tasks and known failure cases. Score outcomes against the product contract, then retain the examples and evaluator version so a later release can be compared with the same baseline.

# Replace these names and thresholds with checks from your own evaluation runner.
steps:
  - name: Run deterministic tests
    run: make test
  - name: Run offline AI evaluation
    run: make eval-ci
  - name: Enforce release thresholds
    run: python scripts/check_eval_results.py --policy eval/release-policy.json
# The policy should fail closed for critical safety or quality regressions.

• Task completion, answer correctness, and groundedness where sources are required
• Citation validity and correct abstention when evidence is missing
• Tool choice, argument validity, authorization, and approval behavior
• Prompt-injection resistance and refusal behavior for disallowed requests
• Latency, token or inference cost, and failure rates against stated budgets

Separate validation from Azure deployment

Use GitHub OpenID Connect (OIDC) with Microsoft Entra workload identity federation instead of storing a long-lived client secret. Configure the federated credential to trust only the repository and the exact protected branch, tag, or GitHub Environment used by the deployment. Grant id-token permission only to the job that needs it, and use a production Environment with branch restrictions and required reviewers where your repository plan supports them. The YAML below is a security-oriented skeleton: replace its final script path with your reviewed deployment implementation, and configure the federated credential and Environment protections before enabling the workflow.

name: deploy-production

on:
  push:
    branches: [main]
    paths:
      - "app/**"
      - "prompts/**"
      - "infra/**"
      - ".github/workflows/deploy-production.yml"

permissions:
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    concurrency:
      group: production-deployment
      cancel-in-progress: false
    permissions:
      contents: read
      id-token: write
    steps:
      - name: Check out the protected main branch
        uses: actions/checkout@v4
      - name: Sign in to Azure with GitHub OIDC
        uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      - name: Deploy the reviewed release
        run: ./scripts/deploy-production.sh

• Keep pull-request jobs read-only and cloud-credential-free
• Use separate identities for test and production, with the narrowest practical Azure roles
• Store the client, tenant, and subscription identifiers as environment-scoped GitHub secrets if required by the action; OIDC removes the client secret, not the need to configure identity
• Pin third-party actions to reviewed full commit SHAs in production workflows
• Never build or execute untrusted pull-request code inside a privileged deployment job

Treat Terraform state and plans as sensitive

Terraform has two Azure authentication surfaces to configure: the provider that manages resources and the backend that reads and writes state. Configure OIDC for both when using the AzureRM backend, and grant the state identity only the required data-plane access, commonly Storage Blob Data Contributor scoped to the state container. Separate plan and apply identities when practical; a pull-request plan must never inherit production write access.

• Use remote state with locking and recovery/versioning appropriate to the environment
• Do not commit state, plan files, credentials, or generated Terraform directories
• Treat saved plan files and logs as potentially sensitive because they can expose values
• Do not publish a plan artifact broadly; if a reviewed plan must cross jobs, tightly restrict artifact access and retention
• Run apply only from a trusted ref after the appropriate deployment protection rules pass

Promote the same release and keep evidence

After test deployment, run smoke checks against the deployed service with synthetic or approved test inputs. Promote the exact tested build and configuration rather than rebuilding from a moving branch. Attach the commit, container or package digest, prompt and evaluation versions, deployment identifier, and rollback target to the release record.

• Verify health, authentication, retrieval, and required safety behavior
• Compare smoke-evaluation results with the approved baseline
• Use staged rollout or traffic shifting when the service supports it
• Watch latency, error rate, quality signals, and cost after release
• Define a human owner and an explicit rollback trigger before production traffic

Roll out in small, verifiable steps

A useful first pipeline does not need a large platform team or a complicated agent framework. Add one trust boundary at a time and make every gate visible in the pull request or deployment record.

• Stage 1: lint, unit tests, schema checks, and Terraform validation
• Stage 2: a small offline evaluation suite with critical safety cases
• Stage 3: a test deployment using a dedicated OIDC identity and smoke checks
• Stage 4: a protected production Environment, reviewed action pins, monitoring, and a tested rollback

Choose your next step

Use the roadmap as a loop, not a reading list. Pick the next action that produces evidence.

Frequently asked questions