Skip to content

test-framework-dev-guide.md

Latest commit

 

History

History
806 lines (623 loc) · 42.2 KB

test-framework-dev-guide.md

File metadata and controls

806 lines (623 loc) · 42.2 KB
Raw

Developer Guide for the Integration and E2E Testing Framework

The agent testing framework allows running integration and end-to-end tests against real agents installed on remotely provisioned virtual machines. The current set of integration tests can be found in https://github.com/elastic/elastic-agent/tree/main/testing/integration. Each test must start with a define.Require declaration describing how the test can be run. The diagram below provides a high level overview of the testing framework.

Elastic Agent Testing Framework

Prerequisites

Dependencies

Go version

Go version should be at least the same than the one in .go-version file at the root of this repository.

GCloud CLI

The integration testing framework spins up resources in GCP. To achieve this, it needs the GCloud CLI to be installed on the system where the tests are initiated from.

Beats

The Elastic Agent package that is used for integration tests packages Beats from the beats submodule in this repository. The submodule must be explicitly intialized with git submodule update --init on first checkout. The commit of beats packaged and tested in this repository is the commit of beats pinned in the submodule.

Helm & Helm charts

To run the Kubernetes integration tests you need to install helm. Then add and update some helm charts repositories:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Make sure to run helm repo update before running K8s integration tests otherwise the tests will fail stating a chart/version cannot be found.

Configuration

ESS (production) API Key to create on https://cloud.elastic.co/account/keys Warning: if you never created a deployment on it, you won't have permission to get this key, so you will need to create one first.

Setup Serverless deployment

This process is now automated and runs daily, utilizing the existing oblt-cli framework. Serverless deployments are created each day and automatically destroyed every three days.

The automation is configured in the serverless-project.yml file located in the .github/workflows directory.

If necessary, you can create a new serverless deployment manually; the previous deployments will be destroyed automatically, but not immediately. To do so, you need to run the GitHub action called serverless-project.yml.

Credentials for these deployments are securely stored in Google and can only be accessed by Buildkite pipelines. The access control is set using OpenID Connect in Google Cloud Platform. And that's managed by the Robots team.

Running tests

Some integration and E2E tests are safe to run locally. These tests set Local: true in their test functions' define.Require directive. Tests that don't set Local: true or explicitly set Local: false are not considered safe to run locally and will be executed on remote VMs instead.

The framework will look for the agent version defined by the AGENT_VERSION environment variable, even for local tests, regardless of what was defined in the test Fixture. If AGENT_VERSION isn't set, it'll default to the current version without SNAPSHOT.

Setup

One-time setup is required to run any integration and E2E tests. Run mage integration:auth to perform this setup.

You'll also need to separately authenticate to Elastic's docker registry. Go to https://docker-auth.elastic.co/ and authenticate with Okta to receive your credentials.

Packaging

Before you can run any test, you first need to package the Elastic Agent version you want to test. For that you'll need to run mage package, for example:

DEV=true mage -v package

The packaging process has many levers that you may need to set depending on what kind of test you're running. A more advanced invocation might look like:

SNAPSHOT=false PLATFORMS=linux/arm64 PACKAGES=docker DOCKER_VARIANTS=complete mage package
  • DEV=true|false: Build with debug symbols
  • PLATFORMS: Comma separated list of platforms you want to build. If not set, it defaults to the host platform only. Selecting specific platform contains a list of common values. For a full list look at elastic-agent/dev-tools/mage/platforms.go.
  • PACKAGES: Comma separated list of packages you want to build. If not set, defaults to tar.gz for non-Windows platforms and zip for Windows platforms (based on PLATFORMS). Use PACKAGES=all to build all package types. The packages are defined in elastic-agent/dev-tools/mage/pkgtypes.go. To run Linux tests you need to package tar.gz, deb and rpm. The possible options are:
    • tar.gz
    • zip (Windows only)
    • deb
    • rpm
    • docker
  • DOCKER_VARIANTS: When building Docker images, which variants to build. The variants are defined on elastic-agent/dev-tools/mage/dockervariants.go, possible values are:
    • basic
    • ubi
    • wolfi
    • complete
    • complete-wolfi
    • cloud
    • service
    • elastic-otel-collector
    • slim
    • elastic-otel-collector-wolfi
    • slim-wolfi

Choosing dependencies and metadata

By default, packaging will build the binaries defined in this repository:

  • elastic-agent
  • otel-collector (includes agentbeat)
  • osquery-extension

It will pull the remaining dependencies from a manifest specified in .package-version. This process can be controlled through the following environment variables:

  • AGENT_CORE_SOURCE=local|manifest: Build the aforementioned binaries locally or pull from the manifest. Default is local. Manifest is defined either by .package-version or the MANIFEST_URL environment variable.
  • USE_PACKAGE_VERSION=true|false: Use the content of .package-version for configuring the package, most importantly the manifest url, but the version is effective as well. Default is true. Only targets that need consistency with the published snapshot build read this file (package and the targets built on it, downloadManifest, ironbank, the cloud:* image targets, and the integration:* test runners); it has no effect on any other target (for example helm:package or plain builds). Providing MANIFEST_URL also disables it, since the manifest is then authoritative; explicitly setting both MANIFEST_URL and USE_PACKAGE_VERSION=true is an error.
  • MANIFEST_URL: The manifest url from which to pull the dependencies. Takes precedence over .package-version. Default is empty.
  • SNAPSHOT=true|false: Create a snapshot build. This is just versioning metadata indicating that the package doesn't contain a release build. Read from the manifest if present, otherwise defaults to true. For targets that read .package-version, an explicit value that contradicts the file is an error, because mixing snapshot dependencies with a release package (or vice versa) produces a broken artifact.
  • EXTERNAL=true|false: If you want to build with dependencies you've provided locally (you have a custom build of endpoint, for example), then set this to false and USE_PACKAGE_VERSION=false. Default is true.

For example, if you want to create a package the same way as the unified release job, you'd run:

MANIFEST_URL=... AGENT_CORE_SOURCE=manifest mage package

Running the tests

The test are run with mage using the integration namespace, they share similar leavers as the packaging process.

  • AGENT_VERSION: The version to test, in the format 9.2.0-SNAPSHOT. It is REQUIRED to be set when packages were built with SNAPSHOT=true (the default on main). Currently there is no way to tell the integration tests framework to use snapshot versions if testing on VMs.

  • SNAPSHOT=true|false: Use snapshot build when running Kubernetes tests.

  • INSTANCE_PROVISIONER: Sets the provisioner used to create instances, possible values are:

    • gcloud: Uses the gcloud CLI to create VMs on GCP, if not set, that's the default.
    • multipass: Uses Multipass to create local VMs.
    • kind: Uses Kind to run Kubernetes in Docker. This needs to be set if running Kubernetes integration tests.
    • docker: Runs each test batch in a local, systemd-enabled Docker container (with sshd) instead of a VM. It builds an Ubuntu image on first use and the runner drives the container over SSH exactly like a VM. Requires Docker.

  • STACK_PROVISIONER: Sets the provisioner used to create the Elastic Stack (Elasticsearch, Kibana, Fleet Server) the tests talk to. Possible values are:

    • stateful: Provisions a cloud ESS deployment (the default). Requires an ESS API key (mage integration:auth).

    • serverless: Provisions a cloud serverless project. Also requires an ESS API key.

    • external: Uses an existing stack supplied via environment variables (ELASTICSEARCH_HOST, KIBANA_HOST, ELASTICSEARCH_USERNAME, ELASTICSEARCH_PASSWORD). It creates and deletes nothing, and needs no cloud account or local stack. Useful in CI, where a stack is provisioned separately before the tests run.

    • local: Brings up a fully local stack with elastic-package stack up — no cloud account needed. Requires the elastic-package binary on PATH (go install github.com/elastic/elastic-package@latest, or point ELASTIC_PACKAGE_BIN at it). It uses a dedicated elastic-package profile (elastic-agent-integration, override with ELASTIC_PACKAGE_PROFILE) so it does not disturb a stack you may run yourself in the default profile.

      The stack pulls Elastic images from docker.elastic.co, including the package-registry image, which requires authentication. Run a one-time docker login docker.elastic.co before first use. (elastic-package has no supported way to skip the package-registry — Kibana depends on it — and its image has no override, so authenticating is currently the simplest path.)

    The local stack is designed to pair with the docker instance provisioner. The local stack serves Elasticsearch/Kibana/Fleet over HTTPS with a self-signed CA, so the runner:

    • installs that CA into the test container's system trust store, so both the test clients and the agent under test trust the stack with no --insecure or per-test certificate configuration (both fall back to the system trust store when no CA is configured); and
    • attaches the test container to elastic-package's compose network, so the container reaches the stack by service name (https://elasticsearch:9200, https://kibana:5601, https://fleet-server:8220) — the names covered by each service certificate's SANs, so TLS hostname verification passes; and
    • relaxes a couple of Elasticsearch cluster settings on the single-node local stack (disables disk-watermark-based allocation and raises the per-node shard cap), since the suites create many data streams and would otherwise hit 503 no_shard_available_action_exception.

    It also works with mage integration:local: host-mode tests use the stack's published loopback endpoints and trust its CA via SSL_CERT_FILE, without modifying the host's system trust store.

    Remote stack provisioners work with both local and remote instance provisioners. The local stack provisioner requires a local instance provisioner with an implemented connectivity bridge; currently docker and the host-local runner are supported. Multipass and Kind are rejected before provisioning until their local-stack networking support is implemented.

    On macOS you need to build a Linux package on the same arch as your host.

    Example (no cloud account required):

    PLATFORMS=linux/$(go env GOARCH) mage package
STACK_PROVISIONER=local INSTANCE_PROVISIONER=docker \
  TEST_PLATFORMS="linux/$(go env GOARCH)/ubuntu/24.04" AGENT_VERSION="9.5.0-SNAPSHOT" \
  TEST_PACKAGES=tar.gz mage integration:single TestSystemMetricsWithLogstashOutput

When running local mode integration tests, BUILD_AGENT=true will build the agent for the current platform before running.

An example for running a single test, including packaging the artifacts for it is:

DEV=true PACKAGES="tar.gz,rpm,deb" PLATFORMS="linux/amd64" mage package # create elastic-agent snapshot package (EXTERNAL=true and snapshot state from .package-version by default)
INSTANCE_PROVISIONER="multipass" TEST_PLATFORMS="linux/amd64" mage integration:single $TEST_NAME # Run TEST_NAME on a multipass VM

TL;DR: Packaging and running tests

Package the Elastic Agent

# SNAPSHOT state comes from .package-version (true on main, false on release branches); pass SNAPSHOT=false to override.
DEV=true PACKAGES="tar.gz,deb,rpm" PLATFORMS=linux/amd64 mage -v package

# If running Kubernetes tests. Adjust the variants according to your tests
DEV=true PACKAGES="tar.gz,deb,rpm" DOCKER_VARIANTS="basic,complete,elastic-otel-collector" PLATFORMS=linux/amd64 mage -v package

Run the tests

# If testing on VMs. Adjust the AGENT_VERSION according to your build.
AGENT_VERSION="9.2.0-SNAPSHOT" mage -v integration:single TestLogIngestionFleetManaged

# If testing on Kubernetes
SNAPSHOT=true INSTANCE_PROVISIONER=kind mage -v integration:testKubernetesSingle TestKubernetesJournaldInput

ESS oriented tests

  • mage integration:test to execute all tests under the testing/integration/ess folder. All tests are executed on remote VMs, including those that set Local: true.

  • mage integration:local [testName|all] to execute only those tests under the testing/integration/ess folder that set Local: true. It'll run all the tests if all is passed as argument, or it'll pass [testName] to go test as --run=[testName]. These tests are executed on your local machine.

  • mage integration:single [testName] to execute a single test under the testing/integration/ess folder. Only the selected test will be executed on remote VMs.

  • mage integration:matrix to run all tests under the testing/integration/ess folder on the complete matrix of supported operating systems and architectures of the Elastic Agent.

Kubernetes oriented tests

  • INSTANCE_PROVISIONER=kind mage integration:testKubernetes to run kubernetes tests under the testing/integration/k8s folder for the default image on the default version of kubernetes (all previous commands will not run any kubernetes tests).

  • INSTANCE_PROVISIONER=kind mage integration:testKubernetesMatrix to run a matrix of kubernetes tests under the testing/integration/k8s folder for all image types and supported versions of kubernetes.

  • INSTANCE_PROVISIONER=kind mage integration:testKubernetesSingle [testName|all] to execute a single test under the testing/integration/k8s folder. Only the selected test will be executed.

Serverless oriented tests

  • mage integration:testServerless to execute all tests under the testing/integration/serverless folder. All tests are executed on remote VMs, including those that set Local: true.

  • mage integration:testServerlessSingle [testName|all] to execute a single test under the testing/integration/serverless folder. Only the selected test will be executed on remote VMs.

Resource leaks tests

  • mage integration:testForResourceLeaks to execute all tests under the testing/integration/leak folder. All tests are executed on remote VMs.

  • mage integration:testForResourceLeaksSingle [testName|all] to execute a single test under the testing/integration/leak folder. Only the selected test will be executed on remote VMs.

You can list all available mage targets by running mage -l

Selecting specific platform

By default, the runner will deploy to every combination of operating system and architecture that the tests define as supporting. When working on tests and debugging an issue it's better to limit the operating system and architecture to a specific one. This can be done inside a test but requires the test code to be modified. An easier way is available using the TEST_PLATFORMS="linux/amd64" environment variable. This variable can take multiple definitions with a space between, and it can be very specific or not very specific.

  • TEST_PLATFORMS="linux" mage integration:test to execute tests only on Linux using both AMD64 and ARM64.
  • TEST_PLATFORMS="linux/amd64" mage integration:test to execute tests only on Linux AMD64.
  • TEST_PLATFORMS="linux/arm64/ubuntu" mage integration:test to execute tests only on Ubuntu ARM64.
  • TEST_PLATFORMS="linux/amd64/ubuntu/20.04" mage integration:test to execute tests only on Ubuntu 20.04 ARM64.
  • TEST_PLATFORMS="windows/amd64/2022" mage integration:test to execute tests only on Windows Server 2022.
  • TEST_PLATFORMS="linux/amd64 windows/amd64/2022 mage integration:test to execute tests on Linux AMD64 and Windows Server 2022.
  • INSTANCE_PROVISIONER="kind" TEST_PLATFORMS="kubernetes/arm64/1.36.1/wolfi" mage integration:testKubernetes to execute kubernetes tests on Kubernetes version 1.36.1 with wolfi docker variant under kind cluster.

Note

This only filters down the tests based on the platform. It will not execute a tests on a platform unless the test defines as supporting it.

Selecting specific group

By default, the runner will run all test groups. Each group runs on a dedicated machine instance. When working on groups of tests it's better to limit to a specific group of tests instead of running all tests. This can be done by using the TEST_GROUPS="default upgrade-standalone" environment variable. This variable can take multiple groups with a space between.

  • TEST_GROUPS="default" mage integration:test to execute only tests in the "default" group.
  • TEST_GROUPS="default upgrade-standalone" mage integration:test to execute only tests in the "default" or "upgrade-standalone" group.

Passing additional go test flags

When running the tests we can pass additional go test flag using the env variable GOTEST_FLAGS.

These flags are passed also when calculating batches for remote execution of integration tests. This allows for selecting a subset of test in a convenient way (see examples below)

This feature is intended mostly for integration tests debugging/development without the need for new mage targets corresponding to a new set of test flags.

A few examples:

Run a single test with an exact match

We want to run only the test named "TestStandaloneUpgrade" GOTEST_FLAGS="-test.run ^TestStandaloneUpgrade$" mage integration:test

Run a tests matching a partial expression

We want to run any test with "Upgrade" in the name GOTEST_FLAGS="-test.run Upgrade" mage integration:test

Run a single test and signal that we want the short version

We pass a -test.short flag along with the name match GOTEST_FLAGS="-test.run ^TestStandaloneUpgrade$ -test.short" mage integration:test

Run a single test multiple times

We pass a -test.count flag along with the name match GOTEST_FLAGS="-test.run ^TestStandaloneUpgrade$ -test.count 10" mage integration:test

Run specific tests

We pass a -test.run flag along with the names of the tests we want to run in OR GOTEST_FLAGS="-test.run ^(TestStandaloneUpgrade|TestFleetManagedUpgrade)$" mage integration:test

Selecting specific upgrade versions

By default, upgrade tests read the list of versions from testing/integration/testdata/.upgrade-test-agent-versions.yml. When developing or debugging upgrade tests it's useful to limit to specific versions using the TEST_UPGRADE_VERSIONS environment variable. This variable takes a comma-separated list of versions and is passed to the remote test runner.

  • TEST_UPGRADE_VERSIONS="9.3.1" mage integration:single TestStandaloneUpgrade to test upgrading from 9.3.1 only.
  • TEST_UPGRADE_VERSIONS="9.3.1,9.2.6" mage integration:single TestStandaloneUpgrade to test upgrading from 9.3.1 and 9.2.6.
Run Serverless tests

The test framework includes a smoke test suite to check elastic-agent in a serverless environment. The suite can be run via the integration:TestServerless mage target.

Run Extended Runtime Leak Test

The test framework includes a "long running" test to check for resource leaks and stability. The runtime of the test can be set via the LONG_TEST_RUNTIME environment variable. The test itself can be run via the integration:TestForResourceLeaks mage target.

Limitations

Due to the way the parameters are passed to devtools.GoTest the value of the environment variable is split on space, so not all combination of flags and their values may be correctly split.

Test output / live progress (GOTESTSUM_FORMAT)

Tests are driven by gotestsum. By default remote runs use its quiet format, which prints nothing until a package finishes — so a long-running test appears to hang with no feedback. Set GOTESTSUM_FORMAT to change the output format (see gotestsum --help for the full list); the most useful values are:

  • standard-verbose — streams the full go test -v output live, including each test's t.Log progress. Best for watching a single long test.
  • testname — prints one line per test as it finishes. Less noisy for large batches.

GOTESTSUM_FORMAT is honored locally and propagated to the remote host, so it works for integration:test/integration:single as well. For the local provisioners (multipass/kind/docker) and mage integration:local, standard-verbose is the default (a human is watching); set GOTESTSUM_FORMAT explicitly to override. CI (gcloud) keeps the quiet default unless the variable is set.

Cleaning up resources

The test run will keep provisioned resources (instances and stacks) around after the tests have been ran. This allows following mage integration:* commands to re-use the already provisioned resources.

  • mage integration:clean will de-provision the allocated resources and cleanup any local state.

Tests with external dependencies might need more environment variables to be set when running them manually, such as ELASTICSEARCH_HOST, ELASTICSEARCH_USERNAME, ELASTICSEARCH_PASSWORD, KIBANA_HOST, KIBANA_USERNAME, KIBANA_PASSWORD, and ELASTIC_APM_SERVER_URL.

TEST_INTEG_CLEAN_ON_EXIT=true|false will determine whether mage artifacts and .integration-cache are cleaned on exit automatically. Defaults to false (no automatic cleanup) for local runs; CI sets this to true explicitly.

Debugging tests

Connecting to VMs

All VMs (including Windows) support connections via SSH, the framework generates and stores the necessary SSH keys to access the VMs, the easiest way to connect to them is using the SSH command returned by mage integration:SSH. It will list the VMs and ask to select one.

On a Unix shell you can run $(mage integration:SSH), the menu is printed to stderr and the SSH command to stdout. After selecting the VM you will have shell connected to it.

Credentials for cloud stack/projects

All cloud deployments and projects can be listed with mage integration:listStacks, they can be used to manually connect to Kibana and Elasticsearch.

If you need to manually run tests against any deployments, mage integration:GenerateEnvFile will generate a file called env.sh that exports environment variables for Unix compatible shells, you can load them into your shell by running source ./env.sh.

To easily deploy the credentials to any VM, just run mage integration:DeployEnvFile. A menu will ask for the desired Stack and VM.

Manually running the tests (using go test)

If you want to run the tests manually, skipping the test runner, set the TEST_DEFINE_PREFIX environment variable to any value and run your tests normally with go test. E.g.:

TEST_DEFINE_PREFIX=gambiarra go test -v -tags integration -run TestProxyURL ./testing/integration/

You will need the environment variables containing the stack URL and credentials for the tests to succeed.

Installing debug/build tools

mage integration:DeployDebugTools will install a few tools necessary to build the Elastic-Agent in the VM and debug tests:

  • Docker
  • Delve
  • Mage

When called, it will show a menu to select a VM and then install the tools listed above. It will also create the ~/elastic-agent folder containing the Git repository (required o package from within the VM) and the last version of the code uploaded to the VM. This allows you to easily build/package the Elastic-Agent from within the VM as well as run any tests.

In the VM there are two important folders:

  • agent: that is created by the integration test framework and used by mage to run the tests, it gets updated every time you run an integration test from your machine.
  • elastic-agen: that is a copy agent with Git information created by mage integration:DeployDebugTools, the Git information there is not a copy from your machine, but it will work if you need to package the Elastic-Agent from the VM. Most of the time you won't need it.

Step-by-Step commands

## Install DebugTools
mage -v integration:DeployDebugTools

## Generate and deploy env file
mage -v integration:DeployEnvFile

## SSH into the VM
$(mage integration:SSH)

## From inside the VM, the test needs root
sudo su
source ./env.sh
cd agent # That's the folder the mage automation uses to run the tests

## Then run the test using `go test`

TEST_DEFINE_PREFIX=gambiarra AGENT_VERSION="8.16.0-SNAPSHOT" go test -tags=integration -v ./testing/integration/ -run TestLogIngestionFleetManaged

## Run the test using delve:
## Any flags passed to the test binary go after the '--', they also need to
## include the `test.` prefix if they're for `go test`
TEST_DEFINE_PREFIX=gambiarra dlv test ./testing/integration/ --build-flags="-tags integration" -- -test.v -test.run TestLogIngestionFleetManaged

A Delve trick: If you didn't build the Elastic-Agent directly on the machine you're debugging, it is very likely the location of the source code is different, hence delve cannot show you the code it is running. To solve this, once on Delve shell, run: config substitute-path /go/src/github.com/elastic/elastic-agent /home/ubuntu/agent where:

  • /go/src/github.com/elastic/elastic-agent is the path annotated in the binary you are debugging (the one Delve shows).
  • /home/ubuntu/agent is where Delve should read the source code form.

Other useful mage targets:

  • integration:stacks lists all stack deployments and connection information in a human readable table.
  • integration:listInstances lists all VMs and their connection command in a human readable table. It also lists the URL for the VM page on GCP, which is helpful to verify if the VM still exists (GCE VMs are automatically deleted)
  • integration:printState is a shortcut for running the two commands above.

Auto diagnostics retrieval

When an integration test fails the testing fixture will try its best to automatically collect the diagnostic information of the installed Elastic Agent. In the case that diagnostics is collected the test runner will automatically transfer any collected diagnostics from the instance back to the running host. The results of the diagnostic collection are placed in build/diagnostics.

Gather diagnostics manually

In the case that you want to run the integration testing suite and have it gather the diagnostics at the end of every tests you can use the environment variable AGENT_COLLECT_DIAG=true. When that environment variable is defined it will cause the testing fixture to always collect diagnostics before the uninstall in the cleanup step of a test.

Keeping Elastic Agent installed

When the testing fixture installs the Elastic Agent, it will automatically uninstall the Elastic Agent during the cleanup process of the test. In the event that you do not want this to occur, you can disable the auto-uninstallation using the AGENT_KEEP_INSTALLED=true environment variable. If the test succeeds, the agent will be uninstalled regardless of the value of AGENT_KEEP_INSTALLED.

  • AGENT_KEEP_INSTALLED=true mage integration:single [testName]

Run until failure

In the case that you're tracking down a flaky test it is helpful to have it keep running until it fails. The testing suite has this ability built into it. Using the TEST_RUN_UNTIL_FAILURE=true will keep running the testing suite until it reports a failure.

  • TEST_RUN_UNTIL_FAILURE=true mage integration:single [testName]

Running integration tests with local changes in beats (for otel receivers)

If you've made local changes to Beats-OTel and want to run the Agent's integration tests against those changes, follow these steps:

  1. Update go.mod and add a replace directive as follows (change the path to match your local machine):
replace github.com/elastic/beats/v7 => /Users/vihasmakwana/Desktop/Vihas/elastic/beats
  1. Package the Agent with EXTERNAL=true:
EXTERNAL=true PLATFORMS=darwin/arm64 mage package
  1. Run integration tests as per the instructions

Note

Make sure you add an absolute path in replace directive

Note

Old agent might be cached at .agent-testing directory. Run mage integration:clean to clean it.

Writing tests

Write integration and E2E tests by adding them to the testing/integration folder.

// TODO: Replace with a comprehensive write-up of define.* directives, // environment variables, etc. useful when writing tests. Until then...

Look at existing tests under the testing/integration for examples of how to write tests using the integration and E2E testing framework. Also look at the github.com/elastic/elastic-agent/pkg/testing/define package for the test framework's API and the github.com/elastic/elastic-agent/pkg/testing/tools package for helper utilities.

Test group

Every define.Require must define a Group that it belongs too. Each group is executed on a separate instance with all tests with in the same group executed on the same instance. Placing similar tests in the same group allows those tests to run on its own instance as well as provides a way for a developer to select a specific group of tests with TEST_GROUP="{group-name}".

Grouping tests is another way of spreading out the testing load across multiple instances. The more groups that are defined the more instances will be provisioned to complete all tests. A balance between a small good set of groups is better than a ton of groups each executing a small set of tests, as the time to set up an instance can out weight the benefits of creating another group.

Creating a new test group and Buildkite integration tests

When creating a new test group, it is important to add the new group to the job in the .buildkite/bk.integration.pipeline.yml file. This will ensure that the new group is executed in the CI pipeline.

Add the new group to the matrix in the corresponding steps. The matrix is a list of all test groups that are executed in the step. Example:

- label: "x86_64:sudo: {{matrix}}"
      command: |
        ...
      artifact_paths:
        - build/**
      agents:
        provider: "gcp"
        machineType: "n1-standard-8"
        image: "family/platform-ingest-elastic-agent-ubuntu-2404"
      plugins:
        - elastic/vault-secrets#v0.1.0:
            path: "kv/ci-shared/platform-ingest/buildkite_analytics_token"
            field: "token"
            env_var: "BUILDKITE_ANALYTICS_TOKEN"
        - test-collector#v1.11.0:
            files: "build/TEST-*.xml"
            format: "junit"
            debug: true
            annotation-link: true
      matrix:
        - default
        - container
        - fleet-upgrade-to-pr-build
        - upgrade
        - fleet

This requirement is temporary and will be removed once the Buildkite pipeline is updated to automatically detect new test groups.

CI tiers for extended/stateful testing

.buildkite/bk.integration.pipeline.yml runs the "Stateful" (ESS-backed) Linux and Windows integration tests. To keep PR feedback fast while still getting broad OS coverage, the pipeline splits these tests into three tiers:

  • Tier 1 — Runs on every pull request (if: build.pull_request.id != null). It exercises only the default variant on the newest supported OS per platform (currently Ubuntu 24.04 for Linux, Windows Server 2022 for Windows). This tier must stay small and fast, since it gates every PR.
  • Tier 2 — Runs when relevant files change (if_changed, e.g. .buildkite/**, magefile.go, dev-tools/**, go.mod/go.sum). It adds one additional OS per package family (for example Debian 13 for .deb and RHEL 10 for .rpm) to catch packaging/tooling regressions without running on every PR.
  • Tier 3 — Runs only on a schedule or when triggered from the scheduler pipeline (build.source == "schedule" or BUILDKITE_TRIGGERED_FROM_BUILD_PIPELINE_SLUG == "elastic-agent-pipeline-scheduler"). This is the broadest matrix: it exercises the full set of supported OS versions and package variants (including stress), and covers arm64 in addition to amd64. Because it's the most expensive tier, it does not block PRs.

When a new OS image is added to the support matrix (see the IMAGE_* environment variables at the top of bk.integration.pipeline.yml), it should generally be added to the tier 3 matrix.setup.os list first. Promote it to tier 2 or tier 1 only when it needs to run more frequently (e.g. it's becoming the new default, or it validates something that changes often).

Image versions are timestamped VM image names (for example platform-ingest-elastic-agent-ubuntu-2604-1789378451) that are bumped automatically across all .buildkite/*.yml files by the updatecli-bump-vm-images.yml Updatecli pipeline (.ci/updatecli/updatecli-bump-vm-images.yml).

Test namespaces

Every test has access to its own unique namespace (a string value). This namespace can be accessed from the info.Namespace field, where info is the struct value returned from the define.Require(...) call made at the start of the test.

Namespaces should be used whenever test data is being written to or read from a persistent store that's shared across all tests. Most commonly, this store will be the Elasticsearch cluster that Agent components may index their data into. All tests share a single stack deployment and, therefore, a single Elasticsearch cluster as well.

Some examples of where namespaces should be used:

  • When creating a policy in Fleet. The Create Policy and Update Policy APIs takes a namespace parameter.
  • When searching for documents in logs-* or metrics-* data streams. Every document in these data streams has a data_stream.namespace field.

⚠️ Not using namespaces when accessing data in a shared persistent store can cause tests to be flaky.

Alternative Providers

Multipass Instance Provisioner

By default the integration testing suite uses the gcloud CLI to provision GCE instances. In the case that you want to use a local VM instead of a remote VM, you can use the Multipass provisioner.

  • INSTANCE_PROVISIONER="multipass" mage integration:test

It is always best to run mage integration:clean before changing the provisioner because the change will not cause already provisioned resources to be replaced with an instance created by a different provisioner.

Kind Instance Provisioner

Use only when running Kubernetes tests. Uses local installed kind to create Kubernetes clusters on the fly.

  • INSTANCE_PROVISIONER="kind" mage integration:testKubernetes

Docker Instance Provisioner

Runs each test batch in a local, systemd-enabled Docker container (running sshd) instead of a VM. Because the container runs systemd as PID 1, the privileged ("sudo") tests that install the Elastic Agent as a systemd service work locally without provisioning a cloud VM. The existing Linux runner drives the container over SSH exactly like a VM.

  • INSTANCE_PROVISIONER="docker" mage integration:test

Notes:

  • Requires a running Docker daemon. An Ubuntu image (elastic-agent-test-systemd) is built on first use and reused afterwards. The image bakes in the build toolchain (build-essential, unzip) and the exact Go version from .go-version, so the runner's per-instance Prepare step is skipped — containers start ready to build. It also pre-installs mage and gotestsum (at the versions pinned in go.mod) with warm Go module/build caches, so the per-run make mage && mage integration:prepareOnRemote resolves from cache instead of downloading and compiling. The Go/mage/gotestsum versions and a hash of the Dockerfile are part of the image tag, so bumping any of them (.go-version or go.mod) or editing the Dockerfile rebuilds the image.
  • The image also installs Docker Engine and runs a nested daemon (docker-in-docker) so tests that start helper containers via testcontainers' docker-compose (e.g. the Kafka and Logstash output tests in testing/integration/ess/otel_test.go) work. Those helper containers publish their ports on the test container's own localhost, which is what the test code and agent expect (e.g. KAFKA_ADVERTISED_HOST=localhost); a mounted host socket could not provide that. The provisioner waits for the nested daemon to be ready before starting tests. The non-root test user (ubuntu) is in the docker group, so non-sudo tests reach the daemon too. /var/lib/docker and /var/lib/containerd are backed by volumes (removed with the container via docker rm -fv) because the nested daemon's overlay storage cannot stack on the container's own overlay rootfs.
  • Known limitation — kernel audit tests (TestAuditdCorrectBinaries): The Linux kernel audit subsystem (NETLINK_AUDIT) restricts control operations such as AUDIT_GET and AUDIT_ADD_RULE to processes running in the initial PID namespace. Docker containers always run in their own PID namespace; even a fully --privileged container cannot access the audit subsystem. Running with --pid=host would grant access but is incompatible with the container design (systemd must be PID 1). TestAuditdCorrectBinaries therefore cannot pass with the Docker instance provisioner and must be run against a real VM (e.g. INSTANCE_PROVISIONER=multipass), a cloud environment, or locally as root via mage integration:local (which runs directly on the host in the initial PID namespace).

Troubleshooting Tips

Error: GCE service token missing; run 'mage integration:auth'

If you encounter this error when running mage integration:test, it's because the test runner is unable to create VMs on GCP to execute the tests.

As the error message suggests, run mage integration:auth to resolve this error.

Error: missing required Elastic Agent package builds for integration runner to execute: ...

If you encounter this error when running mage integration:test or mage integration:local, it's because the test runner couldn't find the appropriate Agent packages in the build/distributions folder.

Run mage package with the appropriate value(s) in PLATFORMS, as suggested by the error message, to build the necessary Agent packages first.

If the issue is that the built Agent packages contain -SNAPSHOT in their versions, whereas the package names in the error message do not, either pass SNAPSHOT=false to the mage package command OR set the AGENT_VERSION environment variable to a version that includes the -SNAPSHOT suffix when running mage integration:test or mage integration:local.

Failures on reused resources

The integration framework tries to re-use resource when it can. This improves the speed at which the tests can run, but also means its possible for a failed test to leave state behind that can break future runs.

Run mage integration:clean before running mage integration:test to ensure the tests are being run with fresh instances and stack.

Provisioner-related errors

If you encounter any errors during instance provisioning, try running mage integration:clean and then re-running whatever mage integration:* target you were trying to run originally when you encountered the error.

Tests seemingly using a stale Elastic Agent package

If your integration tests seem to be using a stale or outdated version of Elastic Agent, it might be due to a cached copy in the .agent-testing directory. To fix this, run mage clean to remove cached artifacts, then rebuild the agent with mage package, and finally run the integration tests again.

Using a different agent version from the stack version

The agent version is used as a fallback for the stack version to use in integration tests if no other version is specified.

If we need to use a different version between agent and stack we can specify the stack version using a separate env variable AGENT_STACK_VERSION like in this example (we used a custom package version for the agent):

AGENT_VERSION="8.10.0-testpkgversion.1-SNAPSHOT" AGENT_STACK_VERSION="8.10.0-SNAPSHOT" mage integration:test

K8s: pod not read (timeout waiting for k8s object elastic-agent-standalone-xxxxx to be ready/client rate limiter Wait returned an error: context deadline exceeded)

Your Kind cluster might be in a broken state. Run:

% kubectl get nodes
NAME                               STATUS     ROLES           AGE   VERSION
v1.33.0-kubernetes-control-plane   NotReady   control-plane   58m   v1.33.0

And check the status of your node. If it is different than Ready, delete the cluster and create a new one.