Linux hosted agents

Size

Buildkite offers a selection of instance sizes, allowing you to tailor your hosted agents' resources to the demands of your jobs. Below is a breakdown of the available sizes.

Cache volumes

Cache volumes are external volumes attached to hosted agent instances. These volumes are attached on a best-effort basis depending on their locality, expiration and current usage, and therefore, should not be relied upon as durable data storage.

By default, cache volumes:

• Are disabled, although you can enable them by providing a list of paths to cache at the pipeline- or step-level.
• Are scoped to a pipeline and are shared between all steps in the pipeline.

Cache volumes act as regular disks with the following properties:

• The volumes use NVMe storage, delivering high performance.
• The volumes are formatted as a regular Linux filesystem (e.g. ext4)—therefore, these volumes support any Linux use-cases.

Cache configuration

Cache paths can be defined in your pipeline.yml file. Defining cache paths for a step will implicitly create a cache volume for the pipeline.

When cache paths are defined, the cache volume is mounted under /cache in the agent instance. The agent links subdirectories of the cache volume into the paths specified in the configuration. For example, defining cache: "node_modules" in your pipeline.yml file will link ./node_modules to /cache/bkcache/node_modules in your agent instance.

Custom caches can be created by specifying a name for the cache, which allows you to use multiple cache volumes in a single pipeline.

When requesting a cache volume, you can specify a size. The cache volume provided will have a minimum available storage equal to the specified size. In the case of a cache hit (most of the time), the actual volume size is: last used volume size + the specified size.

Defining a top-level cache configuration (as opposed to one within a step) sets the default cache volume for all steps in the pipeline. Steps can override the top-level configuration by defining their own cache configuration.

cache:
  paths:
    - "node_modules"
  size: "100g"

steps:
  - command: "yarn run build"
    cache: ".build"

  - command: "yarn run test"
    cache:
      - ".build"

  - command: "rspec"
    cache:
      paths:
        - "vendor/bundle"
      size: 20g
      name: "bundle-cache"

Required attributes:

Optional attributes:

Lifecycle

At any point in time, multiple versions of a cache volume may be used by different jobs.

The first request creates the first version of the cache volume, which is used as the parent of subsequent forks until a new parent version is committed. A fork in this context is a "moment", or a readable/writable "snapshot", version of the cache volume in time.

When requesting a cache volume, a fork of the previous cache volume version is attached to the agent instance. This is the case for all cache volumes, except for the first request, which starts empty, with no cache volumes attached.

Each job gets its own private copy of the cache volume, as it existed at the time of the last cache commit.

Version commits follow a "last write" model: whenever a job terminates successfully (that is, exits with exit code 0), cache volumes attached to that job have a new parent committed: the final flushed volume of the exiting agent instance.

Whenever a job fails, the cache volume versions attached to the agent instance are abandoned.

Billing model

Cache volumes are charged at an initial fixed cost per pipeline build when a cache path (for example, cache: "node_modules") is defined at least once in the pipeline's pipeline.yml file. This fixed cost is the same, regardless of the number of times a cache path is defined/used in the pipeline.yml file.

An additional (smaller) charge is made per gigabyte of active cache, where active cache is defined as any cache volume used in the last 24 hours.

Git mirror cache

The Git mirror cache is a special type of cache volume that is used to speed up Git operations by caching the Git repository between builds. This is useful for large repositories that are slow to clone.

Git mirror caching can be enabled on the cluster's cache volumes settings page. Once enabled, the Git mirror cache will be used for all hosted jobs in that cluster. A separate cache volume will be created for each repository.

Container cache

The container cache can be used to cache Docker images between builds.

Container caching can be enabled on the cluster's cache volumes settings page. Once enabled, a container cache will be used for all hosted jobs in that cluster. A separate cache volume will be created for each pipeline.

Agent images

Buildkite provides a Linux agent image pre-configured with common tools and utilities to help you get started quickly. This image also provides tools required for running jobs on hosted agents.

The image is based on Ubuntu 20.04 and includes the following tools:

• docker
• docker-compose
• docker-buildx
• git-lfs
• node
• aws-cli

You can customize the image that your hosted agents use by creating an agent image or by providing a URL of an existing image.

Create an agent image

Creating an agent image requires you to define a Dockerfile that installs the tools and utilities you require. This Dockerfile should be based on the Buildkite hosted agent base image.

An example Dockerfile that installs the awscli and kubectl:

# Set the environment variable to avoid interactive prompts during awscli installation
ENV DEBIAN_FRONTEND=noninteractive

# Install AWS CLI
RUN apt-get update && apt-get install -y awscli

# Install kubectl using pkgs.k8s.io
RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl \
    && chmod +x kubectl \
    && mv kubectl /usr/local/bin/

You can create an agent image:

1. Select Agents in the global navigation to access the Clusters page.

2. Select the cluster in which to create the new agent image.

   Note: Before continuing, ensure you have created a hosted agent queue (based on Linux architecture) within this cluster. Learn more about how to do this in Create a queue.

3. Select Agent Images to open the Agent Images page.

4. Select New Image to open the New Agent Image dialog.

5. Enter the Name for your agent image.

6. In the Dockerfile field, enter the contents of your Dockerfile.

   Note: The top of the Dockerfile contains the required FROM instruction, which cannot be changed. This instruction obtains the required Buildkite hosted agent base image.

7. Select Create Agent Image to create your new agent image.

Use an agent image

Once you have created an agent image, you can set it as the default image for any hosted agent queues based on Linux architecture within this cluster. Once you do this for such a hosted agent queue, any agents in the queue will use this agent image in new jobs.

To set a Linux-architecture hosted agent queue to use an agent image:

1. Select Agents in the global navigation to access the Clusters page.
2. Select the cluster in with the Linux-architecture hosted agent queue to configure with the agent image.
3. On the Queues page, select the hosted agent queue based on Linux architecture.
4. Select the Base Image tab to open its settings.
5. In the Agent image dropdown, select your agent image.
6. Select Save settings to save this update.

Delete an agent image

To delete a previously created agent image, it must not be used by any hosted agent queues.

To delete an agent image:

1. Select Agents in the global navigation to access the Clusters page.
2. Select the cluster in which to delete the agent image.
3. Select Agent Images to open the Agent Images page.

4. Select the agent image to delete > Delete.

   Note: If you are prompted that the agent image is currently in use, follow the link/s to each hosted agent queue on the Delete Image message to change the queue's Agent image (from the Base Image tab) to another agent image.

5. On the Delete Image message, select Delete Image and the agent image is deleted.

Use an image URL

A full URL for an image can be provided that will be used as the agent image to run jobs. This image can be hosted with any public registry, or by using the internal registry that's accessible from within the hosted agent cluster.

The default agent image URL is docker.io/buildkite/hosted-agent-base:ubuntu-v1.0.1.

Create a private image

An internal registry is managed for a Buildkite hosted agent cluster that you are able to push to from within a pipeline. Having a dedicated pipeline that can manage an image in this registry will allow other queues within the same cluster to utilize it. A pipeline to build and push to this registry could look like the following:

steps:
  - label: ":docker: Build and push image"
    command: |
      export REGISTRY="$(nsc workspace describe -o json -k registry_url)"
      export TAG="$(date +%s)"
      export IMAGE="$${REGISTRY}/base-image:$${TAG}"
      docker buildx build \
        --platform linux/amd64,linux/arm64 \
        --tag "$${IMAGE}" \
        --push \
        -f Dockerfile \
        .
      buildkite-agent annotate \
        --style "success" \
        ":rocket: :docker: Image pushed to $${IMAGE} :rocket:"

When this pipeline runs it will build and push the image for the repository, adding an annotation to the build with the URL to be used. There are a few points to specifically make note of within this code block:

1. It is assumed that the repository the pipeline runs for will contain a file named Dockerfile at the root level. This can be modified by replacing the content of the -f flag in the command.
2. The TAG value is the current timestamp, which may not matter specifically for this scenario, though can be changed as required.
3. The platform architectures are specified as linux/amd64,linux/arm64 to include support for both architectures. This can also be changed to support only 1 if both are not required.
4. An annotation is used to surface the generated URL to make it easy, this is also not required.