Kerberos (SPNEGO) authentication support for private locations

This doc describes how to configure the synthetics job manager (SJM) to run monitors with Kerberos authentication support. It also includes an optional walkthrough for standing up a Kerberos Key Distribution Center (KDC) and a SPNEGO-protected web application on an Amazon Linux 2023 EC2 instance, so you can validate the setup before you point the job manager at your own realm.

Placeholder values

Before you begin, identify the following values for your environment. Replace each placeholder with your own value wherever it appears in this doc.

Placeholder	Description	Example
`<REALM_NAME>`	Kerberos realm (must be uppercase)	`KERBTEST.LOCAL`
`<EC2_PRIVATE_IP>`	Private IPv4 address of the EC2 instance	`10.8.9.172`
`<EC2_INTERNAL_HOSTNAME>`	Fully qualified domain name (FQDN) of the web app	`ip-X-X-X-X.ec2.internal`
`<CLIENT_PRINCIPAL>`	Kerberos identity assigned to synthetics workers	`synthetics-test`
`<CLIENT_KEYTAB_PATH>`	Full path on the host where the client keytab is stored	`/home/ec2-user/kerb-test/synthetics-test.keytab`
`<PRIVATE_LOCATION_KEY>`	New Relic private location key	`NRSP-us...`

Requirements and compatibility

Kerberos authentication depends on direct network access to your Key Distribution Center (KDC), internal DNS resolution, and closely synchronized clocks. These constraints limit where you can run it, so review the following requirements before you begin.

We support Kerberos authentication in these environments:

Private locations that run the synthetics job manager in a Docker container environment
Private locations that run the synthetics job manager in a Podman container environment

We don't support Kerberos authentication in these environments:

Public locations, which have no network route to your internal KDC
Kubernetes and OpenShift job manager environments

The job manager provides Kerberos credentials to browser monitor jobs only. Scripted API monitors and ping monitors run without Kerberos authentication.

Each browser runtime container acquires its own Kerberos ticket (via kinit, before Chrome launches), so your private location must also meet these requirements:

Reach your KDC on TCP and UDP port 88 from inside the runtime containers.
Resolve your KDC and target application hostnames from inside the runtime containers. On Docker deployments, if internal DNS doesn't resolve them, map them with RUNTIME_EXTRA_HOSTS — this variable is Docker-only. Podman deployments depend on internal DNS.
Keep the private location host's clock within 5 minutes of the KDC clock, using a service such as chrony. Runtime containers share the host's kernel clock, so synchronizing the host is sufficient.

Part 1: Set up a Kerberos test lab (optional)

Tip

Part 1 builds a self-contained Kerberos realm and a protected test application so you can validate the job manager configuration end to end. It isn't the production path, and New Relic doesn't manage your KDC. If you already run a Kerberos KDC or an Active Directory domain, get the realm name, the KDC address, and a client keytab from whoever administers it, then skip to Part 2.

Install the Kerberos KDC server and administration tools on the host EC2 instance:

bash

$sudo dnf install -y krb5-server krb5-workstation

Configure /etc/krb5.conf to direct realm requests to your KDC:

bash

$sudo tee /etc/krb5.conf > /dev/null <<'EOF'
$[libdefaults]
$    default_realm = <REALM_NAME>
$    dns_lookup_realm = false
$    dns_lookup_kdc = false
$    rdns = false
$
$[realms]
$    <REALM_NAME> = {
$        kdc = <EC2_PRIVATE_IP>
$        admin_server = <EC2_PRIVATE_IP>
$    }
$
$[domain_realm]
$    <EC2_INTERNAL_HOSTNAME> = <REALM_NAME>
$EOF

Define the port and encryption rules for the KDC service:

bash

$sudo tee /var/kerberos/krb5kdc/kdc.conf > /dev/null <<'EOF'
$[kdcdefaults]
$    kdc_ports = 88
$    kdc_tcp_ports = 88
$
$[realms]
$    <REALM_NAME> = {
$        acl_file = /var/kerberos/krb5kdc/kadm5.acl
$        dict_file = /usr/share/dict/words
$        admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
$        supported_enctypes = aes256-cts-hmac-sha1-96:normal aes128-cts-hmac-sha1-96:normal
$    }
$EOF

Create the database and start the KDC background services:

bash

$# Create the Kerberos database (assign a master password when prompted)
$sudo kdb5_util create -r <REALM_NAME> -s
$
$# Grant admin permissions and enable KDC services on system boot
$echo "*/admin@<REALM_NAME> *" | sudo tee /var/kerberos/krb5kdc/kadm5.acl
$sudo systemctl enable --now krb5kdc kadmin

Generate keytab files holding non-interactive credentials for both the client (synthetics worker) and the target web server service principal name (SPN):

bash

$mkdir -p /home/ec2-user/kerb-test
$
$# 1. Create client principal & keytab for synthetics runners
$sudo kadmin.local -q "addprinc -randkey <CLIENT_PRINCIPAL>@<REALM_NAME>"
$sudo kadmin.local -q "ktadd -k <CLIENT_KEYTAB_PATH> <CLIENT_PRINCIPAL>@<REALM_NAME>"
$sudo chmod 644 <CLIENT_KEYTAB_PATH>
$
$# 2. Create HTTP service principal name (SPN) keytab for the web server
$sudo kadmin.local -q "addprinc -randkey HTTP/<EC2_INTERNAL_HOSTNAME>@<REALM_NAME>"
$sudo kadmin.local -q "ktadd -k /home/ec2-user/kerb-test/http.keytab HTTP/<EC2_INTERNAL_HOSTNAME>@<REALM_NAME>"
$sudo chown 33:33 /home/ec2-user/kerb-test/http.keytab
$sudo chmod 644 /home/ec2-user/kerb-test/http.keytab

Important

The client keytab's file mode matters less than it looks: The primary trust boundary here is Docker socket access — SJM itself requires -v /var/run/docker.sock:/var/run/docker.sock:rw, and anyone with that access can already read any file on the host as root regardless of its permission bits. Keep the client keytab root-owned, as created above, with permissions 644 — this matches the browser runtime container's execution identity (UID 2000, GID 0) via the group-read bit. Don't chown it to a different user, Doing so can break that group match and leave access depending entirely on the world-read bit. The HTTP service keytab is the test web server's own identity: Give it the UID and GID that Apache runs as inside the container from step 6 — 33, the www-data user on Debian — and keep its permissions at 644 too. You don't need the HTTP keytab to configure the job manager.

Tip

You only need this step if you want to run a test web server on the same host to validate the setup. You don't need it to configure the SJM.

Deploy an Apache container configured with mod_auth_gssapi:

bash

$mkdir -p ~/kerb-test/httpd-image && cd ~/kerb-test/httpd-image
$
$cat > kerb-test.conf <<'EOF'
$<Location /kerb-test>
$    AuthType GSSAPI
$    AuthName "Kerberos Login"
$    GssapiCredStore keytab:/etc/krb5-http.keytab
$    Require valid-user
$</Location>
$EOF
$
$cat > Dockerfile <<'EOF'
$FROM debian:bookworm-slim
$RUN apt-get update && \
>    apt-get install -y --no-install-recommends apache2 libapache2-mod-auth-gssapi krb5-user && \
>    a2enmod auth_gssapi && \
>    mkdir -p /var/www/html/kerb-test && \
>    echo '<h1>Kerberos SSO worked</h1>' > /var/www/html/kerb-test/index.html && \
>    rm -rf /var/lib/apt/lists/*
$COPY kerb-test.conf /etc/apache2/conf-enabled/kerb-test.conf
$EXPOSE 80
$CMD ["apache2ctl", "-D", "FOREGROUND"]
$EOF
$
$docker build -t kerb-test-httpd .
$
$docker run -d --name kerb-test-httpd \
>  -p 8080:80 \
>  -v /etc/krb5.conf:/etc/krb5.conf:ro \
>  -v /home/ec2-user/kerb-test/http.keytab:/etc/krb5-http.keytab:ro \
>  kerb-test-httpd

Ensure the internal domain resolves locally on the host, and verify that Kerberos authentication completes successfully:

bash

$# 1. Map internal hostname in /etc/hosts
$echo "<EC2_PRIVATE_IP> <EC2_INTERNAL_HOSTNAME>" | sudo tee -a /etc/hosts
$
$# 2. Verify unauthenticated challenge returns 401 Unauthorized
$curl -i http://<EC2_INTERNAL_HOSTNAME>:8080/kerb-test/
$
$# 3. Test ticket acquisition and end-to-end Kerberos SSO
$kinit -kt <CLIENT_KEYTAB_PATH> <CLIENT_PRINCIPAL>@<REALM_NAME>
$curl --negotiate -u : -i http://<EC2_INTERNAL_HOSTNAME>:8080/kerb-test/

A successful test responds with HTTP/1.1 200 OK and displays the protected web page content.

Part 2: Configure the synthetics job manager for Kerberos

To allow synthetics browser runner containers to automatically resolve the KDC, acquire Kerberos tickets, and authenticate against protected endpoints, add the following Kerberos environment variables and keytab volume mount to the command that starts your job manager.

Caution

When you enable Kerberos on a private location, every browser monitor job at that location receives the keytab mount and the Kerberos environment variables. The job manager doesn't filter credentials per monitor or per URL. Chrome decides which hosts receive a Kerberos ticket based only on KERBEROS_HOST_ALLOWLIST, so keep that allowlist as narrow as possible.

Important

Set all four KERBEROS_* variables, or set none of them. If you set only some, the synthetics job manager fails to start and logs an incomplete Kerberos configuration error.

Standard command (without Kerberos)

bash

$docker run -e PRIVATE_LOCATION_KEY=<PRIVATE_LOCATION_KEY> \
>  -d --restart unless-stopped \
>  -v /var/run/docker.sock:/var/run/docker.sock:rw \
>  newrelic/synthetics-job-manager

Required command (with Kerberos support enabled)

Add the four KERBEROS_* environment variables (-e) and the keytab volume mount (-v) to the command you already use to start the job manager.

bash

$docker run -d \
>  --name sjm-container \
>  --restart unless-stopped \
>  -e PRIVATE_LOCATION_KEY=<PRIVATE_LOCATION_KEY> \
>  -e KERBEROS_REALM=<REALM_NAME> \
>  -e KERBEROS_KDC=<EC2_PRIVATE_IP> \
>  -e KERBEROS_HOST_ALLOWLIST=<EC2_INTERNAL_HOSTNAME> \
>  -e KERBEROS_KEYTAB_HOST_PATH=<CLIENT_KEYTAB_PATH> \
>  -e RUNTIME_EXTRA_HOSTS=<EC2_INTERNAL_HOSTNAME>:<EC2_PRIVATE_IP> \
>  -v /var/run/docker.sock:/var/run/docker.sock:rw \
>  -v <CLIENT_KEYTAB_PATH>:<CLIENT_KEYTAB_PATH>:ro \
>  newrelic/synthetics-job-manager

Add the same Kerberos variables and keytab mount to the podman run command from Install the job manager. Podman runs the job manager in a pod rather than mounting the Docker socket, so keep your existing CONTAINER_ENGINE, PODMAN_API_SERVICE_PORT, and PODMAN_POD_NAME settings.

bash

$podman run -d \
>  --pod SYNTHETICS \
>  --restart unless-stopped \
>  -e PRIVATE_LOCATION_KEY=<PRIVATE_LOCATION_KEY> \
>  -e CONTAINER_ENGINE=PODMAN \
>  -e PODMAN_API_SERVICE_PORT=8000 \
>  -e PODMAN_POD_NAME=SYNTHETICS \
>  -e KERBEROS_REALM=<REALM_NAME> \
>  -e KERBEROS_KDC=<EC2_PRIVATE_IP> \
>  -e KERBEROS_HOST_ALLOWLIST=<EC2_INTERNAL_HOSTNAME> \
>  -e KERBEROS_KEYTAB_HOST_PATH=<CLIENT_KEYTAB_PATH> \
>  -v <CLIENT_KEYTAB_PATH>:<CLIENT_KEYTAB_PATH>:ro,Z \
>  newrelic/synthetics-job-manager

Tip

On SELinux hosts, add a relabel flag so the container can read the keytab. The job manager and every browser runtime container it launches run inside the same pod (PODMAN_POD_NAME), and Podman shares one SELinux label across all containers in a pod — so :ro,Z works here even though multiple containers read the keytab. Use :Z rather than :ro,Z, since the keytab never needs to be shared with containers outside this pod. For more information, see the Podman volume documentation.

Part 3: Troubleshoot Kerberos authentication

If Kerberos initialization fails inside the runtime container, the container's startup pipeline still launches the browser and runs your monitor. The monitor then fails against a protected endpoint with an HTTP 401 challenge or a navigation timeout, and we tag the result with a KerberosInitWarning root cause. Because the failure looks like an ordinary timeout, check your job logs for the following error codes before you investigate the monitor script.

These codes are separate from the startup validation in Part 2 — if you set only some of the KERBEROS_* variables, the job manager fails to start outright, and none of these codes will ever appear in a job's logs.

Error code	Cause	What to do
`KERBEROS_KDC_UNREACHABLE`	The runtime container can't reach the KDC over the network.	Check the firewall rules for TCP and UDP port 88 between your private location host and the KDC.
`KERBEROS_BAD_CREDENTIAL`	The principal doesn't match the keytab, or the keytab is damaged.	Verify the keytab on the host by running `klist -kt <CLIENT_KEYTAB_PATH>`.
`KERBEROS_CLOCK_SKEW`	The host clock differs from the KDC clock by more than 5 minutes.	Synchronize the host clock with NTP, using a service such as `chrony`.
`KERBEROS_DNS_LOOKUP_FAILED`	The container can't resolve the KDC hostname or the realm.	Check the internal DNS resolver settings for your job manager container, or map the hosts with `RUNTIME_EXTRA_HOSTS`.
`KERBEROS_CONFIG_INVALID`	The runtime container can't read the mounted keytab, or no principal could be parsed from it (an empty or malformed keytab).	Verify the keytab is readable inside the runtime container and contains at least one principal: `klist -kt <CLIENT_KEYTAB_PATH>`.
`KERBEROS_UNKNOWN_FAILURE`	A system-level GSSAPI or cryptographic failure stopped the handshake.	Compare the encryption types in your client keytab (`klist -kt -e <CLIENT_KEYTAB_PATH>` lists each entry's enctype) against the `supported_enctypes` value configured on your KDC in step 3. A mismatch between what the keytab offers and what the KDC accepts causes most of these failures.

Kerberos (SPNEGO) authentication support for private locations

Placeholder values

Requirements and compatibility

Part 1: Set up a Kerberos test lab (optional)

Tip

Step 1: Install Kerberos packages

Step 2: Configure system Kerberos (`/etc/krb5.conf`)

Step 3: Configure the KDC daemon (`kdc.conf`)

Step 4: Initialize the master database and KDC daemons

Step 5: Generate client and server keytabs

Important

(Optional) Step 6: Deploy a test web container

Tip

Step 7: Verify local DNS and the Kerberos handshake

Part 2: Configure the synthetics job manager for Kerberos

Caution

Important

Standard command (without Kerberos)

Required command (with Kerberos support enabled)

Docker

Podman

Tip

Part 3: Troubleshoot Kerberos authentication

Kerberos (SPNEGO) authentication support for private locations

Placeholder values .css-21sua1{background:none;border:none;width:0;padding:0;}

Requirements and compatibility

Part 1: Set up a Kerberos test lab (optional)

Tip

Step 2: Configure system Kerberos (/etc/krb5.conf)

Step 3: Configure the KDC daemon (kdc.conf)

Step 4: Initialize the master database and KDC daemons

Step 5: Generate client and server keytabs

(Optional) Step 6: Deploy a test web container

Step 7: Verify local DNS and the Kerberos handshake

Part 2: Configure the synthetics job manager for Kerberos

Caution

Important

Standard command (without Kerberos)

Required command (with Kerberos support enabled)

Docker

Podman

Part 3: Troubleshoot Kerberos authentication

Placeholder values

Step 2: Configure system Kerberos (`/etc/krb5.conf`)

Step 3: Configure the KDC daemon (`kdc.conf`)