--- title: Kerberos (SPNEGO) authentication support for private locations source: https://docs.newrelic.com/docs/synthetics/synthetic-monitoring/private-locations/synthetics-kerberos-auth-support --- This doc describes how to configure the [synthetics job manager](https://docs.newrelic.com/docs/synthetics/synthetic-monitoring/private-locations/install-job-manager) (SJM) to run monitors with Kerberos authentication support. It also includes an optional walkthrough for standing up a Kerberos Key Distribution Center (KDC) and a SPNEGO-protected web application on an Amazon Linux 2023 EC2 instance, so you can validate the setup before you point the job manager at your own realm. ## Placeholder values [#variables] Before you begin, identify the following values for your environment. Replace each placeholder with your own value wherever it appears in this doc. | Placeholder | Description | Example | | ------------------------- | ------------------------------------------------------- | ------------------------------------------------- | | `` | Kerberos realm (must be uppercase) | `KERBTEST.LOCAL` | | `` | Private IPv4 address of the EC2 instance | `10.8.9.172` | | `` | Fully qualified domain name (FQDN) of the web app | `ip-X-X-X-X.ec2.internal` | | `` | Kerberos identity assigned to synthetics workers | `synthetics-test` | | `` | Full path on the host where the client keytab is stored | `/home/ec2-user/kerb-test/synthetics-test.keytab` | | `` | New Relic private location key | `NRSP-us...` | ## Requirements and compatibility [#requirements] Kerberos authentication depends on direct network access to your Key Distribution Center (KDC), internal DNS resolution, and closely synchronized clocks. These constraints limit where you can run it, so review the following requirements before you begin. We support Kerberos authentication in these environments: - Private locations that run the synthetics job manager in a Docker container environment - Private locations that run the synthetics job manager in a Podman container environment We don't support Kerberos authentication in these environments: - Public locations, which have no network route to your internal KDC - Kubernetes and OpenShift job manager environments The job manager provides Kerberos credentials to browser monitor jobs only. Scripted API monitors and ping monitors run without Kerberos authentication. Each browser runtime container acquires its own Kerberos ticket (via `kinit`, before Chrome launches), so your private location must also meet these requirements: - Reach your KDC on TCP and UDP port 88 from inside the runtime containers. - Resolve your KDC and target application hostnames from inside the runtime containers. On Docker deployments, if internal DNS doesn't resolve them, map them with [`RUNTIME_EXTRA_HOSTS`](https://docs.newrelic.com/docs/synthetics/synthetic-monitoring/private-locations/job-manager-configuration#mtls-docker) — this variable is Docker-only. Podman deployments depend on internal DNS. - Keep the private location host's clock within 5 minutes of the KDC clock, using a service such as `chrony`. Runtime containers share the host's kernel clock, so synchronizing the host is sufficient. ## Part 1: Set up a Kerberos test lab (optional) [#host-setup] > #### 💡 TIP > > Part 1 builds a self-contained Kerberos realm and a protected test application so you can validate the job manager configuration end to end. It isn't the production path, and New Relic doesn't manage your KDC. If you already run a Kerberos KDC or an Active Directory domain, get the realm name, the KDC address, and a client keytab from whoever administers it, then skip to [Part 2](#sjm-configuration). **Step 1: Install Kerberos packages** Install the Kerberos KDC server and administration tools on the host EC2 instance: ````bash sudo dnf install -y krb5-server krb5-workstation ``` ```` **Step 2: Configure system Kerberos (`/etc/krb5.conf`)** Configure `/etc/krb5.conf` to direct realm requests to your KDC: ````bash sudo tee /etc/krb5.conf > /dev/null <<'EOF' [libdefaults] default_realm = dns_lookup_realm = false dns_lookup_kdc = false rdns = false [realms] = { kdc = admin_server = } [domain_realm] = EOF ``` ```` **Step 3: Configure the KDC daemon (`kdc.conf`)** Define the port and encryption rules for the KDC service: ````bash sudo tee /var/kerberos/krb5kdc/kdc.conf > /dev/null <<'EOF' [kdcdefaults] kdc_ports = 88 kdc_tcp_ports = 88 [realms] = { acl_file = /var/kerberos/krb5kdc/kadm5.acl dict_file = /usr/share/dict/words admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab supported_enctypes = aes256-cts-hmac-sha1-96:normal aes128-cts-hmac-sha1-96:normal } EOF ``` ```` **Step 4: Initialize the master database and KDC daemons** Create the database and start the KDC background services: ````bash # Create the Kerberos database (assign a master password when prompted) sudo kdb5_util create -r -s # Grant admin permissions and enable KDC services on system boot echo "*/admin@ *" | sudo tee /var/kerberos/krb5kdc/kadm5.acl sudo systemctl enable --now krb5kdc kadmin ``` ```` **Step 5: Generate client and server keytabs** Generate keytab files holding non-interactive credentials for both the client (synthetics worker) and the target web server service principal name (SPN): ````bash mkdir -p /home/ec2-user/kerb-test # 1. Create client principal & keytab for synthetics runners sudo kadmin.local -q "addprinc -randkey @" sudo kadmin.local -q "ktadd -k @" sudo chmod 644 # 2. Create HTTP service principal name (SPN) keytab for the web server sudo kadmin.local -q "addprinc -randkey HTTP/@" sudo kadmin.local -q "ktadd -k /home/ec2-user/kerb-test/http.keytab HTTP/@" sudo chown 33:33 /home/ec2-user/kerb-test/http.keytab sudo chmod 644 /home/ec2-user/kerb-test/http.keytab ``` The client keytab's file mode matters less than it looks: The primary trust boundary here is Docker socket access — SJM itself requires `-v /var/run/docker.sock:/var/run/docker.sock:rw`, and anyone with that access can already read any file on the host as root regardless of its permission bits. Keep the client keytab root-owned, as created above, with permissions `644` — this matches the browser runtime container's execution identity (UID 2000, GID 0) via the group-read bit. Don't chown it to a different user, Doing so can break that group match and leave access depending entirely on the world-read bit. The HTTP service keytab is the test web server's own identity: Give it the UID and GID that Apache runs as inside the container from step 6 — `33`, the `www-data` user on Debian — and keep its permissions at `644` too. You don't need the HTTP keytab to configure the job manager. ```` **(Optional) Step 6: Deploy a test web container** > #### 💡 TIP > > You only need this step if you want to run a test web server on the same host to validate the setup. You don't need it to configure the SJM. Deploy an Apache container configured with `mod_auth_gssapi`: ````bash mkdir -p ~/kerb-test/httpd-image && cd ~/kerb-test/httpd-image cat > kerb-test.conf <<'EOF' AuthType GSSAPI AuthName "Kerberos Login" GssapiCredStore keytab:/etc/krb5-http.keytab Require valid-user EOF cat > Dockerfile <<'EOF' FROM debian:bookworm-slim RUN apt-get update && \ apt-get install -y --no-install-recommends apache2 libapache2-mod-auth-gssapi krb5-user && \ a2enmod auth_gssapi && \ mkdir -p /var/www/html/kerb-test && \ echo '

Kerberos SSO worked

' > /var/www/html/kerb-test/index.html && \ rm -rf /var/lib/apt/lists/* COPY kerb-test.conf /etc/apache2/conf-enabled/kerb-test.conf EXPOSE 80 CMD ["apache2ctl", "-D", "FOREGROUND"] EOF docker build -t kerb-test-httpd . docker run -d --name kerb-test-httpd \ -p 8080:80 \ -v /etc/krb5.conf:/etc/krb5.conf:ro \ -v /home/ec2-user/kerb-test/http.keytab:/etc/krb5-http.keytab:ro \ kerb-test-httpd ``` ```` **Step 7: Verify local DNS and the Kerberos handshake** Ensure the internal domain resolves locally on the host, and verify that Kerberos authentication completes successfully: ````bash # 1. Map internal hostname in /etc/hosts echo " " | sudo tee -a /etc/hosts # 2. Verify unauthenticated challenge returns 401 Unauthorized curl -i http://:8080/kerb-test/ # 3. Test ticket acquisition and end-to-end Kerberos SSO kinit -kt @ curl --negotiate -u : -i http://:8080/kerb-test/ ``` A successful test responds with `HTTP/1.1 200 OK` and displays the protected web page content. ```` ## Part 2: Configure the synthetics job manager for Kerberos [#sjm-configuration] To allow synthetics browser runner containers to automatically resolve the KDC, acquire Kerberos tickets, and authenticate against protected endpoints, add the following Kerberos environment variables and keytab volume mount to the command that starts your job manager. > #### ⚠️ CAUTION > > When you enable Kerberos on a private location, every browser monitor job at that location receives the keytab mount and the Kerberos environment variables. The job manager doesn't filter credentials per monitor or per URL. Chrome decides which hosts receive a Kerberos ticket based only on `KERBEROS_HOST_ALLOWLIST`, so keep that allowlist as narrow as possible. > #### ⚠️ IMPORTANT > > Set all four `KERBEROS_*` variables, or set none of them. If you set only some, the synthetics job manager fails to start and logs an incomplete Kerberos configuration error. ### Standard command (without Kerberos) [#standard-sjm-command] ```bash docker run -e PRIVATE_LOCATION_KEY= \ -d --restart unless-stopped \ -v /var/run/docker.sock:/var/run/docker.sock:rw \ newrelic/synthetics-job-manager ``` ### Required command (with Kerberos support enabled) [#kerberos-sjm-command] Add the four `KERBEROS_*` environment variables (`-e`) and the keytab volume mount (`-v`) to the command you already use to start the job manager. **Docker** ````bash docker run -d \ --name sjm-container \ --restart unless-stopped \ -e PRIVATE_LOCATION_KEY= \ -e KERBEROS_REALM= \ -e KERBEROS_KDC= \ -e KERBEROS_HOST_ALLOWLIST= \ -e KERBEROS_KEYTAB_HOST_PATH= \ -e RUNTIME_EXTRA_HOSTS=: \ -v /var/run/docker.sock:/var/run/docker.sock:rw \ -v ::ro \ newrelic/synthetics-job-manager ``` ```` **Podman** Add the same Kerberos variables and keytab mount to the `podman run` command from [Install the job manager](https://docs.newrelic.com/docs/synthetics/synthetic-monitoring/private-locations/install-job-manager). Podman runs the job manager in a pod rather than mounting the Docker socket, so keep your existing `CONTAINER_ENGINE`, `PODMAN_API_SERVICE_PORT`, and `PODMAN_POD_NAME` settings. ````bash podman run -d \ --pod SYNTHETICS \ --restart unless-stopped \ -e PRIVATE_LOCATION_KEY= \ -e CONTAINER_ENGINE=PODMAN \ -e PODMAN_API_SERVICE_PORT=8000 \ -e PODMAN_POD_NAME=SYNTHETICS \ -e KERBEROS_REALM= \ -e KERBEROS_KDC= \ -e KERBEROS_HOST_ALLOWLIST= \ -e KERBEROS_KEYTAB_HOST_PATH= \ -v ::ro,Z \ newrelic/synthetics-job-manager ``` On SELinux hosts, add a relabel flag so the container can read the keytab. The job manager and every browser runtime container it launches run inside the same pod (`PODMAN_POD_NAME`), and Podman shares one SELinux label across all containers in a pod — so `:ro,Z` works here even though multiple containers read the keytab. Use `:Z` rather than `:ro,Z`, since the keytab never needs to be shared with containers outside this pod. For more information, see the [Podman volume documentation](https://docs.podman.io/en/latest/markdown/podman-run.1.html#volume-v-source-volume-host-dir-container-dir-options). ```` ## Part 3: Troubleshoot Kerberos authentication [#troubleshooting] If Kerberos initialization fails inside the runtime container, the container's startup pipeline still launches the browser and runs your monitor. The monitor then fails against a protected endpoint with an HTTP 401 challenge or a navigation timeout, and we tag the result with a `KerberosInitWarning` root cause. Because the failure looks like an ordinary timeout, check your job logs for the following error codes before you investigate the monitor script. These codes are separate from the startup validation in [Part 2](#sjm-configuration) — if you set only some of the `KERBEROS_*` variables, the job manager fails to start outright, and none of these codes will ever appear in a job's logs. | Error code | Cause | What to do | | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `KERBEROS_KDC_UNREACHABLE` | The runtime container can't reach the KDC over the network. | Check the firewall rules for TCP and UDP port 88 between your private location host and the KDC. | | `KERBEROS_BAD_CREDENTIAL` | The principal doesn't match the keytab, or the keytab is damaged. | Verify the keytab on the host by running `klist -kt `. | | `KERBEROS_CLOCK_SKEW` | The host clock differs from the KDC clock by more than 5 minutes. | Synchronize the host clock with NTP, using a service such as `chrony`. | | `KERBEROS_DNS_LOOKUP_FAILED` | The container can't resolve the KDC hostname or the realm. | Check the internal DNS resolver settings for your job manager container, or map the hosts with `RUNTIME_EXTRA_HOSTS`. | | `KERBEROS_CONFIG_INVALID` | The runtime container can't read the mounted keytab, or no principal could be parsed from it (an empty or malformed keytab). | Verify the keytab is readable inside the runtime container and contains at least one principal: `klist -kt `. | | `KERBEROS_UNKNOWN_FAILURE` | A system-level GSSAPI or cryptographic failure stopped the handshake. | Compare the encryption types in your client keytab (`klist -kt -e ` lists each entry's enctype) against the `supported_enctypes` value configured on your KDC in step 3. A mismatch between what the keytab offers and what the KDC accepts causes most of these failures. |