How do I monitor OpenShift Container Platform?

Based on Docker and Kubernetes, Red Hat OpenShift is a next generation platform for developing, deploying, and running containerized applications, conveniently and at scale. Using the Red Hat OpenShift Container Platform (formerly OpenShift Enterprise), you can manage OpenShift on either physical or virtual infrastructure.

To monitor applications running within an OpenShift cluster, install Dynatrace OneAgent on each cluster node, as described below.

Before you begin

A Red Hat Customer Portal login is required to access the enterprise-ready dynatrace/oneagent image from the Red Hat Container Catalog (RHCC). If you do not already have a Red Hat Customer Portal login, you can register here.

Locate your Dynatrace OneAgent installer URL

The Dynatrace OneAgent installation described below is configured to download the installer from the location specified in the user-provided ONEAGENT_INSTALLER_SCRIPT_URL environment variable.

If you plan to install Dynatrace OneAgent on more than 50 hosts, please consider serving the installer script via a dedicated server, such as Amazon S3. Otherwise, with more than 50 concurrent connections, Dynatrace Server may throttle requests.

The first step is to obtain the location for ONEAGENT_INSTALLER_SCRIPT_URL. This information is presented to you on the Dynatrace OneAgent installation page.

  1. Select Deploy Dynatrace from the navigation menu.
  2. Click the Start installation button.
  3. Select Linux.
  4. Locate your ONEAGENT_INSTALLER_SCRIPT_URL, as shown below.

Installation

The following OpenShift Template uses a Dynatrace OneAgent Docker image with a DaemonSet to install Dynatrace OneAgent for full-stack monitoring on each node of an OpenShift cluster.

Note that enforcing the below dynatrace-oneagent.yml template requires a Service Account named dynatrace that can create privileged pods. See below for detailed instructions.

kind: Template
apiVersion: v1
name: dynatrace-oneagent
labels:
  template: dynatrace-oneagent
metadata:
  name: dynatrace-oneagent
  annotations:
    openshift.io/display-name: Dynatrace OneAgent
    description: Installs Dynatrace OneAgent for all-in-one, full-stack monitoring of OpenShift with Dynatrace. Requires privileged access.
objects:
- apiVersion: extensions/v1beta1
  kind: DaemonSet
  metadata:
    name: dynatrace-oneagent
  spec:
    template:
      metadata:
        labels:
          name: dynatrace-oneagent
      spec:
        containers:
        - name: dynatrace-oneagent
          image: registry.connect.redhat.com/dynatrace/oneagent
          imagePullPolicy: Always
          env:
          - name: ONEAGENT_INSTALLER_SCRIPT_URL
            value: "${ONEAGENT_INSTALLER_SCRIPT_URL}"
          - name: ONEAGENT_INSTALLER_SKIP_CERT_CHECK
            value: "${ONEAGENT_INSTALLER_SKIP_CERT_CHECK}"
          volumeMounts:
          - name: host-root
            mountPath: /mnt/root
          securityContext:
            privileged: true
        volumes:
        - name: host-root
          hostPath:
            path: /
        hostIPC: true
        hostNetwork: true
        hostPID: true
        serviceAccountName: dynatrace
parameters:
- name: ONEAGENT_INSTALLER_SCRIPT_URL
  description: "A URL that points to your cluster's OneAgent download location (Select \"Deploy Dynatrace\" from the Dynatrace navigation menu to access your URL). Example: https://EnvironmentID.live.dynatrace.com/installer/oneagent/unix/latest/AbCdEfGhIjKlMnOp."
  required: true
- name: ONEAGENT_INSTALLER_SKIP_CERT_CHECK
  description: "Must be true if the SSL certificate check upon OneAgent download will be omitted, otherwise false (default). If you're using a Dynatrace Managed cluster with a self-signed certificate, set this to true."
  value: "false"
  required: false

Log into your OpenShift cluster as system:admin:

$ oc login -u system:admin

Select an OpenShift project to run the Dynatrace OneAgent image:

$ oc project openshift-infra

In this project, create a service account named dynatrace:

$ oc create serviceaccount dynatrace

Allow the dynatrace service account to pull images from the RHCC via registry.connect.redhat.com. Be sure to replace [username], [password] and [email] with your Red Hat Customer Portal's account credentials:

$ oc secrets new-dockercfg rhcc \
    --docker-server=registry.connect.redhat.com \
    --docker-username=[username] \
    --docker-password=[password] \
    --docker-email=[email]
$ oc secrets link dynatrace rhcc --for=pull

Grant the dynatrace service account permissions to run Dynatrace OneAgent as a privileged container:

$ oc adm policy add-scc-to-user privileged -z dynatrace

Deploy Dynatrace OneAgent using the above dynatrace-oneagent.yml OpenShift template. Be sure to replace [oneagent-installer-script-url] with an appropriate download location:

$ oc process -f dynatrace-oneagent.yml -p=ONEAGENT_INSTALLER_SCRIPT_URL=[oneagent-installer-script-url] | oc create -f -
daemonset "dynatrace-oneagent" created

Verify that the dynatrace-oneagent daemon set has been created successfully:

$ oc status
In project openshift-infra on server https://127.0.0.1:8443

pod/dynatrace-oneagent-abcde runs dynatrace/oneagent
$ oc get pods
NAME                       READY     STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   1/1       Running             0          1m
$ oc logs -f dynatrace-oneagent-abcde
09:46:18 Deploying agent to /tmp/Dynatrace-OneAgent-Linux.sh via https://EnvironmentID.live.dynatrace.com/installer/oneagent/unix/latest/AbCdEfGhIjKlMnOp
...
09:46:24 Validating agent installer in /tmp/Dynatrace-OneAgent-Linux.sh
Verification successful
09:46:24 Started agent deployment as docker image, PID 1234.
09:46:24 Container version: 1.119.162.20170515-161517
09:46:24 Checking root privileges...
09:46:24 OK
09:46:27 Installation started, version 1.119.162.65984, build date: 15.05.2017, PID 5678.
...

Uninstallation

Uninstalling Dynatrace OneAgent from each node of an OpenShift cluster can be achieved as follows:

Select the project that runs the dynatrace-oneagent daemon set:

$ oc project openshift-infra

Delete the dynatrace-oneagent daemon set:

$ oc delete ds/dynatrace-oneagent

Updating

Whenever a new version of Dynatrace OneAgent becomes available in Dynatrace, you can re-deploy Dynatrace OneAgent as explained in the steps below. Your dynatrace/oneagent image will automatically fetch the latest version of Dynatrace OneAgent. If you've specified a default OneAgent install version for new hosts and applications in your OneAgent updates settings, the dynatrace/oneagent image will automatically fetch the defined default version of Dynatrace OneAgent:

Delete the dynatrace-oneagent daemon set:

$ oc delete ds/dynatrace-oneagent

Deploy Dynatrace OneAgent using the above dynatrace-oneagent.yml OpenShift template. Be sure to replace [oneagent-installer-script-url] with an appropriate download location:

$ oc process -f dynatrace-oneagent.yml -p=ONEAGENT_INSTALLER_SCRIPT_URL=[oneagent-installer-script-url] | oc create -f -
daemonset "dynatrace-oneagent" created

Enabling Additional Features

OpenShift Labels

Dynatrace supports the processing of OpenShift labels for containers running on Kubernetes. The following steps describe how to grant Dynatrace OneAgent sufficient privileges to read these metadata from the OpenShift masters' REST API:

Select the OpenShift project that hosts your application (assuming foo in the following examples):

$ oc project foo

In this project, create a service account named dynatrace:

$ oc create serviceaccount dynatrace

Allow the dynatrace service account to view metadata about your project via the OpenShift masters' REST API:

$ oc policy add-role-to-user view -z dynatrace

Add the dynatrace service account to the Kubernetes Pod configuration of your application:

kind: DeploymentConfig
apiVersion: v1
spec:
  ...
  template
    ...
    spec:
      containers: [...]
      ...
      serviceAccountName: dynatrace

Pitfalls

Find out how to solve common problems that you may encounter.

Deployment seems successful, the dynatrace/oneagent image cannot be pulled

$ oc get pods
NAME                       READY   STATUS         RESTARTS   AGE
dynatrace-oneagent-abcde   0/1     ErrImagePull   0          3s
$ oc logs -f dynatrace-oneagent-abcde
Error from server (BadRequest): container "dynatrace-oneagent" in pod "dynatrace-oneagent-abcde" is waiting to start: image can't be pulled

This is typically the case if the dynatrace service account has not been allowed to pull images from the RHCC (please see the installation steps above):

Deployment seems successful, but the dynatrace-oneagent container does not produce meaningful logs

$ oc get pods
NAME                       READY   STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   0/1     ContainerCreating   0          3s
$ oc logs -f dynatrace-oneagent-abcde
Error from server (BadRequest): container "dynatrace-oneagent" in pod "dynatrace-oneagent-abcde" is waiting to start: ContainerCreating

This is typically the case if the container has not yet fully started. Simply wait a couple of seconds.

Deployment seems successful, but the dynatrace-oneagent container is not running

$ oc process -f dynatrace-oneagent.yml -p=ONEAGENT_INSTALLER_SCRIPT_URL=[oneagent-installer-script-url] | oc create -f -
daemonset "dynatrace-oneagent" created
$ oc get pods
No resources found.

This is typically the case if the dynatrace service account has not been configured to run privileged pods (please see the installation steps above):

$ oc describe ds/dynatrace-oneagent
Name:   dynatrace-oneagent
Image(s): dynatrace/oneagent
Selector: name=dynatrace-oneagent
Node-Selector:  <none>
Labels:   template=dynatrace-oneagent
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Events:
  FirstSeen LastSeen  Count From    SubObjectPath Type    Reason    Message
  --------- --------  ----- ----    ------------- --------  ------    -------
  6m    3m    17  {daemon-set }     Warning   FailedCreate  Error creating: pods "dynatrace-oneagent-" is forbidden: unable to validate against any security context constraint: [spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.securityContext.hostIPC: Invalid value: true: Host IPC is not allowed to be used spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed spec.containers[0].securityContext.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.containers[0].securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.containers[0].securityContext.hostIPC: Invalid value: true: Host IPC is not allowed to be used]