Detect data volume drops

A drop in data volume might mean you don't receive all the necessary information about your system, compromising the quality of the decisions made on this data. Observing the stable data volume helps you ensure you have the best data set. Detection of volume drops varies depending on the frequency of the incoming data.

High ingestion frequencies

For real-time streamed data records, the Dynatrace out-of-the-box anomaly detection automatically learns a baseline for the ingested data counts and alerts proactively in case of baseline violations or missing data.

The image below shows a typical anomaly detector configuration that uses a baseline to alert when the volume of ingested data on ActiveGates drops.

Every anomaly detection configuration includes an alerting preview, enabling you to adapt the configuration sensitivity to avoid over-alerting.

Low ingestion frequencies

Many data pipelines aggregate data records into batches, sending them in scheduled intervals. Depending on the data source, these intervals could range from hourly to daily transmissions. This varying submission frequency requires a workflow to monitor and alert on abnormal data drops. Coupled with the Davis forecast, the workflow is capable of proactive alerting on a noticeable decline or surge in the ingested data records.

Let's consider an example of a data pipeline that regularly receives data records in hourly batches. In such a scenario, generating forecasts for hourly totals to predict the expected number of records in the next hour becomes essential.

1. Set up a workflow

To observe the incoming data and create a forecast, we use a workflow

0 * * * *

2. Analyze with Davis

In this step, we extract the count of data records via a DQL query and use a forecast to predict the expected data volume for the next hour.

fetch events
| filter event.name == "YOUR EVENTS"
| makeTimeseries count(), interval:1m
| fieldsRename value = `count()`

3. Retrieve current data records

Now, we need to count data records in the most recent batch. We can do it via the DQL query:

fetch events, from:now()-60min
| filter event.name == "YOUR INTERESTING EVENTS"
| summarize count = count()

4. Compare predicted data volume to current volume and alert

The final step of the workflow is to compare the predicted data volume with the actual value and raise an event in case the actual value is outside of the forecast boundaries.

import { execution } from '@dynatrace-sdk/automation-utils';
import { eventsClient, EventIngestEventType } from "@dynatrace-sdk/client-classic-environment-v2";

const PREDICTION_TASK_ID = 'predict_data_volume';
const VOLUME_TASK_ID = 'get_latest_volume';

export default async function ({ execution_id }) {
  const e = await execution(execution_id);
  const prediction = (await e.result(PREDICTION_TASK_ID))['result']['output'][0];
  // get the current data volume of the last 60min
  const volume = await e.result(VOLUME_TASK_ID);
  const current_volume = volume['records'][0]['count'];
  // get the predicted lower bound volume
  const prediction_lower_bounds = prediction['timeSeriesDataWithPredictions']['records'][0]['dt.davis.forecast:lower'][0];

  console.log('Forecast status: ' + prediction['analysisStatus']);
  console.log('Forecast quality: ' + prediction['forecastQualityAssessment']);
  console.log('Forecast lower bound: ' + prediction_lower_bounds);
  console.log('Current value: ' + current_volume);
  if(current_volume < prediction_lower_bounds) {
    console.log("Data volume anomaly detected!");
    eventsClient.createEvent({
      body : {
        eventType: EventIngestEventType.AvailabilityEvent,
        title: 'Abnormal data volume drop',
        properties: {
      }
      }
      });
      }
}

5. Observe low data volume alerts

Whenever the observed data volume falls below the lower threshold of the forecast, Dynatrace triggers a Low data volume notification. The image below shows examples of such alerts.