How do I analyze memory dumps for Java and Node.js?

When your application experiences memory leaks or high object churn, it’s important that you get your hands on the memory dumps so you can analyze these issues. In production environments, this is often quite a challenge as you most likely can’t log into the environment, and you may have no other means of triggering memory dumps. Dynatrace enables you to both trigger and securely download memory dumps to subsequently analyze them in your analysis tool of choice.

Triggering a memory dump

To trigger a memory dump

Navigate to the Host page, Service page, Process page, or Process group page of the entity that you want to analyze.

Click the Browse [] button and select Memory dump details.

The Memory dumps page displays a list of the memory dumps that occurred during the selected time frame for the selected entity (host, service, process, or process group).

Click the Trigger new dump button to generate a new memory dump (you’ll need the Configure capture of sensitive data user permission to do this). To trigger a memory dump first select the process.

Note:
Dynatrace currently supports memory dumps for Java and Node.js processes only.

Click the Trigger dump button.

It takes a few minutes to generate a memory dump. The time required varies widely based on application type. Java applications that have multiple GBs of heap memory may take several minutes; smaller dumps will be available almost immediately.

Return to the Memory dumps page. The newly created dump now appears in the list.

Click the memory dump Download link to view the memory dump (see example below).

Memory dumps are first stored locally on the disk of the monitored application-server machine. Dynatrace OneAgent can subsequently upload the memory dumps automatically to a Security Gateway, which can act as a long-term storage center for memory dumps. The provided download links direct you to the Security Gateway that received that memory dump. This approach ensures that memory dumps are only available to users who have access to the network location of your Security Gateway. This precaution provides an additional security layer that ensures that no sensitive data leaves your data center without you explicitly configuring it that way.

In the case of Java applications, the download provides the memory dump in hprof format, which can be analyzed using a number of tools, including Eclipse Memory analyzer and VisualVM (see example below).

The process works the same for Node.js applications. Node.js memory dumps can be opened in Google Chrome’s integrated memory heap snapshot analysis tool (see example below).

Configure a Security Gateway for memory dump storage

Note:
If you don’t already have a Dynatrace Security Gateway setup, you need to install a Security Gateway.

Access the config.properties file of your Security Gateway to make a few modifications. This file is located in the config directory of your Security Gateway:

  • Linux filepath: /var/lib/dynatrace/gateway/config
  • Windows filepath: %ProgramData%\dynatrace\gateway\config
    Note: On Windows, the programdata directory is usually hidden by default. Paste the file path above into Windows Explorer to access the file.

Add the following code to the config.properties file and adjust the values based on your requirements.

[collector]
...
# collector supports collecting of memory dumps from agents (default: false)
# Add this to the [collector] section
DumpSupported = true
...

# Add this section to the end of the file
[dump]
# relative to security gateway installation directory or absolute path
# The Security Gateway runs with reduced permissions to the file system. Ensure that the user that runs the Security gateway has write permissions to this directory
dumpDir = dump
# maximum size in gb for stored memory dump
maxSizeGb = 100
# maximum age in days for keeping stored memory dumps
maxAgeDays = 7
# maximum number of concurrent file uploads supported
maxConcurrentUploads = 5

The configured directory must be created manually and the permissions must be set so that the user who runs the Security Gateway has read and write permissions.

  • On Linux, this requires that the user have the Read & execute permission on the directory.
  • On Windows, this requires that the Users user group have the Read, Write, and Modify permissions.

Restart the Security Gateway. All instances of Dynatrace OneAgent will automatically reconnect and recognize that the Security Gateway can now store memory dumps. Going forward, whenever a memory dump is successfully triggered, the dump will be uploaded automatically to the correct Security Gateway. Once safely stored on the Security Gateway, OneAgent deletes the local file to clean up its storage.

Note:
This feature requires OneAgent v1.127 or higher and Security Gateway v1.127 or higher.

Limitations

  • Memory dumps are not available for IBM Java.
  • Memory dumps are not yet available for PaaS integrations.
  • Your application server must have adequate space available to store the heap dump.