Java

How to setup recording & monitoring for Java applications

Requirements

• Maven / Gradle project configuration:

  • RevDeBug is compatible with Maven 2.0.1 or newer

  • RevDeBug is compatible with Gradle 5.0 or newer

• Java application version 8 or higher

  • See RevDeBug Java Spring Boot Demo application as an example

• Configured & installed RevDeBug Server

To enable RevDeBug in your Java application you will need to:

1. Add RevDeBug repository

2. Add RevDeBug dependency

3. Configure the connection to RevDeBug Server

4. Add Java Agent

Adding RevDeBug repository

Add the following lines to the super pom.xml / build.gradle file in the Java application main node:

Maven

pom.xml

<project>
 ...
  <repositories>
     <repository>
        <id>rdb-repo</id>
        <name>RevDeBug Maven Repo</name>
        <url>https://release.revdebug.com/repository/maven</url>
     </repository>
  </repositories>
 ...
</project>

Gradle

build.gradle

repositories {
    maven {
       url 'https://nexus.revdebug.com/repository/maven'
   }
 }

Adding RevDeBug dependency

Add the following lines to the super pom.xml / build.gradle file in the Java application main node:

Maven

pom.xml

 <project>
    ...
    <dependencies>
            <dependency>
                <groupId>com.revdebug</groupId>
                <artifactId>revdebug-compiler</artifactId>
                <version>7.2.0</version>
                <scope>provided</scope>
            </dependency>
            <dependency>
                <groupId>com.revdebug</groupId>
                <artifactId>revdebug-runtime</artifactId>
                <version>7.2.0</version>
            </dependency>
    ...
</project>

Gradle

build.gradle

 dependencies {
    compileOnly 'com.revdebug:revdebug-compiler:7.2.0'
    annotationProcessor 'com.revdebug:revdebug-compiler:7.2.0' // Required after Gradle v4.10.2
    implementation 'com.revdebug:revdebug-runtime:7.2.0'
 }
  
 //* Use compile keyword instead of implementation for Gradle version less than 3.0
 //* Use compile keyword instead of compileOnly for Gradle version less than 2.12

For spring-boot projects additional exclusion in pom.xml is needed (without it build time dependencies will be packaged with application):

pom.xml

<project>
    ...
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <excludes>
                        <exclude>
                            <groupId>com.revdebug</groupId>
                            <artifactId>revdebug-compiler</artifactId>
                        </exclude>
                    </excludes>
                </configuration>
            </plugin>
        ...
        </plugins>
    </build>    
    ...
</project>

Configure the connection to RevDeBug Server

RevDeBug can be easily added to all projects in the working directory. But if you want, you can add RevDeBug to the selected subset. In the examples below substitute [SERVER_ADDRESS] for actual fully qualified domain address of RevDeBug server.

Maven - single project

pom.xml

<project>
 ...
 <plugins>
   <plugin>
       <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-compiler-plugin</artifactId>
       <version>3.8.1</version>
       <configuration>
           <compilerArgs>
               <arg>-Xplugin:RevDeBugCompiler</arg>
               <arg>-AREVDEBUG_SERVER_HOSTNAME=[SERVER_ADDRESS]</arg>
               <arg>-AREVDEBUG_SECURE_CONNECTION=false</arg>
           </compilerArgs>
           <source>1.8</source>
           <target>1.8</target>
       </configuration>
   </plugin>
</plugins>
...
</project>

Maven - many projects

pom.xml

<project>
 <build>
 ...
 <pluginManagement>
        <plugins>
             <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <compilerArgs>
                        <arg>-Xplugin:RevDeBugCompiler</arg>
                        <arg>-AREVDEBUG_SERVER_HOSTNAME=[SERVER_ADDRESS]</arg>
                        <arg>-AREVDEBUG_SECURE_CONNECTION=false</arg>
                    </compilerArgs>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
     </plugins>
 </pluginManagement>
 ...
 </build>
 </project>

Gradle

build.gradle

   compileJava {
       options.compilerArgs = [
               "-Xplugin:RevDeBugCompiler",
               "-AREVDEBUG_SERVER_HOSTNAME=[SERVER_ADDRESS]",
               "-AREVDEBUG_SECURE_CONNECTION=false"
   ]}

Spring Boot Starter Parent

If you are using a spring-boot-starter-parent, make sure its version is equal to or greater than 1.4.0, earlier versions are not supported.

pom.xml

<parent>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-parent</artifactId>
   <version>1.4.0.RELEASE</version>
   <relativePath/> <!-- lookup parent from repository -->
</parent>

Additional configuration for Java 17+

Starting from Java 17 you need to add additional exports to JVM arguments.

Requirements:

• Gradle 7+ or Maven

For Gradle, create or edit gradle.properties in your project's root directory by adding:

gradle.properties

org.gradle.jvmargs = --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
                     --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
                     --add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED \
                     --add-exports=jdk.compiler/com.sun.tools.javac.model=ALL-UNNAMED \
                     --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED \
                     --add-exports=jdk.compiler/com.sun.tools.javac.processing=ALL-UNNAMED \
                     --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED \
                     --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED \
                     --add-exports=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED \
                     --add-exports=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED

Adding --add-exports directly in build.gradle file wouldn't work - it is a known Gradle issue (link).

For Maven, create .mvn directory in the project's root directory (or in the user's $HOME directory) and create a jvm.config file in it with the following contents:

jvm.config

--add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED
--add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED
--add-exports jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED
--add-exports jdk.compiler/com.sun.tools.javac.model=ALL-UNNAMED
--add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED
--add-exports jdk.compiler/com.sun.tools.javac.processing=ALL-UNNAMED
--add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED
--add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
--add-exports jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED
--add-exports jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED

Settings overview

There are configurable fields in the connection section of RevDeBug DevOps Monitor:

Name	Explanation
-AREVDEBUG_SERVER_HOSTNAME	Usage: Fully qualified domain name (recommended) or an IP address of RevDeBug Server. RevDeBug compile process sends metadata needed to replay recordings of program execution to this server - controlled by 'UPLOAD_METADATA' settings (by default it sends the metadata). When several RevDeBug Servers should receive the emitted metadata, put all of their fully qualified domain names separated by a comma. Example value: 'revdebug.my-company.com'. Can be overridden at runtime using environment variable 'recordServerAddress'.
-AREVDEBUG_SERVER_PORT	Usage: TCP/IP port number of RevDeBug Server. Defaults to 42734 and should not be changed in most cases. Example value: '42734'. Can be overridden at runtime using environment variable 'recordServerPort'.
-AREVDEBUG_APPLICATION_NAME	Usage: Additional distinguishing name for desktop tray companion application. In most cases, not set up during compile time but overridden at the runtime level through environment variables. Example value: 'MainProcessingService-user1'. Can be overridden at runtime using environment variable 'applicationName'.
-AREVDEBUG_SOLUTION_NAME	Usage: Main application name - application will be listed under this name on the applications list on the RevDeBug Server 'Monitor' tab. Usually not needed to be set as it is automatically determined from on-disk directory name of the project. Example value: 'MainProcessingService'. Can be overridden at runtime using environment variable 'solutionName'.
-AREVDEBUG_RELEASE_NAME	Usage: Human readable release name corresponding to current commit ID. Could be set to a tag name or manually incremented version number in the build file. Example value: '1.3'. Can be overridden at runtime using environment variable 'releaseDisplayName'.
-AREVDEBUG_GIT_HASH	Usage: First eight characters of current git commit id (hash). Determined automatically, but in case it would not find the correct repository eg. when using submodules, it can be overridden by configuration in the build file. Example value: 'f937b564'. Can be overridden at runtime using environment variable 'releaseId'.
-AREVDEBUG_METADATA_PATH	Usage: Absolute directory path where RevDeBug metadata files will be placed. Determined automatically, no need to change in most cases. Example value: (in gradle build file) '${projectDir}/build/'.
-AREVDEBUG_UPLOAD_METADATA	Usage: Control whether the metadata generated during compilation will be sent to the server at the end of compilation. Sending metadata is required to be able to open and replay recordings. Defaults to 'true'. Example value: 'false'.
-AREVDEBUG_SECURE_CONNECTION	Usage: Forces and expects a secure connection to RevDeBug Server. This value needs to correspond to TLS settings on the RevDeBug server. On values mismatch, RevDeBug will fail to operate. Defaults to 'false'. Example value: 'true'. Can be overridden at runtime using environment variable 'forceTLS'.
-AREVDEBUG_METADATA_SERVER_HOSTNAME	Usage: Fully qualified domain name (recommended) or an IP address of RevDeBug Server used to send metadata during compile time. RevDeBug compile process sends metadata needed to replay recordings of program execution to this server. When this option is set, metadata will be sent to this server instead of the default set by REVDEBUG_SERVER_HOSTNAME option. When left empty, metadata will be sent to the default RevDeBug Server. Controlled by 'UPLOAD_METADATA' settings (by default it sends the metadata). Example value: 'revdebug.my-company.com'
-AREVDEBUG_DISABLE_ALL	Usage: Disables RevDeBug compilation. If set to true it would make project compilation behave as RevDeBug compile-time dependency would not be added at all. Could be useful for local development and can be set via environment variable on a developers' machines. Defaults to 'false'. Example value: 'false'. Can be overridden at runtime using environment variable 'REVDEBUG_OFF'.
-AREVDEBUG_TRACE_LOG_ERROR	Usage: Enables or disables (default) treating log.error(...) calls as if exception/error would be thrown. The message passed to log.error call will be used as an exception/error message and have associated stacktrace corresponding to the log.error call. Required to set up also REVDEBUG_LOG_ERROR_CLASS to a fully qualified name of the logger class used and REVDEBUG_LOG_ERROR_METHOD to a method used to denote an error message (ie. 'error' or 'severe'). Disabled automatically when REVDEBUG_TRACE_BEFORE_LINE is disabled. Example value: 'false'.
-AREVDEBUG_LOG_ERROR_CLASS	Define logger class to use for log.error() recording. Must be a fully qualified name, such as ‘org.apache.log4j.Logger’
-AREVDEBUG_LOG_ERROR_METHOD	Usage: Method name used to log error message while using the selected logger class (using REVDEBUG_LOG_ERROR_CLASS option). Example value: 'error' or 'severe'.
-AREVDEBUG_USAGE	Usage: Displays RevDeBug Java library settings and their descriptions and immediately stops the compilation. Example value: 'true'.
-AREVDEBUG_RDBDISPLAY_AUTOGEN	Usage: Instructs RevDeBug Compiler to automatically generate RDBDisplay rules for every class in the compiled project (i.e. which compiler compiles) that overrides toString() method. This requires a performance consideration as RevDeBug will use implemented toString() methods to serialize the state of the object during runtime. Defaults to 'false' (do not generate RDBDisplay rules automatically) Example value: 'true'.
-AREVDEBUG_RDBDISPLAY_AUTOGEN_ARRAYS	Usage: Instructs RevDeBug Compiler to automatically generate RDBDisplay rules for displaying first elements of arrays of basic types (Integers, String, etc). This requires a performance consideration as RevDeBug will serialize the state of those arrays during runtime. Defaults to 'false' (do not generate RDBDisplay rules automatically) Example value: 'true'.
-AREVDEBUG_RDBDISPLAY_AUTOGEN_CLASSES	Usage: Instructs RevDeBug Compiler to automatically generate RDBDisplay rules for all classes matching this regexp list granting they have standard toString() method implemented. The classes does not need to have source code available in the current project (i.e., it works for libraries as well). Defaults: none. Example value: 'com\.mycompany\.dto\..*'.
-AREVDEBUG_RDBDISPLAY_AUTOGEN_COLLECTIONS	Usage: Instructs RevDeBug Compiler to automatically generate RDBDisplay rules for displaying first elements of collections of basic types (Integers, String, etc). This requires a performance consideration as RevDeBug will serialize the state of those/ collections during runtime. Defaults to 'false' (do not generate RDBDisplay rules automatically) Example value: 'true'.
-AREVDEBUG_LARGE_METHODS	Usage: Comma separated list of fully qualified method names that are too large for fullRevDeBug instrumentation. Methods matching the list will only receive basic instrumentation of recording input parameters and method enter and method exit events. Methods declared in an inline declared classes cannot be matched, ie. Object obj = new ClassXYZ() { public method() { ... } }. List items should be regular expressions. White spaces around separating commas are ignored. Example: 'com.hello.World.getApplicationStatus()' - will match getApplicationStatus() method in the 'World' class of 'com.hello package'.
-AREVDEBUG_EXCLUDE_METHODS	Usage: Comma separated list of fully qualified methods to be excluded from being recorded. Methods declared in an inline declared classes cannot be matched, ie. Object obj = new ClassXYZ() { public method() { ... } }. List items should be regular expressions. White spaces around separating commas are ignored. Example: 'com.hello.World.getApplicationStatus()' - will match getApplicationStatus() method in the 'World' class of 'com.hello package'.
-AREVDEBUG_EXCLUDE_CLASSES	Usage: Comma separated list of classes to be excluded from being recorded. List items should be regular expressions. White spaces around separating commas are ignored. Example: 'com.hello.World' will match World class file or '.*Options$' will match every class name ending with 'Options' (keep in mind the dollar sign is escaped - in most cases it needs to be).
-AREVDEBUG_JAVAC_VERSION	Usage: Major version of Java C Compiler RevDeBug should assume. Normally it is automatically determined, but can be overridden. Example value: '9'.
-AREVDEBUG_ANNOTATED_ONLY	Usage: Instruments every class and method (the default and a 'false' value) or (when 'true') only classes that are annotated with @RevDeBugConfig.Annotation.Include.
-AREVDEBUG_TRACE_ON_ERROR	Usage: Enables (default) or disables catching and recording of unhandled exceptions. Example value: 'true'.
-AREVDEBUG_TRACE_RETURN	Usage: Enables (default) or disables recording of values of return statements. Example value: 'true'.
-AREVDEBUG_TRACE_VALUES	Usage: Enables (default) or disables recording of changes/mutations or variables' values. Example value: 'true'.
-AREVDEBUG_TRACE_BEFORE_LINE	Usage: Enables (default) or disables recording of before line execution events. Disabling will also disable recording of log.error(...) calls as errors (separately toggleable using REVDEBUG_TRACE_LOG_ERROR) Example value: 'true'.
-AREVDEBUG_TRACE_METHOD_CALLS	Usage: Enables (default) or disables recording of method calls and their parameters. Example value: 'true'.
-AREVDEBUG_TRACE_LOOPS	Usage: Enables (default) or disables recording of before line execution events. Disabling will also disable recording of log.error(...) calls as errors (separately toggleable using REVDEBUG_TRACE_LOG_ERROR) Example value: 'true'.
-AREVDEBUG_TRACE_SWITCH	Usage: Enables (default) or disables recording of switch statement variables. Example value: 'true'.
-AREVDEBUG_TRACE_COND	Usage: Enables (default) or disables recording of conditional statements. Example value: 'true'.
-AREVDEBUG_TRACE_METHOD_PARAMS	Usage: Enables (default) or disables recording of methods/lambdas parameters. Example value: 'true'.
-AREVDEBUG_TRACE_LAMBDAS	Usage: Enables (default) or disables recording of lambdas in general. Example value: 'true'.
-AREVDEBUG_LOG_LEVEL	Usage: Sets the verbosity of RevDeBug logging during compilation process. Defaults to INFO. Increase to VERBOSE or even ALL when encountering compilation process failures with RevDeBug. Can be overridden at runtime using environment variable 'REVDEBUG_VERBOSE'.
-AREVDEBUG_STOP_ON_METADATA_NOT_SENT	Usage: Control whether the compilation process should hard stop when metadata would not be sent to the server Sending metadata is required to be able to open and replay recordings. Defaults to 'false'. Example value: 'true'.
-AREVDEBUG_AUTH_TOKEN	Usage: Authorization token obtained from RevDeBug Server. If not set or with a wrong value and RevDeBug Server was set to require authorization from application will cause RevDeBug metadata and recordings to fail when sending to the Server. Example value: '2c2f4a6ea2ea467382fabaee59106907'. Can be overridden at runtime using environment variable 'authorizationToken'.
-AREVDEBUG_MAX_RECORDING_SIZE	Usage: Size of the recording buffer limiting how many 'statements' are kept in the recording counting back from the moment the recording was stored (i.e. an error usually). Defaults to 1000. Example value: '5000'. Can be overridden at runtime using environment variable 'backlogSize'.
-AREVDEBUG_SAMPLING	Usage: When set to 'true' RevDeBug will record exceptions only when there is a corresponding trace so if sampling per 3 sec is set on APM side, Flight Recorder will also sample recordings. Defaults to 'false'. Example value: 'false'.
-AREVDEBUG_SKIP_SETUP_GENERATION	Usage: Disables creation of RevDeBug.setup resource containing default runtime configuration for application. Only useful when used in conjunction with RevDeBug APM Agent which can supply the configuration from its own or when setting all required settings manually by environment variables (not recommended). Defaults to 'false'. Example value: 'true'.
-DREVDEBUG_MAX_STRING_LENGTH	Usage: Length of recorded strings. Defaults to: 256. Example value: 128.
-AREVDEBUG_EXCLUDE_CALLS	Usage: Comma separated list of fully qualified method signatures to which calls won't get instrumented. Return values of the calls will still be recorded. List items should be regular expressions. White spaces around separating commas are ignored. Defaults: 'java.util.stream.Collectors..*, java.util.stream.Stream..*,java.util.Collection.stream()'
-AREVDEBUG_DISABLE_RECORDING_METHODS	Usage: Comma separated list of fully qualified methods to be excluded from being recorded. Methods declared in an inline declared classes cannot be matched, ie. Object obj = new ClassXYZ() { public method() { ... } }. List items should be regular expressions. White spaces around separating commas are ignored. Defaults to .*\\.toString\\(\\)\$,.*\\.clone\\(\\)\$,.*\\.hashCode\\(\\)\$,.*\\.get.*\$,.*\\.set.*\$,.*\\.is.*\$ Example: 'com.hello.World.getApplicationStatus()' - will match getApplicationStatus() method in the 'World' class of 'com.hello package'.
-AREVDEBUG_DISABLE_RECORDING_CLASSES	Usage: Comma separated list of fully qualified method names that will have their recording disabled. Methods matching the list will temporarily disable recording till their end, so even recording-enabled methods called from them won't get recorded (i.e. disabling with 'inheritance'). Methods declared in an inline declared classes cannot be matched, ie. Object obj = new ClassXYZ() { public method() { ... } }. List items should be regular expressions. White spaces around separating commas are ignored. Example: 'com.hello.World.getApplicationStatus()' - will match getApplicationStatus() method in the 'World' class of 'com.hello package'.

Add RevDeBug Java Agent

• Download java-agent from nexus repo:

Get Java Agent

• Unzip the archive and modify config/agent.config file

  • agent.service_name set to your app name

    • If you want to group applications in the services view, add a group name before your application name, according to this scheme: group_name::application_name.

  • collector.backend_service set to point to your record server instance

  • agent.force_tls set to true if your RevDeBug DevOps Monitor instance has been configured to work with an SSL certificate.

  • agent.instance_name set to your instance name. The instance name serves as the unique identifier for an instance within the service. In case it is left empty, the agent will generate a 32-bit UUID. By default, the instance name is set as UUID@hostname. The maximum allowed length is 50 characters (UTF-8 encoding). For example, if you want to name your instance my-instance-001 you can use the environment variable: SW_AGENT_INSTANCE_NAME=my-instance-001

• Attach unzipped agent to your app with java argument: -javaagent:[Path-to-unzipped-agent]/agent/skywalking-agent.jar

Trace sampling

• agent.sample_n_per_3_secs is a parameter related to an agent that employs a sampling method within a specific time interval. It determines the number of sampled traces the agent will collect within a duration of 3 seconds.

  Defaults to: 100

  Example value: 12

  Range: -1 to 2147483647

• agent.sample_percentage, similar to sample_n_per_3_secs, is responsible for trace sampling. However, in this case, sampling is not time-dependent; instead, it is based solely on a percentage probability.

  Defaults to: -1

  Example value: 30

  Range: -1 to 100

To turn off agent.sample_n_per_3_secs or agent.sample_percentage set the value to 0 or -1.

In case both limits are set sample_n_per_3_secs acts as the top limit whilst the traces below this limit will still be sampled according to the percentage set in sample_percentage

Ignoring Paths in Distributed Tracing

The entire trace ignore process is defined in the ./config/apm-trace-ignore-plugin.config file

To understand and configure the option for ignoring paths in distributed tracing, we must comprehend the syntax and path matching patterns.

Syntax

The trace.ignore_path option allows you to define a list of path patterns that should be ignored by the distributed tracing system. Here is the syntax of this option:

trace.ignore_path=pattern1,pattern2,pattern3,...

Where pattern1, pattern2, etc., are the path patterns to be ignored. These patterns are separated by commas.

Path Matching Patterns

Path matching patterns are used to specify which paths should be ignored by the tracing system. Here are the available patterns:

• ** - Matches any sequence of characters (including empty strings) in the path and supports multi-level directories.

Default Values

If the trace.ignore_path an option is not configured, it takes the following default values:

trace.ignore_path=/webjars/**,/resources/**

This means that by default, the distributed tracing system will ignore all requests starting with /webjars/ or /resources/ and followed by any characters.

Custom Configurations

You can customize this behavior by configuring the trace.ignore_path option according to your needs. For example, if you want to add additional patterns to ignore, append them to the list, separated by commas:

trace.ignore_path=/webjars/**,/resources/**,/eureka/**,/consul/**

In the above example, additional patterns were added also to ignore requests starting with /eureka/ and /consul/.

Make sure to thoroughly test your changes to avoid unexpected behavior in the tracing system.

In general, the trace.ignore_path option allows for more precise control over which requests are ignored by the tracing system, which can be useful in certain scenarios, such as requests related to static resources.

Controlling RevDeBug with environment variables

Put RevDeBug in Standby mode

Once the application is compiled with RevDeBug it reports exceptions and traces to the RevDeBug server. If you want to put RevDeBug in standby mode you have to set those environment variables and restart your application:

SW_AGENT_ACTIVE=false

REVDEBUG_RECORDING_MODE=noop

Disable RevDeBug from instrumenting application

If you want to completely disable RevDeBug from attaching to your application during compilation you can set the following environment variable on your build server and recompile:

REVDEBUG_DISABLE_ALL=true