“How-to” Guides

This section provides answers to some common “how do I do that…?” questions that often arise when using Spring Cloud Sleuth. Its coverage is not exhaustive, but it does cover quite a lot.

If you have a specific problem that we do not cover here, you might want to check out stackoverflow.com to see if someone has already provided an answer. Stack Overflow is also a great place to ask new questions (please use the spring-cloud-sleuth tag).

We are also more than happy to extend this section. If you want to add a “how-to”, send us a pull request.

1. How to Set Up Sleuth with Brave?

Add the Sleuth starter to the classpath.

Maven

<dependencyManagement>
      <dependencies>
          <dependency>
              <groupId>org.springframework.cloud</groupId>
              <artifactId>spring-cloud-dependencies</artifactId>
              <version>${release.train-version}</version>
              <type>pom</type>
              <scope>import</scope>
          </dependency>
      </dependencies>
</dependencyManagement>

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>

Gradle

dependencyManagement {
    imports {
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
    }
}

dependencies {
    implementation "org.springframework.cloud:spring-cloud-starter-sleuth"
}

2. How to Set Up Sleuth with OpenTelemetry?

Add the Sleuth starter, exlcude Brave and add OpenTelemetry dependency to the classpath.

Maven

<dependencyManagement>
      <dependencies>
          <dependency>
              <groupId>org.springframework.cloud</groupId>
              <artifactId>spring-cloud-dependencies</artifactId>
              <version>${release.train-version}</version>
              <type>pom</type>
              <scope>import</scope>
          </dependency>
      </dependencies>
</dependencyManagement>

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-sleuth</artifactId>
    <exclusions>
        <exclusion>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-sleuth-brave</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-sleuth-otel</artifactId>
</dependency>

Gradle

dependencyManagement {
    imports {
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
    }
}

dependencies {
    implementation("org.springframework.cloud:spring-cloud-starter-sleuth") {
        exclude group: 'org.springframework.cloud', module: 'spring-cloud-sleuth-brave'
    }
}

3. How to Set Up Sleuth with Brave & Zipkin via HTTP?

Add the Sleuth starter and Zipkin to the classpath.

Maven

<dependencyManagement>
      <dependencies>
          <dependency>
              <groupId>org.springframework.cloud</groupId>
              <artifactId>spring-cloud-dependencies</artifactId>
              <version>${release.train-version}</version>
              <type>pom</type>
              <scope>import</scope>
          </dependency>
      </dependencies>
</dependencyManagement>

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-sleuth-zipkin</artifactId>
</dependency>

Gradle

dependencyManagement {
    imports {
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
    }
}

dependencies {
    implementation "org.springframework.cloud:spring-cloud-starter-sleuth"
    implementation "org.springframework.cloud:spring-cloud-sleuth-zipkin"
}

4. How to Set Up Sleuth with OpenTelemetry & Zipkin via HTTP?

Add the Sleuth starter, exclude Brave, add OTel and Zipkin to the classpath.

Maven

<dependencyManagement>
      <dependencies>
          <dependency>
              <groupId>org.springframework.cloud</groupId>
              <artifactId>spring-cloud-dependencies</artifactId>
              <version>${release.train-version}</version>
              <type>pom</type>
              <scope>import</scope>
          </dependency>
      </dependencies>
</dependencyManagement>

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-sleuth</artifactId>
    <exclusions>
        <exclusion>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-sleuth-brave</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-sleuth-otel</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-sleuth-zipkin</artifactId>
</dependency>

Gradle

dependencyManagement {
    imports {
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
    }
}

dependencies {
    implementation("org.springframework.cloud:spring-cloud-starter-sleuth") {
        exclude group: 'org.springframework.cloud', module: 'spring-cloud-sleuth-brave'
    }
    implementation "org.springframework.cloud:spring-cloud-sleuth-zipkin"
}

5. How to Set Up Sleuth with Brave & Zipkin via Messaging?

If you want to use RabbitMQ, Kafka or ActiveMQ instead of HTTP, add the spring-rabbit, spring-kafka or org.apache.activemq:activemq-client dependency. The default destination name is Zipkin.

If using Kafka, you must add the Kafka dependency.

Maven

<dependencyManagement>
      <dependencies>
          <dependency>
              <groupId>org.springframework.cloud</groupId>
              <artifactId>spring-cloud-dependencies</artifactId>
              <version>${release.train-version}</version>
              <type>pom</type>
              <scope>import</scope>
          </dependency>
      </dependencies>
</dependencyManagement>

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-sleuth-zipkin</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

Gradle

dependencyManagement {
    imports {
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
    }
}

dependencies {
    implementation "org.springframework.cloud:spring-cloud-starter-sleuth"
    implementation "org.springframework.cloud:spring-cloud-sleuth-zipkin"
    implementation "org.springframework.kafka:spring-kafka"
}

Also, you need to set the property spring.zipkin.sender.type property accordingly:

spring.zipkin.sender.type: kafka

If you want Sleuth over RabbitMQ, add the spring-cloud-starter-sleuth, spring-cloud-sleuth-zipkin and spring-rabbit dependencies.

Maven

<dependencyManagement>
      <dependencies>
          <dependency>
              <groupId>org.springframework.cloud</groupId>
              <artifactId>spring-cloud-dependencies</artifactId>
              <version>${release.train-version}</version>
              <type>pom</type>
              <scope>import</scope>
          </dependency>
      </dependencies>
</dependencyManagement>

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-sleuth-zipkin</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.amqp</groupId>
    <artifactId>spring-rabbit</artifactId>
</dependency>

Gradle

dependencyManagement {
    imports {
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
    }
}

dependencies {
    implementation "org.springframework.cloud:spring-cloud-starter-sleuth"
    implementation "org.springframework.cloud:spring-cloud-sleuth-zipkin"
    implementation "org.springframework.amqp:spring-rabbit"
}

If you want Sleuth over RabbitMQ, add the spring-cloud-starter-sleuth, spring-cloud-sleuth-zipkin and activemq-client dependencies.

Maven

<dependencyManagement>
      <dependencies>
          <dependency>
              <groupId>org.springframework.cloud</groupId>
              <artifactId>spring-cloud-dependencies</artifactId>
              <version>${release.train-version}</version>
              <type>pom</type>
              <scope>import</scope>
          </dependency>
      </dependencies>
</dependencyManagement>

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-sleuth-zipkin</artifactId>
</dependency>
<dependency>
    <groupId>org.apache.activemq</groupId>
    <artifactId>activemq-client</artifactId>
</dependency>

Gradle

dependencyManagement {
    imports {
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
    }
}

dependencies {
    implementation "org.springframework.cloud:spring-cloud-starter-sleuth"
    implementation "org.springframework.cloud:spring-cloud-sleuth-zipkin"
    implementation "org.apache.activemq:activemq-client"
}

Also, you need to set the property spring.zipkin.sender.type property accordingly:

spring.zipkin.sender.type: activemq

6. How to Set Up Sleuth with OpenTelemetry & Zipkin via Messaging?

If you want to use RabbitMQ, Kafka or ActiveMQ instead of HTTP, add the spring-rabbit, spring-kafka or org.apache.activemq:activemq-client dependency. The default destination name is Zipkin.

If using Kafka, you must add the Kafka dependency.

Maven

<dependencyManagement>
      <dependencies>
          <dependency>
              <groupId>org.springframework.cloud</groupId>
              <artifactId>spring-cloud-dependencies</artifactId>
              <version>${release.train-version}</version>
              <type>pom</type>
              <scope>import</scope>
          </dependency>
      </dependencies>
</dependencyManagement>

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-sleuth</artifactId>
    <exclusions>
        <exclusion>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-sleuth-brave</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-sleuth-otel</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-sleuth-zipkin</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

Gradle

dependencyManagement {
    imports {
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
    }
}

dependencies {
    implementation("org.springframework.cloud:spring-cloud-starter-sleuth") {
        exclude group: 'org.springframework.cloud', module: 'spring-cloud-sleuth-brave'
    }
    implementation "org.springframework.cloud:spring-cloud-sleuth-zipkin"
    implementation "org.springframework.kafka:spring-kafka"
}

Also, you need to set the property spring.zipkin.sender.type property accordingly:

spring.zipkin.sender.type: kafka

If you want Sleuth over RabbitMQ, add the spring-cloud-sleuth-otel (exclude spring-cloud-sleuth-brave), spring-cloud-sleuth-zipkin and spring-rabbit dependencies.

Maven

<dependencyManagement>
      <dependencies>
          <dependency>
              <groupId>org.springframework.cloud</groupId>
              <artifactId>spring-cloud-dependencies</artifactId>
              <version>${release.train-version}</version>
              <type>pom</type>
              <scope>import</scope>
          </dependency>
      </dependencies>
</dependencyManagement>

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-sleuth-zipkin</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.amqp</groupId>
    <artifactId>spring-rabbit</artifactId>
</dependency>

Gradle

dependencyManagement {
    imports {
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
    }
}

dependencies {
    implementation "org.springframework.cloud:spring-cloud-starter-sleuth"
    implementation "org.springframework.cloud:spring-cloud-sleuth-zipkin"
    implementation "org.springframework.amqp:spring-rabbit"
}

If you want Sleuth over RabbitMQ, add the spring-cloud-sleuth-otel (exclude spring-cloud-sleuth-brave), spring-cloud-sleuth-zipkin and activemq-client dependencies.

Maven

<dependencyManagement>
      <dependencies>
          <dependency>
              <groupId>org.springframework.cloud</groupId>
              <artifactId>spring-cloud-dependencies</artifactId>
              <version>${release.train-version}</version>
              <type>pom</type>
              <scope>import</scope>
          </dependency>
      </dependencies>
</dependencyManagement>

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-sleuth</artifactId>
    <exclusions>
        <exclusion>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-sleuth-brave</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-sleuth-otel</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-sleuth-zipkin</artifactId>
</dependency>
<dependency>
    <groupId>org.apache.activemq</groupId>
    <artifactId>activemq-client</artifactId>
</dependency>

Gradle

dependencyManagement {
    imports {
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}"
    }
}

dependencies {
    implementation("org.springframework.cloud:spring-cloud-starter-sleuth") {
        exclude group: 'org.springframework.cloud', module: 'spring-cloud-sleuth-brave'
    }
    implementation "org.springframework.cloud:spring-cloud-sleuth-zipkin"
    implementation "org.apache.activemq:activemq-client"
}

Also, you need to set the property spring.zipkin.sender.type property accordingly:

spring.zipkin.sender.type: activemq

7. How to See Spans in an External System?

If you can’t see spans get reported to an external system (e.g. Zipkin), then it’s most likely due to the following causes:

Your span is not being sampled
You have forgotten to add the dependency to report to an external system (e.g. spring-cloud-sleuth-zipkin)
You have misconfigured the connection to the external system

7.1. Your Span Is Not Being Sampled

In order to check if the span is not being sampled it’s enough to see if the exportable flag is being set. Let’s look at the following example:

2020-10-21 12:01:16.285  INFO [backend,0b6aaf642574edd3,0b6aaf642574edd3,true] 289589 --- [nio-9000-exec-1] Example              : Hello world!

If the boolean value in the section [backend,0b6aaf642574edd3,0b6aaf642574edd3,true] is true means that the span is being sampled and should be reported.

7.2. Missing Dependency

Up till Sleuth 3.0.0 the dependency spring-cloud-starter-zipkin included the spring-cloud-starter-sleuth dependency and the spring-cloud-sleuth-zipkin dependency. With 3.0.0 spring-cloud-starter-zipkin was removed, so you need to change it to spring-cloud-sleuth-zipkin.

If you’re working with OpenTelemetry you need to provide the dependency with your exporter. For example if you want to use the OpenTelemetry Zipkin Exporter just add a dependency.

Maven

<dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-exporter-zipkin</artifactId>
</dependency>

Gradle

dependencies {
    implementation "io.opentelemetry:opentelemetry-exporter-zipkin"
}

Then add the SpanExporter bean.

@Bean SpanExporter zipkinExporter() {
    return ZipkinSpanExporter.builder()
            .setEndpoint("http://localhost/api/v2/spans")
            .setServiceName("my-service")
            .build();
}

7.3. Connection Misconfiguration

Double check if the remote system address is correct (e.g. spring.zipkin.baseUrl) and that if trying to communicate over the broker, your broker connection is set up properly.

8. How to Make RestTemplate, WebClient, etc. Work?

If you’re observing that the tracing context is not being propagated then cause is one of the following:

We are not instrumenting the given library
We are instrumenting the library, however you misconfigured the setup

In case of lack of instrumentation capabilities please file an issue with a request to add such instrumentation.

In case of the misconfiguration please ensure that the client you’re using to communicate is a Spring bean. If you create the client manually via the new operator the instrumentation will not work.

Example where instrumentation will work:

import org.springframework.context.annotation.Configuration;
import org.springframework.web.client.RestTemplate;

@Configuration(proxyBeanMethods = false)
class MyConfiguration {
    @Bean RestTemplate myRestTemplate() {
        return new RestTemplate();
    }
}

@Service
class MyService {
    private final RestTemplate restTemplate;

    MyService(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    String makeACall() {
        return this.restTemplate.getForObject("http://example.com", String.class);
    }

}

Example where instrumentation will NOT work:

@Service
class MyService {

    String makeACall() {
        // This will not work because RestTemplate is not a bean
        return new RestTemplate().getForObject("http://example.com", String.class);
    }

}

9. How to Add Headers to the HTTP Server Response?

import org.springframework.cloud.sleuth.http.HttpResponseParser;
import org.springframework.cloud.sleuth.instrument.web.HttpServerResponseParser;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration(proxyBeanMethods = false)
class MyConfig {

    @Bean(name = HttpServerResponseParser.NAME)
    HttpResponseParser myHttpResponseParser() {
        return (response, context, span) -> {
            Object unwrap = response.unwrap();
            if (unwrap instanceof HttpServletResponse) {
                HttpServletResponse resp = (HttpServletResponse) unwrap;
                resp.addHeader("MyCustom", "Header");
            }
        };
    }

}

Your spans need to be sampled for the parser to work. That means that you need to be able to export spans to e.g. Zipkin.

10. How to Customize HTTP Client Spans?

Register a bean of HttpRequestParser type whose name is HttpClientRequestParser.NAME to add customization for the request side. Register a bean of HttpResponseParser type whose name is HttpClientRequestParser.NAME to add customization for the response side.

@Configuration(proxyBeanMethods = false)
public static class ClientParserConfiguration {

    // example for Feign
    @Bean(name = HttpClientRequestParser.NAME)
    HttpRequestParser myHttpClientRequestParser() {
        return (request, context, span) -> {
            // Span customization
            span.name(request.method());
            span.tag("ClientRequest", "Tag");
            Object unwrap = request.unwrap();
            if (unwrap instanceof feign.Request) {
                feign.Request req = (feign.Request) unwrap;
                // Span customization
                span.tag("ClientRequestFeign", req.httpMethod().name());
            }
        };
    }

    // example for Feign
    @Bean(name = HttpClientResponseParser.NAME)
    HttpResponseParser myHttpClientResponseParser() {
        return (response, context, span) -> {
            // Span customization
            span.tag("ClientResponse", "Tag");
            Object unwrap = response.unwrap();
            if (unwrap instanceof feign.Response) {
                feign.Response resp = (feign.Response) unwrap;
                // Span customization
                span.tag("ClientResponseFeign", String.valueOf(resp.status()));
            }
        };
    }

}

11. How to Customize HTTP Server Spans?

Register a bean of HttpRequestParser type whose name is HttpServerRequestParser.NAME to add customization for the request side. Register a bean of HttpResponseParser type whose name is HttpServerResponseParser.NAME to add customization for the response side.

@Configuration(proxyBeanMethods = false)
public static class ServerParserConfiguration {

    @Bean(name = HttpServerRequestParser.NAME)
    HttpRequestParser myHttpRequestParser() {
        return (request, context, span) -> {
            // Span customization
            span.tag("ServerRequest", "Tag");
            Object unwrap = request.unwrap();
            if (unwrap instanceof HttpServletRequest) {
                HttpServletRequest req = (HttpServletRequest) unwrap;
                // Span customization
                span.tag("ServerRequestServlet", req.getMethod());
            }
        };
    }

    @Bean(name = HttpServerResponseParser.NAME)
    HttpResponseParser myHttpResponseParser() {
        return (response, context, span) -> {
            // Span customization
            span.tag("ServerResponse", "Tag");
            Object unwrap = response.unwrap();
            if (unwrap instanceof HttpServletResponse) {
                HttpServletResponse resp = (HttpServletResponse) unwrap;
                // Span customization
                span.tag("ServerResponseServlet", String.valueOf(resp.getStatus()));
            }
        };
    }

}

Your spans need to be sampled for the parser to work. That means that you need to be able to export spans to e.g. Zipkin.

12. How to See the Application Name in Logs?

Assuming that you haven’t changed the default logging format set the spring.application.name property in bootstrap.yml, not in application.yml.

With the new Spring Cloud configuration bootstrap this should no longer be required since there will be no Bootstrap Context anymore.

13. How to Change The Context Propagation Mechanism?

To use the provided defaults you can set the spring.sleuth.propagation.type property. The value can be a list in which case you will propagate more tracing headers.

For Brave we support AWS, B3, W3C propagation types. For OpenTelemetry we support AWS, B3, JAEGER, W3C, OT_TRACER and W3C propagation types.

If you want to provide a custom propagation mechanism set the spring.sleuth.propagation.type property to CUSTOM and implement your own bean (Propagation.Factory for Brave and TextMapPropagator for OpenTelemetry). Below you can find the examples:

Brave

@Component
class CustomPropagator extends Propagation.Factory implements Propagation<String> {

    @Override
    public List<String> keys() {
        return Arrays.asList("myCustomTraceId", "myCustomSpanId");
    }

    @Override
    public <R> TraceContext.Injector<R> injector(Setter<R, String> setter) {
        return (traceContext, request) -> {
            setter.put(request, "myCustomTraceId", traceContext.traceIdString());
            setter.put(request, "myCustomSpanId", traceContext.spanIdString());
        };
    }

    @Override
    public <R> TraceContext.Extractor<R> extractor(Getter<R, String> getter) {
        return request -> TraceContextOrSamplingFlags.create(TraceContext.newBuilder()
                .traceId(HexCodec.lowerHexToUnsignedLong(getter.get(request, "myCustomTraceId")))
                .spanId(HexCodec.lowerHexToUnsignedLong(getter.get(request, "myCustomSpanId"))).build());
    }

    @Override
    public <K> Propagation<K> create(KeyFactory<K> keyFactory) {
        return StringPropagationAdapter.create(this, keyFactory);
    }

}

OpenTelemetry

@Component
class CustomPropagator implements TextMapPropagator {

    @Override
    public List<String> fields() {
        return Arrays.asList("myCustomTraceId", "myCustomSpanId");
    }

    @Override
    public <C> void inject(Context context, C carrier, Setter<C> setter) {
        SpanContext spanContext = Span.fromContext(context).getSpanContext();
        if (!spanContext.isValid()) {
            return;
        }
        setter.set(carrier, "myCustomTraceId", spanContext.getTraceIdAsHexString());
        setter.set(carrier, "myCustomSpanId", spanContext.getSpanIdAsHexString());
    }

    @Override
    public <C> Context extract(Context context, C carrier, Getter<C> getter) {
        String traceParent = getter.get(carrier, "myCustomTraceId");
        if (traceParent == null) {
            return Span.getInvalid().storeInContext(context);
        }
        String spanId = getter.get(carrier, "myCustomSpanId");
        return Span.wrap(SpanContext.createFromRemoteParent(traceParent, spanId, TraceFlags.getSampled(),
                TraceState.builder().build())).storeInContext(context);
    }

}

14. How to Implement My Own Tracer?

Spring Cloud Sleuth API contains all necessary interfaces to be implemented by a tracer. The project comes with OpenZipkin Brave and OpenTelemetry implementations. You can check how both tracers are bridged to the Sleuth’s API by looking at the org.springframework.cloud.sleuth.brave.bridge and org.springframework.cloud.sleuth.otel.bridge modules respectively.