Ask Sendai

Grafana Labs works everyday to break traditional data boundaries with metric-visualization tools accessible across entire organizations. It began as a pure open-source project and has since expanded into supported subscription services. The Grafana open-source project is a platform for monitoring and analyzing time series data. There are also subscription offerings such as the supported Grafana Enterprise version. Grafana Labs’ engineers service more than 150,000 active installations. Users include companies such as PayPal, eBay and Booking.com.

Recently, OpenTelemetry has been announced as a new CNCF sandbox project resulting from a merger of OpenTracing and OpenCensus [1], [2], [3], [4]. Several people have already asked me what OpenTelemetry means for the Jaeger project (incubating at CNCF), and whether it is going to replace Jaeger. I will attempt to answer these questions in this post. I’ve been working on OpenTracing from its inception at a Zipkin workshop back in the Fall of 2015. We had just begun deploying distributed tracing at Uber, and I knew that we needed an open, vendor-neutral API to incorporate into the source code of Uber’s rapidly growing microservices ecosystem. Today, OpenTracing APIs exist in 9+ languages, enjoys wide support from vendors, and has over 100 integrations with various open source projects (https://opentracing.io/registry).

Kiali has the ability to show traces obtained from Istio. Jaeger collects traces for monitoring and troubleshooting microservices-based distributed systems, and both Istio and Kiali use the data that Jaeger provides. Originally this was done via a separate tab in the UI. But that turned out to be impractical. So the Kiali team has been working with the Jaeger team on the Jaeger integration. By improving the Jaeger UI components to make them embeddable, there is better integration between a Kiali selection and a Jaeger selection, making it easier to enrich Jaeger with Kiali information.

Jaeger is composed of different components, each potentially running in its own host. It might be the case that one of these moving parts isn’t working properly, causing spans to not be processed and stored. When something goes wrong, make sure to check the items listed here. Before everything else, make sure to confirm what is the sampling strategy. By default, Jaeger uses a probabilistic sampling strategy, with a 1/1000th chance that the span will be reported. Typically, the sampling strategy can be set via the environment variables JAEGER_SAMPLER_TYPEand JAEGER_SAMPLER_PARAM, but refer to the Jaeger Client’s documentation for the language you are using for more details about which sampling strategies are available.

In OpenTracing the fundamental concept for representing distributed traces is the (time) span: something that starts and then finishes, can be annotated with key-value pairs and can be “causally” related. This representation gained popularity with Google’s Dapper paper and triggered open-source tracing implementations like Zipkin and Jaeger and eventually the OpenTracing specification, but according to the academic literature it is not the only one. In spite its popularity, the span based trace representation has some shortcomings that limit its applicability.

If you’ve been following the evolution of the Kubernetes templates for Jaeger, you might have noticed an important change recently: the Jaeger Agent is not being deployed as a DaemonSet anymore. Instead, instructions are now being provided on how to deploy it as a “Sidecar”. The Agent component was developed to act as a “buffer” between the tracer and the collector. This buffer should sit very close to the tracer, usually on localhost. Because of this basic assumption, the tracer can just send the collected spans to the agent via UDP, for an optimal balance between performance and reliability. While this assumption works great for a deployment on bare metal, a question arises when we start to think about cloud environments and containers: what is localhostfor your Kubernetes pod?