Getting Started With Graphite

Graphite is a highly scalable free open-source software tool that monitors, stores, and graphs numeric time-series data in real-time or on demand

Overview of Graphite

If you want to track values of anything that could potentially change over time, and you want to represent those values over time on a graph, then Graphite is just the right tool you need. Whether it’s a few time-series data or thousands of them from numerous sources, Graphite will meet your needs. Graphite is ideal for monitoring the performance of IT infrastructure, cloud, and on-prem business applications, the operating temperature of a machine, flight arrival, and departure statistics, stock price changes over time, road traffic fluctuations, number of customers who walk in and out of a retail store, and more.

Figure 1.0 | Graphite software home page

A practical example of a real-life application of Graphite would be the case of a retail store that uses Graphite to determine the number of customers that walk out of a store due to long checkout queues. The retail store uses motion detectors to count people as they enter and exit the store, and uploads that count to the Graphite server once per minute. Another detector keeps track of how many people are waiting in the checkout line. Because the store manager knows how many people enter the store, leave the store, and are queued up at check-out, it can potentially extrapolate how many people walk out of the store because the queue at the check-out point is too long. This data would enable a store manager to know when to open an additional point of sale to prevent customers from leaving due to long queues.

System Architecture

Graphite consists of three main software components: carbon, whisper, and graphite webapp.

Carbon A network service installed on the client machine that listens for inbound time-series metric submissions, data, or TCP/UDP packets sent over the network, and stores them in Graphite’s specialized database.
Whisper A simple database that stores time-series data collected from machines and other sources.
Graphite webapp A Django web app that renders graphs on demand. It enables data visualization through graphite’s web interfaces.

Graphite does not in itself collect data. Data collection is done via a vast array of tools such as Diamond, Bucky, Collectd, and others used to automate the process of the metric collection with minimal code changes. The collected metrics get fed into the stack via the Carbon service, which writes the data to the Whisper database for storage. Users interact with the Graphite web UI or API, which in turn queries Carbon and Whisper for the data needed to generate the requested graphs. Graphite’s web platform offers several ways, output styles, and formats to create and display graphs, including raw images, CSV, XML, and JSON, allowing anyone to easily embed custom graphs in other web pages or dashboards.

The ability to swiftly store and retrieve time-series data at high volume is key to the success of Graphite. Without the ability to scale, it would be impractical to use Graphite for anything besides small teams and hobby projects. The scalability of Graphite enables it to support the processing of millions of data points per second, making it a suitable visualization tool for virtually any scenario where time-series analysis is needed.

Figure 2.0 | Graphite system architecture

Installing and Configuring Graphite

Graphite is designed to run efficiently on almost any type of environment you need to run it in including cloud, on-premise, and hybrid environments. Several installation options exist for Graphite such as installing from Source, Pip, Virtualenv, and others depending on your skill and experience. These options can be quite complex, especially for a newbie. Thankfully, other methods simplify the installation process.

One such is the use of the open-source containerization platform known as Docker. You can use Docker to install the Graphite image to get it up and running in a matter of minutes. This makes for a fast and extremely portable installation. To install Graphite using Docker, simply run the following command:

docker run -d\

 --name graphite\

 --restart=always\

 -p 80:80\

 -p 2003-2004:2003-2004\

 -p 2023-2024:2023-2024\

 -p 8125:8125/udp\

 -p 8126:8126\

 graphiteapp/graphite-statsd

This command would generate a container named graphite and this container would have the following components:

Graphite: Front-end dashboard
Grafana: A more modern front-end dashboard
Nginx: Reverse proxies the graphite dashboard
Carbon: A daemon that listens for inbound time-series data
Statsd: A simple daemon for easy stats aggregation

For detailed instructions, please refer to the official Graphite docker image repo on Github.

Another method that simplifies the installation process is by installing from Synthesize. Synthesize is a script that allows the installation and configuration of a range of components to automate the configuration of Graphite. Synthesize is built to run on Ubuntu 18.04 LTS only. It will not run on other Ubuntu releases or Linux distributions. For other Linux distributions such as CentOS (8.2 and older), use the Resynthesize script—a fork of Synthesize designed to make Graphite easy to install in CentOS distributions.

Synthesize and Resynthesize scripts make it as easy as possible for newbies to get started with Graphite with a minimal learning curve. The resulting Graphite web interface according to the official documentation “listens only on https port 443 and has been configured to collect metrics specifically for helping profile the performance of your Graphite and Carbon services”. It comes with a Grafana dashboard—a modern alternative to Graphite’s built-in Composer, and a default dashboard for monitoring Carbon’s internal statistics. For detailed instructions, please refer to the official Synthesize and Resynthesize project documentation on Github.

Graphite does not directly run on Windows, but you can run it on Windows in Docker or installing via Synthesize. To leverage this capability, you will need to install Vagrant—a tool for building and managing virtual machine environments. If you run into any installation issues with Graphite, you can submit your issue to the Questions forum on Launchpad for assistance.

Getting Metrics into Graphite

Carbon is the software component of Graphite that listens for inbound time-series metric submissions, and sends them to the Graphite database. There are three main methods for sending data to Graphite: Plaintext, Pickle, and AMQP.

For a singular script, the plaintext protocol is the most straightforward method supported by Carbon. The data sent must be in the following format:

 <metric path> <metric value> <metric timestamp>

All that’s required is a dot-delimited metric name, a numeric value, and a metric timestamp. Using this format, you can issue a Unix command in your shell terminal to your application to send a data string to Graphite such as shown in the example below from Graphite documentation:

$ echo "foo.bar 41 `date +%s`" | nc graphite-server.example.com 2003

In the above example, we’re using the Unix echo command to print the metric string “foo.bar” along with the metric value and the metric timestamp generated by the date command. The output is piped into netcat (nc)—a network utility that connects to the Carbon service at graphite-server.example.com TCP port 2003 (by default, ‘plaintext’ runs on port 2003), and sends the data string. On Unix, the netcat (nc) program can be used to create a socket and send data to Carbon. Carbon will then translate this line of text into a metric which is then stored in a Whisper database on your Graphite server’s filesystem that the web interface understands.

The process described above is entirely manual. But in situations where you don’t want to manually feed metrics into Graphite, you can use several methods and tools such as Cron and Shell scripts, StatsD, and Collectd to automate the entire process. For example, you can use the shell script below from Graphite website, make it executable, and add it to your crontab to run at routine intervals:

#!/bin/bash

CARBON_HOST="10.10.10.5"

CARBON_PORT="2003"

METRIC_NAME="debug.jdixon.procs.bad_java_app"

TIMESTAMP=`date +%s`

NUMBER_OF_WAYWARD_PROCS=`ps auwx | grep MY_BAD_JAVA_APP | wc -l`

echo ${METRIC_NAME} ${NUMBER_OF_WAYWARD_PROCS} ${TIMESTAMP} | nc ${CARBON_HOST} ${CARBON_PORT}

When it comes to sending large amounts of data, the plaintext protocol fails woefully. For that, you’ll need a much more efficient protocol like Pickle which supports sending batches of metrics to Carbon in one go. Graphite also supports Advanced Message Queuing Protocol (AMQP), an open standard for passing business messages between applications or organizations. Through AMQP, Carbon can listen to a message bus.

Getting Metrics out of Graphite

There are several ways and tools you can use to create graphs from Graphite. One such is the use of the in-built Graphite Composer. The Graphite Composer provides the fastest means to familiarize yourself with Graphite’s visualization capabilities.

The main Composer window shown below shows a blank graph that acts as a canvas for your graph. On the left pane, you’ll find a navigation tree containing all of your metrics and saved graphs in a hierarchy mapping to their dot-delimited namespace. Clicking on any of the metrics allows you to add or remove it to the graph as shown in figure 3.0 below. The vertical axis tracks the values recorded, and the horizontal axis displays the time range.

Figure 4.0 | Clicking on metrics allows you to add or remove them to the graph

Selecting the other memory metrics will add them to the graph too. This is a great way to correlate different metrics at the same time.

Figure 5.0 | Clicking other memory metrics to add them to the graph

The Graphite render API is also a great way to retrieve graphs or metrics from Graphite. The graphite webapp provides a /render endpoint for generating graphs and retrieving raw data. It accepts various arguments via query string parameters which are separated by an ampersand (&). These parameters are supplied in the format: &name=value

To visualize specific metrics, you’d have to pass one or more target parameters (which specify a path identifying one or several metrics), and specify a time window for the graph via from/until. For details on the Graphite render API capabilities, please check out the official render API documentation as well as the huge library of Graphite’s transformative and statistical functions.

Similarly, the Grafana project offers more modern visualization for Graphite. But since it is not native, you will be required to integrate with Graphite. Thankfully, Grafana comes with a built-in plugin that can be directly used to connect and fetch data from graphite instances located on the same or different servers. It also offers an interactive sandbox that you can use to try out the tool. The Grafana documentation provides detailed information to help you get started.

Tools that Work with Graphite

The table below lists some examples of popular tools that can be used to extend the capabilities of Graphite:

Tool Name	Purpose	Description
Brubeck	Metric Collection	statsd-compatible stats aggregator written in C
Bucky	Metric Collection	Small service implemented in Python for collecting and translating metrics for Graphite. It can currently collect metric data from CollectD daemons and StatsD clients.
Collectd	Metric Collection	Daemon collects system performance statistics periodically and provides mechanisms to store the values in a variety of ways, including RRD. To send collectd metrics into carbon/graphite, use collectd’s write-graphite plugin
Diamond	Metric Collection	Python daemon that collects system metrics and publishes them to Graphite. It is capable of collecting cpu, memory, network, I/O, load, and disk metrics. Additionally, it features an API for implementing custom collectors for gathering metrics from almost any source.
Backstop	Metric Forwarding	Simple endpoint for submitting metrics to Graphite. It accepts JSON data via HTTP POST and proxies the data to one or more Carbon/Graphite listeners.
Eventflow	Metric Forwarding	Simple service for submitting sFlow datagrams to Graphite. It accepts sFlow datagrams from multiple network devices and proxies the data to a Carbon listener.
Graphite-Newrelic	Metric Forwarding	Get your graphite data into New Relic via a New Relic Platform plugin
Statsd	Metric Forwarding	Simple daemon for easy stats aggregation
Gdash	Metric Visualization	Simple Graphite dashboard built using Twitter's Bootstrap driven by a small DSL.
Grafana	Metric Visualization	General-purpose graphite dashboard replacement with feature-rich graph editing and dashboard creation interface.
Pencil	Metric Visualization	Monitoring frontend for graphite. It runs a web server that dishes out pretty Graphite URLs in interesting and intuitive layouts.
Cabot	Monitoring	Self-hosted monitoring and alerting server that watches Graphite metrics and can alert them by phone, SMS, Hipchat, or email.
Icinga	Monitoring	Icinga 2 will directly write metrics to the defined Graphite Carbon daemon tcp socket if the graphite feature is enabled.
Shinken	Monitoring	System monitoring solution compatible with Nagios which emphasizes scalability, flexibility, and ease of setup. It provides complete integration with Graphite for processing and display of performance data.
Ceres	Whisper Database Alternative	Alternate storage backend provided by the Graphite Project that will eventually replace Whisper. It is intended to be a distributable time-series database
Hisser	Whisper Database Alternative	Time series database, backend for graphite, fast alternative to carbon + whisper.
Kenshin	Whisper Database Alternative	Time-series database alternative to Graphite Whisper with 40x improvement in IOPS. It integrates with Graphite as a plugin.