Playing with Dalmatiner DB

Posted by : Steven Acreman | Engineering, Dalmatimer DB

blog-dalmatiner.png

Dalmatimer DB is an open source time series database built on top of riak-core and ZFS. It re-uses the logic from riak-core to handle the logic of where data is located but implements its very own database optimised for metrics

It sits alongside other time series databases like Graphite, InfluxDB, OpenTSDB, Prometheus and a host of alternatives built on top of databases like Cassandra. You can send metrics into it via Graphite or StatsD ports and display them in Grafana as you would expect.

Currently it is more of a direct replacement for Graphite as the data model doesn't include key value pair dimensions like OpenTSDB, Prometheus and InfluxDB do. However, we are working on adding an optional service for that.

How is it different? You can read some of the design ideas here:

https://docs.dalmatiner.io/en/latest/tradeoffs.html

Essentially the main difference is that it relies on some very well tested software as the foundation. Clustering is a first class citizen and complicated compression and checksumming code is kept out of the database where ZFS can do it instead.

Currently Dalmatiner is a great database if you are looking for a reliable time series metrics store that scales and is 100% open source and not backed by a commercial company. Dataloop will continue to contribute to Dalmatiner with Heinz Gies as its benevolent dictator for life.

However, it is a distributed system and will only get more distributed in future when we add a metadata service for metric dimensions etc. Therefore you will trade some of your time for the scalability vs something simpler to setup and run.

Also, at scale the shape of the data often determines how the database performs. Dataloop has thousands of metric sources with a moderate number of metrics under each. Other use cases may involve different ratios. We have a team of Erlangers who constantly optimise Dalmatiner for our use case. At a certain scale you may hit problems that we haven't seen before and you'll either need to dust off the Erlang book or wait for a solution, which may be less favourable than picking a database written in Go or Java depending on your skills.

Originally Dalmatiner was built for Project Fifo which means it began life on SmartOS and the only way to get data in was to use an Erlang client library. We've been contributing to it for the past 5 months and now have it running successfully in production albeit in a slightly different configuration to what Heinz runs. Hopefully this blog makes it a bit more accessible to non Erlangers.

We've made a few changes to fit in with the world of DevOps.

  • We run Dalmatiner on Linux
  • Dalmatiner accepts float values rather than just ints
  • We've made a few tools in Python to make pushing data in easier

To have a play around with it there's a docker container with all of the components inside.

docker run \
  -d \
  -p 3000:3000 \
  -p 2003:2003 \
  -p 8125:8125 \
  -p 5555:5555 \
  -p 8080:8080 \
  --name=dalmatiner-docker \
  dataloop/dalmatiner-docker

That should pull down the dataloop/dalmatiner-docker container from docker hub and start it. Alternatively you can clone this repo and build it locally:

https://github.com/dataloop/dalmatiner-docker

Grafana runs on port 3000 (the default) and comes pre-setup with a Dalmatiner data source. You may need to change this if you are running Docker-Machine on OSX to point to the IP address of your virtualbox VM.

Dalmatiner actually consists of two parts. DDB which is the database and DFE which is the front end used for queries. DFE comes with its own web interface running on port 8080 for browsing buckets and metrics and running queries. It's good for troubleshooting Dalmatiner itself but you're better off using Grafana for browsing your own data.

Grafana uses a plugin that connects to DFE and pulls the data out over the http api. Getting data into Dalmatiner is slightly different from other databases. It was designed to handle very high write throughput so it uses a binary protocol.

Internally at Dataloop we use the Erlang client library to send data into port 5555 of DDB. You can also do that in the container if you are so inclined. However, that is incredibly tricky so I've made a couple of Python packages that are installed in the container.

The first is ddbpy which is a client library that converts string values into the binary protocol.

https://github.com/dataloop/dalmatiner-python-client

Then there's ddbgraphite which listens on port 2003 TCP (the default Graphite port) and uses ddbpy to send data over the binary protocol into DDB.

https://github.com/dataloop/dalmatiner-graphite

These work but could use some optimisation if anyone fancies playing with them and submitting some pull requests. They do at least shed some light on what is happening in the Erlang client.

If you want to set this up properly the best bet is to follow the docs and run Dalmatiner on SmartOS. If you'd prefer to run on Linux then for now you'll need to reverse engineer the Dockerfile. We run Dalmatiner on Ubuntu 12.04 with ZFS on Linux and will provide updated packages for this platform every few weeks when we consider them stable.

Hopefully this opens up Dalmatiner to a wider audience. It has been a very well kept secret up until this point but we feel it deserves a bit more attention.

You may be wondering about the performance of reads and writes. I'm planning to do a future blog on the topic but haven't had time so far. The Dalmatiner docs have a section on performance and hopefully by providing this blog more people can give it a test themselves.

As mentioned previously Dataloop is merely contributing to Dalmatiner. Any bug tickets or pull requests should be submitted to Heinz's upstream repos that live here:

https://github.com/dalmatinerdb

Although we are happy to answer any questions on the topic. Just email steven.acreman@dataloop.io or ping me in our slack channel:

https://slack.dataloop.io (look for @steven)

Go Beyond Cloud Monitoring

Scale your infrastructure monitoring for Cloud, SaaS, Microservices and IoT-ready deployments with Outlyer. We’ve designed a solution for the agile organization. Start now and see your operational metrics in minutes.

Start Your 14 Day Trial Now