Monitoring Posts

How to Monitor Kubernetes with Outlyer (part 3)

Posted by : Jorge Acetozi | Monitoring, Kubernetes, Redis

This is the third installment of our 4-part "Monitoring Kubernetes" series.

Part 1 discusses the components of Kubernetes and why it is hard to monitor Kubernetes and more broadly, containerized applications.

Part 2 (Collecting Metrics with Heapster & Prometheus) covers:

- Where to find metrics and health checks for each Kubernetes component
- Kubernetes monitoring with Heapster and the Kubenertes Web UI add-on
- Kubernetes monitoring with Prometheus and Grafana.

This post approaches how to monitor a Kubernetes cluster and a Redis cluster running on top of it as well as how to create custom dashboards and alerts to monitor a web application in Outlyer.

Outlyer Agent Installation

The agent is a lightweight container that runs on each Kubernetes node in a DaemonSet. It collects and reports metrics from all discovered containers and the underlying hosts into your Outlyer account. Once you are registered for your Outlyer account you will get your agent key, which is required in the agent configuration. This can be found under the integrations page.

All Outlyer configuration is in the kube-system namespace in Kubernetes. We recommend that the agent key is exposed as a Kubernetes Secret:

$ kubectl create secret generic outlyer-agent-key --namespace=kube-system --from-literal=accountKey='<PUT YOUR ACCOUNT AGENT KEY HERE>'

Setting up the agent on Kubernetes is straightforward as we make use of a Kubernetes DaemonSet, which ensures that an Outlyer Pod is always running on each schedulable Node.

You can apply the config straight from GitHub, or checkout the repository to make local changes.

Apply the RBAC (Role-Based Access Control) permissions required for the Outlyer Agent to run:

$ kubectl apply -f https://raw.githubusercontent.com/outlyerapp/outlyer-containers/master/kubernetes/rbac.yaml

And apply the DaemonSet to launch Outlyer across all nodes, including any masters.

$ kubectl apply -f https://raw.githubusercontent.com/outlyerapp/outlyer-containers/master/kubernetes/daemonset.yaml

If your cluster doesn’t have RBAC enabled, the additional RBAC-related manifests will not affect the running of the Outlyer DaemonSet. 

Check that the DaemonSet is properly deployed.

$ kubectl get daemonsets outlyer-agent

In the Outlyer UI, click on Status Views and filter by the label k8s-role: node. You should see all your discovered Kubernetes Nodes. Select a Node to see its details.

Interested in trying Outlyer for monitoring Kubernetes? This blog refers to our new release which is currently in private preview. If you want to try it out please contact us. [This doc can help with Kubernetes deployment.]

Deploy kube -state-metrics

The kube-state-metrics is a simple add-on service that listens to the API Server and generates metrics about the state of the Kubernetes objects such as Deployments, Nodes, and Pods. In order to deploy it to your Kubernetes cluster, clone the kube -state-metrics repository and create the objects:

$ https://github.com/kubernetes/kube-state-metrics.git
$ cd kube-state-metrics

$ kubectl apply -f kubernetes/

Check out the second post of the Monitoring Kubernetes series to learn how to scrape metrics from kube -state-metrics.

Outlyer Kubernetes Integration

An Outlyer integration is a package that contains templates of dashboards, alerts, and plugins for commonly used services (check our integrations repository to see all integrations available). When the Kubernetes integration is installed, you can start using our beautiful out-of-the-box Kubernetes dashboards to see the state of your cluster. They give you all the visibility you need to ensure your cluster is healthy and operating properly.

Kubernetes Cluster Overview Dashboard: Nodes & Pods section

Besides the Nodes and Pods sections, it also shows the status of components of the Control Plane and capacity planning information.

The Hosts Dashboard, shown below, provides memory, CPU, disk, and network usage information as well as and other relevant metrics for each Kubernetes Node.


Hosts Dashboard

The Kubernetes integration provides others Kubernetes-specific dashboards that give visibility to Kubernetes resources like Pods, Requests/Limits, Deployments, StatefulSets, and much more.

With just a few clicks you can detect scenarios like "Pod A, running on Node B, is almost crossing its memory resource limits" or "Nodes A and C are overcommited".

In part 4 of the Monitoring Kubernetes series, we will approach the most important metrics to be aware of when monitoring a Kubernetes cluster. You will be able to easily identify these situations.

Monitoring a Redis Cluster running on K8s with Outlyer

Using Outlyer to monitor a Redis cluster deployed in Kubernetes is a simple job of installing our Redis integration. Just like the Kubernetes integration, it provides a default Redis dashboard containing all the information you need to make sure your Redis cluster is working as intended.

 

Create Custom Dashboards in Outlyer

Outlyer integrations provide an easy way to install default monitoring dashboards, alerts, and plugins for the most common open-source services your company probably use. However, there will be times when you will need to build your own custom dashboards and alerts.

When the agents are running on your Kubernetes cluster, you can start using our simple and flexible drag-and-drop based web UI to add widgets (charts, numbers, status, etc) and create amazing dashboards with just a few clicks.

As an example, let's create a dashboard for a web application ( Kuard , from the book Kubernetes Up and Running) with a status widget that shows whether Kuard is healthy and then set up an alert to trigger an email notification whenever Kuard goes down. First, deploy a  Kuard Pod to your cluster:

$ kubectl run --restart=Never --port=8080 --image=gcr.io/kuar-demo/kuard-amd64:1 kuard

Create a Check

Now that Kuard is up and running, create a new check in your Outlyer account (Checks -> Create new). Checks are used, as the name indicates, to check the status and the performance of a service.

A check can be written in any language available on your server (Python, Golang, Javascript, Bash, Ruby, etc). For this example, we'll stick to Python. The following Python script just issues an HTTP request to  Kuard's  health check endpoint. The script exit code is then interpreted as a Nagios exit code.

#!/usr/bin/env python3

import requests
import sys
import os

HOST = os.environ['ip'] if 'ip' in os.environ else 'localhost'
PORT = os.environ['KUARD_PORT']
HEALTH_ENDPOINT = os.environ['KUARD_HEALTH']

try:
  metrics = requests.get(f'http://{HOST}:{PORT}/{HEALTH_ENDPOINT}').text
  print(metrics)
except:
  print(f"Error connecting to http://{HOST}:{PORT}/{HEALTH_ENDPOINT}")
  sys.exit(2)

Checks work using Selectors that allow you to target checks against any host or container label collected by or applied to the agent. Regardless of whether you're monitoring a standard host or container, checks are configured exactly the same using selectors, making it really simple to standardise your monitoring on both container and non-container environments.

As shown below, add the environment variables KUARD_PORT (8080) and KUARD_HEALTH (healthy), pick the container: kuard  selector and the Nagios format and click on Save.

Test the check by clicking on  Run check; you should see the return code 0 and output "ok". Enable the check on the top menu (in green) so that the agent starts running it against  Kuard . It should look  like this:

Create a Dashboard

Create a new Dashboard (Dashboards -> Create new) named Kuard , then click on Add Widget and select the Status widget.

Note that dashboards (as well as alerts) can be exported to and imported from YAML files, so you can always keep a copy under version control, tracking every change.

Edit the status widget and give it the title Kuard Health. Click on the Queries tab, select the check kuard -health-check and save it. There you are! The status widget immediately becomes green, which shows that Kuard  is healthy. 

As you add more and more widgets to a dashboard, you can move them around by using the built-in drag-and-drop feature to create useful, organized and beautiful dashboards.

Create an Alert

Create a new Alert (Alerts -> Create new) named  Kuard , then add a   new criteria by selecting the Service Status Criteria:

  

Add the  kuard -health-check to the service field, give it the name Kuard Health and save it.

Add a new action to specify where Outlyer should send notifications when the configured alert criteria thresholds are crossed. Save it.

That's it! From now on, if Kuard is down, everyone on the email list will get notified.

When it comes to alert notifications, Outlyer integrates with well-known messaging tools like Slack, HipChat, and others, making it ideal for implementing ChatOps practices.

Conclusion

In this article you have learned how to use Outlyer to monitor Kubernetes and a Redis cluster running on top of it. You have learned how to create custom dashboards and alerts in just a few clicks.

In the next post, Part 4, you will learn which are the key metrics you should be aware of when monitoring a Kubernetes cluster.

Interested in trying Outlyer for monitoring Kubernetes? This blog refers to our new release which is currently in private preview. If you want to try it out please contact us. [This doc can help with Kubernetes deployment.]

Read More

Introducing JMXQuery to Monitor Java via Python

Posted by : David Gildeh | Monitoring, Java, JMX, open-source

Back in December 2014, we had a team Christmas hackathon and I decided I wanted to make Java Monitoring really simple via our agent and integration plugins. At that time we were recommending new users monitor Java services using solutions like Jolokia, listed in this blog we wrote back then. It was frustrating that our users could monitor non-Java services in a few clicks but the moment they wanted to monitor Java we had to get them to install 3rd party agents, and there were so many ways of doing it there was no consistent way we could rely on to make it a one click setup like our non-Java integrations. It broke the whole setup experience.

Read More

Monitoring Kubernetes (part 1)

Posted by : David Gildeh | Docker Containers, Monitoring Wisdom, Monitoring, Kubernetes

In 2017, we saw the rise and eventual takeover of the container orchestration tool, Kubernetes. Although a lot of great alternatives exist such as Docker Swarm, Mesos and Amazon ECS, Kubernetes became the leader for running containers at scale. Its benefits are that it’s Cloud native (it was designed to run applications at scale on Cloud platforms) and the ability to provide a virtual abstraction layer on top of Cloud providers enabling users to deploy their applications consistently between Cloud providers.

Read More

#DOXLON October 2017

Posted by : David Gildeh | Monitoring, DOXLON, DevOps Exchange (DOXLON)

Our October DOXLON featured a solid DevOps Exchange line up, covering some very current topics in the DevOps world. We started with Rob Elkin, CTO @ Busuu talking about Serverless, then Matthew Macdonald-Wallace, a seasoned DevOps consultant ranting about how DevOps has been hijacked by marketing and sales teams, and last but not least, Connon MacRae, VP TechOps at Ticketmaster, talking about their journey to DevOps.

Read More

#DOXLON August 2017

Posted by : Lisa Hong | Monitoring, DOXLON, DevOps Exchange (DOXLON), API, AWS

Our August DOXLON featured a solid DevOps Exchange line up, covering some very current topics in the DevOps world. We started with Owain Perry and his monitoring solutions at Just Eat. Following Richard Clark took us through how to build a resilient VPC transit network. We closed with Graeme Forbes on the topic of SSL. 

Read More

Java Monitoring Using JMX

Posted by : Vince Power | Monitoring, Java

When you are working with Java applications, or any application running inside a Java Virtual Machine, JMX provides an easy and platform-native way to extract details from the runtime without making any code changes. This article provides some examples of how to access monitoring information using JMX.

Read More

Why You Should Monitor Services Rather Than Containers

Posted by : Chris Tozzi | Docker Containers, Monitoring, Docker

Docker containers are a game-changer when it comes to delivering software. They make development, testing and deployment faster and more consistent. That you already know.

Read More

Build vs. Buy (Prometheus)

Posted by : Steven Acreman | Monitoring

Running an online service isn't easy. Every day you make complex decisions about how to solve problems and often there is no right or wrong answer—just different ways with different results. On the infrastructure side, you have to weigh up where everything will be hosted: on a cloud service like AWS, or in your own data centres, or a combination of the two, or any number of other options. 

Read More

Metrics: Nagios, Graphite, Prometheus & InfluxDB

Posted by : Steven Acreman | Monitoring

Open source monitoring can be quite confusing for those who haven't spent a lot of time reading about the options. At Outlyer, we track the most popular standards and then offer wire-compatible endpoints. This helps new customers migrate onto Outlyer with very little effort, and it means we don't invent yet another proprietary standard and create vendor lock-in.

Read More

StatsD, Tags and Outlyer

Posted by : Steven Acreman | Monitoring

PLEASE NOTE:  in February 2017, we rebranded and changed our name from Dataloop.IO to Outlyer. Our agent is still called “dataloop agent”, and relevant code reflects the old name (Dataloop) as well. Thank you for your patience as we update everything.

We had a request today from one of our customers to look into supporting tags on StatsD metrics in Outlyer. The answer was so surprisingly easy I thought I'd do a blog post about it.

Read More

Go Beyond Cloud Monitoring

Scale your infrastructure monitoring for Cloud, SaaS, Microservices and IoT-ready deployments with Outlyer. We’ve designed a solution for the agile organization. Start now and see your operational metrics in minutes.

Get Started for FREE