Redis Posts

How to Monitor Kubernetes with Outlyer (part 3)

Posted by : Jorge Acetozi | Monitoring, Kubernetes, Redis

This is the third installment of our 4-part "Monitoring Kubernetes" series.

Part 1 discusses the components of Kubernetes and why it is hard to monitor Kubernetes and more broadly, containerized applications.

Part 2 (Collecting Metrics with Heapster & Prometheus) covers:

- Where to find metrics and health checks for each Kubernetes component
- Kubernetes monitoring with Heapster and the Kubenertes Web UI add-on
- Kubernetes monitoring with Prometheus and Grafana.

This post approaches how to monitor a Kubernetes cluster and a Redis cluster running on top of it as well as how to create custom dashboards and alerts to monitor a web application in Outlyer.

Outlyer Agent Installation

The agent is a lightweight container that runs on each Kubernetes node in a DaemonSet. It collects and reports metrics from all discovered containers and the underlying hosts into your Outlyer account. Once you are registered for your Outlyer account you will get your agent key, which is required in the agent configuration. This can be found under the integrations page.

All Outlyer configuration is in the kube-system namespace in Kubernetes. We recommend that the agent key is exposed as a Kubernetes Secret:

$ kubectl create secret generic outlyer-agent-key --namespace=kube-system --from-literal=accountKey='<PUT YOUR ACCOUNT AGENT KEY HERE>'

Setting up the agent on Kubernetes is straightforward as we make use of a Kubernetes DaemonSet, which ensures that an Outlyer Pod is always running on each schedulable Node.

You can apply the config straight from GitHub, or checkout the repository to make local changes.

Apply the RBAC (Role-Based Access Control) permissions required for the Outlyer Agent to run:

$ kubectl apply -f https://raw.githubusercontent.com/outlyerapp/outlyer-containers/master/kubernetes/rbac.yaml

And apply the DaemonSet to launch Outlyer across all nodes, including any masters.

$ kubectl apply -f https://raw.githubusercontent.com/outlyerapp/outlyer-containers/master/kubernetes/daemonset.yaml

If your cluster doesn’t have RBAC enabled, the additional RBAC-related manifests will not affect the running of the Outlyer DaemonSet. 

Check that the DaemonSet is properly deployed.

$ kubectl get daemonsets outlyer-agent

In the Outlyer UI, click on Status Views and filter by the label k8s-role: node. You should see all your discovered Kubernetes Nodes. Select a Node to see its details.

Interested in trying Outlyer for monitoring Kubernetes? This blog refers to our new release which is currently in private preview. If you want to try it out please contact us. [This doc can help with Kubernetes deployment.]

Deploy kube -state-metrics

The kube-state-metrics is a simple add-on service that listens to the API Server and generates metrics about the state of the Kubernetes objects such as Deployments, Nodes, and Pods. In order to deploy it to your Kubernetes cluster, clone the kube -state-metrics repository and create the objects:

$ https://github.com/kubernetes/kube-state-metrics.git
$ cd kube-state-metrics

$ kubectl apply -f kubernetes/

Check out the second post of the Monitoring Kubernetes series to learn how to scrape metrics from kube -state-metrics.

Outlyer Kubernetes Integration

An Outlyer integration is a package that contains templates of dashboards, alerts, and plugins for commonly used services (check our integrations repository to see all integrations available). When the Kubernetes integration is installed, you can start using our beautiful out-of-the-box Kubernetes dashboards to see the state of your cluster. They give you all the visibility you need to ensure your cluster is healthy and operating properly.

Kubernetes Cluster Overview Dashboard: Nodes & Pods section

Besides the Nodes and Pods sections, it also shows the status of components of the Control Plane and capacity planning information.

The Hosts Dashboard, shown below, provides memory, CPU, disk, and network usage information as well as and other relevant metrics for each Kubernetes Node.


Hosts Dashboard

The Kubernetes integration provides others Kubernetes-specific dashboards that give visibility to Kubernetes resources like Pods, Requests/Limits, Deployments, StatefulSets, and much more.

With just a few clicks you can detect scenarios like "Pod A, running on Node B, is almost crossing its memory resource limits" or "Nodes A and C are overcommited".

In part 4 of the Monitoring Kubernetes series, we will approach the most important metrics to be aware of when monitoring a Kubernetes cluster. You will be able to easily identify these situations.

Monitoring a Redis Cluster running on K8s with Outlyer

Using Outlyer to monitor a Redis cluster deployed in Kubernetes is a simple job of installing our Redis integration. Just like the Kubernetes integration, it provides a default Redis dashboard containing all the information you need to make sure your Redis cluster is working as intended.

 

Create Custom Dashboards in Outlyer

Outlyer integrations provide an easy way to install default monitoring dashboards, alerts, and plugins for the most common open-source services your company probably use. However, there will be times when you will need to build your own custom dashboards and alerts.

When the agents are running on your Kubernetes cluster, you can start using our simple and flexible drag-and-drop based web UI to add widgets (charts, numbers, status, etc) and create amazing dashboards with just a few clicks.

As an example, let's create a dashboard for a web application ( Kuard , from the book Kubernetes Up and Running) with a status widget that shows whether Kuard is healthy and then set up an alert to trigger an email notification whenever Kuard goes down. First, deploy a  Kuard Pod to your cluster:

$ kubectl run --restart=Never --port=8080 --image=gcr.io/kuar-demo/kuard-amd64:1 kuard

Create a Check

Now that Kuard is up and running, create a new check in your Outlyer account (Checks -> Create new). Checks are used, as the name indicates, to check the status and the performance of a service.

A check can be written in any language available on your server (Python, Golang, Javascript, Bash, Ruby, etc). For this example, we'll stick to Python. The following Python script just issues an HTTP request to  Kuard's  health check endpoint. The script exit code is then interpreted as a Nagios exit code.

#!/usr/bin/env python3

import requests
import sys
import os

HOST = os.environ['ip'] if 'ip' in os.environ else 'localhost'
PORT = os.environ['KUARD_PORT']
HEALTH_ENDPOINT = os.environ['KUARD_HEALTH']

try:
  metrics = requests.get(f'http://{HOST}:{PORT}/{HEALTH_ENDPOINT}').text
  print(metrics)
except:
  print(f"Error connecting to http://{HOST}:{PORT}/{HEALTH_ENDPOINT}")
  sys.exit(2)

Checks work using Selectors that allow you to target checks against any host or container label collected by or applied to the agent. Regardless of whether you're monitoring a standard host or container, checks are configured exactly the same using selectors, making it really simple to standardise your monitoring on both container and non-container environments.

As shown below, add the environment variables KUARD_PORT (8080) and KUARD_HEALTH (healthy), pick the container: kuard  selector and the Nagios format and click on Save.

Test the check by clicking on  Run check; you should see the return code 0 and output "ok". Enable the check on the top menu (in green) so that the agent starts running it against  Kuard . It should look  like this:

Create a Dashboard

Create a new Dashboard (Dashboards -> Create new) named Kuard , then click on Add Widget and select the Status widget.

Note that dashboards (as well as alerts) can be exported to and imported from YAML files, so you can always keep a copy under version control, tracking every change.

Edit the status widget and give it the title Kuard Health. Click on the Queries tab, select the check kuard -health-check and save it. There you are! The status widget immediately becomes green, which shows that Kuard  is healthy. 

As you add more and more widgets to a dashboard, you can move them around by using the built-in drag-and-drop feature to create useful, organized and beautiful dashboards.

Create an Alert

Create a new Alert (Alerts -> Create new) named  Kuard , then add a   new criteria by selecting the Service Status Criteria:

  

Add the  kuard -health-check to the service field, give it the name Kuard Health and save it.

Add a new action to specify where Outlyer should send notifications when the configured alert criteria thresholds are crossed. Save it.

That's it! From now on, if Kuard is down, everyone on the email list will get notified.

When it comes to alert notifications, Outlyer integrates with well-known messaging tools like Slack, HipChat, and others, making it ideal for implementing ChatOps practices.

Conclusion

In this article you have learned how to use Outlyer to monitor Kubernetes and a Redis cluster running on top of it. You have learned how to create custom dashboards and alerts in just a few clicks.

In the next post, Part 4, you will learn which are the key metrics you should be aware of when monitoring a Kubernetes cluster.

Interested in trying Outlyer for monitoring Kubernetes? This blog refers to our new release which is currently in private preview. If you want to try it out please contact us. [This doc can help with Kubernetes deployment.]

Read More

Go Beyond Cloud Monitoring

Scale your infrastructure monitoring for Cloud, SaaS, Microservices and IoT-ready deployments with Outlyer. We’ve designed a solution for the agile organization. Start now and see your operational metrics in minutes.

Get Started for FREE