Kubernetes Posts

Top Kubernetes Metrics to Monitor (part 4)

Posted by : Jorge Acetozi | Monitoring, Metrics, DevOps, Kubernetes

This is the final installment of our 4-part "Monitoring Kubernetes" series.

Read More

How to Monitor Kubernetes with Outlyer (part 3)

Posted by : Jorge Acetozi | Monitoring, Kubernetes, Redis

This is the third installment of our 4-part "Monitoring Kubernetes" series.

Part 1 discusses the components of Kubernetes and why it is hard to monitor Kubernetes and more broadly, containerized applications.

Part 2 (Collecting Metrics with Heapster & Prometheus) covers:

- Where to find metrics and health checks for each Kubernetes component
- Kubernetes monitoring with Heapster and the Kubenertes Web UI add-on
- Kubernetes monitoring with Prometheus and Grafana.

This post approaches how to monitor a Kubernetes cluster and a Redis cluster running on top of it as well as how to create custom dashboards and alerts to monitor a web application in Outlyer.

Outlyer Agent Installation

The agent is a lightweight container that runs on each Kubernetes node in a DaemonSet. It collects and reports metrics from all discovered containers and the underlying hosts into your Outlyer account. Once you are registered for your Outlyer account you will get your agent key, which is required in the agent configuration. This can be found under the integrations page.

All Outlyer configuration is in the kube-system namespace in Kubernetes. We recommend that the agent key is exposed as a Kubernetes Secret:

$ kubectl create secret generic outlyer-agent-key --namespace=kube-system --from-literal=accountKey='<PUT YOUR ACCOUNT AGENT KEY HERE>'

Setting up the agent on Kubernetes is straightforward as we make use of a Kubernetes DaemonSet, which ensures that an Outlyer Pod is always running on each schedulable Node.

You can apply the config straight from GitHub, or checkout the repository to make local changes.

Apply the RBAC (Role-Based Access Control) permissions required for the Outlyer Agent to run:

$ kubectl apply -f https://raw.githubusercontent.com/outlyerapp/outlyer-containers/master/kubernetes/rbac.yaml

And apply the DaemonSet to launch Outlyer across all nodes, including any masters.

$ kubectl apply -f https://raw.githubusercontent.com/outlyerapp/outlyer-containers/master/kubernetes/daemonset.yaml

If your cluster doesn’t have RBAC enabled, the additional RBAC-related manifests will not affect the running of the Outlyer DaemonSet. 

Check that the DaemonSet is properly deployed.

$ kubectl get daemonsets outlyer-agent

In the Outlyer UI, click on Status Views and filter by the label k8s-role: node. You should see all your discovered Kubernetes Nodes. Select a Node to see its details.

Interested in trying Outlyer for monitoring Kubernetes? This blog refers to our new release which is currently in private preview. If you want to try it out please contact us. [This doc can help with Kubernetes deployment.]

Deploy kube -state-metrics

The kube-state-metrics is a simple add-on service that listens to the API Server and generates metrics about the state of the Kubernetes objects such as Deployments, Nodes, and Pods. In order to deploy it to your Kubernetes cluster, clone the kube -state-metrics repository and create the objects:

$ https://github.com/kubernetes/kube-state-metrics.git
$ cd kube-state-metrics

$ kubectl apply -f kubernetes/

Check out the second post of the Monitoring Kubernetes series to learn how to scrape metrics from kube -state-metrics.

Outlyer Kubernetes Integration

An Outlyer integration is a package that contains templates of dashboards, alerts, and plugins for commonly used services (check our integrations repository to see all integrations available). When the Kubernetes integration is installed, you can start using our beautiful out-of-the-box Kubernetes dashboards to see the state of your cluster. They give you all the visibility you need to ensure your cluster is healthy and operating properly.

Kubernetes Cluster Overview Dashboard: Nodes & Pods section

Besides the Nodes and Pods sections, it also shows the status of components of the Control Plane and capacity planning information.

The Hosts Dashboard, shown below, provides memory, CPU, disk, and network usage information as well as and other relevant metrics for each Kubernetes Node.


Hosts Dashboard

The Kubernetes integration provides others Kubernetes-specific dashboards that give visibility to Kubernetes resources like Pods, Requests/Limits, Deployments, StatefulSets, and much more.

With just a few clicks you can detect scenarios like "Pod A, running on Node B, is almost crossing its memory resource limits" or "Nodes A and C are overcommited".

In part 4 of the Monitoring Kubernetes series, we will approach the most important metrics to be aware of when monitoring a Kubernetes cluster. You will be able to easily identify these situations.

Monitoring a Redis Cluster running on K8s with Outlyer

Using Outlyer to monitor a Redis cluster deployed in Kubernetes is a simple job of installing our Redis integration. Just like the Kubernetes integration, it provides a default Redis dashboard containing all the information you need to make sure your Redis cluster is working as intended.

 

Create Custom Dashboards in Outlyer

Outlyer integrations provide an easy way to install default monitoring dashboards, alerts, and plugins for the most common open-source services your company probably use. However, there will be times when you will need to build your own custom dashboards and alerts.

When the agents are running on your Kubernetes cluster, you can start using our simple and flexible drag-and-drop based web UI to add widgets (charts, numbers, status, etc) and create amazing dashboards with just a few clicks.

As an example, let's create a dashboard for a web application ( Kuard , from the book Kubernetes Up and Running) with a status widget that shows whether Kuard is healthy and then set up an alert to trigger an email notification whenever Kuard goes down. First, deploy a  Kuard Pod to your cluster:

$ kubectl run --restart=Never --port=8080 --image=gcr.io/kuar-demo/kuard-amd64:1 kuard

Create a Check

Now that Kuard is up and running, create a new check in your Outlyer account (Checks -> Create new). Checks are used, as the name indicates, to check the status and the performance of a service.

A check can be written in any language available on your server (Python, Golang, Javascript, Bash, Ruby, etc). For this example, we'll stick to Python. The following Python script just issues an HTTP request to  Kuard's  health check endpoint. The script exit code is then interpreted as a Nagios exit code.

#!/usr/bin/env python3

import requests
import sys
import os

HOST = os.environ['ip'] if 'ip' in os.environ else 'localhost'
PORT = os.environ['KUARD_PORT']
HEALTH_ENDPOINT = os.environ['KUARD_HEALTH']

try:
  metrics = requests.get(f'http://{HOST}:{PORT}/{HEALTH_ENDPOINT}').text
  print(metrics)
except:
  print(f"Error connecting to http://{HOST}:{PORT}/{HEALTH_ENDPOINT}")
  sys.exit(2)

Checks work using Selectors that allow you to target checks against any host or container label collected by or applied to the agent. Regardless of whether you're monitoring a standard host or container, checks are configured exactly the same using selectors, making it really simple to standardise your monitoring on both container and non-container environments.

As shown below, add the environment variables KUARD_PORT (8080) and KUARD_HEALTH (healthy), pick the container: kuard  selector and the Nagios format and click on Save.

Test the check by clicking on  Run check; you should see the return code 0 and output "ok". Enable the check on the top menu (in green) so that the agent starts running it against  Kuard . It should look  like this:

Create a Dashboard

Create a new Dashboard (Dashboards -> Create new) named Kuard , then click on Add Widget and select the Status widget.

Note that dashboards (as well as alerts) can be exported to and imported from YAML files, so you can always keep a copy under version control, tracking every change.

Edit the status widget and give it the title Kuard Health. Click on the Queries tab, select the check kuard -health-check and save it. There you are! The status widget immediately becomes green, which shows that Kuard  is healthy. 

As you add more and more widgets to a dashboard, you can move them around by using the built-in drag-and-drop feature to create useful, organized and beautiful dashboards.

Create an Alert

Create a new Alert (Alerts -> Create new) named  Kuard , then add a   new criteria by selecting the Service Status Criteria:

  

Add the  kuard -health-check to the service field, give it the name Kuard Health and save it.

Add a new action to specify where Outlyer should send notifications when the configured alert criteria thresholds are crossed. Save it.

That's it! From now on, if Kuard is down, everyone on the email list will get notified.

When it comes to alert notifications, Outlyer integrates with well-known messaging tools like Slack, HipChat, and others, making it ideal for implementing ChatOps practices.

Conclusion

In this article you have learned how to use Outlyer to monitor Kubernetes and a Redis cluster running on top of it. You have learned how to create custom dashboards and alerts in just a few clicks.

In the next post, Part 4, you will learn which are the key metrics you should be aware of when monitoring a Kubernetes cluster.

Interested in trying Outlyer for monitoring Kubernetes? This blog refers to our new release which is currently in private preview. If you want to try it out please contact us. [This doc can help with Kubernetes deployment.]

Read More

Monitoring Kubernetes: Collecting Metrics with Heapster & Prometheus (part 2)

Posted by : Jorge Acetozi | Kubernetes, heapster, prometheus

In the first post of this series (Monitoring Kubernetes) we covered:

  • Why is it hard to monitor Kubernetes and applications running on top of it;
  • Kubernetes main components;
  • Which monitoring layers have to be covered by your monitoring solution.

This post covers where to find Kubernetes metrics and how to monitor it using two well-established open-source monitoring tools: Heapster and Prometheus.

Read More

Monitoring Kubernetes (part 1)

Posted by : David Gildeh | Docker Containers, Monitoring Wisdom, Monitoring, Kubernetes

In 2017, we saw the rise and eventual takeover of the container orchestration tool, Kubernetes. Although a lot of great alternatives exist such as Docker Swarm, Mesos and Amazon ECS, Kubernetes became the leader for running containers at scale. Its benefits are that it’s Cloud native (it was designed to run applications at scale on Cloud platforms) and the ability to provide a virtual abstraction layer on top of Cloud providers enabling users to deploy their applications consistently between Cloud providers.

Read More

DOXSFO September 2017: Kubernetes

Posted by : David Gildeh | Docker Containers, DevOps Exchange (DOXSFO), Kubernetes

 

Although this started out as a general DevOps meetup for September, after our call for speakers it turned out that everyone wanted to talk about Kubernetes! Kubernetes deployments, Kubernetes Networking and Kubernetes storage. This was my first DOXSFO after being away all summer in London and reconnecting with our original meetup we started there, DOXLON, and it was great to see how far our San Francisco meetup has come along since I last attended!

Read More

#DOXLON July 2017

Posted by : Lisa Hong | Docker Containers, DOXLON, DevOps Exchange (DOXLON), Kubernetes

During our July DOXLON we had three awesome speakers talking about varied topics. Jeremie Vallee from Babylon Health, discussed his journey towards orchestration with a Minimum Viable Docker approach. We also had Federico Fregosi, who talked about Kubernetes and microservices at Curve while diving into the future of the DevOps model. Finally we heard from Jonas Chevalier, who discussed how you can use nix to replace and improve upon Docker for packaging microservices. 

Read More

#DOXLON June 2017: Container Orchestration Technology

Posted by : Lisa Hong | Docker Containers, Meetup, DOXLON, DevOps Exchange (DOXLON), Docker, Kubernetes

This month's DOXLON had another great lineup for those interested in Container Orchestration and Microservices, including a headline talk all the way from the US via Video link from the legendary Kelsey Hightower, talking about Istio, a powerful service mesh framework for Kubernetes and Microservices. In addition we had great talks from James Rasell talking about the container solution they built at Elsevier using Nomad, and Graem Forbes at Infitity Works who walked us through productionising a new application on Docker on a very tight deadline.

As usual, another great meetup, followed by the usual beers and pizza and pub down the road! Our next meetup will be on July 27th, see you there!

Read More

#DOXLON May 2017: Using CodeBuild with Ansible and CloudFormation, and more

Posted by : Lisa Hong | Docker Containers, Meetup, DOXLON, DevOps Exchange (DOXLON), Docker, Kubernetes

 

This month, we at Outlyer will be doing their first talk since our rebrand, on Docker monitoring solutions and how the container monitoring solution was built at Outlyer.

We are also hearing from our hosts Sainsbury’s, and how they have been using CodeBuild with Ansible and CloudFormation for their complex builds.

You’ll also get to hear how Flypay have been using Alexa (Amazon Echo) to deploy to Kubernetes! After setting up a powerful continuous deployment pipeline that allows you to deploy your products to Kubernetes clusters just using your voice.

Read More

Go Beyond Cloud Monitoring

Scale your infrastructure monitoring for Cloud, SaaS, Microservices and IoT-ready deployments with Outlyer. We’ve designed a solution for the agile organization. Start now and see your operational metrics in minutes.

Get Started for FREE