Published by 
Today, Pluribus released Netvisor 7, which marks another major step forward in our mission to radically simplify deployment and operations for distributed cloud networking. One of the most innovative features of this release is a new suite of monitoring and visibility tools, including FlowTracker and KubeTracker™ fabric services.
In prior releases, Netvisor ONE OS and the Adaptive Cloud Fabric software could capture flow telemetry for TCP flows only. With the introduction of FlowTracker in R7, Pluribus now provides telemetry on every flow traversing the fabric, including TCP, UDP, ICMP and even infrastructure services flows like DCHP, DNS and more.
Amazingly, this comprehensive flow telemetry is achieved without the need for an expensive external TAP and TAP aggregation overlay infrastructure. The cost of procuring and deploying TAPS to capture packet flows for analysis can be daunting and often results in cost/benefit tradeoffs where TAPS are only installed at certain points in the network. With FlowTracker, that expense and those tradeoffs are eliminated, every flow in the fabric is captured, and flow metadata is exported to tools like our UNUM Insight Analytics platform.
The KubeTracker fabric service is a powerful new capability delivered by the Adaptive Cloud Fabric specifically for network operators supporting Kubernetes applications. Similar to FlowTracker, it does not require an expensive parallel TAP infrastructure.
KubeTracker – Why now?
Kubernetes has been around since 2014, but traction for the open source container orchestration platform has been accelerating over the past several years, with a stream of public announcements by numerous IT solution providers. Furthermore, research from 451 shows that cloud native applications based on containers have been fully adopted or partially adopted by 66% of enterprises surveyed.
Q. What is your organization’s adoption status for the indicated technologies?
Source: Jay Lyman and Voice of the Enterprise: DevOps, Workloads & Key Projects 2021,
Advisory Report, 451 Research, a part of S&P Market Intelligence
As enterprises adopt hybrid cloud architectures and increasingly deploy cloud native applications, and as more tier-2 cloud providers expand their Kubernetes offerings, Pluribus Networks sees a major opportunity to leverage the Adaptive Cloud Fabric to deliver a comprehensive set of observability features. The time is right to empower network teams with a fabric service so they can rapidly root cause application performance and availability issues in a Kubernetes environment – thus KubeTracker.
Kubernetes applications challenges from a NetOps perspective
To understand the challenges of supporting Kubernetes applications from a physical networking point of view, take the following scenario: assume we have a NetOps team responsible for providing a high-performance, highly available physical network to support multiple Kubernetes clusters. Also assume we have a DevOps team responsible for managing the CI/CD pipeline integrating Kubernetes.
The DevOps team opens a service ticket flagging that, over the past 24 hours, a specific application based on Kubernetes micro-services experienced performance issues and slow response and they suspect the problem was in the network fabric. Whether this suspicion is right or wrong, the NetOps team is responsible for getting to the bottom of this problem and, until now, they simply have not had the right set of tools at their disposal.
To begin with, the network operation engineer would like to understand specific IP addresses associated with the container endpoints (or pods in Kubernetes) that the DevOps team is complaining about. NetOps engineering would also like to understand the location of these containers in the physical network, to be able to quickly narrow down the investigation to a handful of switches and endpoints in the fabric.
After a few phone calls with the DevOps team, our network engineer learns a few harsh facts about Kubernetes:
- The lifespan of these containers is measured in minutes, therefore some of the containers they are looking for no longer exist.
- At the time when the performance problem occurred, the containers backing the micro-service were distributed across the network and the DevOps team is not able to track when and where they were connected into the fabric.
- To make matters even more complicated, the network traffic of the containers is encapsulated directly on the servers with VXLAN and is therefore completely hidden to the physical network.
- Lastly, the network engineer learns that it is not possible to reach or probe specific pods as IP addresses are only reachable from inside the cluster. The only way to communicate with a pod is through a service, which is effectively a stable virtual IP address and not associated with any specific network interface! In other words Kubernetes’ internal networking acts as a distributed load balancer that is a complete black box from the physical network perspective.
NetOps teams historically have had no visibility into container-conatiner E-W network traffic inside a Kubernetes cluster, making performance monitoring and troubleshooting difficult
NetOps suddenly realizes that they’re facing an almost impossible task of looking for objects which (a) may no longer exists, (b) whose location in the network is unknown, (c) cannot directly be reached from outside the cluster, and (d) whose traffic is also “hidden” in a network overlay. Unfortunately, this reality puts NetOps in a difficult position where they do not know whether the network is a problem or is not a problem. And as we all know, fingers always point to the network first.
Tweet courtesy of @DCgubbins https://twitter.com/DCgubbins/status/1425820987644289030?s=20
This is where the KubeTracker service comes into play – assisting the NetOps team to resolve a seemingly impossible task in a matter of minutes.
“Kubernetes-Aware” Fabric
KubeTracker is a service that is supported on any switch that is part of a Pluribus fabric (enabled on any of the fabric nodes) to subscribe to the events published by the Kubernetes API server and make the entire fabric aware of how Kubernetes micro-services are distributed across the network and how their deployments dynamically change over time.
Specifically, the KubeTracker service empowers the NetOps team with the following capabilities:
- KubeTracker allows NetOps to search and filter pods by micro-service or application name and easily determine the identity (IP address) and location in the fabric.
- KubeTracker’s “time machine” enables NetOps to track how the deployments of a micro-service evolves over time, how it elastically expands and contracts to respond to Kubernetes deployment changes. The network engineer can pinpoint which pods are backing a specific micro-service at any point in time and know exactly where they’re connected. Pods can come and go very frequently, but the fabric preserves memory of each pod and micro-service so it is easy to correlate Kubernetes events with networking events (e.g. loss of connectivity, traffic congestion, etc.)
- While it is not possible from outside the Kubernetes cluster to communicate directly with any pod end-points, KubeTracker provides NetOps with the external virtual IPs associated with each micro-services. This way the network engineer has some a basic way to communicate with a micro-services end-points during the trouble-shooting.
- Finally, KubeTracker’s flow telemetry capability provides visibility into every pod-to-pod east-west conversations inside the Kubernetes cluster – even if the traffic is VXLAN encapsulated. This functionality enables NetOps with deeper awareness of how micro-services communicate with each other within the cluster, and with the outside network via “services” load balancing. The actual communication flows can reveal potential issues related to pods or services not responding on certain ports, or being overloaded with a high rate of connections, causing the micro-service to perform poorly or respond slowly.
KubeTracker’s Unique Architecture To Monitor Kubernetes
While DevOps has a broad array of options to monitor a Kubernetes environment from inside the cluster, KubeTracker is a unique tool designed to monitor Kubernetes applications from outside the cluster. With KubeTracker, no agents or special software need to run on the Kubernetes cluster.
Not only is KubeTracker completely transparent to the compute environment, it is also transparent to the networking environment because it does not require any tap, probe or packet broker equipment to monitor the network traffic.
KubeTracker is an extension of Pluribus Adaptive Cloud Fabric integrated visibility architecture, and it is designed to empower NetOps with the intelligence to minimize the mean-time to resolve any Kubernetes performance or availability issues potentially related to the network.
For more information:
- Watch this video from the OCP summit where we talked about our early direction on KubeTracker
- Read the press release
If you would like to talk to us to learn more about Pluribus Networks’ approach to modern data center network architectures, including how to better support Kubernetes environments, reach out to us for a demo.
