Skip to main content
Account
Watch Klustered
Tutorial

Hans Kristian Flaatten & Zach Wachtel

Date
Runtime
1:28:33
Share

About this video

What You'll Learn

  1. Debug cluster access problems by checking kubeconfig endpoints, kubelet and API server logs, and transport socket errors.
  2. Identify broken control-plane components by removing stale encryption provider config and validating etcd and control-plane health.
  3. Locate performance kill-switches in cron jobs, disable harmful system services, and restore startup with Cilium and Kyverno fixes.

Hans Kristian Flaatten and Zach Wachtel tackle two broken Kubernetes clusters, debugging a bad kubeconfig IP, an encryption provider config, a disk IO throttle cron job, plus Kyverno webhooks and a Cilium egress policy.

Jump to a chapter

View 39 chapters
  1. 2:41 Introduction
  2. 2:58 Guest Introductions (Hans & Zach)
  3. 3:56 Discussion on Breaking Kubernetes Clusters
  4. 5:13 Show Mechanics & Sponsors (Teleport, Equinix Metal)
  5. 5:50 Starting with Zach's Cluster (Hans Fixes First)
  6. 6:08 Teleport Connection Issues Debugging
  7. 9:55 Hans Shares Screen & Initial Cluster Access
  8. 10:16 Inspecting KubeConfig (Wrong IP Address)
  9. 12:11 Checking Kubelet Status and Logs
  10. 13:28 Debugging API Server Logs (Socket Error)
  11. 15:40 Examining Kube API Server Static Pod Manifest
  12. 17:55 Investigating Encryption Config File
  13. 19:06 Removing Encryption Config Reference
  14. 20:25 API Server Logs: Etcd Timeout Errors
  15. 21:30 Slow kubectl Commands and Pod CrashLoopBackOff
  16. 30:21 Diagnosing Slowness (Hint: Etcd/IO)
  17. 35:11 Identifying Suspicious Defunct Process
  18. 44:56 Discovering the IO Throttle Script
  19. 45:56 Disabling the Throttle Systemd Service
  20. 46:47 Cluster Control Plane Recovery
  21. 48:20 Pods Pending, Cilium CrashLoopBackOff
  22. 49:09 Attempting to Deploy Custard v2
  23. 50:09 Zach's Cluster Fixed
  24. 51:07 Starting Hans' Cluster (Zach Fixes Second)
  25. 51:12 Zach Takes Over Screen Sharing
  26. 53:27 Hans' Cluster: Pod CrashLoopBackOff & Slow kubectl
  27. 55:08 Identifying Mutating Webhooks (Kyverno Annotations)
  28. 57:06 Removing Kyverno (Dealing with Finalizers)
  29. 59:03 Post-Kyverno Status & Custard Service Issue
  30. 59:29 Database Failure & DNS Resolution Error
  31. 1:21:32 Checking Network Policies
  32. 1:22:39 Discovering and Deleting Cilium Egress Policy
  33. 1:23:07 Pod Still CrashLooping (App Check Error)
  34. 1:24:38 App-Level Check: v2 Image Already Present
  35. 1:25:30 Forcing Image Pull Policy Always
  36. 1:25:50 Hans' Cluster Fixed
  37. 1:26:02 Recap of the Breaks Found
  38. 1:26:24 Cron Job Location Revealed
  39. 1:26:45 Conclusion and Thanks

Technologies featured

Kubernetes Teleport Cilium Kyverno etcd CoreDNS

Meet the Cast

HOST

David Flanagan

@rawkode
GUEST

Hans Kristian Flaatten

@Starefossen
GUEST

Zach Wachtel

@zwachtel11

Comments and resources

Additional Resources

Ship It podcast

More from Klustered

View all 45 episodes
1:17:01

Alex Jones & Alistair Hey

1:29:15

Jetstack & CrashBeerBackOff

1:39:56

The Community Vs. Rawkode

1:33:14

IBM & Nisum

1:39:25

Marino Wijay & John Anderson

1:29:44

Adobe & Zapier

Rawkode Academy newsletter

Academy publications by email

Get an email when Rawkode Academy publishes new tutorials, articles, and videos.

Loading email signup. If it does not load, manage email preferences in account settings.