This is just a post of random, important details of using an AWS NLB that I couldn’t easily find all written down in one place.
The context is I’m trying to put an NLB in front of a Kubernetes cluster (in this case, an AWS managed Kubernetes cluster (aka “EKS”)), and I’m trying to follow the AWS recommendation of directly registering the Kubernetes pods as backend-end targets of the load-balancer itself:
Register Pods as targets using IP Target-Type
EKS Best Practices - Load Balancing
Doing this means that your pod-lifecycle is tied to the particulars of NLB operations.
Some details that aren’t exactly documented:
This means that if you are doing a rollout of your back-end service your deployment-rollout process may start cleaning up the old instances of you service before the new ones have started receiving traffic.
You’ll want to enabled AWS Loadbalancer Controller pod-readiness gates so that Kubernetes controllers understand what’s happening up in your front-end.
This is separate from the “deregistration delay” in AWS documentation. The “deregistration delay” is a configurable amount of time after a target stops receiving new connections that the NLB allows existing connections to drain.
For this, you’ll want some sort of pre-shutdown delay in whatever connection-draining your service does. And again - this needs to be a delay after your pod starts terminating but before your service stops accepting new connections - a pre stop hook is probably great.