Kubernetes is a complex system with multiple components working in tandem. Though AKS is a managed Kubernetes offering, it still requires operators to take care of the security across the components. In this context, the document outlines the various security measures recommended across the Host, Cluster, API Server, Pods, and the supply chain for images.
Access Control
Authentication and Authorization are the primary mechanism to restrict access to a cluster. It is essential to use strong authentication and authorization to limit user and administrator access as well as to limit the attack surface.
Authentication
AKS-managed Azure AD integration simplifies the Azure AD integration process
Local accounts are enabled by default, and it is good best practice to disable them on an AAD enabled cluster
In the event where local accounts are required, it is recommended to limit access to the kubeconfig file
Authorization
AKS supports Azure RBAC as well as Kubernetes RBAC for authorization
Leverage Azure RBAC for all identities that exist inside Azure AD, and Kubernetes RBAC for those outside of AAD (i.e., Kubernetes Service Accounts)
Leverage Azure RBAC built-in roles
Identity
Though AKS supports Service principals it is recommended to use Managed Identities for simplified management.
The identity could be system-assigned or user-assigned (BYO)
User assigned identities can be managed separately and works great with IaC
Control Plane identity controls the creation of supporting Azure resources like load balancers, disks etc.
Kubelet Identity controls authenticating with ACR
Enable AAD Pod Identities to let workloads access cloud resources securely (i.e. KeyVault, databases etc.)
Traffic Control
Use firewalls to limit unneeded network connectivity.
Namespace should be used to define boundaries to apply security controls in an easy manner
Do not deploy any application workloads to the kube-* namespaces
Network policies control East-West traffic (i.e. pod to pod communication) through Network policies applied at the namespace level.
Avoid public exposure to the API server using private AKS cluster or limit exposure by using the authorized IP ranges
Control Egress traffic through a firewall
Encryption
Consider encryption to protect confidentiality
By default, all disks in AKS are encrypted at rest with Microsoft-managed keys. Consider BYO Keys for control over encryption keys.
AKS encrypts Etcd at rest and does not allow a KMS integration to bring your own key.
Use Sealed Secret to store the secrets in source control (for ex. gitops)
Use Azure Key Vault secrets in Azure DevOps pipelines or Github Actions
Use TLS for all ingress definitions through a trusted CA like Let’s Encrypt
Container Security
Kubernetes cluster security involves securing the host, cluster, and pods. Each of these layers should be hardened. Apart from it, the container image supply chain should be secured by scanning for known vulnerabilities and misconfigurations.
Host/Cluster
Azure Defender for Kubernetes provides both Host level and cluster level protection
Host Level has a limitation with AKS cluster on VMSS as there is a dependency on Log Analytics agent which is currently not supported in this setup. An enhanced daemonset approach will be released soon to address this limitation.
Integrated alerts for Cluster level and Host Level
Advanced analysis through AKS data connector for Azure Sentinel
Workload protection
Protection enabled through Kubernetes admission control via Azure Policy
Ensure Policy Add-on is auto provisioned on AKS clusters
Review and Configure the built-in policy bundle
Pod Security Standard Considered to be the replacement for PSP. Review and deploy the built-in baseline and restricted initiatives
Library of policies and tests that can be used with Custom Policy
Image Security
Static image scanning for all images in ACR. Image scans are performed by Qualys and includes all images when they’re pushed to the registry, imported into the registry, or pulled within the last 30 days.
Shift left in image scanning by integrating scans in the CI/CD pipelines. Azure Defender’s CI/CD scanning is powered by Aqua Trivy
Task available for GitHub workflows. ADO tasks are in the roadmap
Continuous image scanning and Drift detection is in development
Use opensource Falco or other commercial offerings for immediate needs
Pod Security
Explicitly control the resource utilization of the pods
Ensure that the containers are run in a specific security context
privileged: ability to run privileged container should be false unless required
hostPID, hostIPC: ability to share host process namespace should be false unless required
hostNetwork: ability to use host network namespace should be false unless required
allowedHostPaths: restricts to host paths and should be set to a dummy path for it to be effective
readOnlyRootFilesystem: Should be set to true by default and exempt writes as required
runAsUser: set to a non-root use
runAsGroup: set group to avoid defaulting to root i.e., 0
fsGroup: set group to specify ownership of files
allowPrivilegeEscalation: defines if the pod can assume root privileges
seLinux labels: lets you define access policies for services, processes, and filesystem access
AppArmor Profiles: Limit container actions
seccomp Profiles: filter the syscalls allowed
Enforce the security context to Policy initiatives
Automation of loading the AppArmor and SecComp profiles should ensure that they are applied to the nodes before any pods get scheduled. Consider a daemonset approach to overcome the race condition.
Opt out of automounting API credentials for a service account by setting automountServiceAccountToken: false on the service account
Audit
Kubernetes emits variety of logs across various components, and it is essential to log, monitor and alert on all of these to identify potential threats. Monitoring the logs ensures that the services are secure and operating as intended.
Monitoring
Effective logging and monitoring is essential to secure any environment, and enabling Container Insights on an AKS cluster provides the required observability.
It is recommended to monitor the below logs
Alerts
Enable the recommended metrics alerts bundle
Create log alert rules for Resource utilization and Resource availability
Create custom alerts as required. Refer sample log queries for table information
Conclusion
Securing the complete surface of the Kubernetes is essential and the above guidance provides a decent coverage. Though most of the concepts are specific to azure services, the security considerations are generic and applies to any Kubernetes distribution. Also, the concepts can be implemented through third party/OSS components in an AKS cluster