3.2.7 - Ensure that the --eventRecordQPS argument is set to 0 or a level which ensures appropriate event capture (Automated)

Views:

Profile Applicability: Level 2

Security relevant information should be captured. The --eventRecordQPS flag on the Kubelet can be used to limit the rate at which events are gathered. Setting this too low could result in relevant events not being logged, however the unlimited setting of 0 could result in a denial of service on the kubelet.

It is important to capture all events and not restrict event creation. Events are an important source of security information and analytics that ensure that your environment is consistently monitored using the event data.

Note

See the Azure AKS documentation for the default value.

Audit

Audit Method 1:

SSH to the relevant node and run the following command on each node to find the Kubelet process:
```
ps -ef | grep kubelet
```
In the output of the above command, review the value set for the --eventRecordQPS argument and determine whether this has been set to an appropriate level for the cluster. The value of 0 can be used to ensure that all events are captured.
If the --eventRecordQPS argument does not exist, check that there is a Kubelet config file specified by --config and review the value in this location.

The output of the above command should return something similar to --config /etc/kubernetes/kubelet/kubelet-config.json which is the location of the Kubelet config file.

Open the Kubelet config file:

cat /etc/kubernetes/kubelet/kubelet-config.json

Verify that "eventRecordQPS" is set to 0 or another appropriate level for the cluster.

Audit Method 2:

If using the API configz endpoint, search for the status of eventRecordQPS by extracting the live configuration from the nodes running Kubelet.:

Set the local proxy port and the following variables and provide proxy port number and node name:
- HOSTNAME_PORT="localhost-and-port-number"
- NODE_NAME="The-Name-Of-Node-To-Extract-Configuration" from the output of "kubectl get nodes"
```
kubectl proxy --port=8001 & 

export HOSTNAME_PORT=localhost:8001 (example host and port number) 
export NODE_NAME=ip-192.168.31.226.aks.internal (example node name 
from "kubectl get nodes") 

curl -sSL "http://${HOSTNAME_PORT}/api/v1/nodes/${NODE_NAME}/proxy/configz"
```

Remediation

Remediation Method 1:

If modifying the Kubelet config file, edit /etc/kubernetes/kubelet/kubelet-config.json and set the below parameter to 5 or a value greater or equal to 0:

"eventRecordQPS": 5

Check that /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf does not define an executable argument for eventRecordQPS because this would override your Kubelet config.

Remediation Method 2:

If using executable arguments, edit the Kubelet service file /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf on each worker node and add the below parameter at the end of the KUBELET_ARGS variable string:

--eventRecordQPS=5

Remediation Method 3:

If using the API configz endpoint, search for the status of "eventRecordQPS" by extracting the live configuration from the nodes running Kubelet.

See step-by-step configmap procedures in the Kubernetes documentation, and rerun the curl command from the audit process to check for kubelet configuration changes:

     kubectl proxy --port=8001 &
     
     export HOSTNAME_PORT=localhost:8001 (example host and port number) 
     export NODE_NAME=ip-192.168.31.226.aks.internal (example node name from 
     "kubectl get nodes")
     
     curl -sSL "http://${HOSTNAME_PORT}/api/v1/nodes/${NODE_NAME}/proxy/configz"

For all three remediations:

Restart the kubelet service and check status:

     systemctl daemon-reload
     systemctl restart kubelet.service
     systemctl status kubelet -l