Tuned Operator does not activate custom profile (Openshift 4.6.18)

Solution 1:

Documentation for tuned operator can be found at https://docs.openshift.com/container-platform/4.7/scalability_and_performance/using-node-tuning-operator.html.

Generic Information about Tuned Operator:

• Namespace/Project : openshift-cluster-node-tuning-operator
• Operator : cluster-node-tuning-operator
• DaemonSet : tuned
• CRD : tuneds.tuned.openshift.io
• CR : Tuned/default & Tuned/rendered

The documentation says that we can create our own custom resources of kind=Tuned apart from default resources provided by openshift named "Tuned/default & Tuned/rendered".

These resources provide default profiles named "openshift", "openshift-node" and "openshift-control-plane".

More information can be seen using below command:

oc get Tuned/default -n openshift-cluster-node-tuning-operator -o yaml

Now, we can create our own custom profile as part of custom resource to tune our own settings.

The trick here is that the configuration in custom resource yaml file regarding custom profile should be correct. If it is correct, tuned operator will load the profile and activate it. If it is incorrect, then tuned operator will NOT activate it and it will ignore any future correct configuration also.

This is a bug in tuned operator which is addressed as part of https://bugzilla.redhat.com/show_bug.cgi?id=1919970.

Fix: Upgrade openshift cluster version to 4.7 and above.

Workaround: Delete the tuned pod so that operator will create new pod. Once new pod is created, it will activate correct profile. (Hoping configuration in your CR.yaml was corrected).

Important Commands:

• To find out pod on which tuned operator itself is running:

oc get pod -n openshift-cluster-node-tuning-operator -o wide

• To check logs of the operator pod: (Actual pod name can be found from above command)

oc logs pod/cluster-node-tuning-operator-6644cd48bb-z2qxn -n openshift-cluster-node-tuning-operator

• To check which all Custom Resouces of kind=Tuned are present:

oc get Tuned -n openshift-cluster-node-tuning-operator

• To describe and check default profiles:

oc get Tuned/default -n openshift-cluster-node-tuning-operator -o yaml

• To find out all tuned pods and nodes on which they are running in the cluster:

oc get pod -n openshift-cluster-node-tuning-operator -o wide

• To check logs of a particular tuned pod: (Actual pod name can be found from above command)

oc logs tuned-h8xgh -n openshift-cluster-node-tuning-operator -f

• To login into tuned pod and manually confirm tuning is applied or not: (Actual pod name can be found from previous-to-above command)

oc exec -it tuned-h8xgh -n openshift-cluster-node-tuning-operator --bash

• You can execute below commands after login into tuned pod using above command to verify tuning settings:

  bash-4.4# cat /etc/tuned/infra-nodes/tuned.conf
  [main] summary=Optimize systems running OpenShift Infra nodes
  [sysctl] fs.inotify.max_user_watches = 1048576 vm.swappiness = 1

  bash-4.4# tuned-adm recommend Cannot talk to Tuned daemon via DBus. Is Tuned daemon running? infra-nodes bash-4.4#

bash-4.4# tuned-adm active 
Cannot talk to Tuned daemon via DBus. Is Tuned daemon running? 
Current active profile: openshift-control-plane 
bash-4.4#

Note: The above sample code exactly depicts the issue asked in this question. If you notice, active profile is "openshift-control-plane" whereas recommended/loaded one is "infra-nodes". This is due to the existing bug as mentioned previously. Once you delete tuned pod (tuned-h8xgh), operator will recover and activate correct profile.

Sample issue in custom profile configuration: If profile priorities are same as default profiles, then operator will give warning something similar as below:

W0722 04:24:25.490704       1 profilecalculator.go:480] profiles openshift-control-plane/infra-node have the same priority 30, please use a different priority for your custom profiles!