Dynamic, individual nodeAffinity for each pod in a StatefulSet
I have two node pools: NP1
and NP2
.
I have a StatefulSet
with pods using ordinal index to retrieve their own configuration from external configuration source.
On certain conditions, I need to reschedule individual pods to either NP1
or NP2
. Actual information about which pod should be run on which node pool is stored in a Consul KV object.
I can write a Kubernetes Operator which subscribes on that object and modifies StatefulSet
in a way that it reschedules individual pods to a correct node pool when Consul KV object is updated. My question is how can I define different nodeAffinity
for each pod?
I've seen some examples of using topologyKey
with podAntiAffinity
but in my case the decision which node pool a pod should be scheduled on is based on data completely external from kubernetes cluster.
As far as I know, you can use mutating webhook as mdaniel has mentioned in the comment:
I'm pretty sure only a Mutating Webhook will allow you to do dynamic tricks like that; I think OPA has some concessions toward that use case.
You can also find here more information about Dynamic Admission Control.