Dynamic, individual nodeAffinity for each pod in a StatefulSet

I have two node pools: NP1 and NP2. I have a StatefulSet with pods using ordinal index to retrieve their own configuration from external configuration source.

On certain conditions, I need to reschedule individual pods to either NP1 or NP2. Actual information about which pod should be run on which node pool is stored in a Consul KV object.

I can write a Kubernetes Operator which subscribes on that object and modifies StatefulSet in a way that it reschedules individual pods to a correct node pool when Consul KV object is updated. My question is how can I define different nodeAffinity for each pod?

I've seen some examples of using topologyKey with podAntiAffinity but in my case the decision which node pool a pod should be scheduled on is based on data completely external from kubernetes cluster.


As far as I know, you can use mutating webhook as mdaniel has mentioned in the comment:

I'm pretty sure only a Mutating Webhook will allow you to do dynamic tricks like that; I think OPA has some concessions toward that use case.

You can also find here more information about Dynamic Admission Control.