Spikes on External Metric scales HPA when it shouldn't

Posting this answer as a community wiki. Feel free to expand it.

Focusing on the part of scaling behavior:

As said in the question. Support for configurable scaling behavior is starting from the Kubernetes version 1.18:

Support for configurable scaling behavior

Starting from v1.18 the v2beta2 API allows scaling behavior to be configured through the HPA behavior field. Behaviors are specified separately for scaling up and down in scaleUp or scaleDown section under the behavior field. A stabilization window can be specified for both directions which prevents the flapping of the number of the replicas in the scaling target. Similarly specifying scaling policies controls the rate of change of replicas while scaling.

Kubernetes.io: Docs: Tasks: Run application: Horizontal pod autoscale: Support for configurable scaling behavior

Kubernetes version 1.18 which includes the support for above scaling is included in the release notes of a Regular channel:

January 8, 2021 (2021-R1)

Version updates

GKE cluster versions have been updated.

New versions available for upgrades and new clusters

The following Kubernetes versions are now available for new clusters and for opt-in master upgrades and node upgrades for existing clusters. See these instructions for more information on the Kubernetes versioning scheme.

Regular channel

Note: Your clusters might not have these versions available. Rollouts begin on the day of the note and take four or more business days to be completed across all Google Cloud zones. For more information, see the Rollout schedule.

• Version 1.17.14-gke.400 is now available in the Regular channel. This version is now the default.
• Version 1.18.12-gke.1201 is now available in the Regular channel.
• Version 1.17.13-gke.2600 is no longer available in the Regular channel.

Auto-upgrading nodes in the Regular channel automatically upgrade from version 1.17 to version 1.17.14-gke.400 with this release.

I encourage everyone to review the Release channels documentation and the properties of each channel(trade-off between feature availability and update churn):

• Cloud.google.com: Kubernetes Engine: Docs: Concepts: Release channels

You could also make a feature request for backporting this functionality to be included in previous GKE versions here:

• Issuetracker.google.com: Issues: New: Google Kubernetes Engine (Feature Request)