Why do AWS spot-instance prices spike above the "on demand" pricing?
There is a paper by folks from Israel Institute of Technology which analyzes the spot market. Their conclusion is that spot prices are not bid driven but rather generated with AR(1) function. So in such case it wouldn't be due to other's bids, but due to algorithm used.
http://www.cs.technion.ac.il/~ladypine/spotprice-ieee.pdf
Though, I guess it could have changed if spot market really reached the level where they could get nice prices from demand/supply. As Matt suggest it become a common practice to drive costs down with spot instances and many overbid on-demand prices to avoid down time. In longer run it was still cost effective ... but is it still? The real problem is that apart of Amazon nobody really knows how the prices are created.
This is not a bug, this is an priority / availability issue because on-demand instances have priority over spot-instances regardless of spot-bids. This stems from Amazon EC2 statement that you "bid on spare Amazon EC2 instances".
So when the demand for on-demand instances increases there are less spares available, and the availability of spot-instances decreases driving spot pricing higher according to competing spot-bids.
Many spot-bidders have failed to understand the higher priority of on-demand instances and have bid above on-demand pricing in an attempt to gain priority over on-demand instance requests, but this fails because on-demand instances have higher priority regardless of spot biding prices.
In turn these above-the-on-demand pricing bids have been competing against each other driving the highest prices multiple times higher than on-demand pricing when spot-instances availability is low.
Although the following was not asked, my recommendations for lowest-cost spot instances management of long-term workloads is:
- Design your application for failures (continuous backup / replication), and rapid recovery
- Set spot-instance bid price to that of on-demand pricing
- When a spot instance is terminated, restart it as an on-demand instance or in another region/zone where spot pricing is lower.
- Monitor spot-pricing until it goes lower than on-demand pricing
- Stop on-demand instance and goto 2. above
EC2 no longer use the artificial AR(1) process to control the minimal price. Even when they did, they only changed the minimal price - the AR(1) process was anchored, and never caused prices to jump above the on-demand price. Such jumps can come either from users specifying high prices (possibly more than they really intended to pay), or from Amazon wishing to shut down a portion of the spot instances, like it seems they did for m1.small in the summer of 2010.
You can read more in the journal version of our paper, in the last section which is more updated (the postmortem section):
It's not a bug in the reporting. Sometimes, the spot prices do spike. In the past, they've spiked to enormous amounts. Since then, Amazon has improved the algorithm in which spot instances are bid on and allocated, but even still, spot prices do spike from time to time.
Back in September 2011, spot prices for m2.2xlarge in one us-east-1 AZ spiked to $999/hour.
The reason is simple: people are specifying maximum spot prices that are higher than the on-demand price, and demand is pushing the prices that high.
Many people use spot instances as a way to minimize the cost of the running EC2 instances. Quite often, they will specify high maximum spot prices in order to reduce the chances that their spot instance will be out-bid and terminated. It's a gamble that may pay off in the long-run, but runs the risk of getting very expensive very fast if they're not careful.