HFT Alert Pro enabled us to catch the price arbs in real-time, take notice of the areas within the red rectangles:
One minute chart highlighting the problem area
NYMEX Black Swan ~ The March 2012 Crude Oil Futures Quote Loop
On February 13, 2012, starting 13:59:57, quotes for crude oil began queuing. At 14:00:35, all of the queued quotes were sent at once. Again at 14:01:08 the same 38 second block of quotes sent earlier was sent again -- old timestamps and all plus a few new quotes. Again at 14:01:18, all quotes since 13:59:57 were sent again. This repeated 12 times.
From a programmers perspective, it looks like a system problem caused a blast of quotes that corrupted a memory queue causing the software to believe the queue was full all the time.
Tick chart of bid prices (red) along with quote age (blue).
Note that as the cycle repeats, it includes a few more quotes (the new quotes + those since 13:59:57). There are 500 quotes between time axis labels.
500 millisecond chart of ETF U.S. Oil Fund (USO) showing massive quote traffic as it reacts to stale futures quotes.
500 millisecond chart of ETF U.S. Oil Fund (USO) showing massive trade executions in reaction to stale futures quotes.