How accurate is the iPhone 5s CPU performance chart?

Apple's keynotes have become famous for the amount of detail and attention put into the slide decks, the interactive demos, and the perfectly scripted speeches. I've watched every keynote religiously since the original iPhone announcement in 2007. 

The team at Apple is great at telling stories. The press and fans are engaged every second of the keynote because the story always captivates them. There's a fine balance between the introduction ("we believe in innovation"), rising action ("our competitors are confused"), the climax ("new launches + one more thing"), and the ending ("live band"). To understand the way the plot works, check out How to Present like Steve Jobs from the folks at KissMetrics. Great read!

During the iPhone 5S keynote, the CPU Performance chart immediately sounded alarm bells. I wanted to investigate. I had recently finished reading Edward Tufte's The Visual Display of Quantitive Information (which I highly recommend to any designer), so I had a fresh perspective on the art of chart-making.

 The two biggest red flags that were obvious to me: 

  1. No labels and timescale on the X-Axis. This creates a false impression of a long period of steady CPU performance in the first 4 generations of iPhone (iPhone to iPhone 4), with a huge ramp up in the past 3 generations (iPhone 4s to iPhone 5s). This misrepresents the time span between launch dates of each device. It also creates a visual focus biased toward the right of the chart.
  2. A smooth connecting line between devices. This implies the data extends "below" the plotted points, in effect pushing the data "down".  In other words, there were no iPhones between iPhone 4 and iPhone 4S that would fall anywhere on the blue line. This creates a false impression that the increase in performance from device-to-device is more significant than it actually is. 

 

  iPhone CPU Performance Chart - Apple Keynote 2013

iPhone CPU Performance Chart - Apple Keynote 2013

Using CPU benchmarks from PassMark and device release dates from Wikipedia, I re-created the chart, in an attempt to correct the flaws I mentioned above. Here's the result:

 iPhone CPU Performance Chart - Adjusted

iPhone CPU Performance Chart - Adjusted

And finally, when overlapping the two, you can spot the differences more easily.

 iPhone CPU Performance Chart - Overlapped

iPhone CPU Performance Chart - Overlapped

Some evident misrepresentations I noticed:

  • The iPhone 3G, iPhone 3GS, iPhone 4 and iPhone 4S are pushed back in terms of release dates, sometimes by an entire year!
  • All devices from iPhone 3GS to iPhone 5 are performing lower than actual data from PassMark

Most importantly to note is that in Apple's chart, the iPhone 5S seems to improve over the iPhone 5 by much more than half, when in fact the data from PassMark indicate an improvement of roughly 50%.

Below is the data I used for creating the chart.

Device Release Date (No. of days from January 1, 2007) CPU Mark Rating
iPhone 179 671
iPhone 3G 557 671
iPhone 3GS 900 2,504
iPhone 4 (GSM) 1,270 3,452
iPhone 4S 1,748 8,845
iPhone 5 (GSM) 2,090 23,517
iPhone 5s (6.1) 2,455 35,271

The take-away is to always question the accuracy of charts, and to try to spot the holes in the story. The people behind the charts always want to paint a nice rosy picture, but the data always speaks the truth.