Dragster Benchmarking

I’ve ranted on this more than once. Benchmarks are 99% of the time utter bull… and tell you nothing about what the solution’s real possibilities are, let alone what they mean to your environment. The dragster benchmarks (i.e. SPC-1) are just a show-off competition with little to no value to you. Allow me to bring up a couple of points why;

• Generally speaking the dragster benchmark is based on 100% 4k-reads. Let me assure you that there is not a single system out there – certainly not yours – that does 100% 4k reads, let alone 4k in general. It’s when these blocks are bigger (16k, 64k and towards 128k/265k) when things get interesting and these machines will show different proportions towards those 4k reads.
• Nowadays you’ll see that vendors will add ‘random’ reads, where it historically always was ‘sequential’ reads. This is mostly seen when the working set will come from flash where random and sequental hardly matter.
• Writes will also show huge differences in the proportions towards those 4k reads as we now have a full IO-ack path to follow. In a 2 controller system you’ll have to take note of the system’s controller caching (& CPU) capabilities where in a scale-out architecture you’ll see penalties from traversing the network once or multiple times.

Bring in the data reduction

Some vendors are even excluded from performing [publishing] the standard benchmarks in the likes of SPC-1 because these benchmarks don’t allow datareduction to take place. A lot of new technologies have, because of the power of flash, put datareduction at the very foundation of their architecture. I’m thinking of SimpliVity for example with their “the best IO is the one you never have to write” and in the case of this post PureStorage.

I was quite pleased to read PureStorage’s blogpost last week where they prepared a version of the vdbench performance tool for you with a lenghtly post about the merits/necessities of a [new] tool like this. It is definitely worth a read! Of course this whole thing is self-serving for them but it certainly can’t do harm in moving the needle towards a more transparent and honest marketing.

I praise where possible, heckle where necessary. I will time and time again ridicule PureStorage when they go after EMC with another cheap-shot campaign (like others do as well where I heckle a well), but this is already the second time I have to praise them for doing what I feel is the right thing to help answer the real question the customer is asking; what is your solution going to do in my environment. And this, my friends, is what most vendors keep ignoring time and time again. So please, if you do yet another benchmark, tell them what that means … for ME.

Disclaimer: I am by no means compensated by PureStorage for this post. That being said both PureStorage and SimpliVity have used my consulting services in the past.