What is AoE?
AoE is a purpose built protocol built on top of Ethernet. AoE is a connectionless network protocol that does not rely on a session or transport layer. Coraid’s AoE driver recognizes every port and will use all in parallel. When an AoE target presents itself to the initiator, the driver will sense multiple unicast lanes between the server and storage without the need for a MPIO driver. When the OS drops an IO request (read or write) onto AoE, the driver will segment the IO into multiple “max MTU” Ethernet frames. These frames will be sent across every available HBA port as unicast AoE datagrams to the AoE target. Each of these datagrams must be acknowledged (with the unique correlation tag) prior to acknowledging the IO to the operating system, ensuring reliable delivery. By using multiple unicast lanes, Coraid’s implementation of AoE is able to achieve much higher throughput and lower latency (response times).
Source: http://www.coraid.com/solutions/technology
In the end this is only a storage backend. You still need some kind of NAS/SAN frontend to present these physical boxes to the servers/clients.
Coraid ZX NAS
A few weeks ago CORAID announced their own NAS headers: ZX-Series. As they already partner with Nexenta for scale-out NAS solutions, I assumed this was a logical next step for both. Apparently it is not. I took me a few tweets to finally get the following answer: “Coraid ZX-Series is based on Oracle Solaris 11 and RSF-1 from http://www.high-availability.com”. So why did they make this decision? John Gilmartin, VP Product Management clarified this for me:
Coraid is a big fan of Nexenta, and the combined solution provides powerful differentiation over typical white box deployments. We are sister companies in the Menlo Ventures fund, and continue to partner with them in accounts worldwide as part of our OpenFS strategy…In terms of our ZX solution, there were a number of business and technical factors that went into the selection. Oracle Solaris is a great OS which continues to evolve, and it includes all of the powerful features of ZFS that we needed.
Scale-out NAS
But then a few days ago I came across this post that stated the ZX would not be able to cluster. If you read the post it is a bit fallacious. What they really meant was that for now there is no scale-out clustering. As far as I know, a high availability cluster of 2 nodes is also a cluster but maybe that’s a pet peeve of the writer or just a way of drawing some more attention (it worked for me). But where he had a point is that a 2 node cluster (as the Nexenta for example right now also) is not a Scale OUT cluster. Scale out is many-2-many.
Real Scale out NAS
For real scale out you have to look for example to NFSv4.1. In that version pNFS has been added. pNFS stands for Parallel NFS and is an open-standard (!= open-source) supported for many years by Panasas. pNFS has been an answer for quit a while to HPC (high performance computing) as it addresses the problems a regular file-system has when addressing al the IO through a single file header. Instead of reading the IO’s through the storage header, the namespace servers will direct the clients to the repositories:
Source; http://www.panasas.com/products/pnfs-overview
My TakeAways
- I truly believe file-based/object-based storage will drive our storage choices more than block-based solutions. More on this in a next blogpost on a Belgian startup in the object storage: “Amplidata”.
- iSCSI has made a huge step-up in Enterprise storage but one of the downsides to most of the iSCSI products is that after a while multipathing can become struggle. One of my pet products EqualLogic for example can’t handle more than 1048 iSCSI connections per storage group. I’m sure new FW would push that number but not faster than we are scaling now. ANd I never really liked FC for its complexity. With AoE you don’t have these issues so much.
- CORAID AoE is also compatible with Windows Server. I haven’t given this a really deep thought yet but I can imagine the new storage capabilities of Server 2012 can leverage from this backend when it comes to scale (Windows SOFS supports 64 nodes).
OTHER LINKS:
Ray Luchessi: Coraid, first thoughts
Aidan Finn: Scale Out File Server in Windows Server 2012
Martin Glasborow: Clustered NAS versus traditional NAS