The CVM, or controller virtual machine, is the real magic of Nutanix.  Each node will run one CVM on top of the hypervisor.  The Nutanix installers will automagically apply best practices to your hypervisor and deploy the CVM on each one of your hosts (your partner or SE can help you with this).

So what is the CVM?  At it's core, it's simply a CentOS VM with the hard drives directly passed through the hypervisor using VT-d.  This has a couple of implications:

1) The hypervisor cannot see, use, or control the local disks which means the hypervisor needs to send commands through the CVM to access the local disks

2) The CVM directly accesses the hardware, the hypervisor doesn't need to translate commands

3) Each hypervisor needs it's own CVM and this CVM cannot move.  If the host needs to be powered down so does the CVM 

Because the CVMs are directly in the datapath they are in charge of the features normally handled by the array (clustering, dedup, failover, etc.) 

During the installation the CVMs discover each other using IPv6, giving you the option to set IPs and cluster them together.  See the install guide for a detailed overview of this process, however it is pretty straight forward (the goal is to make it as quick and easy as possible so most of the install is automated).

Once the cluster is created, you can point your web browser to the IP of any CVM to access the GUI (Prism):

Default credentials are admin/admin

This beautiful GUI is completely rendered in HTML5 and gives you an overview of your system

In the GUI (Home -> Storage) you can create a Storage Pool consisting of one or more disks (Nutanix recommends creating one large storage pool) and a container inside this pool (once again Nutanix recommends creating one big one).  The container is what the hypervisor will mount (so the ESXi equivalent would be a datastore).  If you are using ESXi the CVM will give you the option of automatically mounting the datastore on the hosts.  Let's see what that looks like:

And that's it!  At this point you can use this datastore like you would any other ESXi shared storage (including features such as vMotion, HA, FT, etc.).  ESXi has no idea that it's actually using storage presented by a VM running on itself (and it doesn't need to).  The CVMs, communicating over the network, now will handle things such as data replication automatically.  ESXi just sees the storage as a massive NFS share.

Now you may be wondering how this particular set of trickery is achieved - after all for ESXi to see this as the same NFS share all hosts must mount the same IP.  How do we make sure the ESXi host sends the commands to the local CVM and not another one on the network?  This is done through a private vSwitch.  Let's take a look at the special vSwitch that was created for us:

This means that each host has a private network (192.168.5.0/25) that is used to talk only between the CVM and the host.  Thus the routing table for 192.168.5.2 points to vmk1 - each host can mount this and we can be sure it's no going across the network.  In the case of a failure of the CVM (such as a crash or a power down) there is a script running (ha.py) that will reroute this traffic out vmk0 to one of the other CVMs.  

The CVMs will handle all the data locating and redundancy themselves.  If a host requests data that is not available locally, the CVM will reach out to the node that does have it (using it's external interface) and request it.  The CVM will then locally cache it so that future reads do not need the network.

The end result?  Shared, clustered, redundant storage without the array and with dead simple configuration.