Nutanix - Advanced-Admin-AOS-v51

Acropolis Advanced Administration Guide
Acropolis 5.1
15-Jun-2017
Notice
Copyright
Copyright 2017 Nutanix, Inc.
Nutanix, Inc.
1740 Technology Drive, Suite 150
San Jose, CA 95110
All rights reserved. This product is protected by U.S. and international copyright and intellectual property
laws. Nutanix is a trademark of Nutanix, Inc. in the United States and/or other jurisdictions. All other marks
and names mentioned herein may be trademarks of their respective companies.
License
The provision of this software to you does not grant any licenses or other rights under any Microsoft
patents with respect to anything other than the file server implementation portion of the binaries for this
software, including no licenses or any other rights in any hardware or any devices or software that are used
to communicate with or in connection with this software.
Conventions
Convention Description
variable_value The action depends on a value that is unique to your environment.
ncli> command The commands are executed in the Nutanix nCLI.
user@host$ command The commands are executed as a non-privileged user (such as nutanix)
in the system shell.
root@host# command The commands are executed as the root user in the vSphere or Acropolis
host shell.
> command The commands are executed in the Hyper-V host shell.
output The information is displayed as output from a command or in a log file.
Default Cluster Credentials

Interface Target Username Password
Nutanix web console Nutanix Controller VM admin Nutanix/4u
vSphere Web Client ESXi host root nutanix/4u
Copyright | Acropolis Advanced Administration Guide | AOS | 2

Interface Target Username Password
vSphere client ESXi host root nutanix/4u
SSH client or console ESXi host root nutanix/4u
SSH client or console AHV host root nutanix/4u
SSH client or console Hyper-V host Administrator nutanix/4u
SSH client Nutanix Controller VM nutanix nutanix/4u
SSH client Nutanix Controller VM admin Nutanix/4u
SSH client or console Acropolis OpenStack root admin

Services VM (Nutanix
OVM)
Version
Last modified: June 15, 2017 (2017-06-15 12:40:54 GMT+5.5)
Copyright | Acropolis Advanced Administration Guide | AOS | 3

Contents
1: Cluster Management............................................................................... 6
Controller VM Access.......................................................................................................................... 6
Port Requirements.................................................................................................................... 6
Admin Access to Controller VM................................................................................................6
Starting a Nutanix Cluster................................................................................................................... 8
Stopping a Cluster............................................................................................................................... 9
Destroying a Cluster.......................................................................................................................... 10
Creating Clusters from a Multiblock Cluster......................................................................................11
Cluster IP Address Configuration........................................................................................... 11
Configuring the Cluster........................................................................................................... 11
Verifying IPv6 Link-Local Connectivity....................................................................................15
Failing from one Site to Another....................................................................................................... 17
Disaster failover.......................................................................................................................17
Planned failover...................................................................................................................... 17
Fingerprinting Existing vDisks............................................................................................................18
2: Changing Passwords............................................................................ 19
Changing User Passwords................................................................................................................ 19
Changing the SMI-S Provider Password (Hyper-V)............................................................... 19
Changing the Controller VM Password............................................................................................. 20
3: Cluster IP Address Configuration........................................................22

Network Configuration (Virtual Interfaces, Virtual Switches, and IP Addresses)...............................22
Changing Controller VM IP Addresses............................................................................................. 24
Preparing to Set IP Addresses............................................................................................... 24
Preparing to Change Controller VM IP Addresses.................................................................26
Changing Controller VM IP Addresses...................................................................................27
Changing a Controller VM IP Address (manual).................................................................... 28
Completing Controller VM IP Address Change...................................................................... 30
4: Creating a Windows Guest VM Failover Cluster................................ 32
5: Acropolis Dynamic Scheduling in AHV.............................................. 35

Disabling Acropolis Dynamic Scheduling.......................................................................................... 35
Enabling Acropolis Dynamic Scheduling........................................................................................... 36
6: Logs........................................................................................................ 37
Sending Logs to a Remote Syslog Server........................................................................................37
Configuring the Remote Syslog Server Settings.................................................................... 38
Common Log Files.............................................................................................................................40
Nutanix Logs Root.................................................................................................................. 40
Self-Monitoring (sysstats) Logs...............................................................................................40
/home/nutanix/data/logs/cassandra.........................................................................................40
Controller VM Log Files.......................................................................................................... 41
4
Correlating the FATAL log to the INFO file....................................................................................... 43
Stargate Logs.....................................................................................................................................44
Cassandra Logs................................................................................................................................. 46
Prism Gateway Log........................................................................................................................... 47
Zookeeper Logs................................................................................................................................. 47
Genesis.out........................................................................................................................................ 48
Diagnosing a Genesis Failure................................................................................................ 49
ESXi Log Files................................................................................................................................... 50
7: Troubleshooting Tools.......................................................................... 51
Nutanix Cluster Check (NCC)........................................................................................................... 51
Installing NCC from an Installer File.......................................................................................52
Upgrading NCC Software....................................................................................................... 54
NCC Usage............................................................................................................................. 55
Diagnostics VMs................................................................................................................................ 56
Running a Test Using the Diagnostics VMs........................................................................... 57
Diagnostics Output.................................................................................................................. 58
Syscheck Utility.................................................................................................................................. 58
Using Syscheck Utility.............................................................................................................58
8: Controller VM Memory Configurations............................................... 60

CVM Memory and vCPU Configurations (G5/Broadwell)..................................................................60
Platform Workload Translation (G5/Broadwell).......................................................................61
CVM Memory and vCPU Configurations (G4/Haswell/Ivy Bridge).................................................... 62
CVM Memory Configurations for Features........................................................................................63
5
1
Cluster Management
Although each host in a Nutanix cluster runs a hypervisor independent of other hosts in the cluster, some
operations affect the entire cluster.
Controller VM Access
Most administrative functions of a Nutanix cluster can be performed through the web console or nCLI.
Nutanix recommends using these interfaces whenever possible and disabling Controller VM SSH access
with password or key authentication. Some functions, however, require logging on to a Controller VM
with SSH. Exercise caution whenever connecting directly to a Controller VM as the risk of causing cluster
issues is increased.
Warning: When you connect to a Controller VM with SSH, ensure that the SSH client does
not import or change any locale settings. The Nutanix software is not localized, and executing
commands with any locale other than en_US.UTF-8 can cause severe cluster issues.
To check the locale used in an SSH session, run /usr/bin/locale. If any environment variables
are set to anything other than en_US.UTF-8, reconnect with an SSH configuration that does not
import or change any locale settings.
Port Requirements
Nutanix uses a number of ports for internal communication. The following unique ports are required for
external access to Controller VMs in a Nutanix cluster.
Table
Purpose Port Numbers

Remote site replication 2009 and 2020
Cluster and IP address configuration 2100
Remote support tunnel (outgoing connection to service centers 80 or 8443
nsc01.nutanix.net and nsc02.nutanix.net)
Management interface (web console, nCLI) 9440
Admin Access to Controller VM

You can access the Controller VM as the admin user (admin user name and password) with SSH. For
security reasons, the password of the admin user must meet complexity requirements. When you log on to
the Controller VM as the admin user for the first time, you are prompted to change the default password.
Following are the default credentials of the admin user:
Cluster Management | Acropolis Advanced Administration Guide | AOS | 6

• User name: admin and Password: Nutanix/4u
The password must meet the following complexity requirements:
• At least 8 characters long
• At least 1 lowercase letter
• At least 1 uppercase letter
• At least 1 number
• At least 1 special character
• At least 4 characters difference from the old password
• Should not be among the last 10 passwords
After you have successfully changed the password, the new password is synchronized across all Controller
VMs and interfaces (Prism web console, nCLI, and SSH).
Note:
• As an admin user, you cannot access nCLI by using the default credentials. If you are logging
in as the admin user for the first time, you must SSH to the Controller VM or log on through the
Prism web console. Also, you cannot change the default password of the admin user through
nCLI. To change the default password of the admin user, you must SSH to the Controller VM or
log on through the Prism web console.
• When you make an attempt to log in to the Prism web console for the first time after you
upgrade to AOS 5.1 from an earlier AOS version, you can use your existing admin user
password to log in and then change the existing password (you are prompted) to adhere to the
password complexity requirements. However, if you are logging in to the Controller VM with
SSH for the first time after the upgrade as the admin user, you must use the default admin user
password (Nutanix/4u) and then change the default password (you are prompted) to adhere to
the password complexity requirements.
By default, the admin user password does not have an expiry date, but you can change the password at
any time.
When you change the admin user password, you must update any applications and scripts using the
admin user credentials for authentication. Nutanix recommends that you create a user assigned with the
admin role instead of using the admin user for authentication. The Prism Web Console Guide describes
authentication and roles.
Following are the default credentials to access a Controller VM.
Controller VM Credentials
Interface Target User Name Password
SSH client Nutanix Controller VM admin Nutanix/4u

nutanix nutanix/4u
Prism web console Nutanix Controller VM admin Nutanix/4u
Accessing the Controller VM Using the Admin Account
Perform the following procedure to log on to the Controller VM by using the admin user with SSH for the
first time.
1. Log on to the Controller VM with SSH by using the management IP address of the Controller VM and
the following credentials.
User name: admin and Password: Nutanix/4u

You are now prompted to change the default password.
2. Respond to the prompts, providing the current and new admin user password.
Changing password for admin.
Old Password:
New password:
Retype new password:
Password changed.
The password must meet the following complexity requirements:

For information about logging on to a Controller VM by using the admin user account through the Prism
web console, see Logging Into The Web Console in the Prism Web Console guide.
Starting a Nutanix Cluster
1. Log on to any Controller VM in the cluster with SSH.
2. Start the Nutanix cluster.

nutanix@cvm$ cluster start
If the cluster starts properly, output similar to the following is displayed for each node in the cluster:
CVM: 10.1.64.60 Up
Zeus UP [5362, 5391, 5392, 10848, 10977, 10992]
Scavenger UP [6174, 6215, 6216, 6217]
SSLTerminator UP [7705, 7742, 7743, 7744]
SecureFileSync UP [7710, 7761, 7762, 7763]
Medusa UP [8029, 8073, 8074, 8176, 8221]
DynamicRingChanger UP [8324, 8366, 8367, 8426]
Pithos UP [8328, 8399, 8400, 8418]
Hera UP [8347, 8408, 8409, 8410]
Stargate UP [8742, 8771, 8772, 9037, 9045]
InsightsDB UP [8774, 8805, 8806, 8939]
InsightsDataTransfer UP [8785, 8840, 8841, 8886, 8888, 8889,
8890]
Ergon UP [8814, 8862, 8863, 8864]
Cerebro UP [8850, 8914, 8915, 9288]
Chronos UP [8870, 8975, 8976, 9031]
Curator UP [8885, 8931, 8932, 9243]
Prism UP [3545, 3572, 3573, 3627, 4004, 4076]
CIM UP [8990, 9042, 9043, 9084]
AlertManager UP [9017, 9081, 9082, 9324]
Arithmos UP [9055, 9217, 9218, 9353]
Catalog UP [9110, 9178, 9179, 9180]
Acropolis UP [9201, 9321, 9322, 9323]
Atlas UP [9221, 9316, 9317, 9318]
Uhura UP [9390, 9447, 9448, 9449]
Snmp UP [9418, 9513, 9514, 9516]

SysStatCollector UP [9451, 9510, 9511, 9518]
Tunnel UP [9480, 9543, 9544]
ClusterHealth UP [9521, 9619, 9620, 9947, 9976, 9977,
10301]
Janus UP [9532, 9624, 9625]
NutanixGuestTools UP [9572, 9650, 9651, 9674]
MinervaCVM UP [10174, 10200, 10201, 10202, 10371]
ClusterConfig UP [10205, 10233, 10234, 10236]
APLOSEngine UP [10231, 10261, 10262, 10263]
APLOS UP [10343, 10368, 10369, 10370, 10502,
10503]
Lazan UP [10377, 10402, 10403, 10404]
Orion UP [10409, 10449, 10450, 10474]
Delphi UP [10418, 10466, 10467, 10468]
What to do next: After you have verified that the cluster is running, you can start guest VMs.
(Hyper-V only) If the Hyper-V failover cluster was stopped, start it by logging on to a Hyper-V host and
running the Start-Cluster PowerShell command.
Warning: By default, Nutanix clusters have redundancy factor 2, which means they can tolerate
the failure of a single node or drive. Nutanix clusters with a configured option of redundancy factor
3 allow the Nutanix cluster to withstand the failure of two nodes or drives in different blocks.
• Never shut down or restart multiple Controller VMs or hosts simultaneously.
• Always run the cluster status command to verify that all Controller VMs are up before
performing a Controller VM or host shutdown or restart.
Stopping a Cluster
Before you begin: Shut down all guest virtual machines, including vCenter if it is running on the cluster.
Do not shut down Nutanix Controller VMs.
Note:
• If you are running Acropolis File Services (AFS), stop AFS before stopping your AOS cluster.
• If you are planning to stop your cluster that has metro availability configured, do not stop the
cluster before performing some remedial actions. For more information, see Data Protection
Guidelines (Metro Availability) topic in the Prism Web Console guide.
(Hyper-V only) Stop the Hyper-V failover cluster by logging on to a Hyper-V host and running the Stop-
Cluster PowerShell command.
Note: This procedure stops all services provided by guest virtual machines, the Nutanix cluster,
and the hypervisor host.
1. Log on to a running Controller VM in the cluster with SSH.
2. Stop the Nutanix cluster.

nutanix@cvm$ cluster stop
Wait to proceed until output similar to the following is displayed for every Controller VM in the cluster.
CVM: 172.16.8.191 Up, ZeusLeader

Zeus UP [3167, 3180, 3181, 3182, 3191, 3201]
Scavenger UP [3334, 3351, 3352, 3353]
ConnectionSplicer DOWN []
Hyperint DOWN []

Medusa DOWN []
DynamicRingChanger DOWN []
Pithos DOWN []
Stargate DOWN []
Cerebro DOWN []
Chronos DOWN []
Curator DOWN []
Prism DOWN []
AlertManager DOWN []
StatsAggregator DOWN []
SysStatCollector DOWN []
Destroying a Cluster
Before you begin: Reclaim licenses from the cluster to be destroyed by following Reclaiming Licenses
When Destroying a Cluster in the Web Console Guide.
Note: If you have destroyed the cluster and did not reclaim the existing licenses, contact Nutanix
Support to reclaim the licenses.
Destroying a cluster resets all nodes in the cluster to the factory configuration. All cluster configuration and
guest VM data is unrecoverable after destroying the cluster.
Note: If the cluster is registered with Prism Central (the multiple cluster manager VM), unregister
the cluster before destroying it. See Registering with Prism Central in the Web Console Guide for
more information.


Zeus UP [3167, 3180, 3181, 3182, 3191, 3201]
Scavenger UP [3334, 3351, 3352, 3353]
Hyperint DOWN []
Medusa DOWN []
Pithos DOWN []
Stargate DOWN []
Cerebro DOWN []
Chronos DOWN []
Curator DOWN []
Prism DOWN []
3. Destroy the cluster.

Caution: Performing this operation deletes all cluster and guest VM data in the cluster.
nutanix@cvm$ cluster destroy

Follow the prompts to confirm destruction of the cluster.
Creating Clusters from a Multiblock Cluster

The minimum size for a cluster is three nodes.
1. Remove nodes from the existing cluster.

→ If you want to preserve data on the existing cluster, remove nodes from the cluster using the
Hardware > Table > Host screen of the web console.
→ If you want multiple new clusters, destroy the existing cluster by following Destroying a Cluster on
page 10.
2. Create one or more new clusters by following Configuring the Cluster on page 11.
Cluster IP Address Configuration
AOS includes a web-based configuration tool that automates assigning IP addresses to cluster
components and creates the cluster.
Requirements
The web-based configuration tool requires that IPv6 link-local be enabled on the subnet. If IPv6 link-local is
not available, you must configure the Controller VM IP addresses and the cluster manually. The web-based
configuration tool also requires that the Controller VMs be able to communicate with each other.
All Controller VMs and hypervisor hosts must be on the same subnet. The hypervisor can be multihomed
provided that one interface is on the same subnet as the Controller VM.
Guest VMs can be on a different subnet.
Configuring the Cluster

Before you begin: Check that the cluster is ready to be configured by following Preparing to Set IP
Addresses on page 24.
Note: This procedure has been deprecated (superseded) in AOS 4.5 and later releases. Instead,
use the Foundation tool to configure a cluster. See the "Creating a Cluster" topics in the Field
Installation Guide for more information.

Video: Click here to see a video (MP4 format) demonstration of this procedure. (The video may
not reflect the latest features described in this section.)
Figure: Cluster IP Address Configuration Page
1. Open a web browser.

Nutanix recommends using Internet Explorer 9 for Windows and Safari for Mac OS.
Note: Internet Explorer requires protected mode to be disabled. Go to Tools > Internet
Options > Security, clear the Enable Protected Mode check box, and restart the browser.
2. In the browser, go to http://[cvm_ipv6_addr]:2100/cluster_init.html.

Replace [cvm_ipv6_addr] with the IPv6 address of any Controller VM that should be added to the
cluster.
Following is an example URL to access the cluster creation page on a Controller VM:
http://[fe80::5054:ff:fea8:8aae]:2100/cluster_init.html
If the cluster_init.html page is blank, then the Controller VM is already part of a cluster. Connect to a
Controller VM that is not part of a cluster.
You can obtain the IPv6 address of the Controller VM by using the ifconfig command.

Example
nutanix@cvm$ ifconfig
eth0 Link encap:Ethernet HWaddr 52:54:00:A8:8A:AE
inet addr:10.1.65.240 Bcast:10.1.67.255 Mask:255.255.252.0
inet6 addr: fe80::5054:ff:fea8:8aae/64 Scope:Link
...etc...
The value of the inet6 addr field up to the / character is the IPv6 address of the Controller VM.
3. Type a meaningful value in the Cluster Name field.

This value is appended to all automated communication between the cluster and Nutanix support. It
should include the customer's name and, if necessary, a modifier that differentiates this cluster from any
other clusters that the customer might have.
Note: This entity has the following naming restrictions:

• The maximum length is 75 characters (for vSphere and AHV) and 15 characters (for Hyper-
V).
• Allowed characters are uppercase and lowercase standard Latin letters (A-Z and a-z),
decimal digits (0-9), dots (.), hyphens (-), and underscores (_).
4. Type a virtual IP address for the cluster in the Cluster External IP field.
This parameter is required for Hyper-V clusters and is optional for vSphere and AHV clusters.
You can connect to the external cluster IP address with both the web console and nCLI. In the event
that a Controller VM is restarted or fails, the external cluster IP address is relocated to another
Controller VM in the cluster.
5. (Optional) If you want to enable redundancy factor 3, set Cluster Max Redundancy Factor to 3.
Redundancy factor 3 has the following requirements:
• A cluster must have at least five nodes for redundancy factor 3 to be enabled.
• For guest VMs to tolerate the simultaneous failure of two nodes or drives in different blocks, the data
must be stored on storage containers with replication factor 3.
• Controller VMs must be configured with 24 GB of memory.
6. Type the appropriate DNS and NTP addresses in the respective fields.
Note: You must enter NTP servers that the Controller VMs can reach in the CVM NTP Servers
field. If reachable NTP servers are not entered or if the time on the Controller VMs is ahead of
the current time, cluster services may fail to start.
For Hyper-V clusters, the CVM NTP Servers parameter must be set to the IP addresses of one or more
Active Directory domain controllers.
The Hypervisor NTP Servers parameter is not used in Hyper-V clusters.
7. Type the appropriate subnet masks in the Subnet Mask row.
8. Type the appropriate default gateway IP addresses in the Default Gateway row.
9. Select the check box next to each node that you want to add to the cluster.
All unconfigured nodes on the current network are presented on this web page. If you are going to
configure multiple clusters, be sure that you only select the nodes that should be part of the current
cluster.
10. Provide an IP address for all components in the cluster.

Ensure that all components satisfy the cluster subnet requirements. See Cluster IP Address
Configuration on page 11. The use of a DHCP server is not supported for Controller VMs, so make
sure to not use DHCP for Controller VMs.
Note: The unconfigured nodes are not listed according to their position in the block. Ensure
that you assign the intended IP address to each node.
11. Click Create.

Wait until the Log Messages section of the page reports that the cluster has been successfully
configured.
Output similar to the following indicates successful cluster configuration.
Configuring IP addresses on node 13SM71450003/A...
Configuring the Hypervisor DNS settings on node 13SM71450003/A...
Configuring the Hypervisor NTP settings on node 13SM71450003/A...
Configuring Zeus on node 13SM71450003/A...
Initializing cluster...
Cluster successfully initialized!
Initializing the CVM DNS and NTP servers...
Successfully updated the CVM NTP and DNS server list
The cluster is started automatically after creation.
13. Verify that all services are up on all Controller VMs.

nutanix@cvm$ cluster status
If the cluster is running properly, output similar to the following is displayed for each node in the cluster:
CVM: 10.1.64.60 Up
Zeus UP [5362, 5391, 5392, 10848, 10977, 10992]
Scavenger UP [6174, 6215, 6216, 6217]
Medusa UP [8029, 8073, 8074, 8176, 8221]
Pithos UP [8328, 8399, 8400, 8418]
Hera UP [8347, 8408, 8409, 8410]
Stargate UP [8742, 8771, 8772, 9037, 9045]
InsightsDB UP [8774, 8805, 8806, 8939]
8890]
Ergon UP [8814, 8862, 8863, 8864]
Cerebro UP [8850, 8914, 8915, 9288]
Chronos UP [8870, 8975, 8976, 9031]
Curator UP [8885, 8931, 8932, 9243]
Prism UP [3545, 3572, 3573, 3627, 4004, 4076]
CIM UP [8990, 9042, 9043, 9084]

AlertManager UP [9017, 9081, 9082, 9324]
Arithmos UP [9055, 9217, 9218, 9353]
Catalog UP [9110, 9178, 9179, 9180]
Acropolis UP [9201, 9321, 9322, 9323]
Atlas UP [9221, 9316, 9317, 9318]
Uhura UP [9390, 9447, 9448, 9449]
Snmp UP [9418, 9513, 9514, 9516]
Tunnel UP [9480, 9543, 9544]
10301]
Janus UP [9532, 9624, 9625]
MinervaCVM UP [10174, 10200, 10201, 10202, 10371]
APLOSEngine UP [10231, 10261, 10262, 10263]
APLOS UP [10343, 10368, 10369, 10370, 10502,
10503]
Lazan UP [10377, 10402, 10403, 10404]
Orion UP [10409, 10449, 10450, 10474]
Delphi UP [10418, 10466, 10467, 10468]
Verifying IPv6 Link-Local Connectivity

The automated IP address and cluster configuration utilities depend on IPv6 link-local addresses, which
are enabled on most networks. Use this procedure to verify that IPv6 link-local is enabled.
1. Connect two Windows, Linux, or Apple laptops to the switch to be used.
2. Disable any firewalls on the laptops.
3. Verify that each laptop has an IPv6 link-local address.

→ Windows (Control Panel)
Start > Control Panel > View network status and tasks > Change adapter settings > Local
Area Connection > Details

→ Windows (command-line interface)
> ipconfig
Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . : corp.example.com

Link-local IPv6 Address . . . . . : fe80::ed67:9a32:7fc4:3be1%12
IPv4 Address. . . . . . . . . . . : 172.16.21.11
Subnet Mask . . . . . . . . . . . : 255.240.0.0
Default Gateway . . . . . . . . . : 172.16.0.1
→ Linux
$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:0c:29:dd:e3:0b

inet6 addr: fe80::20c:29ff:fedd:e30b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2895385616 errors:0 dropped:0 overruns:0 frame:0
TX packets:3063794864 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2569454555254 (2.5 TB) TX bytes:2795005996728 (2.7 TB)
→ Mac OS
$ ifconfig en0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500

ether 70:56:81:ae:a7:47
inet6 fe80::7256:81ff:feae:a747 en0 prefixlen 64 scopeid 0x4
inet 172.16.21.208 netmask 0xfff00000 broadcast 172.31.255.255
media: autoselect
status: active
Note the IPv6 link-local addresses, which always begin with fe80 . Omit the / character and anything
following.
4. From one of the laptops, ping the other laptop.

→ Windows
> ping -6 ipv6_linklocal_addr%interface
→ Linux/Mac OS
$ ping6 ipv6_linklocal_addr%interface
• Replace ipv6_linklocal_addr with the IPv6 link-local address of the other laptop.
• Replace interface with the interface identifier on the other laptop (for example, 12 for Windows, eth0
for Linux, or en0 for Mac OS).
If the ping packets are answered by the remote host, IPv6 link-local is enabled on the subnet. If the
ping packets are not answered, ensure that firewalls are disabled on both laptops and try again before
concluding that IPv6 link-local is not enabled.
5. Reenable the firewalls on the laptops and disconnect them from the network.
Results:
• If IPv6 link-local is enabled on the subnet, you can use automated IP address and cluster configuration
utility.

• If IPv6 link-local is not enabled on the subnet, you have to manually set IP addresses and create the
cluster.
Note: IPv6 connectivity issue might occur if mismatch occurs because of VLAN tagging. This
issue might occur because ESXi that is shipped from the factory does not have VLAN tagging,
hence it might have VLAN tag as 0. The workstation (laptop) that you have connected might be
connected to access port, so it might use different VLAN tag. Hence, ensure that ESXi port must
be in the trunking mode.
Failing from one Site to Another
Disaster failover
Connect to the backup site and activate it.

ncli> pd activate name="pd_name"
This operation does the following:

1. Restores all VM files from last fully-replicated snapshot.
2. Registers VMs on recovery site.
a. All the VMs are registered on a single host in the cluster.
b. The VMs are not powered on automatically. You need to manually start the VMs.
c. It is recommended to enable DRS in the cluster so that the hypervisor migrates the VMs them
once they are powered on.
Caution: The VM registration might fail if the storage container is not mounted on the
selected host.
3. Marks the failover site protection domain as active.
Planned failover
Connect to the primary site and specify the failover site to migrate to.
ncli> pd migrate name="pd_name" remote-site="remote_site_name2"
This operation does the following:

1. Creates and replicates a snapshot of the protection domain.
2. Shuts down VMs on the local site.
3. Creates and replicates another snapshot of the protection domain.
4. Unregisters all VMs and removes their associated files.
5. Marks the local site protection domain as inactive.
6. Restores all VM files from the last snapshot and registers them on the remote site.
7. Marks the remote site protection domain as active.

Fingerprinting Existing vDisks
The vDisk manipulator utility fingerprints vDisks that existed in the cluster before deduplication was
enabled.
Before you begin: The storage container must have fingerprint-on-write enabled.
Run the vDisk manipulator utility from any Controller VM in the cluster.
→ To fingerprint a particular vDisk:
nutanix@cvm$ vdisk_manipulator --operation="add_fingerprints" \
--stats_only="false" --nfs_container_name="ctr_name" \
--nfs_relative_file_path="vdisk_path"
• Replace ctr_name with the name of the storage container where the vDisk to fingerprint resides.
• Replace vdisk_path with the path of the vDisk to fingerprint relative to the storage container path
(for example, Win7-desktop11/Win7-desktop11-flat.vmdk). You cannot specify multiple vDisks in
this parameter.
→ To fingerprint all vDisks in the cluster:

nutanix@cvm$ ncli vdisk list | grep "Name.*NFS" | awk -F: \
'{print $4 ":" $5}' > fingerprint.txt
nutanix@cvm$ for i in `cat fingerprint.txt`; do vdisk_manipulator --vdisk_name=$i \
--operation="add_fingerprints" --stats_only=false; done
Note: You can run vdisk_manipulator in a loop to fingerprint multiple vDisks, but run only
one instance of vdisk_manipulator on each Controller VM at a time. Executing multiple
instances on a Controller VM concurrently would generate significant load on the cluster.

2
Changing Passwords
Changing User Passwords

You can change user passwords, including for the default admin user, in the web console or nCLI.
Changing the password through either interface changes it for both.
To change a user password, do one of the following:
• (Web console) Log on to the web console as the user whose password is to be changed and select
Change Password from the user icon pull-down list of the main menu.
For more information about changing properties of the current users, see the Web Console Guide.
• (nCLI) Specify the username and passwords.

$ ncli -u 'username' -p 'old_pw' user change-password current-password="curr_pw" \
new-password="new_pw"
• Replace username with the name of the user whose password is to be changed.
• Replace curr_pw with the current password.
• Replace new_pw with the new password.
Note: If you change the password of the admin user from the default, you must specify the
password every time you start an nCLI session from a remote system. A password is not
required if you are starting an nCLI session from a Controller VM where you are already logged
on.
Changing the SMI-S Provider Password (Hyper-V)

If you change the password of the Prism admin user, you have to update the Prism run-as account in
SCVMM.
1. Log on to the system where the SCVMM console is installed and start the console.
2. Go to Settings > Security > Run As Account.
3. Right-click the account named cluster_name-Prism and select Properties.
Changing Passwords | Acropolis Advanced Administration Guide | AOS | 19

Figure: Prism Run As Account in SCVMM
4. Update the username and password to include the new credentials and ensure that Validate domain
credentials is not checked.
5. Go to Fabric > Storage > Providers.
6. Right-click the provider with Name cluster_name and select Refresh.
Figure: Storage Provider
Changing the Controller VM Password

Perform these steps on every Controller VM in the cluster.
Note: Nutanix recommends that you set the same password for the nutanix user on all the
Controller VMs.
1. Log on to the Controller VM with SSH.
2. Change the nutanix user password.

nutanix@cvm$ passwd
3. Respond to the prompts, providing the current and new nutanix user password.
Changing password for nutanix.
Old Password:
New password:
Retype new password:
Password changed.

Note: The password must meet the following complexity requirements:

3
Cluster IP Address Configuration
AOS includes a web-based configuration tool that automates the modification of Controller VM IP
addresses and configures the cluster to use these new IP addresses. Other cluster components must be
modified manually.
Network Configuration (Virtual Interfaces, Virtual Switches, and IP Addresses)

By default, Nutanix hosts have the following virtual switches:
Internal Virtual Switch
The internal virtual switch manages network communications between the Controller VM and the
hypervisor host. This switch is associated with a private network on the default VLAN and uses the
192.168.5.0/24 address space. The traffic on this subnet is typically restricted to the internal virtual switch,
but might be sent over the physical wire, through a host route, to implement storage high availability on
ESXi and Hyper-V clusters. This traffic is on the same VLAN as the Nutanix storage backplane.
Note: For guest VMs and other devices on the network, do not use a subnet that overlaps with
the 192.168.5.0/24 subnet on the default VLAN. If you want to use an overlapping subnet for such
devices, make sure that you use a different VLAN.
The following tables list the interfaces and IP addresses on the internal virtual switch on different
hypervisors:
Interfaces and IP Addresses on the Internal Virtual Switch virbr0 on an AHV Host
Device Interface Name IP Address
AHV Host virbr0 192.168.5.1
Cluster IP Address Configuration | Acropolis Advanced Administration Guide | AOS | 22

Controller VM eth1 192.168.5.2
eth1:1 192.168.5.254
Interfaces and IP Addresses on the Internal Virtual Switch vSwitchNutanix on an ESXi Host
ESXi Host vmk1 192.168.5.1
eth1:1 192.168.5.254
Interfaces and IP Addresses on the Internal Virtual Switch InternalSwitch on a Hyper-V Host
Hyper-V Host vEthernet (InternalSwitch) 192.168.5.1
eth1:1 192.168.5.254
External Virtual Switch
The external virtual switch manages communication between the virtual machines, between the virtual
machines and the host, and between the hosts in the cluster. The traffic on this virtual switch also includes
Controller VM–driven replication traffic for the purposes of maintaining the specified replication factor, as
well as any ADSF traffic that cannot be processed locally. The external switch is assigned a NIC team or
bond as the means to provide connectivity outside of the host.
Note: Make sure that the hypervisor and Controller VM interfaces on the external virtual switch
are not assigned IP addresses from the 192.168.5.0/24 subnet.
The following tables list the interfaces and IP addresses on the external virtual switch on different
hypervisors:
Interfaces and IP Addresses on the External Virtual Switch br0 on an AHV Host
AHV Host br0 User-defined
Controller VM eth0 User-defined
Guest VM br0 or user-defined Open vSwitch User-defined

bridge)

Interfaces and IP Addresses on the External Virtual Switch vSwitch0 on an ESXi Host
ESXi Host vmk0 User-defined
Guest VM vSwitch0 or user-defined switch User-defined
Interfaces and IP Addresses on the External Virtual Switch ExternalSwitch on a Hyper-V Host
Hyper-V Host vEthernet (ExternalSwitch) User-defined

Guest VM vEthernet (ExternalSwitch) or user- User-defined

defined switch
Changing Controller VM IP Addresses

Warning: If you are reassigning a Controller VM IP address to another Controller VM, you must
perform this complete procedure twice: once to assign intermediate IP addresses and again to
assign the desired IP addresses.
For example, if Controller VM A has IP address 172.16.0.11 and Controller VM B has IP address
172.16.0.10 and you want to swap them, you would need to reconfigure them with different IP
addresses (such as 172.16.0.100 and 172.16.0.101) before changing them to the IP addresses in
use initially.
Perform one of the following to change the Controller VM IP address:

→ Perform the following if IPv6 is enabled on the subnet:
1. Place the cluster in reconfiguration mode by following Preparing to Change Controller VM IP
2. Change the Controller VM IP addresses by following Changing Controller VM IP Addresses on
page 27.
3. Complete cluster reconfiguration by following Completing Controller VM IP Address Change on
page 30.
→ If IPv6 is not enabled on the subnet, follow Changing a Controller VM IP Address (manual) on
page 28.
Preparing to Set IP Addresses

Before you configure the cluster, check that these requirements are met.
• Confirm that IPv6 link-local is enabled on the subnet.

IPv6 link-local is required only for discovery of nodes. It is not required after cluster creation except to
add nodes to an existing cluster.

• Confirm that the system you are using to configure the cluster meets the following requirements:
• IPv6 link-local enabled.
• Windows 7, Vista, or MacOS.
• (Windows only) Bonjour installed (included with iTunes or downloadable from http://
support.apple.com/kb/DL999).
• (Hyper-V only) Confirm that the hosts have only one type of NIC (10 GbE or 1 GbE) connected during
cluster creation. If the nodes have multiple types of network interfaces connected, disconnect them until
after you join the hosts to the domain.
• Determine the IPv6 service name of any Controller VM in the cluster.

The service name depends on a unique identifier for the system.
Nutanix Serial Number
IPv6 service names are uniquely

generated at the factory and have the
following form (note the final period):
NTNX-block_serial_number-node_location-
CVM.local.
On the right side of the block toward the front

is a label that has the block_serial_number (for
example, 12AM3K520060).
The node_location is A for one-node blocks, A-
B for two-node blocks, and A-D for four-node
blocks.
If you do not have access to get the block serial
number, see the Nutanix support knowledge base
for alternative methods.

Dell Service Tag
IPv6 service names are uniquely

generated at the factory and have the
following form (note the final period):
NTNX-system_service_tag-node_location-
CVM.local.
On the front left side of the system is a slide-out

label that contains the system_service_tag (for
example, B57PW12).
The node_location is A for one-node blocks.
Preparing to Change Controller VM IP Addresses

Before you begin:
• Ensure that the cluster NTP and DNS servers are reachable from the new Controller VM addresses.
If different NTP and DNS servers are to be used, remove the existing NTP and DNS servers from the
cluster configuration and add the new ones. If the new addresses are not known, remove the existing
NTP and DNS servers before cluster reconfiguration and add the new ones afterwards.
Web Console
> Name Servers
> NTP Servers
nCLI ncli>cluster remove-from-name-servers servers="name_servers"

ncli>cluster add-to-name-servers servers="name_servers"
ncli>cluster remove-from-ntp-servers servers="ntp_servers"
ncli>cluster add-to-ntp-servers servers="ntp_servers"
• Log on to a Controller VM in the cluster and check that all hosts are part of the metadata store.
nutanix@cvm$ ncli host ls | grep "Metadata store status"
For every host in the cluster, Metadata store enabled on the node should be shown.
Warning: If Node marked to be removed from metadata store is displayed, do not proceed
with the IP address reconfiguration, and contact Nutanix support to resolve the issue.


Zeus UP [3167, 3180, 3181, 3182, 3191, 3201]
Scavenger UP [3334, 3351, 3352, 3353]
Hyperint DOWN []
Medusa DOWN []
Pithos DOWN []
Stargate DOWN []
Cerebro DOWN []
Chronos DOWN []
Curator DOWN []
Prism DOWN []
3. Put the cluster in reconfiguration mode.

nutanix@cvm$ cluster reconfig
Type y to confirm the reconfiguration.

Wait until the cluster successfully enters reconfiguration mode, as shown in the following example.
INFO cluster:185 Restarted Genesis on 172.16.8.189.

INFO cluster:864 Success!
Changing Controller VM IP Addresses

Before you begin: Check that the cluster is ready to be configured by following Preparing to Set IP
Warning: If you are reassigning a Controller VM IP address to another Controller VM, you must
perform this complete procedure twice: once to assign intermediate IP addresses and again to
assign the desired IP addresses.
For example, if Controller VM A has IP address 172.16.0.11 and Controller VM B has IP address
172.16.0.10 and you want to swap them, you would need to reconfigure them with different IP
addresses (such as 172.16.0.100 and 172.16.0.101) before changing them to the IP addresses in
use initially.
The cluster must be stopped and in reconfiguration mode before changing the Controller VM IP addresses.
1. Open a web browser.

Nutanix recommends using Internet Explorer 9 for Windows and Safari for Mac OS.
Note: Internet Explorer requires protected mode to be disabled. Go to Tools > Internet
Options > Security, clear the Enable Protected Mode check box, and restart the browser.
2. In the browser, go to http://[cvm_ipv6_addr]:2100/ip_reconfig.html.

Replace [cvm_ipv6_addr] with the IPv6 address of any Controller VM in the cluster.
You can obtain the IPv6 address of the Controller VM by using the ifconfig command.
Example
nutanix@cvm$ ifconfig
eth0 Link encap:Ethernet HWaddr 52:54:00:A8:8A:AE
inet6 addr: fe80::5054:ff:fea8:8aae/64 Scope:Link
...etc...
The value of the inet6 addr field up to the / character is the IPv6 address of the Controller VM.
3. Update one or more cells on the IP Reconfiguration page.

Ensure that all components satisfy the cluster subnet requirements. See Cluster IP Address
Configuration on page 22.
4. Click Reconfigure.
5. Wait until the Log Messages section of the page reports that the cluster has been successfully
reconfigured, as shown in the following example.
Configuring IP addresses on node S10264822116570/A...
Success!
Configuring IP addresses on node S10264822116570/C...
Success!
Configuring IP addresses on node S10264822116570/B...
Success!
Configuring IP addresses on node S10264822116570/D...
Success!
Configuring Zeus on node S10264822116570/A...
Configuring Zeus on node S10264822116570/C...
Configuring Zeus on node S10264822116570/B...
Configuring Zeus on node S10264822116570/D...
Reconfiguration successful!
The IP address reconfiguration disconnects any SSH sessions to cluster components. The cluster is
taken out of reconfiguration mode.
Changing a Controller VM IP Address (manual)

If IPv6 is not enabled on the subnet, manually change the Controller VM IP address for each Controller VM
in the cluster.
If IPv6 is enabled on the subnet, you can change the Controller VM IP address by following Changing
Controller VM IP Addresses on page 27.
To manually change the Controller VM IP address, do the following:
1. Stop the cluster.
2. Run the external IP address reconfiguration script (external_ip_reconfig).
3. Restart all the Controller VMs in the cluster after the script has been successfully executed.
4. Start the cluster.
The external IP address reconfiguration script (external_ip_reconfig) performs the following tasks:
1. Puts the cluster in reconfiguration mode.
2. Restarts Genesis.
3. Prompts you to type the new netmask, gateway, and external IP addresses, and updates them.

4. Updates the IP addresses of the Zookeeper hosts.
Perform the following procedure to manually change the Controller VM IP address.

Warning: If you are changing the Controller VM IP addresses to another subnet, network, IP
address range, or VLAN, you should also change the hypervisor management IP addresses to the
same subnet, network, IP address range, or VLAN.
1. Log on to the hypervisor with SSH (vSphere or AHV) or remote desktop connection (Hyper-V), or the
IPMI remote console.
2. Log on to the Controller VM.

→ vSphere or AHV root@host# ssh nutanix@192.168.5.254
→ Hyper-V > ssh nutanix@192.168.5.254
Accept the host authenticity warning if prompted, and enter the Controller VM nutanix password.

Warning: This step affects the operation of a Nutanix cluster. Schedule a down time before
performing this step.
Note: If necessary, change the hypervisor management IP address or IPMI IP address before
you execute the external_ip_reconfig script.
4. Run the external IP address reconfiguration script (external_ip_reconfig) from any one Controller VM in
the cluster.
nutanix@cvm$ external_ip_reconfig
5. Follow the prompts to type the new netmask, gateway, and external IP addresses.
A message similar to the following is displayed after the reconfiguration is successfully completed:
External IP reconfig finished successfully. Restart all the CVMs and start the cluster.
6. Restart each Controller VM in the cluster.

nutanix@cvm$ sudo reboot
Enter the nutanix password if prompted.

CVM: 10.1.64.60 Up
Zeus UP [3704, 3727, 3728, 3729, 3807, 3821]
Scavenger UP [4937, 4960, 4961, 4990]
Hyperint UP [5059, 5082, 5083, 5086, 5099, 5108]
Medusa UP [5534, 5559, 5560, 5563, 5752]
Pithos UP [5877, 5899, 5900, 5962]
Stargate UP [5902, 5927, 5928, 6103, 6108]
Cerebro UP [5930, 5952, 5953, 6106]
Chronos UP [5960, 6004, 6006, 6075]
Curator UP [5987, 6017, 6018, 6261]
Prism UP [6020, 6042, 6043, 6111, 6818]
AlertManager UP [6070, 6099, 6100, 6296]

Arithmos UP [6107, 6175, 6176, 6344]
Tunnel UP [6263, 6312, 6313]
ClusterHealth UP [6317, 6342, 6343, 6446, 6468, 6469, 6604, 6605,
6606, 6607]
Janus UP [6365, 6444, 6445, 6584]
NutanixGuestTools UP [6377, 6403, 6404]
What to do next: Run the following NCC checks to verify the health of the Zeus configuration:
• nutanix@cvm$ ncc health_checks system_checks zkalias_check_plugin
• nutanix@cvm$ ncc health_checks system_checks zkinfo_check_plugin
Completing Controller VM IP Address Change
1. If you changed the IP addresses by modifying the Controller VM configuration files directly rather than
using the Nutanix utility, take the cluster out of reconfiguration mode.
Perform these steps for every Controller VM in the cluster.
a. Log on to the Controller VM with SSH.
b. Take the Controller VM out of reconfiguration mode.

nutanix@cvm$ rm ~/.node_reconfigure
c. Restart genesis.
nutanix@cvm$ genesis restart
If the restart is successful, output similar to the following is displayed:
Stopping Genesis pids [1933, 30217, 30218, 30219, 30241]

Genesis started on pids [30378, 30379, 30380, 30381, 30403]

CVM: 10.1.64.60 Up
Zeus UP [5362, 5391, 5392, 10848, 10977, 10992]
Scavenger UP [6174, 6215, 6216, 6217]
Medusa UP [8029, 8073, 8074, 8176, 8221]
Pithos UP [8328, 8399, 8400, 8418]
Hera UP [8347, 8408, 8409, 8410]
Stargate UP [8742, 8771, 8772, 9037, 9045]
InsightsDB UP [8774, 8805, 8806, 8939]
8890]
Ergon UP [8814, 8862, 8863, 8864]
Cerebro UP [8850, 8914, 8915, 9288]
Chronos UP [8870, 8975, 8976, 9031]
Curator UP [8885, 8931, 8932, 9243]
Prism UP [3545, 3572, 3573, 3627, 4004, 4076]

CIM UP [8990, 9042, 9043, 9084]
AlertManager UP [9017, 9081, 9082, 9324]
Arithmos UP [9055, 9217, 9218, 9353]
Catalog UP [9110, 9178, 9179, 9180]
Acropolis UP [9201, 9321, 9322, 9323]
Atlas UP [9221, 9316, 9317, 9318]
Uhura UP [9390, 9447, 9448, 9449]
Snmp UP [9418, 9513, 9514, 9516]
Tunnel UP [9480, 9543, 9544]
10301]
Janus UP [9532, 9624, 9625]
MinervaCVM UP [10174, 10200, 10201, 10202, 10371]
APLOSEngine UP [10231, 10261, 10262, 10263]
APLOS UP [10343, 10368, 10369, 10370, 10502,
10503]
Lazan UP [10377, 10402, 10403, 10404]
Orion UP [10409, 10449, 10450, 10474]
Delphi UP [10418, 10466, 10467, 10468]

4
Creating a Windows Guest VM Failover Cluster
A guest VM failover cluster allows high availability for cluster-aware workloads within the VM and enables
an application to seamlessly failover to another VM on the same host or on a different host. This feature is
supported on Hyper-V and ESXi hypervisor environments. Windows guest failover clustering is supported
for Windows Server 2008 and Windows Server 2012 R2.
Note: Windows Guest VM Failover Clustering is for guest VMs only.
1. From the Server Manager, add and enable the Multipath I/O feature in Tools > MPIO.
a. Add support for iSCSI devices by checking the box in the Discovered Multipaths tab.
b. Enable multipath for the targets by checking the box in the Microsoft iSCSI Initiator and selecting
the IP addresses for the Target Portal IP.
2. Set the default load balancing policy for all LUNS to Fail Over Only by running the following PowerShell
cmdlet on each Windows Server 2012 VM that is being used for Windows Failover Clustering:
> Set-MSDSMGlobalDefaultLoadBalancePolicy -Policy FOO
3. Log on to any Controller VM in your cluster through an SSH session and access the Acropolis
command line.
nutanix@cvm$ acli
<acropolis>
4. Create a volume group, then add a disk to the newly-created volume group. Verify that the new volume
group and disk were created successfully
a. Create a volume group, where vg_name is the name of the volume group.
<acropolis> vg.create vg_name shared=true
b. Add a disk to the newly-created volume group, where container_name is the name of the storage
container and disk_size is the disk size. (Optional) Use index_number to index the disk within the
cluster, (for example, create_size=1000G creates a disk with a capacity of 1000G). Otherwise, the
system automatically assigns index numbers.
Note: If you allocate disk size, the lower case g (in size=20g) indicates gigabyte and capital
G (in size=1000G) indicates gibibytes.
<acropolis> vg.disk_create vg_name container=container_name \
create_size=disk_size index=index_number
Note: For best results, Nutanix recommends that you configure 1 vDisk per volume group.
c. Verify the volume group.
Creating a Windows Guest VM Failover Cluster | Acropolis Advanced Administration Guide | AOS | 32
Note: If you have more than one disk inside the target, the vg.get command displays all
disks within the target.
<acropolis> vg.get vg_name
Next, log on to your Windows Server VM to perform the following steps.
5. From Windows Server, get the iSCSI initiator name. Then, from the Acropolis CLI, attach the external
initiator to the volume group and verify the connection.
a. From the Windows Server Manager VM, in the Disk Management window, click Tools iSCSI
Initiator, then click the Configuration tab and copy the iSCSI initiator name from the text box.
b. From the Acropolis CLI, attach the external initiators, where initiator_name is the copied initiator
name.
<acropolis> vg.attach_external vg_name initiator_name
c. Repeat this step for any remaining external initiators. Verify that the external initiators are connected.
<acropolis> vg.get vg_name
Note: You can also create a volume group and enable multiple initiators to access the
volume group by using Prism web console. For more information, see Creating a Volume
Group section of Prism Web Console Guide.
6. Allow the Controller VM to be discoverable by the external initiators.
a. From the Windows Server Manager VM, in the iSCSI Initiator Properties window, click Discovery
and add the Controller VM IP address, then click OK.
b. Repeat this step for the remaining Controller VM IP addresses by doing the same to the next target.
c. To verify that the IP addresses are connected, go to the Targets tab and click Refresh.
d. Click OK to exit.
7. From the Server Manager, place the target disks online and create a New Simple Volume.
a. In the Disk Management window, right-click each disk and choose the Online option.
Repeat for any remaining disks.
b. Click New Simple Volume Wizard and verify the information in the following windows until you
reach the Format Platform window.
c. Enter a name for the volume in Volume label and complete the remaining wizard steps.
Note: (Optional) If a formatting window appears, you can format the Simple Volume.
8. From the Server Manager, create a Windows Guest VM Failover Cluster and add the disks.
a. In the Server Manager, click Tools > Failover Cluster Manager and click Create Cluster.
b. Click Browse and in the Select Computer window enter the names of the VMs you want to add, then
click OK. Click Yes to validate configuration tests.
c. Verify the information, then enter a name and IP address for the Windows Failover Cluster. Click OK.
d. In the Failover Cluster Manager, click the volume group and click Storage > Disks. Choose Add
disk.
e. Select the disks you want to add to the cluster and click OK.
The new cluster and disks have been created and configured.
5
Acropolis Dynamic Scheduling in AHV
In an Acropolis managed clusters, the Acropolis Dynamic Scheduling (ADS) feature proactively monitors
your cluster for any compute and/or storage I/O contention or hotspots over a period of time. If a problem is
detected, a migration plan is created and executed thereby eliminating hotspots in the cluster by migrating
VMs from one host to another. This feature only detects the contentions that are currently in progress. You
can monitor these tasks from the Task dashboard of the Prism Web console. You can click the VM link to
view the migration information, which includes the migration path (to the destination AHV host).
Some other advantages of the ADS feature are as follows.
• This feature also improves the initial placement of the VMs depending on the VM configuration.
• The Acropolis block services feature uses the ADS feature for balancing sessions of the externally
visible iSCSI targets.
Note: If you have configured any host or VM-host affinity or VM-VM anti-affinity policies, these
policies are honored.
By default, the feature is enabled and it is recommended you keep this feature enabled. However, you
can disable the feature by using aCLI. For disabling ADS feature, see Disabling Acropolis Dynamic
Scheduling on page 35. Even if you disable the feature, the checks for the contentions or hotspots run
in the background and if any anomalies are detected, an alert is raised in the Alerts dashboard after third
notification. However, no action is taken by the ADS feature to resolve these contentions. You need to
manually take the remedial actions or you can enable the feature. For more information about enabling
ADS feature, see Enabling Acropolis Dynamic Scheduling on page 36
Acropolis Dynamic Scheduling Requirements and Limitations
• Ensure that all the hosts are running AOS 5.0 or later releases.
• The iSCSI targets are displayed as an empty entity. However, if any action is taken on an iSCSI target,
the relevant message is displayed in the Tasks dashboard.
• If a problem is detected and the ADS cannot solve the issue (for example, because of limited CPU or
storage resources), the migration plan might fail. In these cases, an alert is generated. You need to
monitor these alerts from the Alerts dashboard of the Prism Web console and take necessary remedial
actions.
• If the host, firmware, or AOS upgrade is in progress and if any resource contention occurs, during the
period of upgrade no resource contention rebalancing is performed.
Disabling Acropolis Dynamic Scheduling

Perform the following procedure to disable the ADS feature. Note that it is not recommended to disable the
ADS feature.
1. Log into the Controller VM in your cluster through an SSH session and access the Acropolis command
line.
Acropolis Dynamic Scheduling in AHV | Acropolis Advanced Administration Guide | AOS | 35

2. Disable the ADS feature.
acli> ads.update enable=false
Even after you disable the feature, the checks for the contentions or hotspots run in the background and
if any anomalies are detected, an alert is raised in the Alerts dashboard. However, no action is taken by
the ADS to solve the contentions. You need to manually take the remedial actions or you can enable the
feature.
Enabling Acropolis Dynamic Scheduling

If you have disabled the ADS feature and want to enable the feature, perform the following procedure.
1. Log into the Controller VM in your cluster through an SSH session and access the Acropolis command
line.
2. Enable the ADS feature.

acli> ads.update enable=true
Acropolis Dynamic Scheduling in AHV | Acropolis Advanced Administration Guide | AOS | 36

6
Logs
Sending Logs to a Remote Syslog Server

The Nutanix command-line interface (nCLI) command rsyslog-config enables you to send logs from your
Nutanix cluster to a remote syslog server.
• The Command Reference contains more information about rsyslog-config command syntax.
• The Acropolis Advanced Administration Guide troubleshooting topics have more detailed information
about common and AOS logs (such as Stargate and Cassandra logs).
Recommendations and considerations
• As the logs are forwarded from a Controller VM, the logs display the IP address of the Controller VM.
• You can only configure one rsyslog server; you cannot specify multiple servers.
• The remote syslog server is enabled by default.
• Supported transport protocols are TCP and UDP.
• You can also forward logs to a remote syslog server by using Reliable Event Logging Protocol (RELP).
To use RELP logging, ensure that you have installed rsyslog-relp on the remote syslog server.
Note: You can use RELP logging only if the transport protocol is TCP.
• rsyslog-config supports and can report messages from the following Nutanix modules:
AOS Module Names for rsyslog-config
Logs are located in /home/nutanix/data/logs (except SYSLOG_MODULE).
Module name With monitor logs disabled, these logs are With monitor logs enabled, these logs are
forwarded also forwarded
SYSLOG_MODULE Controller VM: /home/log/messages Controller VM: /home/log/messages

AHV host: /var/log/messages AHV host: /var/log/messages
• Forwards all the Controller VM logs that are stored in /home/log/messages to a

remote syslog server.
• Forwards all the AHV host logs that are stored in /var/log/messages to a
remote syslog server.
CASSANDRA cassandra/system.log, cassandra_monitor.loglevel

dynamic_ring_changer.out,
dynamic_ring_changer.loglevel
CEREBRO cerebro.loglevel cerebro.out
Logs | Acropolis Advanced Administration Guide | AOS | 37

Module name With monitor logs disabled, these logs are With monitor logs enabled, these logs are
forwarded also forwarded
CURATOR curator.loglevel curator.out
GENESIS genesis.out genesis.out
PRISM prism_gateway.log prism_monitor.loglevel, prism.out
STARGATE stargate.loglevel stargate.out
ZOOKEEPER zookeeper.out zookeeper_monitor.loglevel
ACROPOLIS acropolis.out acropolis.out
Forwards all the logs related to the ACROPOLIS service to a remote syslog
server.
AOS Log Level Mapping to syslog Log Levels
AOS log levels Contain information from these syslog log levels
INFO DEBUG, INFO
WARNING NOTICE, WARNING
ERROR ERROR
FATAL CRITICAL, ALERT, EMERGENCY

• SYSLOG_MODULE logs Controller VM messages in /home/log/messages and the AHV host messages
in /var/log/messages. Adding the SYSLOG_MODULE module to the rsyslog configuration configures
rsyslog on compatible AHV hosts. A compatible host must be running an AHV release later than
AHV-20160217.2.
• Enable module logs at the ERROR level, unless you require more information. If you enable more
levels, the rsyslogd daemon sends more messages. For example, if you set the SYSLOG_MODULE
level to INFO, your remote syslog server might receive a large quantity of operating system messages.
• CPU usage might reach 10 percent when the rsyslogd daemon is initially enabled and starts
processing existing logs. This is an expected condition on first use of an rsyslog implementation.
Configuring the Remote Syslog Server Settings

Before you begin: Install the Nutanix command-line interface (nCLI) and connect to a Controller VM in
your cluster. See the Command Reference for details.
Note: As the logs are forwarded from a Controller VM, the logs display the IP address of the
Controller VM.
1. As the remote syslog server is enabled by default, disable it while you configure settings.
ncli> rsyslog-config set-status enable=false
2. Create a syslog server (which adds it to the cluster) and confirm it has been created.
ncli> rsyslog-config add-server name=remote_server_name relp-enabled={true | false} ip-
address=remote_ip_address port=port_num network-protocol={tcp | udp}
ncli> rsyslog-config ls-servers
Name : remote_server_name
IP Address : remote_ip_address

Port : port_num
Protocol : TCP or UDP
Relp Enabled : true or false
remote_server_name A descriptive name for the remote server receiving the specified
messages
remote_ip_address The remote server's IP address
port_num Destination port number on the remote server.
tcp | udp Choose tcp or udp as the transport protocol
true | false Choose true to enable RELP and choose false to disable RELP
3. Choose a module to forward log information from and specify the level of information to collect.
ncli> rsyslog-config add-module server-name=remote_server_name module-name=module
level=loglevel include-monitor-logs={ false | true }
• Replace module with one of the following:

• ACROPOLIS
• CASSANDRA
• CEREBRO
• CURATOR
• GENESIS
• PRISM
• STARGATE
• SYSLOG_MODULE
• ZOOKEEPER
• Replace loglevel with one of the following:
• DEBUG
• INFO
• NOTICE
• WARNING
• ERROR
• CRITICAL
• ALERT
• EMERGENCY
Enable module logs at the ERROR level unless you require more information.
• (Optional) Set include-monitor-logs to specify whether the monitor logs are sent. It is enabled
(true) by default. If disabled (false), only certain logs are sent.
Note: If enabled, the include-monitor-logs option sends all monitor logs, regardless of the
level set by the level= parameter.
4. Configure additional modules if desired with rsyslog-config add-module.
5. Enable the server.

ncli> rsyslog-config set-status enable=true
Logs are now forwarded to the remote syslog server.

Common Log Files
Nutanix nodes store log files in different directories, depending on the type of information they contain.
Nutanix Logs Root

The location for Nutanix logs is /home/nutanix/data/logs.
This location of the logs directory contains all the Nutanix process logs at the INFO, WARNING, ERROR
and FATAL levels. It also contains the directories for the system stats (sysstats), and Cassandra system
logs (cassandra).
The most recent FATAL log only contains the reason for the process to fail. More information can be found
in the other types of logs by analyzing the entries leading up to the failure.
Note: The symbolic link component_name.[INFO|WARNING|ERROR|FATAL] points to the most recent
component log. For example:
stargate.FATAL -> stargate.NTNX-12AM3K490006-2-
CVM.nutanix.log.FATAL.201701-141913.30286
.FATAL Logs
If a component fails, it creates a log file named according to the following convention:
component-name.cvm-name.log.FATAL.date-timestamp
• component-name identifies to the component that created the file, such as Curator or Stargate.
• cvm-name identifies to the Controller VM that created the file.
• date-timestamp identifies the date and time when the first failure within that file occurred.
Each failure creates a new .FATAL log file.
Log entries use the following format:

[IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
The first character indicates whether the log entry is an Info, Warning, Error, or Fatal. The next four
characters indicate the day on which the entry was made. For example, if an entry starts with F0820, it
means that at some time on August 20th, the component had a failure.
Tip: The cluster also creates .INFO and .WARNING log files for each component. Sometimes, the
information you need is stored in one of these files.
Self-Monitoring (sysstats) Logs

Self-monitoring logs are in /home/nutanix/data/logs/sysstats.
The node self-monitors itself by running several Linux tools every few minutes, including ping, iostat, sar,
and df.
This directory contains the output for each of these commands, along with the corresponding timestamp.
/home/nutanix/data/logs/cassandra
This is the directory where the Cassandra metadata database stores its logs. The Nutanix process that
starts the Cassandra database (cassandra_monitor) logs to the /home/nutanix/data/logs directory.
However, the most useful information relating to the Cassandra is found in the system.log* files located in
the /home/nutanix/data/logs/cassandra directory.

This directory contains the output for each of these commands, along with the corresponding timestamp.
Controller VM Log Files

These log files are present on Controller VMs.
Location: /home/nutanix/data/logs
Log Contents Frequency
alert_manager.[out, ERROR, FATAL, INFO, Alert manager process output

WARNING]
cassandra_monitor.[out, ERROR, FATAL, INFO] Cassandra database monitor

process output
catalina.out Catalina/Tomcat for Prism
process output
cerebro.[out, ERROR, FATAL] DR and replication activity
check-cores.log Core file processing every 1 min
check-fio fio-status output every 1 hour
check-hardware.log Power supply, fan speed, and every 1 min

DIMM temperature status
check_intel.log Intel PCIe-SSD status every 1 min
check-ip-connectivity.log Network connectivity status to every 1 min

IPMI, hypervisor, and Controller
VM of all nodes in the cluster
chronos_node_main.[INFO, ERROR, FATAL, Write-ahead log (WAL) status

WARNING]
connection_splicer.[out, ERROR, FATAL, INFO, Internal process connection status

WARNING]
cron_avahi_monitor.log Avahi process status
cron_time_check.log Check time difference across every 1 min

Controller VMs
curator.[out, ERROR, FATAL, INFO, WARNING] Metadata health and ILM activity
disk_usage.log Disk and inode usage of all every 1 min

partitions on the Controller VM
dynamic_ring_changer.[out, ERROR, FATAL] Metadata migration across nodes

activity
genesis.out Nutanix software start process

output
hyperint_monitor.[out, ERROR, FATAL, INFO, Hypervisor integration activity

WARNING]
pithos.[out, ERROR, FATAL, INFO, WARNING] vDisk configuration activity

Log Contents Frequency
prism_gateway.[out, ERROR, FATAL, INFO] Prism leader activity
prism_monitor.[out, ERROR, FATAL, INFO] Prism (Web console, nCLI, REST

API) monitor process output
scavenger.out Log and core file clean-up status
send-email.log E-mail alerts sent from the every 1 min

Controller VM
snmp_manager.out SNMP service logs.
ssh_tunnel.log Connect status to

nsc.nutanix.com for the remote
support tunnel
stargate.[out, ERROR, FATAL, INFO, WARNING] NFS interface activity
stats_aggregator.[out, ERROR, FATAL, INFO] Statistics aggregator process

output
support-info.log Daily automated support (ASUP)

alerts
using-gflags.log gflags status
zeus_config_printer.INFO Contents of cluster configuration

database
zookeeper_monitor.[out, ERROR, INFO] Cluster configuration and cluster

state activity
Location: /home/nutanix/data/logs/cassandra
Log Contents
system.log Cassandra system activity
Location: /home/nutanix/data/logs/sysstats
Log Contents Frequency Command
df.info Mounted filesystems every 5 sec df -h
disk_usage.INFO Disk usage across disks every 1 hour du
interrupts.INFO CPU interrupts every 5 sec
iostat.INFO I/O activity for each physical disk every 5 sec sudo iostat
iotop.INFO Current I/O in realtime every 5 sec sudo iotop
lsof.INFO List of open files and processes every 1 min sudo lsof
that open them

Log Contents Frequency Command
meminfo.INFO Memory usage every 5 sec cat /proc/

meminfo
metadata_disk_usage.INFO Disk usage for metadata drives every 5 sec
mpstat.INFO CPU activities per CPU every 5 sec mpstat
ntpq.INFO NTP information every 1 min ntpq -pn
ping_gateway.INFO Pings to the default gateway every 5 sec ping
ping_hosts.INFO Pings to all other Controller VMs every 1 min ping
sar.INFO Network bandwidth every 5 sec sar -n DEV, -n

EDEV
top.INFO Real-time CPU and memory every 5 sec top

activity
Location: /home/nutanix/data/serviceability/alerts
Log Contents
num.processed Alerts that have been processed
Location: /var/log
Log Contents
dmesg OS start messages
kernel OS kernel messages
messages OS messages after starting
Correlating the FATAL log to the INFO file

When a process fails, the reason for the failure is recorded in the corresponding FATAL log. There are two
ways to correlate this log with the INFO file to get more information:
1. Search for the timestamp of the FATAL event in the corresponding INFO files.
a. Determine the timestamp of the FATAL event.
b. Search for the timestamp in the corresponding INFO files.
c. Open the INFO file with vi and go to the bottom of the file (Shift+G).
d. Analyze the log entries immediately before the FATAL event, especially any errors or warnings.
In the following example, the latest stargate.FATAL determines the exact timestamp:
nutanix@cvm$ cat stargate.FATAL

Log file created at: 2013/09/07 01:22:23
Running on machine: NTNX-12AM3K490006-2-CVM
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0907 01:22:23.124495 10559 zeus.cc:1779] Timed out waiting for Zookeeper session
establishment
In the above example, the timestamp is F0907 01:22:23 , or September 7 at 1:22:23 AM.
Next, grep for this timestamp in the stargate*INFO* files:
nutanix@cvm$ grep "^F0907 01:22:23" stargate*INFO* | cut -f1 -
d:stargate.NTNX-12AM3K490006-2-CVM.nutanix.log.INFO.20130904-220129.7363
This tells us that the relevant file to look at is stargate.NTNX-12AM3K490006-2-

CVM.nutanix.log.INFO.20130904-220129.7363.
2. If a process is repeatedly failing, it might be faster to do a long listing of the INFO files and select
the one immediately preceding the current one. The current one would be the one referenced by the
symbolic link.
For example, in the output below, the last failure would be recorded in the file
stargate.NTNX-12AM3K490006-2-CVM.nutanix.log.INFO.20130904-220129.7363.
ls -ltr stargate*INFO*
-rw-------. 1 nutanix nutanix 104857622 Sep 3 11:22 stargate.NTNX-12AM3K490006-2-

CVM.nutanix.log.INFO.20130902-004519.7363
lrwxrwxrwx. 1 nutanix nutanix 71 Sep 7 01:22 stargate.INFO ->
stargate.NTNX-12AM3K490006-2-CVM.nutanix.log.INFO.20130907-012223.11357
Tip: You can use the procedure above for the other types of files as well (WARNING and
ERROR) in order to narrow the window of information. The INFO file provides all messages,
WARNING provides only warning, error, and fatal-level messages, ERROR provides only error
and fatal-level messages, and so on.
Stargate Logs
This section discusses common entries found in Stargate logs and what they mean.
The Stargate logs are located at /home/nutanix/data/logs/stargate.[INFO|WARNING|ERROR|FATAL].
Log Entry: Watch dog fired
F1001 16:20:49.306397 6630 stargate.cc:507] Watch dog fired
This message is generic and can happen for a variety of reasons. While Stargate is initializing, a watch dog
process monitors it to ensure a successful startup process. If it has trouble connecting to other components
(such as Zeus or Pithos) the watch dog process stops Stargate.
If Stargate is running, this indicates that the alarm handler thread is stuck for longer than 30 seconds. The
stoppage could be due to a variety of reasons, such as problems connecting to Zeus or accessing the
Cassandra database.

To analyze why the watch dog fired, first locate the relevant INFO file, and review the entries leading up to
the failure.
Log Entry: HTTP request timed out
E0820 09:14:05.998002 15406 rpc_client.cc:559] Http request timed out
This message indicates that Stargate is unable to communicate with Medusa. This may be due to a
network issue.
Analyze the ping logs and the Cassandra logs.
Log Entry: CAS failure seen while updating metadata for egroup egroupid or Backend returns error
'CAS Error' for extent group id: egroupid
W1001 16:22:34.496806 6938 vdisk_micro_egroup_fixer_op.cc:352] CAS failure seen while

updating metadata for egroup 1917333
This is a benign message and usually does not indicate a problem. This warning message means that
another Cassandra node has already updated the database for the same key.
Log Entry: Fail-fast after detecting hung stargate ops: Operation with id <opid> hung for 60secs
F0712 14:19:13.088392 30295 stargate.cc:912] Fail-fast after detecting hung stargate ops:
Operation with id 3859757 hung for 60secs
This message indicates that Stargate restarted because an I/O operation took more than 60 seconds to
complete.
To analyze why the I/O operation took more than 60 seconds, locate the relevant INFO file and review the
entries leading up to the failure.
Log Entry: Timed out waiting for Zookeeper session establishment
F0907 01:22:23.124495 10559 zeus.cc:1779] Timed out waiting for Zookeeper session
establishment
This message indicates that Stargate was unable to connect to Zookeeper.

Review the sysstats/ping_hosts.INFO log to determine if there were any network issues around that time.
Log Entry: Too many attempts trying to access Medusa
F0601 10:14:47.101438 2888 medusa_write_op.cc:85] Check failed: num_retries_ < 5 (5 vs. 5)

Too many attempts trying to access Medusa
This message indicates that Stargate had 5 failed attempts to connect to Medusa/Cassandra.
Review the Cassandra log (cassandra/system.log) to see why Cassandra was unavailable.
Log Entry:multiget_slice() failed with error: error_code while reading n rows from
cassandra_keyspace
E1002 18:51:43.223825 24634 basic_medusa_op.cc:1461] multiget_slice() failed with error: 4

while reading 1 rows from 'medusa_nfsmap'. Retrying...

This message indicates that Stargate cannot connect to Medusa/Cassandra.
Review the Cassandra log (cassandra/system.log) to see why Cassandra was unavailable.
Log Entry: Forwarding of request to NFS master ip:2009 failed with error kTimeout.
W1002 18:50:59.248074 26086 base_op.cc:752] Forwarding of request to NFS master

172.17.141.32:2009 failed with error kTimeout
This message indicates that Stargate cannot connect to the NFS master on the node specified.
Review the Stargate logs on the node specified in the error.
Cassandra Logs
After analyzing Stargate logs, if you suspect an issue with Cassandra/Medusa, analyze the Cassandra
logs. This topic discusses common entries found in system.log and what they mean.
The Cassandra logs are located at /home/nutanix/data/logs/cassandra. The most recent file is named
system.log. When the file reaches a certain size, it rolls over to a sequentially numbered file (example,
system.log.1, system.log.2, and so on).
Log Entry: batch_mutate 0 writes succeeded and 1 column writes failed for
keyspace:medusa_extentgroupidmap
INFO [RequestResponseStage:3] 2013-09-10 11:51:15,780 CassandraServer.java

(line 1290) batch_mutate 0 writes succeeded and 1 column writes failed for
keyspace:medusa_extentgroupidmap cf:extentgroupidmap row:lr280000:1917645 Failure Details:
Failure reason:AcceptSucceededForAReplicaReturnedValue : 1
This is a common log entry and can be ignored. It is equivalent to the CAS errors in the stargate.ERROR
log. It simply means that another Cassandra node updated the keyspace first.
Log Entry: InetAddress /x.x.x.x is now dead.
INFO [ScheduledTasks:1] 2013-06-01 10:14:29,767 Gossiper.java (line 258) InetAddress /

x.x.x.x is now dead.
This message indicates that the node could not communicate with the Cassandra instance at the specified
IP address.
Either the Cassandra process is down (or failing) on that node or there are network connectivity issues.
Check the node for connectivity issues and Cassandra process restarts.
Log Entry: Caught Timeout exception while waiting for paxos read response from leader: x.x.x.x
ERROR [EXPIRING-MAP-TIMER-1] 2013-08-08 07:33:25,407 PaxosReadDoneHandler.java (line 64)

Caught Timeout exception while waiting for paxos read reponse from leader: 172.16.73.85.
Request Id: 116. Proto Rpc Id : 2119656292896210944. Row no:1. Request start time: Thu Aug
08 07:33:18 PDT 2013. Message sent to leader at: Thu Aug 08 07:33:18 PDT 2013 # commands:1
requestsSent: 1
This message indicates that the node encountered a timeout while waiting for the Paxos leader.
Either the Cassandra process is down (or failing) on that node or there are network connectivity issues.
Check the node for connectivity issues or for the Cassandra process restarts.

Prism Gateway Log
This section discusses common entries found in prism_gateway.log and what they mean. This log is
located on the Prism leader. The Prism leader is the node which is running the web server for the Nutanix
UI. This is the log to analyze if there are problems with the UI such as long loading times.
The Prism log is located at /home/nutanix/data/logs/prism_gateway.log on the Prism leader.
To identify the Prism leader, you can run cluster status | egrep "CVM|Prism" and determine which node
has the most processes. In the output below, 10.3.176.242 is the Prism leader.
nutanix@cvm$ cluster status | egrep "CVM|Prism"
2013-09-10 16:06:42 INFO cluster:946 Executing action status on CVMs

10.3.176.240,10.3.176.241,10.3.176.2422013-09-10
16:06:45 INFO cluster:987 Success!
CVM: 10.3.176.240 Up
Prism UP[32655, 32682, 32683, 32687]
CVM: 10.3.176.241 Up
Prism UP[11371, 25913, 25925, 25926]
Prism UP[4291, 4303, 4304, 19468, 20072, 20074, 20075, 20078,
20113]
Log Entry: Error sending request: java.net.NoRouteToHostException: Cannot assign requested

address
The stats_aggregator component periodically issues an RPC request for all Nutanix vdisks in the cluster. It
is possible that all the ephemeral ports are exhausted.
The ss -s command shows you the number of open ports.

nutanix@cvm$ ss -s
Total: 277 (kernel 360)

TCP: 218 (estab 89, closed 82, orphaned 0, synrecv 0, timewait 78/0), ports
207
Transport Total IP IPv6

* 360 - -
RAW 1 1 0
UDP 23 13 10
TCP 136 84 52
INET 160 98 62
FRAG 0 0 0
If there are issues with connecting to the Nutanix UI, escalate the case and provide the
output of the ss -s command as well as the contents of prism_gateway.log.
Zookeeper Logs
The Zookeeper logs are located at /home/nutanix/data/logs/zookeeper.out.

This log contains the status of the Zookeeper service. More often than not, there is no need to look at this
log. However, if one of the other logs specifies that it is unable to contact Zookeeper and it is affecting
cluster operations, you may want to look at this log to find the error Zookeeper is reporting.
Genesis.out
When checking the status of the cluster services, if any of the services are down, or the Controller VM
is reporting Down with no process listing, review the log at /home/nutanix/data/logs/genesis.out to
determine why the service did not start, or why Genesis is not properly running.
Check the contents of genesis.out if a Controller VM reports multiple services as DOWN, or if the entire
Controller VM status is DOWN.
Like other component logs, genesis.out is a symbolic link to the latest genesis.out instance and has the
format genesis.out.date-timestamp.
An example of steady state output:
nutanix@cvm$ tail -F ~/data/logs/genesis.out
2017-03-23 19:24:00 INFO node_manager.py:2070 Certificate cache in sync
2017-03-23 19:24:00 INFO node_manager.py:4732 Checking if we need to sync the local SVM and
Hypervisor DNS configuration with Zookeeper
2017-03-23 19:24:38 ERROR lcm_zeus.py:96 Failed to read zknode /appliance/logical/lcm/

operation with error: no node
2017-03-23 19:24:39 INFO framework.py:637 No other LCM operation in progress
2017-03-23 19:26:00 INFO node_manager.py:1960 Certificate signing request data is not

available in Zeus configuration
2017-03-23 19:26:00 INFO node_manager.py:1874 No CA certificates found in the Zeus

configuration
2017-03-23 19:26:00 INFO node_manager.py:1877 No Svm certificates found in the Zeus

configuration
2017-03-23 19:26:00 INFO node_manager.py:1880 No Svm certificate maps found in the Zeus
configuration
2017-03-23 19:26:00 INFO node_manager.py:4732 Checking if we need to sync the local SVM and
Hypervisor DNS configuration with Zookeeper
2017-03-23 19:28:00 INFO node_manager.py:1960 Certificate signing request data is not

available in Zeus configuration
2017-03-23 19:28:00 INFO node_manager.py:1874 No CA certificates found in the Zeus

configuration
2017-03-23 19:28:00 INFO node_manager.py:1877 No Svm certificates found in the Zeus

configuration
2017-03-23 19:28:00 INFO node_manager.py:1880 No Svm certificate maps found in the Zeus
configuration

Under normal conditions, the genesis.out file logs the following messages periodically:
Unpublishing service Nutanix Controller
Publishing service Nutanix Controller
Zookeeper is running as [leader|follower]
Prior to these occasional messages, you should see Starting [n]th service. This is an indicator that all
services were successfully started. As of 5.0, there are 34 services.
Tip: You can ignore any INFO messages logged by Genesis by running the command:
grep -v -w INFO /home/nutanix/data/logs/genesis.out
Possible Errors
2017-03-23 19:28:00 WARNING command.py:264 Timeout executing scp -q -o CheckHostIp=no

-o ConnectTimeout=15 -o StrictHostKeyChecking=no -o TCPKeepAlive=yes -o
UserKnownHostsFile=/dev/null -o PreferredAuthentications=keyboard-interactive,password -o
BindAddress=192.168.5.254 'root@[192.168.5.1]:/etc/resolv.conf' /tmp/resolv.conf.esx: 30
secs elapsed
2017-03-23 19:28:00 ERROR node_dns_ntp_config.py:287 Unable to download ESX DNS

configuration file, ret -1, stdout , stderr
2017-03-23 19:28:00 WARNING node_manager.py:2038 Could not load the local ESX configuration
2017-03-23 19:28:00 ERROR node_dns_ntp_config.py:492 Unable to download the ESX NTP

configuration file, ret -1, stdout , stderr
Any of the above messages means that Genesis was unable to log on to the ESXi host using the
configured password.
Diagnosing a Genesis Failure

Determine the cause of a Genesis failure based on the information available in the log files.
1. Examine the contents of the genesis.out file and locate the stack trace (indicated by the CRITICAL
message type).
2. Analyze the ERROR messages immediately preceding the stack trace.

...
2017-03-23 19:30:00 INFO node_manager.py:4170 No cached Zeus configuration found.
2017-03-23 19:30:00 INFO hyperv.py:142 Using RemoteShell ...
2017-03-23 19:30:00 INFO hyperv.py:282 Updating NutanixUtils path
2017-03-23 19:30:00 ERROR hyperv.py:290 Failed to update the NutanixUtils path: [Errno
104] Connection reset by peer
2017-03-23 19:30:00 CRITICAL node_manager.py:3559 File "/home/nutanix/cluster/bin/
genesis", line 207, in <module>
main(args)
File "/home/nutanix/cluster/bin/genesis", line 149, in main
Genesis().run()
File "/home/nutanix/jita/main/28102/builds/build-danube-4.1.3-stable-release/python-
tree/bdist.linux-x86_64/egg/util/misc/decorators.py", line 40, in wrapper
tree/bdist.linux-x86_64/egg/cluster/genesis/server.py", line 132, in run
tree/bdist.linux-x86_64/egg/cluster/genesis/node_manager.py", line 502, in initialize
tree/bdist.linux-x86_64/egg/cluster/genesis/node_manager.py", line 3559, in discover

...
In the example above, the certificates in AuthorizedCerts.txt were not updated, which means that you
failed to connect to the NutanixHostAgent service on the host.
Note: NutanixHostAgent is Hyper-V specific.
ESXi Log Files

These log files are present on ESXi hosts.
Location: /var/logs
Log Contents
hostd.log hostd (daemon to communicate with vmkernel) process output
vmkernel.log vmkernel activity

vpxa.log vpxa (daemon to commmunicate with vCenter) process output
Location: /vmfs/volumes/
Log Contents
datastore/vm_name/vmware.log Virtual machine activity and health

7
Troubleshooting Tools
Nutanix Cluster Check (NCC)

Nutanix Cluster Check (NCC) is a framework of scripts that can help diagnose cluster health. NCC can
be run provided that the individual nodes are up, regardless of cluster state. The scripts run standard
commands against the cluster or the nodes, depending on the type of information being retrieved.
When run from the Controller VM command line, NCC generates a log file with the output of the diagnostic
commands selected by the user.
NCC actions are grouped into plugins and modules.
• Plugins are objects that run the diagnostic commands.
• Modules are logical groups of plugins that can be run as a set.
Note: Some plugins run nCLI commands and might require the user to input the nCLI password.
The password is logged on as plain text. If you change the password of the admin user from
the default, you must specify the password every time you start an nCLI session from a remote
system. A password is not required if you are starting an nCLI session from a Controller VM where
you are already logged on.
Comprehensive documentation of NCC is available in the Nutanix Command Reference.
NCC Output
Each NCC plugin is a test that completes independently of other plugins. Each test completes with one of
these status types.
PASS
The tested aspect of the cluster is healthy and no further action is required.
FAIL
The tested aspect of the cluster is not healthy and must be addressed.
WARN
The plugin returned an unexpected value and must be investigated.
INFO
The plugin returned an expected value that however cannot be evaluated as PASS/FAIL.
Running Health Checks
In addition to running all health checks, you can checks as follows:

Run all or some checks from the Prism Web Console
• From the Prism web console Health page, select Actions > Run Checks. Select All checks and
click Run.
• If you disable a check in the Prism web console, you cannot run it from the NCC command line
unless you enable it again from the web console.
Troubleshooting Tools | Acropolis Advanced Administration Guide | AOS | 51

• You can run NCC checks from the Prism web console for clusters where AOS 5.0 or later and
NCC 3.0 or later are installed. You cannot run NCC checks from the Prism web console for
clusters where AOS 4.7.x or previous and NCC 3.0 are installed.
• For AOS clusters where it is installed, running NCC 3.0 or later from the command line updates
the Cluster Health score, including the color of the score. For some NCC checks, you can clear
the score by disabling and then re-enabling the check.
Run two or more individual checks at a time
• You can specify two or more individual checks from the command line, with each check separated
by a comma. Ensure you do not use any spaces between checks, only a comma character. For
example:
ncc health_checks system_checks \
--plugin_list=cluster_version_check,cvm_reboot_check
Re-run failing checks

• You can re-run any NCC checks or plug-ins that reported a FAIL status.
ncc --rerun_failing_plugins=True
Run NCC health checks in parallel

• You can specify the number of NCC health checks to run in parallel to reduce the amount of time
it takes for all checks to complete. For example, the command ncc health_checks run_all --
parallel=4 runs four of the health checks in parallel (which is the maximum). [ENG-32466]
Use npyscreen to display NCC status

• You can specify npyscreen as part of the ncc command to display status to the terminal window.
Specify --use_npyscreen=true as part of the ncc health_checks command.
ncc health_checks run_all --use_npyscreen=true
Installing NCC from an Installer File

Before you begin:
Note:
• NCC 3.0 is not supported on AOS 4.5.x and previous versions.
• If you are adding one or more nodes to expand your cluster, the latest version of NCC might not
be installed on each newly-added node. In this case, re-install NCC in the cluster after you have
finished adding the one or more nodes.
• This topic describes how to install NCC from the command line. To install NCC software from
the web console, see Upgrading NCC Software on page 54.
Note: To help ensure that Prism Central and each managed cluster are taking advantage of NCC
features, ensure that:
• Each node in your cluster is running the same NCC version.
• Prism Central and each cluster managed by Prism Central are all running the same NCC
version.
To check the NCC version and optionally upgrade the NCC software version on Prism Central, see
the Prism Central Guide, Upgrading Nutanix Cluster Check (NCC) on Prism Central topic.
You can download the NCC installation file from the Nutanix support portal under Downloads > Tools &
Firmware. The file type to download depends on the NCC version:
Tip: Note the MD5 value of the file as published on the support portal.

• Some NCC versions include a single installer file (ncc_installer_filename.sh) that you can download and
run from any Controller VM.
• Some NCC versions include an installer file inside a compressed tar file (ncc_installer_filename.tar.gz)
that you must first extract, then run from any Controller VM.
• The directory to which you copy the installation package should exist on all nodes in the cluster (/home/
nutanix is suggested). Additionally, the folder should be owned by any accounts that uses NCC.
1. Download the installation file to any controller VM in the cluster and copy the installation file to the /
home/nutanix directory.
2. Check the MD5 value of the file. It must match the MD5 value published on the support portal. If the
value does not match, delete the file and download it again from the support portal.
nutanix@cvm$ md5sum ./ncc_installer_filename.sh
3. Perform these steps for NCC versions that include a single installer file (ncc_installer_filename.sh)
a. Make the installation file executable.

nutanix@cvm$ chmod u+x ./ncc_installer_filename.sh
b. Install NCC.
nutanix@cvm$ ./ncc_installer_filename.sh
The installation script installs NCC on each node in the cluster.
NCC installation file logic tests the NCC tar file checksum and prevents installation if it detects file
corruption.
• If it verifies the file, the installation script installs NCC on each node in the cluster.
• If it detects file corruption, it prevents installation and deletes any extracted files. In this case,
download the file again from the Nutanix support portal.
4. Perform these steps for NCC versions that include an installer file inside a compressed tar file
(ncc_installer_filename.tar.gz).
a. Extract the installation package.

nutanix@cvm$ tar xvmf ncc_installer_filename.tar.gz --recursive-unlink
Replace ncc_installer_filename.tar.gz with the name of the compressed installation tar file.
The --recursive-unlink option is needed to ensure old installs are completely removed.
b. Run the install script. Provide the installation tar file name if it has been moved or renamed.
nutanix@cvm$ ./ncc/bin/install.sh [-f install_file.tar]
The installation script copies the install_file.tar tar file to each node and install NCC on each node in
the cluster.
5. Check the output of the installation command for any error messages.
• If installation is successful, a Finished Installation message is displayed. You can check any
NCC-related messages in /home/nutanix/data/logs/ncc-output-latest.log.
• In some cases, output similar to the following is displayed. Depending on the NCC version installed,
the installation file might log the output to /home/nutanix/data/logs/ or /home/nutanix/data/
serviceability/ncc.
Copying file to all nodes [ DONE ]
-------------------------------------------------------------------------------+
+---------------+

| State | Count |
+---------------+
| Total | 1 |
+---------------+
Plugin output written to /home/nutanix/data/logs/ncc-output-latest.log
[ info ] Installing ncc globally.
[ info ] Installing ncc on 10.130.45.72, 10.130.45.73
[ info ] Installation of ncc succeeded on nodes 10.130.45.72, 10.130.45.73.
What to do next:
• As part of installation or upgrade, NCC automatically restarts the cluster health service on each node
in the cluster, so you might observe notifications or other slight anomalies as the service is being
restarted.
Upgrading NCC Software

Before you begin:
Note:
• NCC 3.0 is not supported on AOS 4.5.x and previous versions.
• If you are adding one or more nodes to expand your cluster, the latest version of NCC might not
be installed on each newly-added node. In this case, re-install NCC in the cluster after you have
finished adding the one or more nodes.
• This topic describes how to install NCC software from the Prism web console. To install NCC
from the command line, see Installing NCC from an Installer File on page 52.
Note: To help ensure that Prism Central and each managed cluster are taking advantage of NCC
features, ensure that:
• Each node in your cluster is running the same NCC version.
• Prism Central and each cluster managed by Prism Central are all running the same NCC
version.
To check the NCC version and optionally upgrade the NCC software version on Prism Central, see
the Prism Central Guide, Upgrading Nutanix Cluster Check (NCC) on Prism Central topic.
1. Run the Nutanix Cluster Checks (NCC).

→ From the Prism web console Health page, select Actions > Run Checks. Select All checks and
click Run.
→ Log in to a Controller VM and use the ncc CLI.
nutanix@cvm$ ncc health_checks run_all
If the check reports a status other than PASS, resolve the reported issues before proceeding. If you are
unable to resolve the issues, contact Nutanix support for assistance.
2. Do this step to download and install the NCC upgrade files.
a. Log on to the Prism web console for any node in the cluster.
b. Click Upgrade Software from the gear icon in the Prism web console, then click NCC in the dialog
box.
c. If an update is available, click Upgrade Available and then click Download.
d. When the download is complete, do one of the following:

→ To run only the pre-upgrade installation checks on the controller VM where you are logged on, click
Upgrade > Pre-upgrade. These checks also run as part of the upgrade procedure.
→ Click Upgrade > Upgrade Now, then click Yes to confirm.
The Upgrade Software dialog box shows the progress of your selection, including pre-installation
checks.
As part of installation or upgrade, NCC automatically restarts the cluster health service on each node
restarted.
Upgrading NCC by Uploading Binary and Metadata Files
• Do the following steps to download NCC binary and metadata .JSON files from the Nutanix Support
Portal, then upgrade NCC through Upgrade Software in the Prism web console.
• Typically you would need to perform this procedure if your cluster is not directly connected to the
Internet and you cannot download the binary and metadata .JSON files through the Prism web console.
1. Log on to the Nutanix support portal and select Downloads > Tools & Firmware.
2. Click the download link to save the binary gzipped TAR (.tar.gz) and metadata (.json) files on your local
media.
3. Log on to the Prism web console for any node in the cluster.
4. Click Upgrade Software from the gear icon in the Prism web console, then click NCC in the dialog box.
5. Click the upload the NCC binary link.
6. Click Choose File for the NCC metadata and binary files, respectively, browse to the file locations, and
click Upload Now.
7. When the upload process is completed, click Upgrade > Upgrade Now, then click Yes to confirm.
The Upgrade Software dialog box shows the progress of your selection, including pre-installation
checks.
As part of installation or upgrade, NCC automatically restarts the cluster health service on each node
restarted.
NCC Usage
The general usage of NCC is as follows:
nutanix@cvm$ ncc ncc-flags module sub-module [...] plugin plugin-flags
Alternately, you can run all or individual checks from the Prism web console. Select Actions > Run
Checks. Select All checks and click Run.
Typing ncc with no arguments yields a table listing the next modules that can be run. The Type column
distinguishes between modules (M) and plugins (P). The Impact tag identifies a plugin as intrusive or non-
intrusive. By default, only non-intrusive checks are used if a module is run with the run_all plugin.
nutanix@cvm$ ncc
+-----------------------------------------------------------------------------+
| Type | Name | Impact | Short help |
+-----------------------------------------------------------------------------+
| M | cassandra_tools | N/A | Plugins to help with Cassandra ring analysis |

| M | file_utils | N/A | Utilities for manipulating files on the |
| | | | cluster |
| M | health_checks | N/A | All health checks |
| M | info | N/A | Contains all info modules (legacy |
| | | | health_check.py) |
+-----------------------------------------------------------------------------+
The usage table is displayed for any module specified on the command line. Specifying a plugin runs its
associated checks.
nutanix@cvm$ ncc info
+------------------------------------------------------------------------------+
+------------------------------------------------------------------------------+
| M | cluster_info | N/A | Displays summary of info about this cluster. |
| M | cvm_info | N/A | Displays summary of info about this CVM. |
| M | esx_info | N/A | Displays summary of info about the local esx |
| | | | host. |
| M | ipmi_info | N/A | Displays summary of info about the local IPMI.|
| P | run_all | N/A | Run all the plugins in this module |
+------------------------------------------------------------------------------+
The file_utils module does not run any checks. It exists to help you manage files in the cluster.
nutanix@cvm$ ncc file_utils
+------------------------------------------------------------------------------+
+------------------------------------------------------------------------------+
| P | file_copy | Non-Intrusive | Copies a local file to all CVMs. |
| P | remove_old_cores | Non-Intrusive | Removing cores older than 30 days |
| P | remove_old_fatals | Non-Intrusive | Removing fatals older than 90 days|
| P | run_all | N/A | Run all the plugins in this module|
+------------------------------------------------------------------------------+
Usage Examples
• Run all health checks.

nutanix@cvm$ ncc health_checks run_all
• Display default command flags.

nutanix@cvm$ ncc --ncc_interactive=false module sub-module [...] plugin \
--helpshort
• Run the NCC with a named output file and a non-standard path for ipmitool.
nutanix@cvm$ ncc --ncc_plugin_output_history_file=ncc.out health_checks \
hardware_checks ipmi_checks run_all --ipmitool_path /usr/bin/ipmitool
Note: The flags override the default configurations of the NCC modules and plugins. Do not
run with these flags unless your cluster configuration requires these modifications.
Diagnostics VMs
Nutanix provides a diagnostics capability to allow partners and customers to run performance tests on the
cluster. This is a useful tool in pre-sales demonstrations of the cluster and while identifying the source of

performance issues in a production cluster. Diagnostics should also be run as part of setup to ensure that
the cluster is running properly before the customer takes ownership of the cluster.
The diagnostic utility deploys a VM on each node in the cluster. The Controller VMs control the diagnostic
VM on their hosts and report back the results to a single system.
The diagnostics test provide the following data:
• Sequential write bandwidth
• Sequential read bandwidth
• Random read IOPS
• Random write IOPS
Because the test creates new cluster entities, it is necessary to run a cleanup script when you are finished.
Running a Test Using the Diagnostics VMs

Before you begin:
• Ensure that 10 GbE ports are active on the ESXi hosts using esxtop or vCenter. The tests run very slow
if the nodes are not using the 10 GbE ports. For more information about this known issue with ESXi 5.0
update 1, see VMware KB article 2030006.
2. Set up the diagnostics test.

nutanix@cvm$ ~/diagnostics/diagnostics.py cleanup
(vSphere only) In vCenter, right-click any diagnostic VMs labeled as "orphaned", select Remove from
Inventory, and click Yes to confirm removal.
3. Start the diagnostics test.

nutanix@cvm$ ~/diagnostics/diagnostics.py run
Include the parameter --default_ncli_password='admin_password' if the Nutanix admin user password

has been changed from the default.
If the command fails with ERROR:root:Zookeeper host port list is not set, refresh the environment
by running source /etc/profile or bash -l and run the command again.
The diagnostic may take up to 15 minutes to complete for a four-node cluster. Larger clusters take
longer time.
The script performs the following tasks:
1. Installs a diagnostic VM on each node.
2. Creates cluster entities to support the test, if necessary.
3. Runs four performance tests, using the Linux fio utility.
4. Reports the results.
4. Review the results.
5. Remove the entities from this diagnostic.

nutanix@cvm$ ~/diagnostics/diagnostics.py cleanup
(vSphere only) In vCenter, right-click any diagnostic VMs labeled as "orphaned", select Remove from
Inventory, and click Yes to confirm removal.

Diagnostics Output
System output similar to the following indicates a successful test.
Checking if an existing storage pool can be used ...

Using storage pool sp1 for the tests.
Checking if the diagnostics container exists ... does not exist.
Creating a new container NTNX-diagnostics-ctr for the runs ... done.
Mounting NFS datastore 'NTNX-diagnostics-ctr' on each host ... done.
Deploying the diagnostics UVM on host 172.16.8.170 ... done.
Preparing the UVM on host 172.16.8.170 ... done.
VM on host 172.16.8.170 has booted. 3 remaining.
Waiting for the hot cache to flush ... done.
Running test 'Prepare disks' ... done.
Running test 'Sequential write bandwidth (using fio)' ... bandwidth MBps
Running test 'Sequential read bandwidth (using fio)' ... bandwidth MBps
Running test 'Random read IOPS (using fio)' ... operations IOPS
Running test 'Random write IOPS (using fio)' ... operations IOPS
Tests done.
Note:
• Expected results vary based on the specific AOS version and hardware model used.
• The IOPS values reported by the diagnostics script is higher than the values reported by the
Nutanix management interfaces. This difference is because the diagnostics script reports
physical disk I/O, and the management interfaces show IOPS reported by the hypervisor.
• If the reported values are lower than expected, the 10 GbE ports may not be active. For more
information about this known issue with ESXi 5.0 update 1, see VMware KB article 2030006.
Syscheck Utility
Syscheck is a tool that runs load on a cluster and evaluate its performance characteristics. This tool
provides pass or fail feedback on all the checks. The current checks are network throughput and direct
disk random write performance. Syscheck tracks the tests on a per node basis and prints the result at the
conclusion of the test.
Using Syscheck Utility

Perform the following procedure to run the syscheck utility on AOS clusters.

Note:
• Run this test on a newly created cluster or a cluster that is idle or has minimal load.
• Do not run this test if systems are sharing the network as it may interfere with their operation.
• Do not run this test if the guest VMs have already been deployed.
1. Log into the Controller VM.
2. Run the syscheck utility.

nutanix@cvm$ /usr/local/nutanix/syscheck/bin/syscheck
After executing the command, a message that records all the considerations of running this test is
displayed. When prompted with the message, type yes to run the check.
The test returns either pass or fail result. The latest result is placed under /home/nutanix/data/syscheck
directory. An output tar file is also placed in /home/nutanix/data/ directory after every time you run this
utility.

8
Controller VM Memory Configurations
Controller VM memory allocation requirements differ depending on the models and the features that are
being used.
CVM Memory and vCPU Configurations (G5/Broadwell)

This topic lists the recommended Controller VM memory allocations for workload categories.
Note: If the AOS upgrade process detects that any node hypervisor host has total physical
memory of 64 GB or greater, it automatically upgrades any Controller VM in that node with less
than 32 GB memory by 4 GB. The Controller VM is upgraded to a maximum 32 GB.
If the AOS upgrade process detects any node with less than 64 GB memory size, no memory
changes occur.
For nodes with ESXi hypervisor hosts with total physical memory of 64 GB, the Controller VM
is upgraded to a maximum 28 GB. With total physical memory greater than 64 GB, the existing
Controller VM memory is increased by 4 GB.
• To calculate the number of vCPUs for your model, use the number of physical cores per socket in your
model. The minimum number of vCPUS your Controller VM can have is eight and the maximum number
is 12, unless otherwise noted.
• If your CPU has eight or fewer logical cores, allocate a maximum of 75 percent of the cores of a single
CPU to the Controller VM. For example, if your CPU has 6 cores, allocate 4 vCPUs.
Controller VM Memory Configurations for Base Models
Platform Recommended / Default vCPUs

Memory (GB)
Default configuration for all platforms 20 8
Nutanix Broadwell Models
The following table shows the minimum amount of memory required for the Controller VM on each node
for platforms that do not follow the default. For the workload translation into models, see Platform Workload
Translation (G5/Broadwell) on page 61.
Platform Default Memory (GB)

VDI, server virtualization 20
Storage Heavy 28
Storage Only 28
Large server, high-performance, all-flash 32
Controller VM Memory Configurations | Acropolis Advanced Administration Guide | AOS | 60

Platform Workload Translation (G5/Broadwell)
The following table maps workload types to the corresponding Nutanix and Lenovo models.
Workload Exceptions
Note: Upgrading to 5.1 requires a 4GB memory increase, unless the CVM memory already has
32 GB.
If all the data disks in a platform are SSDs, the node is assigned the High Performance workload except for
the following exceptions.
• Klas Voyager 2 uses SSDs but due to workload balance, this platform workload default is VDI.
• Cisco B-series is expected to have large remote storage and two SSDs as a local cache for the hot tier,
so this platform workload is VDI.
Workload Nutanix Nutanix Lenovo Cisco Dell Klas Crystal

Telecom
Features NX Model SX Model HX Model UCS XC
VDI NX-1065S- SX-1065- HX3310 B200-M4 XC430- VOYAGER2RS2616PS18

G5 G5 Xpress
NX-1065- - HX3310-F C240- - - -

G5 M4L
NX-3060- - HX2310-E C240- - - -

G5 M4S
NX-3155G- - HX3510- C240- - - -

G5 G M4S2
NX-3175- - HX3710 C220- - - -

G5 M4S
- - HX1310 C220- - - -
M4L
- - HX2710-E Hyperflex - - -
HX220C-
M4S
- - HX3510- - - - -
FG
- - HX3710-F - - - -
Storage Heavy NX-6155- - HX5510 - - - -

G5
NX-8035- - HX5510- - - - -
G5 C
NX-6035- - - - - - -
G5
Storage Node NX-6035C- - HX5510- - - - -
G5 C

Workload Nutanix Nutanix Lenovo Cisco Dell Klas Crystal
Telecom
Features NX Model SX Model HX Model UCS XC
High Performance NX-8150- SX-1155- HX7510 C240- XC630-10P- -

and All-Flash G5 G5 M4SX
NX-9060- - HX7510-F Hyperflex XC730xd-12R

- -
G5 HX240C-
M4SX
NX-1155- - - - - - -
G5
NX-6155- - - - - - -
G5
NX-8150- - - - - - -
G5
NX-9060- - - - - - -
G5
CVM Memory and vCPU Configurations (G4/Haswell/Ivy Bridge)

This topic lists the recommended Controller VM memory allocations for models and features.
Controller VM Memory Configurations for Base Models
Platform Default
Platform Recommended/Default vCPUs

Memory (GB)
Default configuration for all platforms unless 20 8
otherwise noted
The following tables show the minimum amount of memory and vCPU requirements and recommendations
for the Controller VM on each node for platforms that do not follow the default.
Nutanix Platforms
Platform Recommended Default Memory vCPUs

Memory (GB) (GB)
NX-1020 16 16 4
NX-6035C 28 28 8
NX-6035-G4 28 20 8
NX-8150 32 32 8
NX-8150-G4 32 32 8
NX-9040 32 20 8

Memory (GB) (GB)
NX-9060-G4 32 32 8
Dell Platforms

Memory (GB) (GB)
XC730xd-24 32 20 8
XC6320-6AF
XC630-10AF
Lenovo Platforms
Platform Recommended/Default vCPUs

Memory (GB)
HX-3500 28 8
HX-5500
HX-7500
CVM Memory Configurations for Features

The following table lists the minimum amount of memory required when enabling features.
The memory size requirements are in addition to the default or recommended memory available for your
platform. The maximum additional memory required is 16 GB even if the total indicated for the features is
more than that.
Note: Total CVM memory required = recommended platform memory + memory required for each
enabled feature (max 12 GB)
Features Memory (GB)

Capacity tier deduplication (includes performance tier deduplication) 12
Redundancy factor 3 8
Performance tier deduplication 8
Cold-tier nodes + capacity tier deduplication 4
Capacity tier deduplication + redundancy factor 3 12

Nutanix - Advanced-Admin-AOS-v51

Uploaded by

Copyright:

Available Formats

Nutanix - Advanced-Admin-AOS-v51

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Nutanix - Advanced-Admin-AOS-v51

Uploaded by

Copyright:

Available Formats

Acropolis Advanced Administration Guide

variable_value The action depends on a value that is unique to your environment.

ncli> command The commands are executed in the Nutanix nCLI.

output The information is displayed as output from a command or in a log file.

Default Cluster Credentials

Nutanix web console Nutanix Controller VM admin Nutanix/4u

vSphere Web Client ESXi host root nutanix/4u

Copyright | Acropolis Advanced Administration Guide | AOS | 2

vSphere client ESXi host root nutanix/4u

SSH client or console ESXi host root nutanix/4u

SSH client or console AHV host root nutanix/4u

SSH client or console Hyper-V host Administrator nutanix/4u

SSH client Nutanix Controller VM nutanix nutanix/4u

SSH client Nutanix Controller VM admin Nutanix/4u

SSH client or console Acropolis OpenStack root admin

Copyright | Acropolis Advanced Administration Guide | AOS | 3

3: Cluster IP Address Configuration........................................................22

4: Creating a Windows Guest VM Failover Cluster................................ 32

5: Acropolis Dynamic Scheduling in AHV.............................................. 35

8: Controller VM Memory Configurations............................................... 60

Purpose Port Numbers

Management interface (web console, nCLI) 9440

Admin Access to Controller VM

Cluster Management | Acropolis Advanced Administration Guide | AOS | 6

Interface Target User Name Password

SSH client Nutanix Controller VM admin Nutanix/4u

Accessing the Controller VM Using the Admin Account

Cluster Management | Acropolis Advanced Administration Guide | AOS | 7

The password must meet the following complexity requirements:

Starting a Nutanix Cluster

1. Log on to any Controller VM in the cluster with SSH.

2. Start the Nutanix cluster.

Cluster Management | Acropolis Advanced Administration Guide | AOS | 8

1. Log on to a running Controller VM in the cluster with SSH.

2. Stop the Nutanix cluster.

CVM: 172.16.8.191 Up, ZeusLeader

Cluster Management | Acropolis Advanced Administration Guide | AOS | 9

1. Log on to any Controller VM in the cluster with SSH.

2. Stop the Nutanix cluster.

CVM: 172.16.8.191 Up, ZeusLeader

3. Destroy the cluster.

nutanix@cvm$ cluster destroy

Cluster Management | Acropolis Advanced Administration Guide | AOS | 10

Creating Clusters from a Multiblock Cluster

1. Remove nodes from the existing cluster.

Cluster IP Address Configuration

Configuring the Cluster

Cluster Management | Acropolis Advanced Administration Guide | AOS | 11

Figure: Cluster IP Address Configuration Page

1. Open a web browser.

2. In the browser, go to http://[cvm_ipv6_addr]:2100/cluster_init.html.

Cluster Management | Acropolis Advanced Administration Guide | AOS | 12

3. Type a meaningful value in the Cluster Name field.

Note: This entity has the following naming restrictions:

7. Type the appropriate subnet masks in the Subnet Mask row.

10. Provide an IP address for all components in the cluster.

Cluster Management | Acropolis Advanced Administration Guide | AOS | 13

11. Click Create.

The cluster is started automatically after creation.

12. Log on to any Controller VM in the cluster with SSH.