@harlem X What's a Keytab Is a container to store Kerberos principal's keys. A key in terms of Kerberos can be some random bytes. You can use container to store a password for a user principal or a password (key) for a service principal. Keytabs are cryptographic files containing an representation of the service and its long-term key.Keytabs are used to either de-crypt the Kerberos service ticket of a Service or user to an hadoop service or authenticate the service itself to another service on the network. The permissions of the keytabs is usually 400 or 440 for services like (hdfs, hbase, ambari-qa, accumulo, and spnego). These keytabs are used by other services rather than just the owner for testing connectivity to the cluster, and other functions on startup. All of the other keytabs are only readable by the service account that owns the file and the group that owns the keytabs (hadoop) group, and should be reserved for service accounts. What's a Kerberos service Represents an application either acting as a client or as a server, it does not really matter which way. Such application may have own arrangement on how it runs (which UID/GID it uses on the operating system level) but it is not important from Kerberos point of view because Kerberos is not responsible for the identity of your application (or user), it only deals with Kerberos principals and their keys. Headless keytab What you call 'headless keytab' is user principal keys. This already makes an assumption that you have a user principal that corresponds to certain POSIX user it will start services without prompting for password e.g(smokeuser,hdfs,hbase). Headless principals are not bound to a specific host or node, they have the syntax: - @ EXAMPLE.COM For Example: Headless: hdfs-mycluster@EXAMPLE.COM Service keytab Service principal is something that does not need to be a POSIX user,they are mostly applications that have own arrangement on how they run on the OS level and need to interact with the Kerberized cluster. Kerberos services have traditional meaning e.g there could be host/fqdn REALM service principal that 'represents' this host in Kerberos realm. The same service principal may be shared by several applications: for example, both SSSD and SSH daemon use host/fqdn REALM for own needs. SSSD uses it as a client when authenticating against IPA LDAP server using SASL GSSAPI, and SSH daemon uses host/fqdn REALM key to represent itself as a server to incoming SSH clients using GSSAPI or Kerberos authentication methods. Service principals are bound to a specific service and host or node, they have the syntax: / @ EXAMPLE.COM e.g: Service: nn/c6601.ambari.apache.org@EXAMPLE.COM For example in FreeIPA it is recommended to create service principals to represent applications as they are not required to have global POSIX identity associated with them and they are usually running on a specific host. In addition, if they are accepting SASL GSSAPI authentication method to access themselves, a client application will usually build up a target principal based on the hostname they run on, e.g. HTTP/fqdn REALM for a web server running on the host fqdn. Thus, there is clear arrangement between client and server applications on what they expect from each other on Kerberos (or SASL GSSAPI) level. User principals For user principals you can store user's password in a 'headless keytab' to allow some impersonation of the user for certain needs but it is irrelevant from Kerberos level what identity is there, both service and user principals can equally be used at a client side to initiate authentication towards a server. Script for generating headless keytabs Generating a headless keytab uses the same utility using kadmin.local you just specify headless assuming you have already a POSIX see above. kadmin: ktadd -k /etc/security/keytabs/solr-headless-keytab host/prodsolr.exmple.com The above will generate solr-headless-keytab for host prodsolr HTH ... View more

@Steve Kiaie A simple answer is NO you cannot remove the SNN in a non NameNode HA setup reasons below: Secondary NameNode(SNN) and Standby NameNode are mutually exclusive,enabling HA is not mandatory. But, when it is enabled, you can't use Secondary Namenode. So, either Secondary Namenode is enabled OR Standby Namenode is enabled. Secondary NameNode downloads the FsImage and EditLogs from the NameNode and then it merges EditLogs with the Fsimage periodically. It keeps edits log size within a limit. After that, it stores the modified FsImage into persistent storage. So we can use FsImage in case of NameNode failure. Secondary namenode is just a helper for Namenode. It gets the edit logs from the namenode in regular intervals and applies to fsimage. Once it has new fsimage, it copies back to namenode. Namenode will use this fsimage for the next restart, which will reduce the startup time. Secondary Namenode's whole purpose is to have a checkpoint in HDFS. Its just a helper node for namenode. That’s why it also known also as checkpoint node,But, It can't replace namenode on namenode's failure.NameNode is single point of Failure (SPOF). If namenode fails, all clients would unable to read/write files. In such event, whole cluster will be out of service until new namenode is up. The standby namenode provides automatic failover in case Active Namenode (can be simply called 'Namenode' if HA is not enabled) fails.You need a Zookeeper cluster (quorum of 3) to add a Standby namenode hence enabling NameNode HA To overcome this issue; Standby NameNode comes into picture. It does three things: Merging fsimage and edits-log files. (Secondary-namenode's work). Receive online updates of the file system meta-data using journalnoodes. Apply the changes to its memory state and persist them on disks just like the name-node does. Thus at any time the Backup node contains an up-to-date image of the namespace both in memory and on local disk(s). Cluster will switch over to the new name-node (this standby-node) if the active namenode dies High availability feature provides an extra NameNode to hadoop architecture,this feature provides automatic failover. If active NameNode fails, then standby-Namenode takes all the responsibility of active node. And cluster continues to work. I have attached 2 visuals to help you understand HTH ... View more