The HDFS Balancer program can be invoked to rebalance HDFS blocks when data nodes are added to or removed from the cluster. For more information about the HDFS balancer, see this HCC article. Since Kerberos tickets are designed to expire, a common question that arises in secure clusters is whether one needs to account for ticket expiration (namely, TGT) when invoking long-running Balancer jobs. To cut to the chase: the answer depends on just how long the job takes to run. Let's discuss some background context (I am referencing Chris Nauroth's excellent answer on Stack Overflow as well as HDFS-9698 below). The primary use case for Kerberos authentication in the Hadoop ecosystem is Hadoop's RPC framework, which uses SASL for authentication. Many daemon processes, i.e., non-interactive processes, call UserGroupInformation#loginUserFromKeytab at process startup, using a keytab to authenticate to the KDC. Moreover, Hadoop implements an automatic re-login mechanism directly inside the RPC client layer. The code for this is visible in the RPC Client#handleSaslConnectionFailure method: // try re-login if (UserGroupInformation.isLoginKeytabBased()) { UserGroupInformation.getLoginUser().reloginFromKeytab(); } else if (UserGroupInformation.isLoginTicketBased()) { UserGroupInformation.getLoginUser().reloginFromTicketCache(); } However, the Balancer is not designed to be run as a daemon (in Hadoop 2.7.3, i.e., HDP 2.6 and earlier)! Please see HDFS-9804, which introduces this capability. With this in place, the Balancer would log in with a keytab and the above re-login mechanism would take care of everything. Since the Balancer is designed to be run interactively, the assumption is that kinit has already run, and there is a TGT sitting in the ticket cache. Now we need to understand some Kerberos configuration settings, in particular the distinction between ticket_lifetime and renew_lifetime. Every ticket, including TGTs, have a ticket_lifetime (usually around 18 hours), this strikes the balance between annoying the user by requiring they log in multiple times during their workday and mitigating the risk of TGTs being stolen (note there is separate support for preventing replay of authenticators). A ticket can be renewed, but only up to its renew_lifetime (usually around 7 days). which it can be renewed to extend to a maximum value of the later. Since a TGT is generated by the user and provided to the balancer (which means in the balancer context, UserGroupInformation.isLoginTicketBased() == true), client#handleSaslConnectionFailure is behaving correctly on extending the ticket_lifetime. But there's no way to extend beyond the renew_lifetime! To summarize, if your balancer job is going to run longer than the configured renew_lifetime in your environment (a week by default), then you need to worry about ticket renewal (or you need HDFS-9804). Otherwise, you will be fine relying on the RPC framework to renew the TGT's ticket_lifetime (as long it doesn't eclipse the renew_lifetime). ... View more

This article assumes you have already identified the GUID associated with your Atlas entity and that the tag in question you wish to associate to this entity already exists. For more information, on how to identify the GUID for your entity, please see this HCC article by Mark Johnson. For example, if we wanted to add a new tag to all hive tables containing the word "claim", we could use the Full Text Search capability to identify all entities (replace admin:admin with the username:password values for an Atlas administrator): curl -u admin:admin http://$ATLAS_SERVER:21000/api/atlas/discovery/search/fulltext?query=claim Now we are ready to assign a new tag, called "PII", to all such entities. We are using the v1 API for Atlas in HDP 2.5. In HDP 2.6, the Atlas API has been revamped and simplified. Please see this HCC article for more details. Let's construct an example using the first GUID, f7a24ec6-5b0c-42d8-ba8a-1ac654d24f45. We will use the traits resource associated with this entity, and POST our payload to this endpoint. curl -u admin:admin http://$ATLAS_SERVER:21000/api/atlas/entities/f7a24ec6-5b0c-42d8-ba8a-1ac654d24f45/traits -X POST -H 'Content-Type: application/json' --data-binary '{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Struct","typeName":"PII","values":{}}' We can now query the traits of this entity using a GET request. curl -u admin:admin http://$ATLAS_SERVER:21000/api/atlas/entities/f7a24ec6-5b0c-42d8-ba8a-1ac654d24f45/traits We can also see the tag now exists in the Atlas UI: ... View more

Q1) What is the difference between using the programs kadmin.local and kadmin, respectively? A1) The difference between kadmin and kadmin.local is that kadmin.local directly accesses the KDC database (a file called principal in /var/kerberos/krb5kdc) and does not use Kerberos for authentication. Since kadmin.local directly accesses the KDC database, it must be run directly on the master KDC as a user with sufficient permissions to read the KDC database. When using kadmin to administer the KDC database, the user is communicating with the kadmind daemon over the network and will authenticate using Kerberos to the KDC master. Hence, the first principal must already exist before connecting over the network—this is the motivation for the existence of kadmin.local. This also means the KDC administrator will need to kinit as the administrator principal (by default, admin/admin@REALM) to run kadmin from another box. Q2) How can we restrict which users can administer the MIT-KDC service? A2) There is an ACL, /var/kerberos/krb5kdc/kadm5.acl, which authorizes access to the KDC database. By default, there is a line: */admin@REALM * This provides authorization for all operations to all Kerberos principals that have an instance matching that form, like admin/admin@REALM Q3) What local (POSIX) permissions are needed by MIT-KDC administrators? It’s important to make a distinction between the user's network account and the associated Kerberos principal. The network account needs permissions to read the KDC database when running kadmin.local, per the above. If this user has sudo access on this box then this is sufficient (the KDC database is usually only readable by root). Tangentially, this is a good motivation to run the KDC on a different box than one of the cluster hosts, for separation of concerns between cluster and KDC administrators. ... View more

Imagine we have a use case of exchanging data with an external party via AWS S3 buckets: we want to push these messages into our internal Kafka cluster, after enriching each message with additional metadata, in an event-driven fashion. In AWS, this is supported by associating notifications with an S3 bucket. These destinations can make use of a few different destinations, namely, SQS, SNS, and Lambda. We'll focus on the SQS approach, and will make use of NiFi's GetSQS processor. To configure this in AWS, navigate to the S3 bucket and then to the Properties tab, and scroll down to Advanced settings > Events. You'll need to create an SQS queue for this purpose. With this configured, a new SQS message will appear any time an object is created within our S3 bucket. We need to configure some IAM policies in order for our NiFi data flow to be authorized to read from the S3 bucket and to read from the SQS queue. We will authenticate from NiFi using the Access Key and Secret Key associated with a particular IAM user. First, the IAM policy for reading from the S3 bucket called nifi: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:ListBucketMultipartUploads" ], "Resource": [ "arn:aws:s3:::nifi" ] }, { "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::nifi/*" ] } ] } Second, the IAM policy for reading from--as well as deleting from since we'll configure GetSQS to auto-delete received messages--the SQS queue. We'll need the ARN and URL associated with our SQS queue, these can be retrieved from the SQS Management Console and navigating to the SQS queue name we created above. Note: we could harden this by restricting the permitted SQS actions further. { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sqs:*", "Resource": "arn:aws:sqs:$REGION:$UUID:$QUEUE_NAME" } ] } You will then need to attach these policies to your IAM user via the IAM Management Console. We are now ready to build the NiFi data flow: For the GetSQS processor, just use the SQS queue URL from the SQS Management console that we retrieved above, the Region, and the Access Key and Secret Key associated with the IAM user. We'll use SplitJSON to extract the name of the file associated with the SQS notification. We'll need this to fetch the object from S3. ExtractText is used to associate the result of the JsonPath expression with a new custom attribute, $filename: Which we'll pass into the FetchS3Object processor: Finally, we can enrich the message with UpdateAttribute using Advanced > Rules and push to our Kafka topic using the PublishKafka processor. I'd like to credit this blog post: https://adamlamar.github.io/2016-01-30-monitoring-an-s3-bucket-in-apache-nifi/ ... View more

In Zeppelin LDAP Authentication with OpenLDAP and How to Set Up OpenLDAP we've shown how to use LDAP Authentication with Zeppelin. In this article, we'll harden that configuration by ensuring that Zeppelin and OpenLDAP communicate over LDAPS. LDAPS is a secure protocol that uses TLS to assure authenticity, confidentiality, and integrity of communications. This prevents man-in-the-middle attacks that sniff traffic to discover LDAP credentials communicated in plaintext, which could compromise the security of the cluster. The first step is to modify the configuration of the OpenLDAP server, as root, to expose LDAPS connectivity, we'll need to modify /etc/openldap/ldap.conf. Please recall that we created /etc/openldap/certs/myldap.field.hortonworks.com.cert in the How to Set Up OpenLDAP article #TLS_CACERTDIR /etc/openldap/certs TLS_CACERT /etc/openldap/certs/myldap.field.hortonworks.com.cert URI ldaps://myldap.field.hortonworks.com ldap://myldap.field.hortonworks.com BASE dc=field,dc=hortonworks,dc=com We also need to modify /etc/sysconfig/slapd : SLAPD_URLS="ldapi:/// ldap:/// ldaps:///" Then restart slapd: systemctl restart slapd You can confirm that slapd is listening on 636: netstat -anp | grep 636 Finally, confirm TLS connectivity and secure ldapsearch (with the appropriate bind user and password from the previous articles): # should succeed openssl s_client -connect myldap.field.hortonworks.com:636 </dev/null # should succeed ldapsearch -H ldaps://myldap.field.hortonworks.com:636 -D cn=ldapadm,dc=field,dc=hortonworks,dc=com -w $password -b "ou=People,dc=field,dc=hortonworks,dc=com" The next step is the client-side configuration changes. Since we are using a self-signed certificate for the OpenLDAP server, we need to import this into the Java truststore, called cacerts, which is in /etc/pki/ca-trust/extracted/java on my CentOS 7 system. Copy the myldap.field.hortonworks.com.cert file from the OpenLDAP server to the Zeppelin server (this file does not contain sensitive key material, only public keys), and run (making sure you set this certificate to be trusted): keytool -import -alias myldap -file /etc/security/certificates/myldap.field.hortonworks.com.cert -keystore cacerts Otherwise, you will see errors like Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed Lastly, in Ambari, we just need to make one small change to the shiro.ini configuration in Zeppelin > Config > Advanced zeppelin-env > shiro_ini_content : ldapRealm.contextFactory.url = ldaps://myldap.field.hortonworks.com:636 Note the protocol change to LDAPS and the port number change to 636. To test, restart the Zeppelin service and confirm that users can still log in to the Zeppelin UI. ... View more

OpenLDAP is an open-source implementation of the Lightweight Directory Access Protocol and is used for central management of accounts (users, hosts, and services) and can be used in concert with a KDC to provide authentication within the Hadoop ecosystem. Fundamentally, LDAP functions like a database in many ways and can be used to store any information. We will assume that you have a fresh CentOS 7 host available that will host OpenLDAP. Make sure you have network connectivity between any clients and this server and that DNS resolution is working. Let's ssh to the host, and install, as root, the packages we need with yum: yum -y install openldap compat-openldap openldap-clients openldap-servers openldap-servers-sql openldap-devel We'll also start the LDAP daemon (called slapd) and enable it to auto-start on system boot: systemctl start slapd.service systemctl enable slapd.service Next, run the slappasswd command to create an LDAP root password. Please take note of this root password, the entire hashed value that is returned as output and starts {SSHA}, as you'll use it throughout this article. We'll now configure the OpenLDAP server in a couple of steps. We'll create LDIF text files and then use the ldapmodify command to push the configuration to the server. These will ultimately land in /etc/openldap/slapd.d but the files shouldn't be edited manually. The first file will update the variables for olcSuffix, the domain name for which your server LDAP server provides account information, and olcRootDN, the root distinguished name (DN) user who has unrestricted administrative access. My domain is field.hortonworks.com, or dc=field,dc=hortonworks,dc=com and my root DN is cn=ldapadm,dc=field,dc=hortonworks,dc=com. Create the following db.ldif file using vi or your favorite editor. dn: olcDatabase={2}hdb,cn=config changetype: modify replace: olcSuffix olcSuffix: dc=field,dc=hortonworks,dc=com dn: olcDatabase={2}hdb,cn=config changetype: modify replace: olcRootDN olcRootDN: cn=ldapadm,dc=field,dc=hortonworks,dc=com dn: olcDatabase={2}hdb,cn=config changetype: modify replace: olcRootPW olcRootPW: {SSHA}theHashedPasswordValueFromSlapPasswd We'll then push this config: ldapmodify -Y EXTERNAL -H ldapi:/// -f db.ldif We'll next restrict monitor access to the ldapadm user: dn: olcDatabase={1}monitor,cn=config changetype: modify replace: olcAccess olcAccess: {0}to * by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external, cn=auth" read by dn.base="cn=ldapadm,dc=field,dc=hortonworks,dc=com" read by * none And push that config change: ldapmodify -Y EXTERNAL -H ldapi:/// -f monitor.ldif In order to communicate securely with the OpenLDAP server, we'll need a certificate and associated private key. These would likely be obtained from our PKI Administrator in a production environment, but a self-signed certificate and associated private key can be created in development environments, using a command like below: openssl req -new -x509 -nodes -out /etc/openldap/certs/myldap.field.hortonworks.com.cert -keyout /etc/openldap/certs/myldap.field.hortonworks.com.key -days 365 Set the owner and group permissions to ldap:ldap for both files. We'll then create certs.ldif to configure OpenLDAP for secure communication over LDAPS: dn: cn=config changetype: modify replace: olcTLSCertificateFile olcTLSCertificateFile: /etc/openldap/certs/myldap.field.hortonworks.com.cert dn: cn=config changetype: modify replace: olcTLSCertificateKeyFile olcTLSCertificateKeyFile: /etc/openldap/certs/myldap.field.hortonworks.com.key We can then push the config file and finally test the configuration: ldapmodify -Y EXTERNAL -H ldapi:/// -f certs.ldif slaptest -u We're now ready to set up the initial LDAP database: First, copy the sample database configuration file to /var/lib/ldap and update the file permissions. cp /usr/share/openldap-servers/DB_CONFIG.example /var/lib/ldap/DB_CONFIG chown ldap:ldap /var/lib/ldap/* Next, add the cosine and nis LDAP schemas. ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/cosine.ldif ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/nis.ldif ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/inetorgperson.ldif Finally, generate base.ldif file for your domain. dn: dc=field,dc=hortonworks,dc=com dc: field objectClass: top objectClass: domain dn: cn=ldapadm,dc=field,dc=hortonworks,dc=com objectClass: organizationalRole cn: ldapadm description: LDAP Manager dn: ou=People,dc=field,dc=hortonworks,dc=com objectClass: organizationalUnit ou: People dn: ou=Group,dc=field,dc=hortonworks,dc=com objectClass: organizationalUnit ou: Group We'll now push these changes to OpenLDAP using the ldapadm user (sometimes referred to as the bind user): ldapadd -x -W -D "cn=ldapadm,dc=field,dc=hortonworks,dc=com" -f base.ldif You'll be prompted for the root password. From here, I prefer to use a GUI to create additional users. Apache Directory Studio is a nice multi-platform tool and can be downloaded here. Within Apache Directory Studio, you can create a new connection in the lower left-hand pane and use the following configuration: With the following authentication information: Once you connect successfully you can create your organizational structure and users accordingly. These steps are based on the valuable tutorial provided here. ... View more

Apache Ranger uses an embedded Tomcat server to provide the Web UI functionality for administration of Ranger. A previous HCC article provided details on maintenance of the log files that are managed by the log4j configuration, including xa_portal.log, ranger_admin_perf.log, xa_portal_sql.log. We're going to focus on maintenance of the access_log* logs that get automatically generated by Tomcat, but which are not managed by this log4j configuration. With embedded Tomcat, the configuration is contained within the code for the AccessLogValve (as you can see, it uses an hourly rotation pattern unless overridden by ranger.accesslog.dateformat). We'll use the logrotate application in CentOS/RHEL to manage these access_log* logs as the number of files can grow large without rotation and removal in place. You can check to see how many of these files you have on your Ranger Admin node by running (there would be one access_log* file per hour for each day during which the service has ran continuously): ls /var/log/ranger/admin | cut -d '.' -f 1 | uniq -c Within /etc/logrotate.d, we'll create a configuration specific to these Ranger logs, as the configuration for logrotate, in /etc/logrotate.conf by default, will include these application-spcific configurations as well. Create a new file (as root) ranger_access in /etc/logrotate.d in your favorite editor and then insert: /var/log/ranger/admin/access_log* { daily copytruncate compress dateext rotate 5 maxage 7 olddir /var/log/ranger/admin/old missingok } This is just an example logrotate configuration. I'll make note of a couple items, please see the man page for details on each of these options and some additional examples. The copytruncate option ensures that Tomcat can keep writing to the same file handle (as opposed to writing to a newly-created file which requires recycling Tomcat) The compress option will use gzip by default Maxage limits how old the files are that will be kept Olddir indicates that logs are moved into the directory for rotation Logrotate will be invoked daily as a cronjob by default, due to the existence of the logrotate file in /etc/cron.daily. You can run logrotate manually by specifying the configuration: sudo /usr/sbin/logrotate /etc/logrotate.conf Note that logrotate keeps the state of files in the /var/lib/logrotate.status, and it uses the date of last execution captured there as the reference of what to do with a logfile. You can also run logrotate with the -d flag to test your configuration (this won't actually do anything, it will just produce output regarding what would happen). sudo /usr/sbin/logrotate -d /etc/logrotate.conf 2> /tmp/logrotate.debug As a result of this configuration, only 5 days worth of logs are kept, they're kept in the ./old directory, and they're compressed. This ensures that the Ranger admin access_log* logs data does not grow unmanageably large. ... View more

In order to secure access to the Zeppelin UI, we will want to enable TLS (as well as authentication) to ensure confidentiality of communication and to assure the identity of the Zeppelin server. Zeppelin uses Jetty as the underlying HTTP server, so we'll refer to Jetty documentation. In this how-to we'll use a self-signed certificate. In Production environments, you will likely obtain a CA-issued certificate or a trusted root certificate from your PKI team specific to your environment. Since self-signed certificates won't be trusted by your browser by default, we'll show how to trust this certificate on OS X 10.11.6 with Chrome version 55.0.2883.95 (other OS/browser combinations are out of the scope of this article). To generate the self-signed certificate, we'll use the openssl and keytool utilities as follows (see this Jetty doc for reference): openssl genrsa -des3 -out zeppelin.key openssl req -new -x509 -key zeppelin.key -out zeppelin.crt keytool -keystore keystore -import -alias zeppelin -file zeppelin.crt -trustcacerts openssl pkcs12 -inkey zeppelin.key -in zeppelin.crt -export -out zeppelin.pkcs12 keytool -importkeystore -srckeystore zeppelin.pkcs12 -srcstoretype PKCS12 -destkeystore keystore These steps, respectively: 1) create a new private key 2) create a new self-signed certificate using this key 3) imports this self-signed certificate into a new keystore (called "keystore") 4) creates a PCKS12 file that combines the private key and certificate chain 5) converts this PCKS12 file to JKS format and imports it into the keystore We'll then need to move this keystore to the appropriate location with the appropriate ownership and permissions: mv keystore /usr/hdp/zeppelin-server/conf chown zeppelin:zeppelin /usr/hdp/zeppelin-server/conf Finally, we'll configure Zeppelin to use TLS in Ambari. There is currently a bug affecting HDP 2.5.0 and 2.5.3 regarding using relative paths for the keystore and truststore. This bug was introduced by ZEPPELIN-1319, namely, when using a relative path like conf/keystore, Zeppelin server is unreachable and the error in the logs is as below. ZEPPELIN-1810 fixes the bug introduced by ZEPPELIN-1319. The error looks like: FAILED SslContextFactory@6cd166b8(/usr/hdp/current/zeppelin-server/conf/null,/usr/hdp/current/zeppelin-server/conf/null): java.io.FileNotFoundException: /etc/zeppelin/2.5.0.0-1245/0/null (No such file or directory) However, with absolute paths for the keystore and truststore paths, such as /usr/hdp/current/zeppelin-server/conf/keystore, Zeppelin server starts normally and is reachable over HTTPS. Now we need to ensure that our Chrome browser trusts this self-signed certificate. We need to copy the certificate to our Desktop (click the broken HTTPS link > Details > View Certificate and drag and drop to the desktop). We can then import the certificate into our OS X keychain and set it as trusted: Make sure you restart Chrome. After doing so, you should see the green lock icon next to the HTTPS URL and should no longer see a browser warning, ... View more