Update: This is an update to my previous article on the same topic. This covers the new features added in HDP 2.6 (and Zeppelin 0.7). Motivation: Starting HDP 2.6, a new Shiro configuration implementation has been added in Zeppelin to handle LDAP/Active Directory authentication and authorization. It fixes lot of known issues (Bind issue, limited search/filter options, Group based authorization etc.) present in earlier versions and this should be used for any kind of LDAP/AD authentication + authorization going forward. Configuration: 1. While most of the configuration steps remain same from the previous article, the following "shiro_init_content" is where the most of the magic happen: Note: Before pasting this configuration in your Zeppelin configuration, please change the Active Directory details to suit your AD environment. # Sample LDAP configuration, for Active Directory user Authentication, currently tested for single Realm [main] ldapRealm=org.apache.zeppelin.realm.LdapRealm ldapRealm.contextFactory.systemUsername=cn=ldap-reader,ou=ServiceUsers,dc=lab,dc=hortonworks,dc=net ldapRealm.contextFactory.systemPassword=SomePassw0rd ldapRealm.contextFactory.authenticationMechanism=simple ldapRealm.contextFactory.url=ldap://ad.somedomain.net:389 # Ability to set ldap paging Size if needed; default is 100 ldapRealm.pagingSize=200 ldapRealm.authorizationEnabled=true ldapRealm.searchBase=OU=CorpUsers,DC=lab,DC=hortonworks,DC=net ldapRealm.userSearchBase=OU=CorpUsers,DC=lab,DC=hortonworks,DC=net ldapRealm.groupSearchBase=OU=CorpUsers,DC=lab,DC=hortonworks,DC=net ldapRealm.userObjectClass=person ldapRealm.groupObjectClass=group ldapRealm.userSearchAttributeName = sAMAccountName # Set search scopes for user and group. Values: subtree (default), onelevel, object ldapRealm.userSearchScope = subtree ldapRealm.groupSearchScope = subtree ldapRealm.userSearchFilter=(&(objectclass=person)(sAMAccountName={0})) ldapRealm.memberAttribute=member # Format to parse & search group member values in 'memberAttribute' ldapRealm.memberAttributeValueTemplate=CN={0},OU=CorpUsers,DC=lab,DC=hortonworks,DC=net # No need to give userDnTemplate if memberAttributeValueTemplate is provided #ldapRealm.userDnTemplate= # Map from physical AD groups to logical application roles ldapRealm.rolesByGroup = "hadoop-admins":admin_role,"hadoop-users":hadoop_users_role # Force usernames returned from ldap to lowercase, useful for AD ldapRealm.userLowerCase = true # Enable support for nested groups using the LDAP_MATCHING_RULE_IN_CHAIN operator ldapRealm.groupSearchEnableMatchingRuleInChain = true sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager ### If caching of user is required then uncomment below lines cacheManager = org.apache.shiro.cache.MemoryConstrainedCacheManager securityManager.cacheManager = $cacheManager securityManager.sessionManager = $sessionManager securityManager.realms = $ldapRealm # 86,400,000 milliseconds = 24 hour securityManager.sessionManager.globalSessionTimeout = 86400000 shiro.loginUrl = /api/login [urls] # This section is used for url-based security. # You can secure interpreter, configuration and credential information by urls. Comment or uncomment the below urls that you want to hide. # anon means the access is anonymous. # authc means Form based Auth Security # To enfore security, comment the line below and uncomment the next one #/api/version = anon /api/interpreter/** = authc, roles[admin_role,hadoop_users_role] /api/configurations/** = authc, roles[admin_role] /api/credential/** = authc, roles[admin_role,hadoop_users_role] #/** = anon /** = authc Lets discuss the new configuration options here: 2. ldapRealm.rolesByGroup = "hadoop-admins":admin_role,"hadoop-users":hadoop_users_role This line maps the AD groups "hadoop-admins" and "hadoop-users" to custom roles which can be used in [urls] section to control access to various Zeppelin users. Note that the short group names are to be used instead of fully qualified names like "cn=hadoop-admins,OU=CorpUsers,DC=lab,DC=hortonworks,DC=net". The role names can be set to any name but the same names should be used in the [urls] section. 3. ldapRealm.groupSearchEnableMatchingRuleInChain = true A very powerful option to search all the groups that a given user is member of in a single query. An LDAP search query with this option traverses the LDAP group hierarchy till the root to find out all the groups. Specially useful for nested groups. More info can be found here. Caution : This option can cause performance overhead (slow to log in etc.) if LDAP hierarchy is not setup optimally. 4. ldapRealm.userSearchFilter=(&(objectclass=person)(sAMAccountName={0})) Use this search filter to limit scope of user results when looking for user's Distinguished Name (DN). This is used only If userSearchBase and userSearchAttributeName are defined. If these two are not defined, then userDnTemplate is used to look for user's DN. ... View more

Motivation: When Knox is configured for perimeter security, the end users need to depend heavily on cURL tool or the browser to access the Hadoop services exposed via Knox. Similarly, Hive queries can be submitted by using WebHCat (Templeton) service via Knox. User can also set various parameters required for Hive job to run correctly. cURL Command Syntax: Here's the cURL command syntax which can be used to submit a Hive Job via Knox: $curl -ivk -u <username>:<password> -d <Hive parameters> [-d ...] https://<knox-server-FQDN>:8443/gateway/<topology>templeton/v1/hive" Complete list of Hive parameters can be found in WebHCat cURL Command Reference. The most important Hive parameters are: Hive Query : -d execute="<Hive-Query>" OR Hive Program : -d file="/hdfs/path/to/hive/program" Specifies a Hive query string using 'execute' OR HDFS file name of Hive program to run using 'file'. It is mandatory to provide either "execute" OR "file" option. Hive Configuration : -d define="NAME=VALUE" Any Hive configuration values like 'hive.execution.engine' or '' can be set by using 'define'. Multiple 'define's can be provided on cURL command. One caveat, cURL can't seem to be processing the double equal symbol in "define=NAME=VALUE" correctly. It would convert that into "defineNAME=VALUE" erroneously. Fix is to escape one equal symbol with URL-encoded equivalent. Meaning, any 'define' should be provided like this: -d define="hive.execution.engine%3Dmr" Output directory in HDFS : -d statusdir="/hdfs/path/to/output/directory" Specifies a HDFS location where the output (and error) of the Hive job execution will be written to. Once the job is finished (either success or failure), this location can be checked for stdout, stderr and exit code of the Hive query / program. Example: With this knowledge, here's a working example cURL command which submits Hive Select query as a job to the cluster via Knox. The output will be a YARN job id which can be used further to track the job progress in Resource Manager UI. # curl -ivk -u hr1:passw0rd -d execute="select+*+from+hivetest;" -d statusdir="/user/hr1/hive.output7" -d define="hive.execution.engine%3Dmr" "https://knox-server.domain.com:8443/gateway/default/templeton/v1/hive" * About to connect() to knox-server.domain.com port 8443 (#0) * Trying 127.0.0.1... connected * Connected to knox-server.domain.com (127.0.0.1) port 8443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * warning: ignoring value of ssl.verifyhost * skipping SSL peer certificate verification * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: * subject: CN=knox-server.domain.com,OU=Test,O=Hadoop,L=Test,ST=Test,C=US * start date: Apr 07 23:02:54 2017 GMT * expire date: Apr 07 23:02:54 2018 GMT * common name: knox-server.domain.com * issuer: CN=knox-server.domain.com,OU=Test,O=Hadoop,L=Test,ST=Test,C=US * Server auth using Basic with user 'hr1' > POST /gateway/default/templeton/v1/hive HTTP/1.1 > Authorization: Basic aHIxOkJhZc3Mjmq== > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: knox-server.domain.com:8443 > Accept: */* > Content-Length: 98 > Content-Type: application/x-www-form-urlencoded > < HTTP/1.1 200 OK HTTP/1.1 200 OK < Date: Fri, 19 May 2017 02:13:58 GMT Date: Fri, 19 May 2017 02:13:58 GMT < Set-Cookie: JSESSIONID=1k52mpj6ot9rm1nwi2dc9qcvu;Path=/gateway/default;Secure;HttpOnly Set-Cookie: JSESSIONID=1k52mpj6ot9rm1nwi2dc9qcvu;Path=/gateway/default;Secure;HttpOnly < Expires: Thu, 01 Jan 1970 00:00:00 GMT Expires: Thu, 01 Jan 1970 00:00:00 GMT < Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Thu, 18-May-2017 02:13:58 GMT Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Thu, 18-May-2017 02:13:58 GMT < Content-Type: application/json; charset=UTF-8 Content-Type: application/json; charset=UTF-8 < Server: Jetty(7.6.0.v20120127) Server: Jetty(7.6.0.v20120127) < Content-Length: 31 Content-Length: 31 < * Connection #0 to host knox-server.domain.com left intact * Closing connection #0 {"id":"job_1495157584958_0016"} Hope this helps you out! ... View more