We use the following topologies right now in HDP 3, shown below. I understand there is authentication and authorization. Prior to using PAM, we used LDAP configurations. Then we ran into... KnoxLdapContextFactory.java (HW had us try identity assertion, null pointer issue hit) KnoxLdapRealm.java (ldap - too many entries, fails, we only get the 'cdisadmin' group as it's in the first page or results, and the code for this does NOT properly page) KnoxPamRealm.java (works, but requires reworking PAM/sssd,). PAM works it seems, but even if I put my own username in the top level Knox policy to deny access, nothing is denied when I access Knox SSO webui's. I don't understand what part of the authorization process is not communicating. In our view, when you pass the PAM authentication stage, you just get access, when what we REALLY want to happen, is Ranger then says yes or no to you getting authorized. PAM module: #%PAM-1.0 auth [user_unknown=ignore success=ok ignore=ignore default=bad] pam_securetty.so #auth       substack     system-auth ############################################### # Imported from: /etc/pam.d/password-auth ############################################### auth        required      pam_env.so auth        sufficient    pam_unix.so nullok try_first_pass auth        sufficient    pam_ldap.so minimum_uid=1000 use_first_pass auth        requisite     pam_succeed_if.so uid >= 1000 quiet #auth        sufficient    pam_sss.so use_first_pass auth        required      pam_deny.so account     required      pam_unix.so broken_shadow account     sufficient    pam_ldap.so minimum_uid=1000 account     sufficient    pam_localuser.so account     sufficient    pam_succeed_if.so uid < 1000 quiet account     [default=bad success=ok user_unknown=ignore] pam_sss.so account     required      pam_permit.so password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type= password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok password    sufficient    pam_ldap.so minimum_uid=1000 try_first_pass #password    sufficient    pam_sss.so use_authtok password    required      pam_deny.so session     optional      pam_keyinit.so revoke session     required      pam_limits.so session     optional      pam_mkhomedir.so umask=0077 -session    optional      pam_systemd.so session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session     required      pam_unix.so session     optional      pam_ldap.so minimum_uid=1000 #session     optional      pam_sss.so knoxsso topology <topology>   <gateway>     <provider>       <role>identity-assertion</role>       <name>Default</name>       <enabled>true</enabled>     </provider>     <provider>       <role>webappsec</role>       <name>WebAppSec</name>       <enabled>true</enabled>       <param>         <name>xframe.options.enabled</name>         <value>true</value>       </param>     </provider>      <provider>       <role>authentication</role>       <name>ShiroProvider</name>       <enabled>true</enabled>       <param>         <name>sessionTimeout</name>         <value>30</value>       </param>       <param>         <name>redirectToUrl</name>         <value>/gateway/knoxsso/knoxauth/login.html</value>       </param>       <param>               <name>restrictedCookies</name>         <value>rememberme,WWW-Authenticate</value>       </param>       <param>         <name>main.pamRealm</name>         <value>org.apache.knox.gateway.shirorealm.KnoxPamRealm</value>       </param>       <param>         <name>main.pamRealm.service</name>         <value>knoxsso</value>        </param>       <param>         <name>urls./**</name>         <value>authcBasic</value>       </param>     </provider>     <provider>        <role>authorization</role>        <name>AclsAuthz</name>        <enabled>true</enabled>      </provider>    </gateway>   <application>     <name>knoxauth</name>   </application>   <service>     <role>KNOXSSO</role>     <param>       <name>knoxsso.cookie.secure.only</name>       <value>true</value>     </param>     <param>       <name>knoxsso.token.ttl</name>       <value>900000</value>     </param>     <param>       <name>knoxsso.redirect.whitelist.regex</name>       <value>^https?:\/\/(.*\.DOMAIN\.COM|localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$</value>     </param>   </service> </topology> default topology: <topology>   <gateway>     <provider>       <role>identity-assertion</role>       <name>Default</name>       <enabled>true</enabled>     </provider>     <provider>       <role>authentication</role>       <name>ShiroProvider</name>       <enabled>true</enabled>       <param>         <name>sessionTimeout</name>         <value>30</value>       </param>       <param>         <name>main.pamRealm</name>         <value>org.apache.hadoop.gateway.shirorealm.KnoxPamRealm</value>       </param>       <param>         <name>main.pamRealm.service</name>         <value>knoxsso</value> </param>       <param>         <name>urls./**</name>         <value>authcBasic</value>       </param>      </provider>      <provider>        <role>authorization</role>        <name>XASecurePDPKnox</name>        <enabled>true</enabled>      </provider>    </gateway>    <service>      <role>AVATICA</role>      <url>http://HOST.DOMAIN.COM:8765</url>    </service>    <service>      <role>DRUID-COORDINATOR-UI</role>      {{druid_coordinator_urls}}    </service>    <service>      <role>DRUID-COORDINATOR</role>      {{druid_coordinator_urls}}    </service>    <service>      <role>DRUID-OVERLORD-UI</role>      {{druid_overlord_urls}}    </service>    <service>      <role>DRUID-OVERLORD</role>      {{druid_overlord_urls}}    </service>    <service>      <role>DRUID-ROUTER</role>      {{druid_router_urls}}    </service>    <service>      <role>DRUID-BROKER</role>      {{druid_broker_urls}}    </service>    <service>      <role>HBASEUI</role>      <url>http://HOST.DOMAIN.COM:16010</url>    </service>    <service>      <role>HDFSUI</role>      <version>2.7.0</version>      <url>http://HOST.DOMAIN.COM:50070/</url>    </service>    <service>      <role>HIVE</role>      <url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url>    </service>    <service>      <role>JOBTRACKER</role>      <url>rpc://{{rm_host}}:{{jt_rpc_port}}</url>    </service>    <service>      <role>JOBHISTORYUI</role>      <url>http://HOST.DOMAIN.COM:19888</url>    </service>    <service>      <role>NAMENODE</role>      <url>{{namenode_address}}</url>    </service>    <service>      <role>OOZIE</role>      <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url>    </service>    <service>      <role>OOZIEUI</role>      <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie/</url>    </service>    <service>      <role>RANGERUI</role>      <url>http://HOST.DOMAIN.COM:6080</url>    </service>    <service>      <role>RESOURCEMANAGER</role>      <url>http://{{rm_host}}:{{rm_port}}/ws</url>    </service>    <service>      <role>SPARKHISTORYUI</role>      <url>http://HOST.DOMAIN.COM:18081</url>    </service>    <service>      <role>WEBHDFS</role>      {{webhdfs_service_urls}}    </service>    <service>      <role>WEBHCAT</role>      <url>http://{{webhcat_server_host}}:{{templeton_port}}/templeton</url>    </service>    <service>      <role>WEBHBASE</role>      <url>http://{{hbase_master_host}}:60080</url>    </service>    <service>      <role>YARNUI</role>      <url>http://HOST.DOMAIN.COM:8088</url>    </service>    <service>      <role>YARNUIV2</role>      <url>http://HOST.DOMAIN.COM:8088</url>    </service>    <service>      <role>ZEPPELINUI</role>      {{zeppelin_ui_urls}}    </service>    <service>      <role>ZEPPELINWS</role>      {{zeppelin_ws_urls}}    </service> </topology> This is all I noticed in gateway.log when testing these scenarios: [mtdeguzis@HOST: knox]$ tailf gateway.log | grep -iE 'mtdeguzis|yarn' 2019-03-22 15:49:07,094 INFO knox.gateway (KnoxPamRealm.java:doGetAuthorizationInfo(126)) - Computed roles/groups: [udaoptst3, udaops, HOST-login, mtdeguzis, cdisadmin] for principal: mtdeguzis 2019-03-22 15:49:07,405 INFO service.knoxsso (WebSSOResource.java:getAuthenticationToken(240)) - About to redirect to original URL: https://HOST.domain.com:8443/gateway/knoxsso-webuis/yarnuiv2 2019-03-22 15:49:07,972 ERROR knox.gateway (UrlRewriteProcessor.java:rewrite(166)) - Failed to rewrite URL: HTTP_ONLY, direction: OUT via rule: YARNUIV2/yarnuiv2/outbound/timeline, status: FAILURE 2019-03-22 15:49:07,973 ERROR knox.gateway (JsonFilterReader.java:filterStreamValue(547)) - Failed to filter value HTTP_ONLY, rule YARNUIV2/yarnuiv2/outbound/timeline: java.lang.NullPointerException 2019-03-22 15:49:08,013 ERROR knox.gateway (UrlRewriteProcessor.java:rewrite(166)) - Failed to rewrite URL: kerberos, direction: OUT via rule: YARNUIV2/yarnuiv2/outbound/timeline, status: FAILURE 2019-03-22 15:49:08,013 ERROR knox.gateway (JsonFilterReader.java:filterStreamValue(547)) - Failed to filter value kerberos, rule YARNUIV2/yarnuiv2/outbound/timeline: java.lang.NullPointerException ... View more

We have searched all over https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/hive-workload/content/hive_workload_management_entity_data_in_sys.html and even the Apache hive code base. How do you monitor what queries are running in what resource pool??? ... View more

In testing out several REST API endpoints for webhcat, as seen at https://cwiki.apache.org/confluence/display/Hive/WebHCat+Reference, it seems that in order to mimic 'show create table' to create a hcatalog table, I would need to combine the extended table info and table properties. Is there a simpler way to do this? Is there an API call that more closely aligns with hive/beeline 'show create table' ? ... View more

We are consuming from Kafka using HDF NiFi in a docker container. We current have this flow attached. The problem is that we wish to bundle up until a max age of 1 hour so that 1 file gets pushed to HDFS. We made this doable by setting min/max files/size of the bin very high, and letting Max Bin Age trigger the bundle completion. As many of you know, HDFS hates small files, especially frequently. The issue with MergeConent, is it holds this in-memory until the bundle is made and pushed to HDFS. If there was a way that the bundle could be built on-disk, that seems ok. Design alternatives? We are thinking of just making a Java application to hanlde Kafka -> HDFS if this fails ... View more

We are consuming from Kafka using HDF NiFi in a docker container. We current have this flow attached. The problem is that we wish to bundle up until a max age of 1 hour so that 1 file gets pushed to HDFS. We made this doable by setting min/max files/size of the bin very high, and letting Max Bin Age trigger the bundle completion. As many of you know, HDFS hates small files, especially frequently. The issue with MergeConent, is it holds this in-memory until the bundle is made and pushed to HDFS. If there was a way that the bundle could be built on-disk, that seems ok. Design alternatives? We are thinking of just making a Java application to hanlde Kafka -> HDFS if this fails. ... View more

Within flume, the definition/channel/sink paradigm I can wrap my head around. It seems with NiFi, you add a syslog listener, ListenSyslog, that mainly connects via a port definition. Does this mean then I need to configure syslog on the NiFi parent host to ingest remove syslogs on a certain port? Docs/links to point me in the right direction? ... View more

Hi, I've implemented a Python wrapper around the webhdfs api (via Knox) for end users to script operations outside the cluster. The one core operation missing I'd like to add is "hadoop fs -copyToLocal'. It seems that only open is supported and possible operations to direct the buffter to a file is possible. Is there any way to copy an HDFS file to a local path using webhdfs? ... View more

It is unfortunate that usersync does not truly sync, and is only additive. I know Ambari has API calls to remove a user from a Group, does Ranger? We would really like to keep a user's Ranger groups in line with their adldap/openldap groups. Struggling to find good information on this. ... View more

Summary. Submitting a PUT request against the Ranger API: https://cwiki.apache.org/confluence/display/RANGER/REST+APIs+for+Policy+Management#RESTAPIsforPolicyManagement-UpdatePolicy. I can create the policy fine, but seems with PUT, I get bad request and cannot determine why. The problem is not the content, as I can copy another policy as test.json and try to PUT that Code snip HEADERS = {'Content-type': 'application/json'} result = requests.put(URL_BASE + URL_POLICY + policy_id, headers=HEADERS, data=open(json_file, 'rb'), auth=requests.auth.HTTPBasicAuth(USERNAME, PASSWORD)) Headers on GET Retrieving and exporting policy 16: <POLICY_NAME> {'transfer-encoding': 'chunked', 'set-cookie': 'RANGERADMINSESSIONID=<ID>; Path=/; HttpOnly, hadoop.auth=; Path=/; Domain=HOST.DOMAIN.COM; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly', 'server': 'Apache-Coyote/1.1', 'date': 'Tue, 29 Aug 2017 14:57:25 GMT', 'x-frame-options': 'DENY', 'content-type': 'application/json', 'www-authenticate': 'Negotiate'} Headers on successful POST {'transfer-encoding': 'chunked', 'set-cookie': 'RANGERADMINSESSIONID=<ID>; Path=/; HttpOnly, hadoop.auth=; Path=/; Domain=HOST.DOMAIN.COM; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly', 'server': 'Apache-Coyote/1.1', 'date': 'Tue, 29 Aug 2017 15:34:04 GMT', 'x-frame-options': 'DENY', 'content-type': 'application/json', 'www-authenticate': 'Negotiate'} Policy applied <POLICY_NUM> Headers on PUT (400 respose) Submitting PUT API request: http://HOST:PORT/service/public/api/policy/<POLICY_NUM>; {'transfer-encoding': 'chunked', 'set-cookie': 'RANGERADMINSESSIONID=<ID>; Path=/; HttpOnly, hadoop.auth=; Path=/; Domain=HOST.DIOMAIN.COM; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly', 'server': 'Apache-Coyote/1.1', 'connection': 'close', 'date': 'Tue, 29 Aug 2017 15:19:58 GMT', 'x-frame-options': 'DENY', 'content-type': 'application/json', 'www-authenticate': 'Negotiate'} The object exists at the endpoint, so a PUT needs made. ## Expected Result Response code 200 ## Actual Result Traceback (most recent call last): File "ranger-policy-manager.py", line 183, in <module> result.raise_for_status() File "/usr/lib/python2.7/site-packages/requests/models.py", line 834, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request ... View more

I've seen Cloudera documentation point out that the URL template should be: URL template: jdbc:hive2://{host}:{port}/{database};AuthMech=1;KrbRealm=FOO.BAR;KrbHostFQDN={server}; But that does not seem to be working. What jars are required? Has anyone gotten this working? Links: http://justnumbersandthings.com/2017-05-06-Dbeaver-Hive-Kerberos.html ... View more

I know how to get the available fields from, parsing them in Python with Element Tree. http://<ip>:6080/service/xusers/users/userName/<userName>; http://<ip>:6080/service/xusers/groups/groupName/<groupName>; http://<ip>:6080/service/xusers/groups/<id>; http://<ip>:6080/service/xusers/users/<id>; <Element 'vxUser' at 0x1453d50> <Element 'createDate' at 0x1453d90> <Element 'id' at 0x1453e10> <Element 'owner' at 0x1453e50> <Element 'updateDate' at 0x1453e90> <Element 'updatedBy' at 0x1453f10> <Element 'description' at 0x1453f50> <Element 'firstName' at 0x1453f90> <Element 'lastName' at 0x1459050> <Element 'isVisible' at 0x1453fd0> <Element 'name' at 0x1459090> <Element 'password' at 0x14590d0> <Element 'userRoleList' at 0x1459110> <Element 'userSource' at 0x1459150> However, the ONE field in Ranger > Users that does not show is the last field in the WebUI: Group. Where is this from? I am looking for the API to get this. I would understand if I had to link some kind of ID from a field above, but I have no idea. The group in our field is an openLDAP group, if that helps. ... View more

We are decommissioning a node, and are stuck with 1 underreplicated block. I checked the metasave report and notcied: Metasave: Blocks waiting for replication: 1 /ranger/audit/hdfs/20170510/hdfs_ranger_audit_<HOST>.log: blk_1286524956_212825612 (replicas: l: 1 d: 0 c: 0 e: 0) 10.230.67.5:50010 : Mis-replicated blocks that have been postponed: Metasave: Blocks being replicated: 0 Metasave: Blocks 0 waiting deletion from 0 datanodes. Corrupt Blocks: Metasave: Number of datanodes: 10 datanode log org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException): Failed to finalize INodeFile hdfs_ranger_audit_<HOST>.log since blocks[12] is non-complete, where blocks=[blk_1286247594_212548250, blk_1286298049_212598705, blk_1286353635_212654291, blk_1286413312_212713968, blk_1286472213_212772869, blk_1286524956_212825612, blk_1286561059_212861715, blk_1286595257_212895934, blk_1286644160_212944837, blk_1286688727_212989404, blk_1286741896_213042573, blk_1286799222_213099899, blk_1286859599_233271569{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=0, replicas=[ReplicaUC[[DISK]DS-cd007f20-570b-4768-afd9-d2dbd92d3c7e:NORMAL:10.230.67.3:50010|RBW]]}]. I pulled up several bug reports, such as https://issues.apache.org/jira/browse/HDFS-11499, but we are having trouble solving this issue. Has anyone gotten this? HDP version: 2.6.0 ... View more

Ambari Version 2.5.0.3 dfsadmin -report shows 2 missing blocks, but apparently hdfs fsck does not show this, nor missing replicas. Is this still a bug? See below links for related topics/bugs. See: https://community.hortonworks.com/questions/6705/ambari-showing-corrupt-blocks-but-not-fsck.html See: https://issues.apache.org/jira/browse/HDFS-8533 ... View more