When submitting a message, such as here, we have access to nice formatting options, but the support portal does not. I especially am looking for code formatting.   ... View more

We use the following topologies right now in HDP 3, shown below. I understand there is authentication and authorization. Prior to using PAM, we used LDAP configurations. Then we ran into... KnoxLdapContextFactory.java (HW had us try identity assertion, null pointer issue hit) KnoxLdapRealm.java (ldap - too many entries, fails, we only get the 'cdisadmin' group as it's in the first page or results, and the code for this does NOT properly page) KnoxPamRealm.java (works, but requires reworking PAM/sssd,). PAM works it seems, but even if I put my own username in the top level Knox policy to deny access, nothing is denied when I access Knox SSO webui's. I don't understand what part of the authorization process is not communicating. In our view, when you pass the PAM authentication stage, you just get access, when what we REALLY want to happen, is Ranger then says yes or no to you getting authorized. PAM module: #%PAM-1.0 auth [user_unknown=ignore success=ok ignore=ignore default=bad] pam_securetty.so #auth       substack     system-auth ############################################### # Imported from: /etc/pam.d/password-auth ############################################### auth        required      pam_env.so auth        sufficient    pam_unix.so nullok try_first_pass auth        sufficient    pam_ldap.so minimum_uid=1000 use_first_pass auth        requisite     pam_succeed_if.so uid >= 1000 quiet #auth        sufficient    pam_sss.so use_first_pass auth        required      pam_deny.so account     required      pam_unix.so broken_shadow account     sufficient    pam_ldap.so minimum_uid=1000 account     sufficient    pam_localuser.so account     sufficient    pam_succeed_if.so uid < 1000 quiet account     [default=bad success=ok user_unknown=ignore] pam_sss.so account     required      pam_permit.so password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type= password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok password    sufficient    pam_ldap.so minimum_uid=1000 try_first_pass #password    sufficient    pam_sss.so use_authtok password    required      pam_deny.so session     optional      pam_keyinit.so revoke session     required      pam_limits.so session     optional      pam_mkhomedir.so umask=0077 -session    optional      pam_systemd.so session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session     required      pam_unix.so session     optional      pam_ldap.so minimum_uid=1000 #session     optional      pam_sss.so knoxsso topology <topology>   <gateway>     <provider>       <role>identity-assertion</role>       <name>Default</name>       <enabled>true</enabled>     </provider>     <provider>       <role>webappsec</role>       <name>WebAppSec</name>       <enabled>true</enabled>       <param>         <name>xframe.options.enabled</name>         <value>true</value>       </param>     </provider>      <provider>       <role>authentication</role>       <name>ShiroProvider</name>       <enabled>true</enabled>       <param>         <name>sessionTimeout</name>         <value>30</value>       </param>       <param>         <name>redirectToUrl</name>         <value>/gateway/knoxsso/knoxauth/login.html</value>       </param>       <param>               <name>restrictedCookies</name>         <value>rememberme,WWW-Authenticate</value>       </param>       <param>         <name>main.pamRealm</name>         <value>org.apache.knox.gateway.shirorealm.KnoxPamRealm</value>       </param>       <param>         <name>main.pamRealm.service</name>         <value>knoxsso</value>        </param>       <param>         <name>urls./**</name>         <value>authcBasic</value>       </param>     </provider>     <provider>        <role>authorization</role>        <name>AclsAuthz</name>        <enabled>true</enabled>      </provider>    </gateway>   <application>     <name>knoxauth</name>   </application>   <service>     <role>KNOXSSO</role>     <param>       <name>knoxsso.cookie.secure.only</name>       <value>true</value>     </param>     <param>       <name>knoxsso.token.ttl</name>       <value>900000</value>     </param>     <param>       <name>knoxsso.redirect.whitelist.regex</name>       <value>^https?:\/\/(.*\.DOMAIN\.COM|localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$</value>     </param>   </service> </topology> default topology: <topology>   <gateway>     <provider>       <role>identity-assertion</role>       <name>Default</name>       <enabled>true</enabled>     </provider>     <provider>       <role>authentication</role>       <name>ShiroProvider</name>       <enabled>true</enabled>       <param>         <name>sessionTimeout</name>         <value>30</value>       </param>       <param>         <name>main.pamRealm</name>         <value>org.apache.hadoop.gateway.shirorealm.KnoxPamRealm</value>       </param>       <param>         <name>main.pamRealm.service</name>         <value>knoxsso</value> </param>       <param>         <name>urls./**</name>         <value>authcBasic</value>       </param>      </provider>      <provider>        <role>authorization</role>        <name>XASecurePDPKnox</name>        <enabled>true</enabled>      </provider>    </gateway>    <service>      <role>AVATICA</role>      <url>http://HOST.DOMAIN.COM:8765</url>    </service>    <service>      <role>DRUID-COORDINATOR-UI</role>      {{druid_coordinator_urls}}    </service>    <service>      <role>DRUID-COORDINATOR</role>      {{druid_coordinator_urls}}    </service>    <service>      <role>DRUID-OVERLORD-UI</role>      {{druid_overlord_urls}}    </service>    <service>      <role>DRUID-OVERLORD</role>      {{druid_overlord_urls}}    </service>    <service>      <role>DRUID-ROUTER</role>      {{druid_router_urls}}    </service>    <service>      <role>DRUID-BROKER</role>      {{druid_broker_urls}}    </service>    <service>      <role>HBASEUI</role>      <url>http://HOST.DOMAIN.COM:16010</url>    </service>    <service>      <role>HDFSUI</role>      <version>2.7.0</version>      <url>http://HOST.DOMAIN.COM:50070/</url>    </service>    <service>      <role>HIVE</role>      <url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url>    </service>    <service>      <role>JOBTRACKER</role>      <url>rpc://{{rm_host}}:{{jt_rpc_port}}</url>    </service>    <service>      <role>JOBHISTORYUI</role>      <url>http://HOST.DOMAIN.COM:19888</url>    </service>    <service>      <role>NAMENODE</role>      <url>{{namenode_address}}</url>    </service>    <service>      <role>OOZIE</role>      <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url>    </service>    <service>      <role>OOZIEUI</role>      <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie/</url>    </service>    <service>      <role>RANGERUI</role>      <url>http://HOST.DOMAIN.COM:6080</url>    </service>    <service>      <role>RESOURCEMANAGER</role>      <url>http://{{rm_host}}:{{rm_port}}/ws</url>    </service>    <service>      <role>SPARKHISTORYUI</role>      <url>http://HOST.DOMAIN.COM:18081</url>    </service>    <service>      <role>WEBHDFS</role>      {{webhdfs_service_urls}}    </service>    <service>      <role>WEBHCAT</role>      <url>http://{{webhcat_server_host}}:{{templeton_port}}/templeton</url>    </service>    <service>      <role>WEBHBASE</role>      <url>http://{{hbase_master_host}}:60080</url>    </service>    <service>      <role>YARNUI</role>      <url>http://HOST.DOMAIN.COM:8088</url>    </service>    <service>      <role>YARNUIV2</role>      <url>http://HOST.DOMAIN.COM:8088</url>    </service>    <service>      <role>ZEPPELINUI</role>      {{zeppelin_ui_urls}}    </service>    <service>      <role>ZEPPELINWS</role>      {{zeppelin_ws_urls}}    </service> </topology> This is all I noticed in gateway.log when testing these scenarios: [mtdeguzis@HOST: knox]$ tailf gateway.log | grep -iE 'mtdeguzis|yarn' 2019-03-22 15:49:07,094 INFO knox.gateway (KnoxPamRealm.java:doGetAuthorizationInfo(126)) - Computed roles/groups: [udaoptst3, udaops, HOST-login, mtdeguzis, cdisadmin] for principal: mtdeguzis 2019-03-22 15:49:07,405 INFO service.knoxsso (WebSSOResource.java:getAuthenticationToken(240)) - About to redirect to original URL: https://HOST.domain.com:8443/gateway/knoxsso-webuis/yarnuiv2 2019-03-22 15:49:07,972 ERROR knox.gateway (UrlRewriteProcessor.java:rewrite(166)) - Failed to rewrite URL: HTTP_ONLY, direction: OUT via rule: YARNUIV2/yarnuiv2/outbound/timeline, status: FAILURE 2019-03-22 15:49:07,973 ERROR knox.gateway (JsonFilterReader.java:filterStreamValue(547)) - Failed to filter value HTTP_ONLY, rule YARNUIV2/yarnuiv2/outbound/timeline: java.lang.NullPointerException 2019-03-22 15:49:08,013 ERROR knox.gateway (UrlRewriteProcessor.java:rewrite(166)) - Failed to rewrite URL: kerberos, direction: OUT via rule: YARNUIV2/yarnuiv2/outbound/timeline, status: FAILURE 2019-03-22 15:49:08,013 ERROR knox.gateway (JsonFilterReader.java:filterStreamValue(547)) - Failed to filter value kerberos, rule YARNUIV2/yarnuiv2/outbound/timeline: java.lang.NullPointerException ... View more

We have searched all over https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/hive-workload/content/hive_workload_management_entity_data_in_sys.html and even the Apache hive code base. How do you monitor what queries are running in what resource pool??? ... View more

We are consuming from Kafka using HDF NiFi in a docker container. We current have this flow attached. The problem is that we wish to bundle up until a max age of 1 hour so that 1 file gets pushed to HDFS. We made this doable by setting min/max files/size of the bin very high, and letting Max Bin Age trigger the bundle completion. As many of you know, HDFS hates small files, especially frequently. The issue with MergeConent, is it holds this in-memory until the bundle is made and pushed to HDFS. If there was a way that the bundle could be built on-disk, that seems ok. Design alternatives? We are thinking of just making a Java application to hanlde Kafka -> HDFS if this fails ... View more

We are consuming from Kafka using HDF NiFi in a docker container. We current have this flow attached. The problem is that we wish to bundle up until a max age of 1 hour so that 1 file gets pushed to HDFS. We made this doable by setting min/max files/size of the bin very high, and letting Max Bin Age trigger the bundle completion. As many of you know, HDFS hates small files, especially frequently. The issue with MergeConent, is it holds this in-memory until the bundle is made and pushed to HDFS. If there was a way that the bundle could be built on-disk, that seems ok. Design alternatives? We are thinking of just making a Java application to hanlde Kafka -> HDFS if this fails. ... View more

Within flume, the definition/channel/sink paradigm I can wrap my head around. It seems with NiFi, you add a syslog listener, ListenSyslog, that mainly connects via a port definition. Does this mean then I need to configure syslog on the NiFi parent host to ingest remove syslogs on a certain port? Docs/links to point me in the right direction? ... View more

Hi, I've implemented a Python wrapper around the webhdfs api (via Knox) for end users to script operations outside the cluster. The one core operation missing I'd like to add is "hadoop fs -copyToLocal'. It seems that only open is supported and possible operations to direct the buffter to a file is possible. Is there any way to copy an HDFS file to a local path using webhdfs? ... View more

Summary. Submitting a PUT request against the Ranger API: https://cwiki.apache.org/confluence/display/RANGER/REST+APIs+for+Policy+Management#RESTAPIsforPolicyManagement-UpdatePolicy. I can create the policy fine, but seems with PUT, I get bad request and cannot determine why. The problem is not the content, as I can copy another policy as test.json and try to PUT that Code snip HEADERS = {'Content-type': 'application/json'} result = requests.put(URL_BASE + URL_POLICY + policy_id, headers=HEADERS, data=open(json_file, 'rb'), auth=requests.auth.HTTPBasicAuth(USERNAME, PASSWORD)) Headers on GET Retrieving and exporting policy 16: <POLICY_NAME> {'transfer-encoding': 'chunked', 'set-cookie': 'RANGERADMINSESSIONID=<ID>; Path=/; HttpOnly, hadoop.auth=; Path=/; Domain=HOST.DOMAIN.COM; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly', 'server': 'Apache-Coyote/1.1', 'date': 'Tue, 29 Aug 2017 14:57:25 GMT', 'x-frame-options': 'DENY', 'content-type': 'application/json', 'www-authenticate': 'Negotiate'} Headers on successful POST {'transfer-encoding': 'chunked', 'set-cookie': 'RANGERADMINSESSIONID=<ID>; Path=/; HttpOnly, hadoop.auth=; Path=/; Domain=HOST.DOMAIN.COM; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly', 'server': 'Apache-Coyote/1.1', 'date': 'Tue, 29 Aug 2017 15:34:04 GMT', 'x-frame-options': 'DENY', 'content-type': 'application/json', 'www-authenticate': 'Negotiate'} Policy applied <POLICY_NUM> Headers on PUT (400 respose) Submitting PUT API request: http://HOST:PORT/service/public/api/policy/<POLICY_NUM>; {'transfer-encoding': 'chunked', 'set-cookie': 'RANGERADMINSESSIONID=<ID>; Path=/; HttpOnly, hadoop.auth=; Path=/; Domain=HOST.DIOMAIN.COM; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly', 'server': 'Apache-Coyote/1.1', 'connection': 'close', 'date': 'Tue, 29 Aug 2017 15:19:58 GMT', 'x-frame-options': 'DENY', 'content-type': 'application/json', 'www-authenticate': 'Negotiate'} The object exists at the endpoint, so a PUT needs made. ## Expected Result Response code 200 ## Actual Result Traceback (most recent call last): File "ranger-policy-manager.py", line 183, in <module> result.raise_for_status() File "/usr/lib/python2.7/site-packages/requests/models.py", line 834, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request ... View more

I've seen Cloudera documentation point out that the URL template should be: URL template: jdbc:hive2://{host}:{port}/{database};AuthMech=1;KrbRealm=FOO.BAR;KrbHostFQDN={server}; But that does not seem to be working. What jars are required? Has anyone gotten this working? Links: http://justnumbersandthings.com/2017-05-06-Dbeaver-Hive-Kerberos.html ... View more

I know how to get the available fields from, parsing them in Python with Element Tree. http://<ip>:6080/service/xusers/users/userName/<userName>; http://<ip>:6080/service/xusers/groups/groupName/<groupName>; http://<ip>:6080/service/xusers/groups/<id>; http://<ip>:6080/service/xusers/users/<id>; <Element 'vxUser' at 0x1453d50> <Element 'createDate' at 0x1453d90> <Element 'id' at 0x1453e10> <Element 'owner' at 0x1453e50> <Element 'updateDate' at 0x1453e90> <Element 'updatedBy' at 0x1453f10> <Element 'description' at 0x1453f50> <Element 'firstName' at 0x1453f90> <Element 'lastName' at 0x1459050> <Element 'isVisible' at 0x1453fd0> <Element 'name' at 0x1459090> <Element 'password' at 0x14590d0> <Element 'userRoleList' at 0x1459110> <Element 'userSource' at 0x1459150> However, the ONE field in Ranger > Users that does not show is the last field in the WebUI: Group. Where is this from? I am looking for the API to get this. I would understand if I had to link some kind of ID from a field above, but I have no idea. The group in our field is an openLDAP group, if that helps. ... View more

We are decommissioning a node, and are stuck with 1 underreplicated block. I checked the metasave report and notcied: Metasave: Blocks waiting for replication: 1 /ranger/audit/hdfs/20170510/hdfs_ranger_audit_<HOST>.log: blk_1286524956_212825612 (replicas: l: 1 d: 0 c: 0 e: 0) 10.230.67.5:50010 : Mis-replicated blocks that have been postponed: Metasave: Blocks being replicated: 0 Metasave: Blocks 0 waiting deletion from 0 datanodes. Corrupt Blocks: Metasave: Number of datanodes: 10 datanode log org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException): Failed to finalize INodeFile hdfs_ranger_audit_<HOST>.log since blocks[12] is non-complete, where blocks=[blk_1286247594_212548250, blk_1286298049_212598705, blk_1286353635_212654291, blk_1286413312_212713968, blk_1286472213_212772869, blk_1286524956_212825612, blk_1286561059_212861715, blk_1286595257_212895934, blk_1286644160_212944837, blk_1286688727_212989404, blk_1286741896_213042573, blk_1286799222_213099899, blk_1286859599_233271569{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=0, replicas=[ReplicaUC[[DISK]DS-cd007f20-570b-4768-afd9-d2dbd92d3c7e:NORMAL:10.230.67.3:50010|RBW]]}]. I pulled up several bug reports, such as https://issues.apache.org/jira/browse/HDFS-11499, but we are having trouble solving this issue. Has anyone gotten this? HDP version: 2.6.0 ... View more

Ambari Version 2.5.0.3 dfsadmin -report shows 2 missing blocks, but apparently hdfs fsck does not show this, nor missing replicas. Is this still a bug? See below links for related topics/bugs. See: https://community.hortonworks.com/questions/6705/ambari-showing-corrupt-blocks-but-not-fsck.html See: https://issues.apache.org/jira/browse/HDFS-8533 ... View more

<RANGER_URL>/index.html#!/reports/audit/bigData It appears we are in the same groups for different access roles in LDAP/AD, so I am trying to figure out why the Audit > Access tab instantly generates for another user, but for me, I get "Unable to connect to Audit store !!". I know it works, based on the other person's login. They can reproduce this from my workstation even. Are there logs I can check out? Is this a problem with a postgres (backing db) setup? The audit logs are going to HDFS, which I can knit + hadoop fs -ls them. ... View more