Member since
01-12-2017
59
Posts
1
Kudos Received
0
Solutions
12-03-2019
04:39 AM
When submitting a message, such as here, we have access to nice formatting options, but the support portal does not. I especially am looking for code formatting.
... View more
03-22-2019
08:16 PM
We use the following topologies right now in HDP 3, shown below. I understand there is authentication and authorization. Prior to using PAM, we used LDAP configurations. Then we ran into... KnoxLdapContextFactory.java (HW had us try identity assertion, null pointer issue hit) KnoxLdapRealm.java (ldap - too many entries, fails, we only get the 'cdisadmin' group as it's in the first page or results, and the code for this does NOT properly page) KnoxPamRealm.java (works, but requires reworking PAM/sssd,). PAM works it seems, but even if I put my own username in the top level Knox policy to deny access, nothing is denied when I access Knox SSO webui's. I don't understand what part of the authorization process is not communicating. In our view, when you pass the PAM authentication stage, you just get access, when what we REALLY want to happen, is Ranger then says yes or no to you getting authorized. PAM module: #%PAM-1.0
auth [user_unknown=ignore success=ok ignore=ignore default=bad] pam_securetty.so
#auth substack system-auth
###############################################
# Imported from: /etc/pam.d/password-auth
###############################################
auth required pam_env.so
auth sufficient pam_unix.so nullok try_first_pass
auth sufficient pam_ldap.so minimum_uid=1000 use_first_pass
auth requisite pam_succeed_if.so uid >= 1000 quiet
#auth sufficient pam_sss.so use_first_pass
auth required pam_deny.so
account required pam_unix.so broken_shadow
account sufficient pam_ldap.so minimum_uid=1000
account sufficient pam_localuser.so
account sufficient pam_succeed_if.so uid < 1000 quiet
account [default=bad success=ok user_unknown=ignore] pam_sss.so
account required pam_permit.so
password requisite pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=
password sufficient pam_unix.so sha512 shadow nullok try_first_pass use_authtok
password sufficient pam_ldap.so minimum_uid=1000 try_first_pass
#password sufficient pam_sss.so use_authtok
password required pam_deny.so
session optional pam_keyinit.so revoke
session required pam_limits.so
session optional pam_mkhomedir.so umask=0077
-session optional pam_systemd.so
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session required pam_unix.so
session optional pam_ldap.so minimum_uid=1000
#session optional pam_sss.so knoxsso topology <topology>
<gateway>
<provider>
<role>identity-assertion</role>
<name>Default</name>
<enabled>true</enabled>
</provider>
<provider>
<role>webappsec</role>
<name>WebAppSec</name>
<enabled>true</enabled>
<param>
<name>xframe.options.enabled</name>
<value>true</value>
</param>
</provider>
<provider>
<role>authentication</role>
<name>ShiroProvider</name>
<enabled>true</enabled>
<param>
<name>sessionTimeout</name>
<value>30</value>
</param>
<param>
<name>redirectToUrl</name>
<value>/gateway/knoxsso/knoxauth/login.html</value>
</param>
<param>
<name>restrictedCookies</name>
<value>rememberme,WWW-Authenticate</value>
</param>
<param>
<name>main.pamRealm</name>
<value>org.apache.knox.gateway.shirorealm.KnoxPamRealm</value>
</param>
<param>
<name>main.pamRealm.service</name>
<value>knoxsso</value>
</param>
<param>
<name>urls./**</name>
<value>authcBasic</value>
</param>
</provider>
<provider>
<role>authorization</role>
<name>AclsAuthz</name>
<enabled>true</enabled>
</provider>
</gateway>
<application>
<name>knoxauth</name>
</application>
<service>
<role>KNOXSSO</role>
<param>
<name>knoxsso.cookie.secure.only</name>
<value>true</value>
</param>
<param>
<name>knoxsso.token.ttl</name>
<value>900000</value>
</param>
<param>
<name>knoxsso.redirect.whitelist.regex</name>
<value>^https?:\/\/(.*\.DOMAIN\.COM|localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$</value>
</param>
</service>
</topology> default topology: <topology>
<gateway>
<provider>
<role>identity-assertion</role>
<name>Default</name>
<enabled>true</enabled>
</provider>
<provider>
<role>authentication</role>
<name>ShiroProvider</name>
<enabled>true</enabled>
<param>
<name>sessionTimeout</name>
<value>30</value>
</param>
<param>
<name>main.pamRealm</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxPamRealm</value>
</param>
<param>
<name>main.pamRealm.service</name>
<value>knoxsso</value> </param>
<param>
<name>urls./**</name>
<value>authcBasic</value>
</param>
</provider>
<provider>
<role>authorization</role>
<name>XASecurePDPKnox</name>
<enabled>true</enabled>
</provider>
</gateway>
<service>
<role>AVATICA</role>
<url>http://HOST.DOMAIN.COM:8765</url>
</service>
<service>
<role>DRUID-COORDINATOR-UI</role>
{{druid_coordinator_urls}}
</service>
<service>
<role>DRUID-COORDINATOR</role>
{{druid_coordinator_urls}}
</service>
<service>
<role>DRUID-OVERLORD-UI</role>
{{druid_overlord_urls}}
</service>
<service>
<role>DRUID-OVERLORD</role>
{{druid_overlord_urls}}
</service>
<service>
<role>DRUID-ROUTER</role>
{{druid_router_urls}}
</service>
<service>
<role>DRUID-BROKER</role>
{{druid_broker_urls}}
</service>
<service>
<role>HBASEUI</role>
<url>http://HOST.DOMAIN.COM:16010</url>
</service>
<service>
<role>HDFSUI</role>
<version>2.7.0</version>
<url>http://HOST.DOMAIN.COM:50070/</url>
</service>
<service>
<role>HIVE</role>
<url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url>
</service>
<service>
<role>JOBTRACKER</role>
<url>rpc://{{rm_host}}:{{jt_rpc_port}}</url>
</service>
<service>
<role>JOBHISTORYUI</role>
<url>http://HOST.DOMAIN.COM:19888</url>
</service>
<service>
<role>NAMENODE</role>
<url>{{namenode_address}}</url>
</service>
<service>
<role>OOZIE</role>
<url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url>
</service>
<service>
<role>OOZIEUI</role>
<url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie/</url>
</service>
<service>
<role>RANGERUI</role>
<url>http://HOST.DOMAIN.COM:6080</url>
</service>
<service>
<role>RESOURCEMANAGER</role>
<url>http://{{rm_host}}:{{rm_port}}/ws</url>
</service>
<service>
<role>SPARKHISTORYUI</role>
<url>http://HOST.DOMAIN.COM:18081</url>
</service>
<service>
<role>WEBHDFS</role>
{{webhdfs_service_urls}}
</service>
<service>
<role>WEBHCAT</role>
<url>http://{{webhcat_server_host}}:{{templeton_port}}/templeton</url>
</service>
<service>
<role>WEBHBASE</role>
<url>http://{{hbase_master_host}}:60080</url>
</service>
<service>
<role>YARNUI</role>
<url>http://HOST.DOMAIN.COM:8088</url>
</service>
<service>
<role>YARNUIV2</role>
<url>http://HOST.DOMAIN.COM:8088</url>
</service>
<service>
<role>ZEPPELINUI</role>
{{zeppelin_ui_urls}}
</service>
<service>
<role>ZEPPELINWS</role>
{{zeppelin_ws_urls}}
</service>
</topology> This is all I noticed in gateway.log when testing these scenarios: [mtdeguzis@HOST: knox]$ tailf gateway.log | grep -iE 'mtdeguzis|yarn'
2019-03-22 15:49:07,094 INFO knox.gateway (KnoxPamRealm.java:doGetAuthorizationInfo(126)) - Computed roles/groups: [udaoptst3, udaops, HOST-login, mtdeguzis, cdisadmin] for principal: mtdeguzis
2019-03-22 15:49:07,405 INFO service.knoxsso (WebSSOResource.java:getAuthenticationToken(240)) - About to redirect to original URL: https://HOST.domain.com:8443/gateway/knoxsso-webuis/yarnuiv2
2019-03-22 15:49:07,972 ERROR knox.gateway (UrlRewriteProcessor.java:rewrite(166)) - Failed to rewrite URL: HTTP_ONLY, direction: OUT via rule: YARNUIV2/yarnuiv2/outbound/timeline, status: FAILURE
2019-03-22 15:49:07,973 ERROR knox.gateway (JsonFilterReader.java:filterStreamValue(547)) - Failed to filter value HTTP_ONLY, rule YARNUIV2/yarnuiv2/outbound/timeline: java.lang.NullPointerException
2019-03-22 15:49:08,013 ERROR knox.gateway (UrlRewriteProcessor.java:rewrite(166)) - Failed to rewrite URL: kerberos, direction: OUT via rule: YARNUIV2/yarnuiv2/outbound/timeline, status: FAILURE
2019-03-22 15:49:08,013 ERROR knox.gateway (JsonFilterReader.java:filterStreamValue(547)) - Failed to filter value kerberos, rule YARNUIV2/yarnuiv2/outbound/timeline: java.lang.NullPointerException
... View more
Labels:
- Labels:
-
Apache Knox
-
Apache Ranger
03-02-2019
12:03 AM
We have searched all over https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/hive-workload/content/hive_workload_management_entity_data_in_sys.html and even the Apache hive code base. How do you monitor what queries are running in what resource pool???
... View more
Labels:
- Labels:
-
Apache Hive
02-05-2019
03:25 PM
I talked to support recently about this. The response I got was "The submission of jobs using HPSQL action JDBC clients is not yet possible as this feature has not been integrated yet."
... View more
01-28-2019
08:43 PM
See also: https://community.hortonworks.com/questions/70182/question-on-hdfs-rebalance-1.html?childToView=236814#answer-236814. The above answer is pretty good, "Need to move / left to move" is a good indicator.
... View more
01-28-2019
08:41 PM
I also noticed you can monitor the "need to move" message for the remaining space to be balanced. This can go up or down depending on how busy the cluster is: cat /tmp/hdfs_rebalancer.log | grep "Need to move" | tail -n 10
19/01/28 12:23:02 INFO balancer.Balancer: Need to move 11.11 TB to make the cluster balanced.
19/01/28 12:43:48 INFO balancer.Balancer: Need to move 11.10 TB to make the cluster balanced.
19/01/28 13:04:38 INFO balancer.Balancer: Need to move 10.89 TB to make the cluster balanced.
19/01/28 13:25:23 INFO balancer.Balancer: Need to move 10.83 TB to make the cluster balanced.
19/01/28 13:45:59 INFO balancer.Balancer: Need to move 10.83 TB to make the cluster balanced.
19/01/28 14:06:30 INFO balancer.Balancer: Need to move 10.78 TB to make the cluster balanced.
19/01/28 14:27:14 INFO balancer.Balancer: Need to move 10.73 TB to make the cluster balanced.
19/01/28 14:47:53 INFO balancer.Balancer: Need to move 10.70 TB to make the cluster balanced.
19/01/28 15:08:42 INFO balancer.Balancer: Need to move 10.66 TB to make the cluster balanced.
19/01/28 15:29:23 INFO balancer.Balancer: Need to move 10.75 TB to make the cluster balanced.
... View more
01-23-2019
03:48 PM
I have the same issue [root@sandbox-hdp /]# su - hive
Last login: Tue Jan 22 20:22:46 UTC 2019 on pts/0
[hive@sandbox-hdp ~]$ export STACK_VERSION=`hdp-select status hive-server2 | awk '{ print $3; }'`
[hive@sandbox-hdp ~]$ echo $STACK_VERSION
2.6.5.0-292
[hive@sandbox-hdp ~]$ export JAVA_HOME=$(java -XshowSettings:properties -version &> /tmp/java_out; cat /tmp/java_out | awk '/java.home/ {print $3}')
[hive@sandbox-hdp ~]$ echo $JAVA_HOME
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-8.b10.el7_5.x86_64/jre
hive@sandbox-hdp ~]$ $JAVA_HOME/bin/java -cp /usr/hdp/$STACK_VERSION/hive2/lib/derby-10.10.2.0.jar:/usr/hdp/$STACK_VERSION/hive2/lib/*:/usr/hdp/$STACK_VERSION/hadoop/*:/usr/hdp/$STACK_VERSION/hadoop/lib/*:/usr/hdp/$STACK_VERSION/hadoop-mapreduce/*:/usr/hdp/$STACK_VERSION/hadoop-mapreduce/lib/*:/usr/hdp/$STACK_VERSION/hadoop-hdfs/*:/usr/hdp/$STACK_VERSION/hadoop-hdfs/lib/*:/usr/hdp/$STACK_VERSION/hadoop/etc/hadoop/*:/tmp/hive-pre-upgrade-3.1.0.3.1.0.0-78.jar:/usr/hdp/$STACK_VERSION/hive/conf/conf.server org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool > /var/log/hive/pre_upgrade_hdp31.log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/hive2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Exception in thread "main" java.lang.IllegalStateException: preUpgrade requires Hive 1.x. Actual: 2.1.2000
at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:147)
... View more
09-10-2018
12:12 PM
We do not yet use this in production due to other items, but I'd suspect your krb.conf should be validated before going further. That is a pretty simple kerberos message.
... View more
05-07-2018
01:55 PM
Another good one for those looking for properties. You can then write known files here via JSON dictionary > XML if needed. Making API connection to: https://host.port/api/v1/clusters/cluster_name/configurations/service_config_versions?is_current=true
... View more
04-26-2018
09:14 PM
Related: https://community.hortonworks.com/questions/33234/how-to-export-all-hdp-configuration-files-xmlprope.html?childToView=189139#answer-189139 Trying to get HIVE_SERVER as well 😕
... View more
04-26-2018
08:50 PM
I was able to get the client config via tarball with python requests: TARBALL_URL = AMBARI_URI + ":" + AMBARI_PORT + "/api/v1/clusters/" + CLUSTER_NAME + '/components?format=client_config_tar' However as others have stated, this is a limited set. I also need the ranger configs like ranger-hive-security.xml. I have been looking at the Ranger API and webpages that describe developing Ranger plugins, as obviously, when something in Hive etc. needs to talk to Ranger, it has to be aware of this config, and this is available under the hive conf.server folder on a give hiveserver2 host: $ sudo ls /usr/hdp/current/hive-client/conf/conf.server/
hadoop-metrics2-hiveserver2.properties hive-env.sh.template hiveserver2-site.xml ranger-hive-audit.xml ranger-security.xml
hive-default.xml.template hive-exec-log4j.properties hive-site.xml ranger-hive-security.xml zkmigrator_jaas.conf
hive-env.sh hive-log4j.properties mapred-site.xml ranger-policymgr-ssl.xml
I need essentially this set from conf.server (Working on a Hive sidecar instance). I do not* want to pull these from a server via rsync or use cp, as it needs to be portable for my purposes. Related: https://community.hortonworks.com/questions/135415/rest-api-to-fetch-server-configs.html
... View more
04-25-2018
09:58 PM
How can one download other configs, such as ranger-security.xml ? Do you need to use other APIs to get these files?
... View more
12-28-2017
08:57 PM
1 Kudo
We have this running in Docker, that is the issue. Because this runs in-memory, that is where we see the issue.
... View more
12-28-2017
02:00 PM
We are consuming from Kafka using HDF NiFi in a docker
container. We current have this flow attached. The problem is that we wish to
bundle up until a max age of 1 hour so that 1 file gets pushed to HDFS. We made
this doable by setting min/max files/size of the bin very high, and letting Max
Bin Age trigger the bundle completion. As many of you know, HDFS hates small files,
especially frequently. The issue with MergeConent, is it holds this in-memory
until the bundle is made and pushed to HDFS. If there was a way that the bundle
could be built on-disk, that seems ok. Design alternatives? We are thinking of
just making a Java application to hanlde Kafka -> HDFS if this fails
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
-
Docker
12-28-2017
01:57 PM
We are consuming from Kafka using HDF NiFi in a docker container. We current have this flow attached. The problem is that we wish to bundle up until a max age of 1 hour so that 1 file gets pushed to HDFS. We made this doable by setting min/max files/size of the bin very high, and letting Max Bin Age trigger the bundle completion. As many of you know, HDFS hates small files, especially frequently. The issue with MergeConent, is it holds this in-memory until the bundle is made and pushed to HDFS. If there was a way that the bundle could be built on-disk, that seems ok. Design alternatives? We are thinking of just making a Java application to hanlde Kafka -> HDFS if this fails.
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
-
Docker
12-06-2017
11:49 PM
Within flume, the definition/channel/sink paradigm I can wrap my head around. It seems with NiFi, you add a syslog listener, ListenSyslog, that mainly connects via a port definition. Does this mean then I need to configure syslog on the NiFi parent host to ingest remove syslogs on a certain port?
Docs/links to point me in the right direction?
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
11-13-2017
12:11 PM
Hi, I've implemented a Python wrapper around the webhdfs api (via Knox) for end users to script operations outside the cluster. The one core operation missing I'd like to add is "hadoop fs -copyToLocal'. It seems that only open is supported and possible operations to direct the buffter to a file is possible. Is there any way to copy an HDFS file to a local path using webhdfs?
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Knox
09-11-2017
05:08 PM
Hmm... so it does* appear you need to provide just* the filename for S1 and S2. interesting
... View more
09-11-2017
04:56 PM
I have the same issue when trying to compute the diff. hadoop distcp -diff s1 s2 -update /data/a /data/a_target /data/a_target is on another cluster. s1 (yesterdays snap) and s2 (todays snap) on the first cluster location are side by side of course. I wonder if the diff needs to the snapshot filename only, and not the absolute path.
... View more
09-09-2017
02:41 PM
Just so everyone is aware: The snapshot created dirs must be named the same on both sides to do the diff distcp: Cannot find the snapshot of directory /group/bti/snapshot with name /group/bti/.snapshot/s20170908-080603.486
#LOF:
/group/bti/snapshot/.snapshot/s20170908-212827.054
Due to default naming conventions, the folders will not be the same. The default folder names created are seemingly time-stamped to the second. Name each created folder with todays day, such as "s20170908" so when the diff distcp runs, it can find and update the same-day folder on the LOF side.
... View more
09-09-2017
12:41 PM
Thanks for the update! I ran into this myself when designing a Python HDFS snapshot manager for two of our clusters.
... View more
08-29-2017
03:56 PM
Summary.
Submitting a PUT request against the Ranger API: https://cwiki.apache.org/confluence/display/RANGER/REST+APIs+for+Policy+Management#RESTAPIsforPolicyManagement-UpdatePolicy. I can create the policy fine, but seems with PUT, I get bad request and cannot determine why. The problem is not the content, as I can copy another policy as test.json and try to PUT that Code snip HEADERS = {'Content-type': 'application/json'}
result = requests.put(URL_BASE + URL_POLICY + policy_id, headers=HEADERS, data=open(json_file, 'rb'), auth=requests.auth.HTTPBasicAuth(USERNAME, PASSWORD))
Headers on GET Retrieving and exporting policy 16: <POLICY_NAME> {'transfer-encoding': 'chunked', 'set-cookie': 'RANGERADMINSESSIONID=<ID>; Path=/; HttpOnly, hadoop.auth=; Path=/; Domain=HOST.DOMAIN.COM; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly', 'server': 'Apache-Coyote/1.1', 'date': 'Tue, 29 Aug 2017 14:57:25 GMT', 'x-frame-options': 'DENY', 'content-type': 'application/json', 'www-authenticate': 'Negotiate'} Headers on successful POST {'transfer-encoding': 'chunked', 'set-cookie': 'RANGERADMINSESSIONID=<ID>; Path=/; HttpOnly, hadoop.auth=; Path=/; Domain=HOST.DOMAIN.COM; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly', 'server': 'Apache-Coyote/1.1', 'date': 'Tue, 29 Aug 2017 15:34:04 GMT', 'x-frame-options': 'DENY', 'content-type': 'application/json', 'www-authenticate': 'Negotiate'} Policy applied <POLICY_NUM> Headers on PUT (400 respose) Submitting PUT API request: http://HOST:PORT/service/public/api/policy/<POLICY_NUM>; {'transfer-encoding': 'chunked', 'set-cookie': 'RANGERADMINSESSIONID=<ID>; Path=/; HttpOnly, hadoop.auth=; Path=/; Domain=HOST.DIOMAIN.COM; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly', 'server': 'Apache-Coyote/1.1', 'connection': 'close', 'date': 'Tue, 29 Aug 2017 15:19:58 GMT', 'x-frame-options': 'DENY', 'content-type': 'application/json', 'www-authenticate': 'Negotiate'} The object exists at the endpoint, so a PUT needs made. ## Expected Result Response code 200 ## Actual Result Traceback (most recent call last): File "ranger-policy-manager.py", line 183, in <module> result.raise_for_status() File "/usr/lib/python2.7/site-packages/requests/models.py", line 834, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request
... View more
Labels:
- Labels:
-
Apache Ranger
08-04-2017
06:52 PM
We got this working. For those using Hortonworks HiveServer2 with Kerberos, this is what you need to do (providing your kerberos / kr5.conf is valid on your target host): Plus signs are for diff representation only. dbeaver.ini: -startup
plugins/org.eclipse.equinox.launcher_1.3.201.v20161025-1711.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.401.v20161122-1740
-showsplash
-vmargs
-Xms128m
-Xmx2048m
+ -Djavax.security.auth.useSubjectCredsOnly=false
+ -Djava.security.krb5.conf="krb5.conf"
Place the krb5.conf in the main installation path, or provide a path to it. After debugging for hours, and checking traces and more, this is wall it took. class name: org.apache.hive.jdbc.HiveDriver Dbeaver URL template: jdbc:hive2://{host}:{port}/{database};principal=hive/{host}.host.com@HOST.COM
... View more
07-26-2017
08:39 PM
I've seen Cloudera documentation point out that the URL template should be: URL template: jdbc:hive2://{host}:{port}/{database};AuthMech=1;KrbRealm=FOO.BAR;KrbHostFQDN={server}; But that does not seem to be working. What jars are required? Has anyone gotten this working? Links: http://justnumbersandthings.com/2017-05-06-Dbeaver-Hive-Kerberos.html
... View more
Labels:
- Labels:
-
Apache Hive
07-10-2017
05:26 PM
Thank you for the confirmation. Until the, I suppose we can just head into postgres on an add-needed basis. I don't want to keep this postgres python code in, since it inhibits who can run the audit tool (only a specific service account).
... View more
07-10-2017
05:02 PM
I can get the following details from: https://host.domain.com:port/api/v1/users/<user>; 1 {
2 "Users": {
3 "active": true,
4 "admin": true,
5 "groups": [
6 "<redacted>"
7 ],
8 "ldap_user": <redacted>,
9 "user_name": "<redacted>",
10 "user_type": "LDAP"
11 },
There is no created date. Is there an API call for this? I don't want to have to resort to using our postgres backing DB, as we have limited users added that, and this hampers who can then run our audit tools.
... View more
Labels:
- Labels:
-
Apache Ambari
07-10-2017
12:55 AM
I know how to get the available fields from, parsing them in Python with Element Tree.
http://<ip>:6080/service/xusers/users/userName/<userName>;
http://<ip>:6080/service/xusers/groups/groupName/<groupName>;
http://<ip>:6080/service/xusers/groups/<id>;
http://<ip>:6080/service/xusers/users/<id>; <Element 'vxUser' at 0x1453d50>
<Element 'createDate' at 0x1453d90>
<Element 'id' at 0x1453e10>
<Element 'owner' at 0x1453e50>
<Element 'updateDate' at 0x1453e90>
<Element 'updatedBy' at 0x1453f10>
<Element 'description' at 0x1453f50>
<Element 'firstName' at 0x1453f90>
<Element 'lastName' at 0x1459050>
<Element 'isVisible' at 0x1453fd0>
<Element 'name' at 0x1459090>
<Element 'password' at 0x14590d0>
<Element 'userRoleList' at 0x1459110>
<Element 'userSource' at 0x1459150>
However, the ONE field in Ranger > Users that does not show is the last field in the WebUI: Group. Where is this from? I am looking for the API to get this. I would understand if I had to link some kind of ID from a field above, but I have no idea. The group in our field is an openLDAP group, if that helps.
... View more
Labels:
- Labels:
-
Apache Ranger
06-12-2017
03:23 PM
We are decommissioning a node, and are stuck with 1 underreplicated block. I checked the metasave report and notcied: Metasave: Blocks waiting for replication: 1
/ranger/audit/hdfs/20170510/hdfs_ranger_audit_<HOST>.log: blk_1286524956_212825612 (replicas: l: 1 d: 0 c: 0 e: 0) 10.230.67.5:50010 :
Mis-replicated blocks that have been postponed:
Metasave: Blocks being replicated: 0
Metasave: Blocks 0 waiting deletion from 0 datanodes.
Corrupt Blocks:
Metasave: Number of datanodes: 10
datanode log org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException): Failed to finalize INodeFile hdfs_ranger_audit_<HOST>.log since blocks[12] is non-complete, where blocks=[blk_1286247594_212548250, blk_1286298049_212598705, blk_1286353635_212654291, blk_1286413312_212713968, blk_1286472213_212772869, blk_1286524956_212825612, blk_1286561059_212861715, blk_1286595257_212895934, blk_1286644160_212944837, blk_1286688727_212989404, blk_1286741896_213042573, blk_1286799222_213099899, blk_1286859599_233271569{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=0, replicas=[ReplicaUC[[DISK]DS-cd007f20-570b-4768-afd9-d2dbd92d3c7e:NORMAL:10.230.67.3:50010|RBW]]}]. I pulled up several bug reports, such as https://issues.apache.org/jira/browse/HDFS-11499, but we are having trouble solving this issue. Has anyone gotten this? HDP version: 2.6.0
... View more
Labels:
- Labels:
-
Apache Hadoop
05-30-2017
01:19 PM
Ambari Version 2.5.0.3 dfsadmin -report shows 2 missing blocks, but apparently hdfs fsck does not show this, nor missing replicas. Is this still a bug? See below links for related topics/bugs. See: https://community.hortonworks.com/questions/6705/ambari-showing-corrupt-blocks-but-not-fsck.html
See: https://issues.apache.org/jira/browse/HDFS-8533
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
05-18-2017
08:57 PM
<RANGER_URL>/index.html#!/reports/audit/bigData It appears we are in the same groups for different access roles in LDAP/AD, so I am trying to figure out why the Audit > Access tab instantly generates for another user, but for me, I get "Unable to connect to Audit store !!". I know it works, based on the other person's login. They can reproduce this from my workstation even. Are there logs I can check out? Is this a problem with a postgres (backing db) setup? The audit logs are going to HDFS, which I can knit + hadoop fs -ls them.
... View more
Labels:
- Labels:
-
Apache Ranger