Similar to this, I have a use case to compare Ansible Code with the Ambari Configs. The reason we are doing this is that we found several inconsistencies w.r.t to Ansible code and Ambari configs. But comparing both is a big task as there are many playbooks where we have Hadoop code so checking all the code base a heck. Any other option to do the comparison..... ... View more

I also met theses errors: Can't get Kerberos realm Cannot locate default realm   It was due to the quotes on parameter -Djava.security.krb5.conf.   I finally manage to connect my DBeaver to Hive with Kerberos SSL.  My final dbeaver.ini config was: --startup plugins/org.eclipse.equinox.launcher_1.5.400.v20190515-0925.jar --launcher.library plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.1000.v20190125-2016 -vmargs -XX:+IgnoreUnrecognizedVMOptions --add-modules=ALL-SYSTEM -Xms64m -Xmx1024m -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true -Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/home/matthieu/jaas.conf   With jaas.conf like that: Client { com.sun.security.auth.module.Krb5LoginModule required debug=true doNotPrompt=true useKeyTab=true keyTab="/path/to/user.REALM.keytab" useTicketCache=true renewTGT=true principal="user@REALM" ; }; and JDBC url:   jdbc:hive2://{host}:{port}/{database};KrbRealm=MY_REALM;principal=hive/{host}@MY_REALM;ssl=true;sslTrustStore=/path/to/trustore;transportMode=http;httpPath=cliservice;trustStorePassword=changeit     ... View more

Thank you for the confirmation. Until the, I suppose we can just head into postgres on an add-needed basis. I don't want to keep this postgres python code in, since it inhibits who can run the audit tool (only a specific service account). ... View more

The second was what I was looking for. Thanks! ... View more

There are 2 ways to do it. 1. Use commands hadoop daemonlog. Like hadoop daemonlog -setlevel <rm host:rm httport> <full class name including package name| or package name itself only> <level> Here are some examples hadoop daemonlog -setlevel <rm host:rm httport> org.apache.hadoop.yarn.server.resourcemanager DEBUG Or to enable debug log on full use something like: hadoop daemonlog -setlevel <rm host:rm httport> org.apache.hadoop.yarn DEBUG Daemonlog reference help "Running Daemonlog make sure are yarn/hdfs cluster admin, e.g. you are part of yarn.admin.acl in case of yarn 2: Via, Namenode/Resourcemanger etc.. access daemon UI like http://<daamonhost>:<httpPort>/logLevel?log, then fill package/classname in test box shown in second row along with desried logging (like DEBUG) here, click "Set Log Level" button. Using either of make sure use have web authenticatio on and you are part cluster admin group or default static web user (which dr.who) is part of cluster admin group ... View more

LISTSTATUS_BATCH is not for recursive listing, but for "iterative" listing. It means that WebHDFS will return the list (of files in the given directory) in (dfs.ls.limit sized) parts. It won't list the files of the subdirectories. Unfortunately hdfs dfs -ls -R is most likely your fastest option, if you really would like to print your whole filesystem. ... View more

@Michael DeGuzis If your cluster has Ambari Metrics Service deployed, then I would suggest using the reference - https://cwiki.apache.org/confluence/display/AMBARI/Metrics+Collector+API+Specification. It is richer and gives you more control on the data you need to fetch. ... View more

The "historical period" or TTL of Yarn logs is controlled by yarn.timeline-service.ttl-enable, and if "true", then by yarn.timeline-service.ttl-ms. By default it's enabled and TTL is 31 days. ... View more

I think this goes back to the whole "dead is bad" theory. If I recall correctly, there was a metric Ambari was monitoring once on HBase - it was for "Dead RegionServers". We incorrectly assumed that "dead" was "bad". Because of this, while decommissioning a RegionServer, alerts would trigger (and not go away for a long time). In the end, it was determined that this metric wasn't really something which needed alerting on. HDFS is a little different - I believe that a DataNode is marked as stale if it hasn't reported in within 30 seconds and marked as dead if it hasn't reported within 1 minute. The problem here is that action is taken by the NameNode in this case - it will begin replicating blocks when it believes a DataNode is dead. So, we alert on it since it's something that is actively causing changes in the cluster data. The NameNode actually has metrics for differentiating "dead" vs "decommissiong dead": "NumLiveDataNodes": 3, "NumDeadDataNodes": 1, "NumDecomLiveDataNodes": 0, "NumDecomDeadDataNodes": 1, In the above example, Ambari won't worry about dead nodes which are marked as known decommissioning, but we will worry about this which are unexpected. ... View more

How can one download other configs, such as ranger-security.xml ? Do you need to use other APIs to get these files? ... View more