I see the problem (my problem). I hope this triggers some answer from the HDP community. My AD user is in CamelCase and so I get this in the yarn logs on the node where the app was running: java.io.IOException: Owner 'myusername' for path /grid/1/hadoop/yarn/log/application_1549437510290_0039/container_e80_1549437510290_0039_01_000001/stderr did not match expected owner 'MyUserName' User names are different in case only and so log aggregation fails and /app-logs/MyUserName/logs/application_1549437510290_0039/ is empty! Please help!
... View more
I have a very similar issue. After integratin the entire cluster, all services including Ranger, with the Active Directory, the YARN application log folders for Active Directory users are empty.
This does not happen when the application is run as the hdfs user, so it has to be a permissions problem.
While an application is running I can see the log folders and files on the node running the container for the application:
As soon as the application stops the log folder is created in hdfs, but it is empty.
The Ranger policies have been set up to give service accounts (e.g. hdfs, hive, spark etc.) full access to the entire hdfs file structure.
AD users who are allowed to submit spark jobs are given access to these folders in an attempt to allow application logs to work:
Any suggestions will be greatly appreciated.
I can post more configuration and settings information if needed.
... View more
After integrating all our cluster's services with Active Directory I was forced to start using the Livy2 interpreter for Zeppelin. Everything is now working as before, but the Livy2 interpreter can not execute Hive queries: %livy2
val testDF = spark.sql("SELECT * FROM business_process_management.awd_bpm_link")
Output: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 7, Node107FQDN, executor 1): java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "Node107FQDN/Node107IPAddress"; destination host is: "PrimaryNameNodeFQDN":8020; This works: %livy2
spark.version Output: res0: String = 22.214.171.124.6.2.0-205 Everything else is working as before while logged in as an Active Directory user that has been granted access to the relevant folders and databases via Ranger:
Hive View 2.0 via Ambari Scala code using spark to query hive (read and write) via spark-shell, spark-submit and Scala applications running in Oozie workflow actions Zeppelin's spark2 interpreter querying databases to which the zeppelin user has been granted access. I can post more information about configuration settings here, but I am not sure which settings are actually relevant. Since everything else is working as expected, here are my Livy2 interpreter settings: Properties name value livy.spark.deploy_mode cluster livy.spark.dynamicAllocation.cachedExecutorIdleTimeout livy.spark.dynamicAllocation.enabled livy.spark.dynamicAllocation.initialExecutors livy.spark.dynamicAllocation.maxExecutors livy.spark.dynamicAllocation.minExecutors livy.spark.executor.cores livy.spark.executor.instances livy.spark.executor.memory 8GB livy.spark.jars [path to jar file],[path to another jar file] livy.spark.jars.packages zeppelin.interpreter.localRepo /usr/hdp/current/zeppelin-server/local-repo/2DSXMN1CA zeppelin.interpreter.output.limit zeppelin.livy.concurrentSQL false zeppelin.livy.displayAppInfo true zeppelin.livy.keytab /etc/security/keytabs/zeppelin.server.kerberos.keytab zeppelin.livy.principal zeppelin-clustername@OUR.REALM.COM zeppelin.livy.pull_status.interval.millis zeppelin.livy.session.create_timeout zeppelin.livy.spark.sql.field.truncate true zeppelin.livy.spark.sql.maxResult zeppelin.livy.url http://LivyNodeFQDN:8999 zeppelin.spark.useHiveContext true ANY help will be appreciated and if I get any solution I will post it here. I am also planning to document the process of getting a Hortonworks cluster to actually work and post it online. I hope I can spare everyone else the pain I had to go through so far, just to get Hortonworks to work like a data lake should work.
... View more