Cloudera Community
ssulav
user rank
user rank
Contributor

Re: hdp-3.0.1 --> hive doesn't honor an s3a endpoi...

by Contributor Contributor in Support Questions
‎10-25-2018 08:49 AM
‎10-25-2018 08:49 AM
Good to know. If the answer helped you, please upvote so that it can help others. ... View more

Re: Enable Intel's Intelligent Storage Acceleratio...

by Contributor Contributor in Community Articles
‎10-25-2018 07:16 AM
‎10-25-2018 07:16 AM
Tested in my cluster as well and was able to add ISA-L libs and remove the WARN messages. ... View more

Re: Ambari REST API to restart all services

by Contributor Contributor in Support Questions
‎10-24-2018 08:58 AM
‎10-24-2018 08:58 AM
This worked for me. Upvoted ... View more

Re: hdp-3.0.1 --> hive doesn't honor an s3a endpoi...

by Contributor Contributor in Support Questions
‎10-23-2018 07:42 AM
‎10-23-2018 07:42 AM
@Sahil Kaw You can follow these steps to get logs from UI : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_security/content/enabling_browser_access_spnego_web_ui.html OR easier way is to get it from the node/servers. Just goto /var/log/hadoop-yarn/yarn/yarn*resourcemanager*log Above file will be log rotated. You can find the relevant file which contains the error stack trace. ... View more

Re: hdp-3.0.1 --> hive doesn't honor an s3a endpoi...

by Contributor Contributor in Support Questions
‎10-11-2018 07:18 AM
‎10-11-2018 07:18 AM
Can you please provide the complete stacktrace. If you aren't sure where to find the logs refer link The exception encountered by you has been reported in a secure cluster as yours. Refer the solution provided - https://community.hortonworks.com/content/supportkb/151796/error-orgapachehadoopsecurityauthenticationclienta.html ... View more

Re: YARN config to skip staging dir ownership chec...

by Contributor Contributor in Support Questions
‎10-09-2018 09:17 AM
‎10-09-2018 09:17 AM
This workaround will mean that for each and every job I will have to delete the staging-dir before submitting any new job and also at a moment single user will be able to run a job. ... View more

YARN config to skip staging dir ownership check

by Contributor Contributor in Support Questions
‎10-09-2018 07:26 AM
‎10-09-2018 07:26 AM
When a YARN/MR Job is submitted it checks the staging directory ownership and if it doesn't matches with the user who is submitting the job, it throws below exception. Staging directory path is referred from YARN config [yarn.app.mapreduce.am.staging-dir = /tmp/hadoop-yarn/staging] java.io.IOException: The ownership on the staging directory /tmp/hadoop-yarn/staging/hdfs/.staging is not as expected. It is owned by . The directory must be owned by the submitter hdfs or hdfs at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:152) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:113) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:151) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588) at org.apache.hadoop.examples.WordCount.main(WordCount.java:87) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232) Is there a YARN config which skips the ownership check for the staging directory. I am facing this issue with OzoneFS, not with HDFS. Ownership check happens in below file : https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java#L144 Any workaround to bypass or skip the check? ... View more
Labels:

Re: hdp-3.0.1 --> hive doesn't honor an s3a endpoi...

by Contributor Contributor in Support Questions
‎10-09-2018 07:26 AM
‎10-09-2018 07:26 AM
I believe you are following proper commands to create the jceks file : hadoop credential create fs.s3a.access.key -value <ACCESS_KEY> -provider jceks://hdfs@<namenode>/tmp/s3a.jceks hadoop credential create fs.s3a.secret.key -value <SECRET_KEY> -provider jceks://hdfs@<namenode>/tmp/s3a.jceks #Verify by running below command hadoop credential list -provider jceks://hdfs@<namenode>/tmp/s3a.jceks Make sure the hive user can access the jceks file. [Check permissions and owners] And then you are adding the mentioned configuration in Ambari UI > HDFS > Configs > Custom core-site I was able to run hive jobs with same scenarios as yours [underlying storage was not AWS] If still it doesn't work can you try once the Method 2. Just to make sure there isn't any other issue. ... View more

Re: hash issue while using S3 as Storage Backend f...

by Contributor Contributor in Support Questions
‎10-08-2018 11:36 AM
‎10-08-2018 11:36 AM
You can try below changes in your submit command as they may be causing the hash value calculated to be different : Submit command : I believe you want to write abc.txt in s3a bucket hadoopsa under sample folder. As you have already set hadoopsa as your defaultFS. So you should use below command hdfs dfs -put abc.txt /sample/ #sample folder should be existing before command run. OR hdfs dfs -put abc.txt s3a://hadoopsa/sample/ In your command when you put a file directly in s3a://sample/ it assumes sample as a bucket and tries to write in the base path. ... View more

Re: hdp-3.0.1 --> hive doesn't honor an s3a endpoi...

by Contributor Contributor in Support Questions
‎10-08-2018 08:36 AM
1 Kudo
‎10-08-2018 08:36 AM
1 Kudo
Above issue is observed cause of https://issues.apache.org/jira/browse/HIVE-20386. Refer bug for more details. As a workaround you can try below methods : Method 1 : Set below config in core-site.xml fs.s3a.bucket.<bucket_name>.security.credential.provider.path = <jceks_file_path> #Replace <bucket_name> and <jceks_file_path> accordingly. Method 2 : Set below configs in core-site.xml fs.s3a.bucket.<bucket_name>.access.key = <s3a access key> fs.s3a.bucket.<bucket_name>.secret.key = <s3a secret key> #Replace <bucket_name> accordingly. Let us know if the resolution works. ... View more
Contact Me
Online
Offline
Last Visited
‎11-20-2024 08:04 AM
Community Statistics
Member Since ‎06-26-2018 05:42 PM
Last Visited ‎11-20-2024 08:04 AM
Posts 26
Kudos received 4