About ssulav

ssulav · ‎09-03-2019

Even I faced a similar problem. Uninstalling MySQL-python (while debugging MySQL issue) removed cloudera-scm-agent as a dependent package. Thought will go by moving the component to other node then adding it back via CM. But figured a simple way : yum install -y cloudera-manager-agent Once installed copy the backed-up config.ini for existing cluster to the new agent config cp /etc/cloudera-scm-agent/config.ini.rpmsave /etc/cloudera-scm-agent/config.ini Just restart the cloudera-scm-agent after confirming above steps : systemctl restart cloudera-scm-agent

ssulav · ‎02-28-2019

Hi, If you don't want SMARTSENSE in your cluster but still it comes as default selected component during install wizard then go through below steps to save yourself some trouble. Tried on HDP Version : 3.0 and 3.1 Goto below path on the ambari-server node: /var/lib/ambari-server/resources/stacks/HDP/3.0/services/SMARTSENSE/metainfo.xml Open the above file in editor mode (e.g. vi) Uncomment or delete the below line [Line 23 may vary in different release] <selection>MANDATORY</selection> After making the above change restart ambari-server and proceed with cluster install wizard. Now SMARTSENSE won't be a mandate component. Thanks for reading.

ssulav · ‎10-30-2018

@Raffaele S You can follow below steps to create Kafka policy in Ranger to limit access per user on topic. Enable Kafka Plugin in Ranger. Go to ranger ui from Quick Links on Ranger component via Ambari : Login with ranger admin user and password. Click on <clustername>_kafka policy list under Kafka. It will list current policies. Click on Add New Policy button. Fill in the Policy name. In topic you can specify each topic name which you want to be controlled or put * for all topics to be governed by this policy. Now come to Allow conditions : You can put your users which you want to allow under 'Select User' similarly for groups. In Add_Permissions you can see all topic related operations. You can further add deny conditions and ip address range as well exclude rom allow conditions. I hope this answers your question. If yes please accept.

ssulav · ‎10-29-2018

@Raffaele S You can Enable Ranger plugin for Kafka. After doing that for each topic level you can control describe/read/write/..etc by logging in Ranger UI and setting the policies.

ssulav · ‎10-29-2018

Can you add exception from metastore and nodemanager logs. You can find them in /var/log/hive/*log and /var/log/hadoop-yarn/yarn/*log on the node where the component is running

ssulav · ‎10-29-2018

If it worked for you, please take a moment to login and "Accept" the answer.

ssulav · ‎10-29-2018

I believe there are other files along with your 9GB file and by coincidence the other files constitute 18GB of data. Other files constitute of many component libraries, ambari data, user data, tmp data. Run the below command to find which files are taking size : hadoop fs -du -s -h /* Go further by putting path in place of * till you find which other files are present which add upto 18G

ssulav · ‎10-25-2018

Good to know. If the answer helped you, please upvote so that it can help others.

ssulav · ‎10-25-2018

Tested in my cluster as well and was able to add ISA-L libs and remove the WARN messages.

ssulav · ‎10-24-2018

This worked for me. Upvoted

ssulav · ‎10-23-2018

@Sahil Kaw You can follow these steps to get logs from UI : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_security/content/enabling_browser_access_spnego_web_ui.html OR easier way is to get it from the node/servers. Just goto /var/log/hadoop-yarn/yarn/yarn*resourcemanager*log Above file will be log rotated. You can find the relevant file which contains the error stack trace.

ssulav · ‎10-11-2018

Can you please provide the complete stacktrace. If you aren't sure where to find the logs refer link The exception encountered by you has been reported in a secure cluster as yours. Refer the solution provided - https://community.hortonworks.com/content/supportkb/151796/error-orgapachehadoopsecurityauthenticationclienta.html

ssulav · ‎10-09-2018

This workaround will mean that for each and every job I will have to delete the staging-dir before submitting any new job and also at a moment single user will be able to run a job.

ssulav · ‎10-09-2018

When a YARN/MR Job is submitted it checks the staging directory ownership and if it doesn't matches with the user who is submitting the job, it throws below exception. Staging directory path is referred from YARN config [yarn.app.mapreduce.am.staging-dir = /tmp/hadoop-yarn/staging] java.io.IOException: The ownership on the staging directory /tmp/hadoop-yarn/staging/hdfs/.staging is not as expected. It is owned by . The directory must be owned by the submitter hdfs or hdfs at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:152) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:113) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:151) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588) at org.apache.hadoop.examples.WordCount.main(WordCount.java:87) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232) Is there a YARN config which skips the ownership check for the staging directory. I am facing this issue with OzoneFS, not with HDFS. Ownership check happens in below file : https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java#L144 Any workaround to bypass or skip the check?

ssulav · ‎10-09-2018

I believe you are following proper commands to create the jceks file : hadoop credential create fs.s3a.access.key -value <ACCESS_KEY> -provider jceks://hdfs@<namenode>/tmp/s3a.jceks hadoop credential create fs.s3a.secret.key -value <SECRET_KEY> -provider jceks://hdfs@<namenode>/tmp/s3a.jceks #Verify by running below command hadoop credential list -provider jceks://hdfs@<namenode>/tmp/s3a.jceks Make sure the hive user can access the jceks file. [Check permissions and owners] And then you are adding the mentioned configuration in Ambari UI > HDFS > Configs > Custom core-site I was able to run hive jobs with same scenarios as yours [underlying storage was not AWS] If still it doesn't work can you try once the Method 2. Just to make sure there isn't any other issue.

ssulav · ‎10-08-2018

You can try below changes in your submit command as they may be causing the hash value calculated to be different : Submit command : I believe you want to write abc.txt in s3a bucket hadoopsa under sample folder. As you have already set hadoopsa as your defaultFS. So you should use below command hdfs dfs -put abc.txt /sample/ #sample folder should be existing before command run. OR hdfs dfs -put abc.txt s3a://hadoopsa/sample/ In your command when you put a file directly in s3a://sample/ it assumes sample as a bucket and tries to write in the base path.

ssulav · ‎10-08-2018

Above issue is observed cause of https://issues.apache.org/jira/browse/HIVE-20386. Refer bug for more details. As a workaround you can try below methods : Method 1 : Set below config in core-site.xml fs.s3a.bucket.<bucket_name>.security.credential.provider.path =<jceks_file_path> #Replace <bucket_name> and <jceks_file_path> accordingly. Method 2 : Set below configs in core-site.xml fs.s3a.bucket.<bucket_name>.access.key =<s3a access key> fs.s3a.bucket.<bucket_name>.secret.key =<s3a secret key> #Replace <bucket_name> accordingly. Let us know if the resolution works.

ssulav · ‎10-08-2018

Above issue is observed cause of https://issues.apache.org/jira/browse/HIVE-20386. Refer bug for more details. As a workaround you can try below methods : Method 1 : Set below config in core-site.xml fs.s3a.bucket.<bucket_name>.security.credential.provider.path = <jceks_file_path> #Replace <bucket_name> and <jceks_file_path> accordingly. Method 2 : Set below configs in core-site.xml fs.s3a.bucket.<bucket_name>.access.key = <s3a access key> fs.s3a.bucket.<bucket_name>.secret.key = <s3a secret key> #Replace <bucket_name> accordingly. Let us know if the resolution works.

Online	Offline
Last Visited	‎09-05-2019 08:49 AM

Date Registered	‎06-26-2018 05:42 PM
Last Visited	‎09-05-2019 08:49 AM
Total Messages Posted	18
Total Kudos Received	4

Re: Datanode replication issue

Re: hdp-3.0.1 --> hive doesn't honor an s3a endpoi...

Re: Accidentally removed cloudera agent RPM....how...

Disabling SMARTSENSE from Install Wizard

Re: Disallow anonymous topic creation in Kerberize...

Re: Disallow anonymous topic creation in Kerberize...

Re: Missing Hive CLI Jar

Re: Datanode replication issue

Re: Datanode replication issue

Re: hdp-3.0.1 --> hive doesn't honor an s3a endpoi...

Re: Enable Intel's Intelligent Storage Acceleratio...

Re: Ambari REST API to restart all services

Re: hdp-3.0.1 --> hive doesn't honor an s3a endpoi...

Re: hdp-3.0.1 --> hive doesn't honor an s3a endpoi...

Re: YARN config to skip staging dir ownership chec...

YARN config to skip staging dir ownership check

Re: hdp-3.0.1 --> hive doesn't honor an s3a endpoi...

Re: hash issue while using S3 as Storage Backend f...

Re: hdp-2.5.0 HIVE doesn't seem to honor an S3a en...

Re: hdp-3.0.1 --> hive doesn't honor an s3a endpoi...