About ssulav

ssulav · ‎10-23-2019

Thanks for reply @rohitmalhotra User Limit for yarn is set to 65536. Is there any recommended highest value or shall I just make it unlimited? (It can have consequences?) Edit : I tried setting unlimited. Still seeing same error.

ssulav · ‎10-23-2019

You can pass arguments to handle null strings : --null-string '\\N' --null-non-string '\\N' Refer https://sqoop.apache.org/docs/1.4.3/SqoopUserGuide.html for more detail

ssulav · ‎10-22-2019

I would suggest you to go through the below docs and verify the outbound rules on port 7180. https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html

ssulav · ‎10-22-2019

Good news. If that resolves your issue, please spare some time in accepting the solution. Thanks.

ssulav · ‎10-22-2019

Seeing below exception on running Hive TPCDS data gen (https://github.com/hortonworks/hive-testbench) for a scale of ~500G. Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread Attached log for complete stacktrace. Cluster Configuration : 16 Nodes / 12 Nodemanagers / 12 Datanodes Per Node Config : Cores : 40 Memory : 392GB Ambari Configs changed from initial configs to improve performance : Decided to set 10G as container size to utilise maximum cores per node (320G/10G = 32 containers using 1 Core/node. Hence ~32 Cores/node utilised) YARN yarn.nodemanager.resource.memory-mb = 329216 MB yarn.scheduler.minimum-allocation-mb = 10240 MB yarn.scheduler.maximum-allocation-mb = 329216 MB MapReduce (All Heap Sizes : -Xmx8192m : 80% of container) mapreduce.map.memory.mb = 10240 MB mapreduce.reduce.memory.mb = 10240 MB mapreduce.task.io.sort.mb = 1 792 MB yarn.app.mapreduce.am.resource.mb = 10240 MB Hive hive.tez.container.size = 10240MB hive.auto.convert.join.noconditionaltask.size = 2027316838 B hive.exec.reducers.bytes.per.reducer = 1073217536 B Tez tez. am. resource. memory. mb = 10240 MB tez.am.resource.java.opts = -server -Xmx8192m tez.task.resource.memory.mb = 10240 MB tez.runtime.io.sort.mb = 2047 MB (~20% of container) tez.runtime.unordered.output.buffer.size-mb = 768 MB (~10% of container) tez.grouping.max-size = 2073741824 B tez. grouping. min-size = 167772160 B Any help would be greatly appreciated. Referred https://community.cloudera.com/t5/Community-Articles/Demystify-Apache-Tez-Memory-Tuning-Step-by-Step/ta-p/245279 for some tuning values.

ssulav · ‎10-22-2019

Regular exception is observed in CM server logs : 2019-10-20 17:32:34,687 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: (11 skipped) Unable to retrieve remote parcel repository manifest java.util.concurrent.ExecutionException: java.net.ConnectException: connection timed out: archive.cloudera.com/151.101.188.167:443 This may happen if you have http_proxy to access public web or you have private network. Currently CM is trying to access the archive url to download parcels as this was the method used while instaling CM and failing to do so. Try running below command on CM node and let us know the output : wget https://archive.cloudera.com/cdh6/6.3.1/parcels/manifest.json If you want to set proxy can be done under Administration > Search for 'Proxy'

ssulav · ‎10-22-2019

Can you verify the proper ownership of the cloudera-scm-server-db folder by running below commands : chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-server-db/ chmod 700 /var/lib/cloudera-scm-server-db/ chmod 700 /var/lib/cloudera-scm-server-db/data service cloudera-scm-server-db start Also verify the selinux status by running sestatus

ssulav · ‎09-03-2019

Even I faced a similar problem. Uninstalling MySQL-python (while debugging MySQL issue) removed cloudera-scm-agent as a dependent package. Thought will go by moving the component to other node then adding it back via CM. But figured a simple way : yum install -y cloudera-manager-agent Once installed copy the backed-up config.ini for existing cluster to the new agent config cp /etc/cloudera-scm-agent/config.ini.rpmsave /etc/cloudera-scm-agent/config.ini Just restart the cloudera-scm-agent after confirming above steps : systemctl restart cloudera-scm-agent

ssulav · ‎02-28-2019

Hi, If you don't want SMARTSENSE in your cluster but still it comes as default selected component during install wizard then go through below steps to save yourself some trouble. Tried on HDP Version : 3.0 and 3.1 Goto below path on the ambari-server node: /var/lib/ambari-server/resources/stacks/HDP/3.0/services/SMARTSENSE/metainfo.xml Open the above file in editor mode (e.g. vi) Uncomment or delete the below line [Line 23 may vary in different release] <selection>MANDATORY</selection> After making the above change restart ambari-server and proceed with cluster install wizard. Now SMARTSENSE won't be a mandate component. Thanks for reading.

ssulav · ‎10-30-2018

@Raffaele S You can follow below steps to create Kafka policy in Ranger to limit access per user on topic. Enable Kafka Plugin in Ranger. Go to ranger ui from Quick Links on Ranger component via Ambari : Login with ranger admin user and password. Click on <clustername>_kafka policy list under Kafka. It will list current policies. Click on Add New Policy button. Fill in the Policy name. In topic you can specify each topic name which you want to be controlled or put * for all topics to be governed by this policy. Now come to Allow conditions : You can put your users which you want to allow under 'Select User' similarly for groups. In Add_Permissions you can see all topic related operations. You can further add deny conditions and ip address range as well exclude rom allow conditions. I hope this answers your question. If yes please accept.

ssulav · ‎10-29-2018

@Raffaele S You can Enable Ranger plugin for Kafka. After doing that for each topic level you can control describe/read/write/..etc by logging in Ranger UI and setting the policies.

ssulav · ‎10-29-2018

Can you add exception from metastore and nodemanager logs. You can find them in /var/log/hive/*log and /var/log/hadoop-yarn/yarn/*log on the node where the component is running

ssulav · ‎10-29-2018

If it worked for you, please take a moment to login and "Accept" the answer.

ssulav · ‎10-29-2018

I believe there are other files along with your 9GB file and by coincidence the other files constitute 18GB of data. Other files constitute of many component libraries, ambari data, user data, tmp data. Run the below command to find which files are taking size : hadoop fs -du -s -h /* Go further by putting path in place of * till you find which other files are present which add upto 18G

ssulav · ‎10-25-2018

Good to know. If the answer helped you, please upvote so that it can help others.

ssulav · ‎10-25-2018

Tested in my cluster as well and was able to add ISA-L libs and remove the WARN messages.

ssulav · ‎10-24-2018

This worked for me. Upvoted

ssulav · ‎10-23-2018

@Sahil Kaw You can follow these steps to get logs from UI : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_security/content/enabling_browser_access_spnego_web_ui.html OR easier way is to get it from the node/servers. Just goto /var/log/hadoop-yarn/yarn/yarn*resourcemanager*log Above file will be log rotated. You can find the relevant file which contains the error stack trace.

ssulav · ‎10-11-2018

Can you please provide the complete stacktrace. If you aren't sure where to find the logs refer link The exception encountered by you has been reported in a secure cluster as yours. Refer the solution provided - https://community.hortonworks.com/content/supportkb/151796/error-orgapachehadoopsecurityauthenticationclienta.html

ssulav · ‎10-09-2018

This workaround will mean that for each and every job I will have to delete the staging-dir before submitting any new job and also at a moment single user will be able to run a job.

ssulav · ‎10-09-2018

When a YARN/MR Job is submitted it checks the staging directory ownership and if it doesn't matches with the user who is submitting the job, it throws below exception. Staging directory path is referred from YARN config [yarn.app.mapreduce.am.staging-dir = /tmp/hadoop-yarn/staging] java.io.IOException: The ownership on the staging directory /tmp/hadoop-yarn/staging/hdfs/.staging is not as expected. It is owned by . The directory must be owned by the submitter hdfs or hdfs at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:152) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:113) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:151) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588) at org.apache.hadoop.examples.WordCount.main(WordCount.java:87) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71) at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232) Is there a YARN config which skips the ownership check for the staging directory. I am facing this issue with OzoneFS, not with HDFS. Ownership check happens in below file : https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java#L144 Any workaround to bypass or skip the check?

ssulav · ‎10-09-2018

I believe you are following proper commands to create the jceks file : hadoop credential create fs.s3a.access.key -value <ACCESS_KEY> -provider jceks://hdfs@<namenode>/tmp/s3a.jceks hadoop credential create fs.s3a.secret.key -value <SECRET_KEY> -provider jceks://hdfs@<namenode>/tmp/s3a.jceks #Verify by running below command hadoop credential list -provider jceks://hdfs@<namenode>/tmp/s3a.jceks Make sure the hive user can access the jceks file. [Check permissions and owners] And then you are adding the mentioned configuration in Ambari UI > HDFS > Configs > Custom core-site I was able to run hive jobs with same scenarios as yours [underlying storage was not AWS] If still it doesn't work can you try once the Method 2. Just to make sure there isn't any other issue.

ssulav · ‎10-08-2018

You can try below changes in your submit command as they may be causing the hash value calculated to be different : Submit command : I believe you want to write abc.txt in s3a bucket hadoopsa under sample folder. As you have already set hadoopsa as your defaultFS. So you should use below command hdfs dfs -put abc.txt /sample/ #sample folder should be existing before command run. OR hdfs dfs -put abc.txt s3a://hadoopsa/sample/ In your command when you put a file directly in s3a://sample/ it assumes sample as a bucket and tries to write in the base path.

ssulav · ‎10-08-2018

Above issue is observed cause of https://issues.apache.org/jira/browse/HIVE-20386. Refer bug for more details. As a workaround you can try below methods : Method 1 : Set below config in core-site.xml fs.s3a.bucket.<bucket_name>.security.credential.provider.path =<jceks_file_path> #Replace <bucket_name> and <jceks_file_path> accordingly. Method 2 : Set below configs in core-site.xml fs.s3a.bucket.<bucket_name>.access.key =<s3a access key> fs.s3a.bucket.<bucket_name>.secret.key =<s3a secret key> #Replace <bucket_name> accordingly. Let us know if the resolution works.

ssulav · ‎10-08-2018

Above issue is observed cause of https://issues.apache.org/jira/browse/HIVE-20386. Refer bug for more details. As a workaround you can try below methods : Method 1 : Set below config in core-site.xml fs.s3a.bucket.<bucket_name>.security.credential.provider.path = <jceks_file_path> #Replace <bucket_name> and <jceks_file_path> accordingly. Method 2 : Set below configs in core-site.xml fs.s3a.bucket.<bucket_name>.access.key = <s3a access key> fs.s3a.bucket.<bucket_name>.secret.key = <s3a secret key> #Replace <bucket_name> accordingly. Let us know if the resolution works.

Online	Offline
Last Visited	‎02-11-2020 12:01 PM

Date Registered	‎06-26-2018 05:42 PM
Last Visited	‎02-11-2020 12:01 PM
Total Messages Posted	25
Total Kudos Received	4

Re: unable to start cloudera-scm-server-db.service...

Re: Datanode replication issue

Re: hdp-3.0.1 --> hive doesn't honor an s3a endpoi...

Re: Hive OutOfMemoryError: unable to create new na...

Re: Scoop import text data problem

Re: Not able to connect to Cloudera Manager Web UI

Re: unable to start cloudera-scm-server-db.service...

Hive OutOfMemoryError: unable to create new native...

Re: Not able to connect to Cloudera Manager Web UI

Re: unable to start cloudera-scm-server-db.service...

Re: Accidentally removed cloudera agent RPM....how...

Disabling SMARTSENSE from Install Wizard

Re: Disallow anonymous topic creation in Kerberize...

Re: Disallow anonymous topic creation in Kerberize...

Re: Missing Hive CLI Jar

Re: Datanode replication issue

Re: Datanode replication issue

Re: hdp-3.0.1 --> hive doesn't honor an s3a endpoi...

Re: Enable Intel's Intelligent Storage Acceleratio...

Re: Ambari REST API to restart all services

Re: hdp-3.0.1 --> hive doesn't honor an s3a endpoi...

Re: hdp-3.0.1 --> hive doesn't honor an s3a endpoi...

Re: YARN config to skip staging dir ownership chec...

YARN config to skip staging dir ownership check

Re: hdp-3.0.1 --> hive doesn't honor an s3a endpoi...

Re: hash issue while using S3 as Storage Backend f...

Re: hdp-2.5.0 HIVE doesn't seem to honor an S3a en...

Re: hdp-3.0.1 --> hive doesn't honor an s3a endpoi...