Member since
06-26-2018
26
Posts
2
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1783 | 10-22-2019 09:24 AM | |
1202 | 10-29-2018 02:28 PM | |
7316 | 10-08-2018 08:36 AM |
09-28-2020
11:49 AM
1 Kudo
Zookeeper does not allow listing or editing znodes if the current ACL doesn't have a set of permissions for the user or group. This is observed as a security authentication of znodes in all Cloudera Distros inherited from Apache Zookeeper. There are few references for the workaround, just compiling them together for Cloudera Managed clusters.
For the following error:
Authentication is not valid
There are two ways to address them:
Disable any ACL validation in Zookeeper (Not recommended):
Add the following config in CM > Zookeeper config > Search for 'Java Configuration Options for Zookeeper Server': -Dzookeeper.skipACL=yes
Then Restart and refresh the stale configs.
Add a Zookeeper super auth:
Skip the part added in <SKIP> if you want to use ‘password' as the auth key. <SKIP> cd /opt/cloudera/parcels/CDH/lib/zookeeper/
java -cp "./zookeeper.jar:lib/*" org.apache.zookeeper.server.auth.DigestAuthenticationProvider super:password Use the last line from the following output on running the above command : SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
super:password->super:DyNYQEQvajljsxlhf5uS4PJ9R28= </SKIP>
Add the following config in CM > Zookeeper config > Search 'Java Configuration Options for Zookeeper Server': -Dzookeeper.DigestAuthenticationProvider.superDigest=super:DyNYQEQvajljsxlhf5uS4PJ9R28=
Restart and refresh the stale configs.
Once connected to zookeeper-client, add the following command before executing any further command: addauth digest super:password
You will be able to run any operation on any znode post this command.
NOTE:
Version of slf4j-api may differ on later builds.
Update the super password to any string you desire. <password>
... View more
Labels:
10-23-2019
05:33 AM
Thanks for reply @rohitmalhotra User Limit for yarn is set to 65536. Is there any recommended highest value or shall I just make it unlimited? (It can have consequences?) Edit : I tried setting unlimited. Still seeing same error.
... View more
10-23-2019
05:17 AM
You can pass arguments to handle null strings : --null-string '\\N' --null-non-string '\\N' Refer https://sqoop.apache.org/docs/1.4.3/SqoopUserGuide.html for more detail
... View more
10-22-2019
08:11 PM
I would suggest you to go through the below docs and verify the outbound rules on port 7180. https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html
... View more
10-22-2019
12:05 PM
Good news. If that resolves your issue, please spare some time in accepting the solution. Thanks.
... View more
10-22-2019
10:44 AM
Seeing below exception on running Hive TPCDS data gen (https://github.com/hortonworks/hive-testbench) for a scale of ~500G.
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread
Attached log for complete stacktrace.
Cluster Configuration :
16 Nodes / 12 Nodemanagers / 12 Datanodes
Per Node Config :
Cores : 40
Memory : 392GB
Ambari Configs changed from initial configs to improve performance :
Decided to set 10G as container size to utilise maximum cores per node (320G/10G = 32 containers using 1 Core/node. Hence ~32 Cores/node utilised)
YARN
yarn.nodemanager.resource.memory-mb = 329216 MB
yarn.scheduler.minimum-allocation-mb = 10240 MB
yarn.scheduler.maximum-allocation-mb = 329216 MB
MapReduce (All Heap Sizes : -Xmx8192m : 80% of container)
mapreduce.map.memory.mb = 10240 MB
mapreduce.reduce.memory.mb = 10240 MB
mapreduce.task.io.sort.mb = 1 792 MB
yarn.app.mapreduce.am.resource.mb = 10240 MB
Hive
hive.tez.container.size = 10240MB
hive.auto.convert.join.noconditionaltask.size = 2027316838 B
hive.exec.reducers.bytes.per.reducer = 1073217536 B
Tez
tez. am. resource. memory. mb = 10240 MB
tez.am.resource.java.opts = -server -Xmx8192m
tez.task.resource.memory.mb = 10240 MB
tez.runtime.io.sort.mb = 2047 MB (~20% of container)
tez.runtime.unordered.output.buffer.size-mb = 768 MB (~10% of container)
tez.grouping.max-size = 2073741824 B
tez. grouping. min-size = 167772160 B
Any help would be greatly appreciated. Referred https://community.cloudera.com/t5/Community-Articles/Demystify-Apache-Tez-Memory-Tuning-Step-by-Step/ta-p/245279 for some tuning values.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hive
-
Apache Tez
10-22-2019
09:38 AM
Regular exception is observed in CM server logs : 2019-10-20 17:32:34,687 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: (11 skipped) Unable to retrieve remote parcel repository manifest
java.util.concurrent.ExecutionException: java.net.ConnectException: connection timed out: archive.cloudera.com/151.101.188.167:443 This may happen if you have http_proxy to access public web or you have private network. Currently CM is trying to access the archive url to download parcels as this was the method used while instaling CM and failing to do so. Try running below command on CM node and let us know the output : wget https://archive.cloudera.com/cdh6/6.3.1/parcels/manifest.json If you want to set proxy can be done under Administration > Search for 'Proxy'
... View more
10-22-2019
09:24 AM
Can you verify the proper ownership of the cloudera-scm-server-db folder by running below commands : chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-server-db/
chmod 700 /var/lib/cloudera-scm-server-db/
chmod 700 /var/lib/cloudera-scm-server-db/data
service cloudera-scm-server-db start Also verify the selinux status by running sestatus
... View more
02-28-2019
11:51 AM
3 Kudos
Hi, If you don't want SMARTSENSE in your cluster but still it comes as default selected component during install wizard then go through below steps to save yourself some trouble. Tried on HDP Version : 3.0 and 3.1 Goto below path on the ambari-server node: /var/lib/ambari-server/resources/stacks/HDP/3.0/services/SMARTSENSE/metainfo.xml Open the above file in editor mode (e.g. vi) Uncomment or delete the below line [Line 23 may vary in different release] <selection>MANDATORY</selection> After making the above change restart ambari-server and proceed with cluster install wizard. Now SMARTSENSE won't be a mandate component. Thanks for reading.
... View more
- Find more articles tagged with:
- FAQ
- smartsense
Labels:
10-30-2018
09:13 AM
@Raffaele S You can follow below steps to create Kafka policy in Ranger to limit access per user on topic. Enable Kafka Plugin in Ranger. Go to ranger ui from Quick Links on Ranger component via Ambari : Login with ranger admin user and password. Click on <clustername>_kafka policy list under Kafka. It will list current policies. Click on Add New Policy button. Fill in the Policy name. In topic you can specify each topic name which you want to be controlled or put * for all topics to be governed by this policy. Now come to Allow conditions : You can put your users which you want to allow under 'Select User' similarly for groups. In Add_Permissions you can see all topic related operations. You can further add deny conditions and ip address range as well exclude rom allow conditions. I hope this answers your question. If yes please accept.
... View more
10-29-2018
04:38 PM
@Raffaele S You can Enable Ranger plugin for Kafka. After doing that for each topic level you can control describe/read/write/..etc by logging in Ranger UI and setting the policies.
... View more
10-29-2018
04:31 PM
1 Kudo
Can you add exception from metastore and nodemanager logs. You can find them in /var/log/hive/*log and /var/log/hadoop-yarn/yarn/*log on the node where the component is running
... View more
10-29-2018
04:21 PM
If it worked for you, please take a moment to login and "Accept" the answer.
... View more
10-29-2018
02:28 PM
I believe there are other files along with your 9GB file and by coincidence the other files constitute 18GB of data. Other files constitute of many component libraries, ambari data, user data, tmp data. Run the below command to find which files are taking size : hadoop fs -du -s -h /* Go further by putting path in place of * till you find which other files are present which add upto 18G
... View more
10-25-2018
08:49 AM
Good to know. If the answer helped you, please upvote so that it can help others.
... View more
10-25-2018
07:16 AM
Tested in my cluster as well and was able to add ISA-L libs and remove the WARN messages.
... View more
10-23-2018
07:42 AM
@Sahil Kaw You can follow these steps to get logs from UI : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_security/content/enabling_browser_access_spnego_web_ui.html OR easier way is to get it from the node/servers. Just goto /var/log/hadoop-yarn/yarn/yarn*resourcemanager*log Above file will be log rotated. You can find the relevant file which contains the error stack trace.
... View more
10-11-2018
07:18 AM
Can you please provide the complete stacktrace. If you aren't sure where to find the logs refer link The exception encountered by you has been reported in a secure cluster as yours. Refer the solution provided - https://community.hortonworks.com/content/supportkb/151796/error-orgapachehadoopsecurityauthenticationclienta.html
... View more
10-09-2018
09:17 AM
This workaround will mean that for each and every job I will have to delete the staging-dir before submitting any new job and also at a moment single user will be able to run a job.
... View more
10-09-2018
07:26 AM
When a YARN/MR Job is submitted it checks the staging directory ownership and if it doesn't matches with the user who is submitting the job, it throws below exception. Staging directory path is referred from YARN config [yarn.app.mapreduce.am.staging-dir = /tmp/hadoop-yarn/staging] java.io.IOException: The ownership on the staging directory /tmp/hadoop-yarn/staging/hdfs/.staging is not as expected. It is owned by . The directory must be owned by the submitter hdfs or hdfs
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:152)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:113)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:151)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232) Is there a YARN config which skips the ownership check for the staging directory. I am facing this issue with OzoneFS, not with HDFS. Ownership check happens in below file : https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java#L144 Any workaround to bypass or skip the check?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache YARN
10-09-2018
07:26 AM
I believe you are following proper commands to create the jceks file : hadoop credential create fs.s3a.access.key -value <ACCESS_KEY> -provider jceks://hdfs@<namenode>/tmp/s3a.jceks
hadoop credential create fs.s3a.secret.key -value <SECRET_KEY> -provider jceks://hdfs@<namenode>/tmp/s3a.jceks
#Verify by running below command
hadoop credential list -provider jceks://hdfs@<namenode>/tmp/s3a.jceks Make sure the hive user can access the jceks file. [Check permissions and owners] And then you are adding the mentioned configuration in Ambari UI > HDFS > Configs > Custom core-site I was able to run hive jobs with same scenarios as yours [underlying storage was not AWS] If still it doesn't work can you try once the Method 2. Just to make sure there isn't any other issue.
... View more
10-08-2018
11:36 AM
You can try below changes in your submit command as they may be causing the hash value calculated to be different : Submit command : I believe you want to write abc.txt in s3a bucket hadoopsa under sample folder. As you have already set hadoopsa as your defaultFS. So you should use below command hdfs dfs -put abc.txt /sample/ #sample folder should be existing before command run.
OR
hdfs dfs -put abc.txt s3a://hadoopsa/sample/
In your command when you put a file directly in s3a://sample/ it assumes sample as a bucket and tries to write in the base path.
... View more
10-08-2018
08:36 AM
1 Kudo
Above issue is observed cause of https://issues.apache.org/jira/browse/HIVE-20386. Refer bug for more details. As a workaround you can try below methods : Method 1 : Set below config in core-site.xml fs.s3a.bucket.<bucket_name>.security.credential.provider.path = <jceks_file_path>
#Replace <bucket_name> and <jceks_file_path> accordingly. Method 2 : Set below configs in core-site.xml fs.s3a.bucket.<bucket_name>.access.key = <s3a access key>
fs.s3a.bucket.<bucket_name>.secret.key = <s3a secret key>
#Replace <bucket_name> accordingly.
Let us know if the resolution works.
... View more