Member since
01-16-2014
336
Posts
43
Kudos Received
31
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1817 | 12-20-2017 08:26 PM | |
1831 | 03-09-2017 03:47 PM | |
1644 | 11-18-2016 09:00 AM | |
2361 | 05-18-2016 08:29 PM | |
2073 | 02-29-2016 01:14 AM |
02-06-2020
04:08 AM
Hi, You also need to check on below configuration (If any). 1. Dynamic Resource Pool Configuration > Resource Pools - Check if jobs are exceeding any max values respective of the queue it's being submitted. 2. Dynamic Resource Pool Configuration > User Limits - Check if t he maximum number of applications a user can submit simultaneously is crossing the default value (5) or the specified value.
... View more
04-29-2019
08:02 AM
Hi Wilfred, Without node labels, is it possible to have a few nodes reserved for some users' exclusive usage? Thanks Vincent
... View more
03-13-2019
06:57 PM
Oozie APIs do not offer a way to do this directly, unfortunately. Its control completes at the point of the launcher invocation, so it does not carry any information about IDs that came later. Hue does special magic [0] to extract this information from the launcher logs, which you can replicate by using both Oozie and JobHistoryServer/RM/AM APIs to retrieve launcher map task logs and parsing it out the same way. [0] - https://github.com/cloudera/hue/blob/master/apps/beeswax/src/beeswax/views.py#L917-L952
... View more
03-05-2019
12:37 AM
I agree sentry is a very poor security managment component in Cloudera. With Sentry enabled Hive cannot impersonate the actual user who run the job. Administrators on Resource Manager UI, will have difficultty to figure out who ran the job. Apache Ranger is better product compare to Sentry. If you are using clouder manager try use option "cluster --> yarn --> Applications" in this UI it provides the actual user who ran the job.
... View more
03-04-2019
06:53 PM
vmem checks have been disabled in CDH almost since their introduction. The vmem check is not stable and highly dependent on Linux version and distro. If you run CDH you are already running with it disabled. Wilfred
... View more
02-20-2019
06:26 AM
When the first attempt fails, it tries to run again the app. So the status changes from "running" to "accepted". If you check the RM webUI you could see several attempts were run.
... View more
10-16-2018
09:49 AM
Hi Guys, I am facing similar issue. I have a new installation of Cloudera and i am trying to run a simple Map reduce Pi Example and also a spark Job. Map Reduce job gets stuck at the map 0% and reduce 0% step as shown below and Spark job is waiting spends lot of time in ACCEPTED state. I checked the user limit and it is blank for me. test@spark-1 ~]$ sudo -u hdfs hadoop jar /data/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100
Number of Maps = 10
Samples per Map = 100
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
18/10/16 12:33:25 INFO input.FileInputFormat: Total input paths to process : 10
18/10/16 12:33:26 INFO mapreduce.JobSubmitter: number of splits:10
18/10/16 12:33:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1539705370715_0002
18/10/16 12:33:26 INFO impl.YarnClientImpl: Submitted application application_1539705370715_0002
18/10/16 12:33:26 INFO mapreduce.Job: The url to track the job: http://spark-4:8088/proxy/application_1539705370715_0002/
18/10/16 12:33:26 INFO mapreduce.Job: Running job: job_1539705370715_0002
18/10/16 12:33:31 INFO mapreduce.Job: Job job_1539705370715_0002 running in uber mode : false
18/10/16 12:33:31 INFO mapreduce.Job: map 0% reduce 0% I made multiple config changes, but cannot find a solution for this. The only error i could trace was in the nodemanager log file as below : ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: RECEIVED SIGNAL 15: SIGTERM I tried checking various properties discussed in this thread, but i still have that issue. Can someone please help in solving this issue? Please let me know what all details i can provide.
... View more
09-19-2018
09:16 PM
I understand this is older post but I am getting same problem. Can you please provide solution if it is resolved for you? Thanks
... View more
09-13-2018
10:27 AM
Please follow the below steps. Options for container size control Now comes the complicated part - there are various overlapping and very poorly documented options for setting the size of Tez containers. According to some links, the following options control how Tez jobs started by Hive behave: hive.tez.container.size – value in megabytes hive.tez.java.opts
... View more
07-31-2018
07:09 AM
@Harsh J No, we rarely run balancer in this environment. I'll set it to 3 for now and observe for a while for any reoccurence of those WARNs if any . (CM recommends to set it between a value equal or greater than replication factor and lesser than number of DNs) Regards
... View more
07-24-2018
07:55 PM
1 Kudo
Hi stigahuang, Cloudera has deprecated the capacity scheduler in CDH 5.8 as per the documentations: deprecated items. We recommend that you move to the FairScheduler which we fully test and support. The difference between the upstream CS and what is in CDH is big and it is really difficult to say if a specific change would fix your issue or if a combination of changes would be needed. Wilfred
... View more
06-18-2018
03:37 AM
I just resolve the same issue, the problem is linked to your script which has windows line endings instead of unix one. try to convert it in the correct format using notepad++ or another tool. best accept my answer if it's working for you also, it will help the community.
... View more
04-06-2018
01:05 AM
Hi, This does not seem to have worked with a latter version of CDH (5.13.1). There we had to set this through - YARN Client Advanced Configuration Snippet (Safety Valve) for yarn-site.xml So, what is the correct way to set this? Is this really changed with newer releases? Thanks, Sumit
... View more
12-29-2017
02:04 AM
hello.everyone. the first step in ssh-action is "source /home/oracle/bash_profile",and then you can run oracle command. thanks
... View more
- Tags:
- ! everyone
12-20-2017
08:36 PM
Thanks for quick reply.
... View more
12-20-2017
07:26 AM
1 Kudo
Hi Wilfried, I'm sorry to ask again, but i'm facing the same problem and I don't understand how to configure Dynamic Ressource Pool Configuration to work using orginal user groups (me not hive). I'm using CDH 5.13 with Kerberos and Sentry. As I am using Sentry, impersonation is disabled. My configuration is root |--A |--B On root, submission ACL are set to allow only "sentry" user to submit in this pool On A, submission ACL are set to allow only group A to submit in this pool On B, submission ACL are set to allow only group B to submit in this pool Placement rules are : 1 - "Use the pool Specified at run time, only if the pool exists." 2 - " Use the pool root.[username] and create the pool if it does not exist. " When I submit a query with a user from the group A, using Hue and setting "set mapred.job.queue.name=A;" I got the error : "User hive cannot submit applications to queue root.A" If I add hive to allowed user on root, the query is working fine but both A and B user's can submit query If I add hive to only "A" resource pool, then user from A and B group can submit query to ressource pool A, but none can submit to resource pool B Maybe I am missing an important part, but I don't have the same behavior as you explained and if I add hive in authorized user it will break the ACL's as every user could use all the resource pool. Can you give us the good configuration to have the same behavior as your's ?
... View more
12-20-2017
07:05 AM
hello . now i can execute sql query by oozie ssh-acion with sqlplus command. it isn't a good way to do this and there are some bugs... thanks
... View more
12-20-2017
05:09 AM
Are you sure that the node (i.e. the HW had crashed) and was no longer reachable? It looks like the NM had crashed and the HW was still up and running and thus the container was still up and running. If you use YARN to kill a task attempt you need the NM to be up and running because the NM handles the container kill and cleanup. If the container does not get told to exit it will sit there and do its work until it is done. The RM removed the container from all its house keeping details there is no guarantee that the container is also removed from the node in this case. Did you use the mapred or yarn command to kill the attempt? Have you collected the container log and checked what was going on? Did you check the application master to see what it thought the attempt was still doing? Wilfred
... View more
12-20-2017
04:45 AM
The easiest way is to check what is set when you login and things work for you. Oracle requirements change per version and the best way forward is copy from a working install and make sure that everything that is set there is set in the action that is executed. Wilfred
... View more
12-08-2017
05:46 AM
What they have done is turn on the partial aggregation via the setting: yarn.nodemanager.log-aggregation.roll -monitoring-interval-seconds. That will allow you to grab some of the logs using the command line. We do not support this in CDH although we have the exact same code in CDH is available upstream. We have tested the setting and found that it breaks log access via the different UIs in multiple ways. So you get a working command line in 99% of the cases but when you try to use the RM or AM UIs it breaks almost always. The way it breaks changes over time for the same application. That is not a feature that we can support in the state it is at the moment. Wilfred
... View more
12-08-2017
05:29 AM
You will need to shade the guava that you use in your application. There is no way to replace the guava that is part of CDH with a later release, it will break a number of things. What it looked like from the previous message is that they did not shade it correctly. Wilfred
... View more
12-08-2017
05:25 AM
I cannot find the message any where in our source code. AppSchedulingInfo even upstream does not have a warning message that gets printed. Can you tell me which CDH release you are on? Wilfred
... View more
09-08-2017
10:10 AM
You need to copy hive-site.xml to SPARK_HOME to make sure you are no using derby. Then it still blocks and does not allow multiple sessions. There is still some problem.
... View more
06-22-2017
02:07 AM
Hi Guys, I was also facing same issue while submitting jobs to Oozie on my local system. I tried to find any solutions on google, but havent succeeded. Then i got to know that oozie always execute a workflow through mapreduce and i tried to run a MAPRED 'hello world' and it got failed. Now configured few properties for MAPRED to works in mapred-site.xml and yarn-site.xml as: For yarn-site.xml: <property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property> For mapred-site.xml: <property>
<name>mapreduce.jobtracker.address</name>
<value>HOST:PORT</value>
</property>
<property>
<name>mapreduce.jobtracker.http.address</name>
<value>HOST:PORT</value>
</property>
<property>
<name>mapreduce.tasktracker.report.address</name>
<value>127.0.0.1:0</value>
</property>
<property>
<name>mapreduce.tasktracker.http.address</name>
<value>0.0.0.0:50060</value>
</property>
<property>
<name>mapreduce.job.queuename</name>
<value>default</value>
</property>
<property>
<name> mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>localhost:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>localhost:19888</value>
</property> And make sure job history server is up and running and values of the properties should be unique on all the nodes of the cluster: mapreduce.jobhistory.done-dir
mapreduce.jobhistory.intermediate-done-dir After these changes, i am able to submit oozie workflow on my local.
... View more
06-05-2017
12:26 PM
In my case, neither the ResourceManager nor the NodeManager were up and running. Phew!
... View more
03-10-2017
06:42 AM
the configuration works fine only issue is that the bind user password is not redacted in the advanced configuration snippet and in clear text in the core-site.xml According to the security guide (sensitive data redaction), v5.8.x (not documented for 5.7.x): Redaction of Advanced Configuration Snippet parameters is based on detecting keywords explicitly defined as sensitive in the contents of these parameters. That is, parameters containing the keywords password, key, aws, or secret, will be redacted for users who do not have the required edit privileges I'll open a case to check how to get this working on 5.7.1
... View more
03-09-2017
04:19 PM
Did you check the tuning guide that we have for Yarn? It takes all this into account when you follow the tuning guide. Yarn uses vCores, in other words they are virtual cores and for simplicity we normally map 1 physical core, taking hyper threading into account, to somewhere between 1 and 4 virtual cores in Yarn. Wilfred
... View more
02-14-2017
01:29 AM
--conf "spark.driver.extraClassPath=/opt/cloudera/parcels/CDH-5.4.7-1.cdh5.4.7.p0.3/jars/htrace-core-3.1.0-incubating.jar:/opt/cloudera/parcels/CDH-5.4.7-1.cdh5.4.7.p0.3/lib/hive/conf:/opt/cloudera/parcels/CDH-5.4.7-1.cdh5.4.7.p0.3/lib/hive/lib/*.jar" \ --conf "spark.executor.extraClassPath=/opt/cloudera/parcels/CDH-5.4.7-1.cdh5.4.7.p0.3/jars/htrace-core-3.1.0-incubating.jar:/opt/cloudera/parcels/CDH-5.4.7-1.cdh5.4.7.p0.3/lib/hive/conf:/opt/cloudera/parcels/CDH-5.4.7-1.cdh5.4.7.p0.3/lib/hive/lib/*.jar" \ --conf "spark.driver.extraJavaOptions=-XX:MaxPermSize=1024m -XX:PermSize=256m" \ --conf "spark.executor.extraJavaOptions=-XX:MaxPermSize=1024m -XX:PermSize=256m" \ work for me
... View more
02-13-2017
08:10 PM
Thank you, you are right, when I create a kadmin user on each linux machine, you can successfully submit the task!
... View more