Member since
02-11-2017
11
Posts
3
Kudos Received
0
Solutions
05-03-2018
05:49 PM
Is there a way to kill a stalled query that is running under LLAP? In regular Hive, you would kill the YARN container and call it a day. In LLAP, doing so would kill the daemons!
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Hive
01-18-2018
08:58 PM
Any ETA for ExecuteSparkInteractive processor to make it into HDF?
... View more
07-13-2017
12:32 PM
Excellent article. Would presto fall into option #2?
... View more
02-27-2017
11:56 PM
Thanks. After realizing you cannot build a reasonable cluster within the Azure trial parameters, i opted to stay on my cluster in GCP.
... View more
02-25-2017
03:15 PM
2 Kudos
All image links in https://hortonworks.com/hadoop-tutorial/deploying-hortonworks-sandbox-on-microsoft-azure/ are broken
... View more
02-20-2017
03:19 PM
The behavior of the -replaceLabelsOnNode of yarn rmadmin is not what I expect. [yarn@master1 ~]$ yarn rmadmin -replaceLabelsOnNode data2=GPU
[yarn@master1 ~]$ echo $?
0 However, looking at the resource manager UI and clicking on nodes, the label is not applied. Therefore, if the command didnt accomplish the task, why didnt it fail? [yarn@master1 ~]$ yarn rmadmin -replaceLabelsOnNode data2.c.hdpfastrack.internal=GPU
Using the full hostname, the command executes and the expected output is seen in the RM UI. At least, if i mistype the node label, yarn rmadmin will complain that the label is not recognized.
... View more
Labels:
- Labels:
-
Apache YARN
02-13-2017
01:28 PM
Thanks guys. I guess a feature request could be done to have the config reversal give you a hint of the other config that could/should be reversed. This was a fun troubleshooting exercise 🙂
... View more
02-11-2017
04:34 PM
for personal use, you dont need to pay for anything unless you want paid support instead of relying on the community to help you. for corporate uses, many organizations opt to use the HDP subscription to obtain paid support. There are varying levels of support & SLA's + varying levels of support based on the components (core vs. premium such as storm, ranger, knox, accumulo, atlas, etc)
... View more
02-11-2017
04:31 PM
1 Kudo
I am currently going through the HDP FastTrack labs, more precisely "Lab 11: Managing YARN containers and queues" Basically, you run 2 concurrent example yarn jobs to understand the behavior of the minimum container size and the ability to run multiple applications concurrently, depending on container availability. Job: yarn jar hadoop-mapreduce-examples.jar pi 5 10 The lab asks you to set the min. container size to be 60% of the maximum container size Before: Min. Container Size: 2560MB, Max Container Size: 10GB After: Min. Container Size: 5888MB, Max Container Size: 10GB After changing the size, Ambari, recommends updating a series of config settings for mapreduce2 service which you have to accept. Then you restart all affected services and resubmit the 2 concurrent applications. The expected result is that one goes into running state and completes; then the 2nd finally gets resources and runs afterwards (unlike before where both would be able to run at the same time. ALL GOOD FOR NOW.... After you complete that exercise, you are asked to revert the changes to the YARN service through the config history and restart all affected services. For my own sanity, I resubmitted the 2 concurrent apps and the behavior was not reverted to before i made the changes. My expectation was that all config changes done during the initial change would be reverted. However, the changes to MR2 configuration dont seem to be linked. After some thinking, I went ahead and reverted the config for MR2 service, restarted all affected, resubmitted concurrent yarn apps...and bingo, the original behavior returned. Am I incorrect to expect configuration changes to holistically revert across services? This could be dangerous if you dont track what other service configs got changed by an initial change. Is there a better way?
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache YARN