Is there a way to kill a stalled query that is running under LLAP? In regular Hive, you would kill the YARN container and call it a day. In LLAP, doing so would kill the daemons! ... View more

We have a data source that is scheduled based on a 4-4-4 calendar (aka every 4 weeks). We are looking to use coordinators and trigger file to notice the data arrival. I'd like to organize the folder structures based on yyyy/period_id (there are 13 periods in a year). We have 3 data feeds per period and each will arrive staggered over 3-4 days...so using mmdd is not ideal. For example, the calendar would look as follows. Start Date Jan 16 End Date Dec 18 Jan 16 Instance 1 Path: 2017/1/* Feb 13 Instance 2 Mar 13 Instance 3 Apr 10 Instance 4 May 8 Instance 5 Jun 5 Instance 6 Jul 3 Instance 7 Jul 31 Instance 8 Aug 28 Instance 9 Sep 25 Instance 10 Oct 23 Instance 11 Nov 20 Instance 12 Dec 18 Instance 13 Path: 2017/13/* Is there a function in OOzie language that would allow to return the incremental instantiation # for the oozie coordinators? I.e, the first job execution in July would return 7 and the 2nd job execution in July would return 8 Am i looking at this the wrong way? Another option would be to have a script lookup a value in a table for a date range and return the period ID which would be a bit more complex Thanks ... View more

The behavior of the -replaceLabelsOnNode of yarn rmadmin is not what I expect. [yarn@master1 ~]$ yarn rmadmin -replaceLabelsOnNode data2=GPU [yarn@master1 ~]$ echo $? 0 However, looking at the resource manager UI and clicking on nodes, the label is not applied. Therefore, if the command didnt accomplish the task, why didnt it fail? [yarn@master1 ~]$ yarn rmadmin -replaceLabelsOnNode data2.c.hdpfastrack.internal=GPU Using the full hostname, the command executes and the expected output is seen in the RM UI. At least, if i mistype the node label, yarn rmadmin will complain that the label is not recognized. ... View more

I am currently going through the HDP FastTrack labs, more precisely "Lab 11: Managing YARN containers and queues" Basically, you run 2 concurrent example yarn jobs to understand the behavior of the minimum container size and the ability to run multiple applications concurrently, depending on container availability. Job: yarn jar hadoop-mapreduce-examples.jar pi 5 10 The lab asks you to set the min. container size to be 60% of the maximum container size Before: Min. Container Size: 2560MB, Max Container Size: 10GB After: Min. Container Size: 5888MB, Max Container Size: 10GB After changing the size, Ambari, recommends updating a series of config settings for mapreduce2 service which you have to accept. Then you restart all affected services and resubmit the 2 concurrent applications. The expected result is that one goes into running state and completes; then the 2nd finally gets resources and runs afterwards (unlike before where both would be able to run at the same time. ALL GOOD FOR NOW.... After you complete that exercise, you are asked to revert the changes to the YARN service through the config history and restart all affected services. For my own sanity, I resubmitted the 2 concurrent apps and the behavior was not reverted to before i made the changes. My expectation was that all config changes done during the initial change would be reverted. However, the changes to MR2 configuration dont seem to be linked. After some thinking, I went ahead and reverted the config for MR2 service, restarted all affected, resubmitted concurrent yarn apps...and bingo, the original behavior returned. Am I incorrect to expect configuration changes to holistically revert across services? This could be dangerous if you dont track what other service configs got changed by an initial change. Is there a better way? ... View more