An easier solution would be to read every row, and then add a check to see if it's a header row by checking one of the columns that you know only appears in a header row. ... View more

Sorting in MR applies to two areas: Sort output by keys: this done "naturally" in the sense that the keys are sorted as they come into the reducer. The compareTo method in the key class determines this natural sorting Secondary sort: both the keys and values are sorted. That involves writing a group comparator class and then registering that class with the MR Job using the setGroupingComparator class The exam objective you listed above is referring to both. The first one is fairly straightforward - you implement the compareTo method in your key class. The secondary sort involves a bit more work. There is a nice blog here that has an example of how to implement a secondary sort: https://vangjee.wordpress.com/2012/03/20/secondary-sorting-aka-sorting-values-in-hadoops-mapreduce-programming-paradigm/ ... View more

@Sanjay Ungarala - thank you for pointing this out. We decided a while back to remove this objective from the HDPCD:Java exam, but the website was not changed accordingly. I just updated the website and removed that objective. LocalResource is used heavily in YARN, but it is beyond the scope of our Java exam. ... View more

@Mahesh Deshmukh I'm not sure what your need is, but null values should be filtered first. The general rule of thumb in Pig is to "filter early and often" to minimize the amount of data that gets shuffled and sorted, so before the foreach: For example: a = "some Pig relation" b = filter a by $1 is not null; //filter out tuples where the $1 field is null c = foreach b generate ... //no need to worry about $1 being null The term "empty" refers to bags typically, and in particular you can use the isEmpty function to check if a bag is empty. You normally do this after a GROUP command: a = "some Pig relation" b = group a by $3; c = filter b by not IsEmpty(group); What are you trying to accomplish? ... View more

Many of the Pig operators have a PARALLEL option for specifying the number of reducers, which also determines the number of output files. For the intent of the practice exam and the real exam, using PARALLEL is all you need to accomplish this task. ... View more