Not sure what you mean with consuming factor. You can see that the reducer took 15 min and had 23m records as input. You can also see that the shuffle had 500MB. Which should not take 15 minutes in the reducer to count by group. So I am wondering if you by any chance do not have enough memory for the reducers and they cannot keep the groups ( 8m ) in memory or something. You should definitely increase the number of reducers. Since you have 8m groups and both tasks took a long time ( so most likely not a single huge group ) you can essentially create as many as you have task slots in the cluster. But I would also look at my hive memory configuration to see if I would increase the task memory and have a look at what happens on the machines running a reducer since aggregating 23m rows should not take 15 minutes. Quick way to test with more reducers: SET MAPRED.REDUCE.TASKS = x; ( where x is the number of task slots in your cluster ) Quick way to test with more RAM: set hive.tez.java.opts="-Xmx3400m"; set hive.tez.container.size = 4096; where the Xmx RAM value is 75-90% of the container size. Depending on your level of conservatism. ... View more

Get date from Filename There are some ways to get at the filename in mapreduce but its difficult. MapReduce by definition abstracts filenames away. You have two options there: 1) Use a little python/java/shell whatever preprocessing script OUTSIDE hadoop that adds a field with the date to each row of each file taken from the filename. Easy but not that scalable 2) Write your own recordreader 3) Pig seems to provide some value called tagsource that can do the same http://stackoverflow.com/questions/9751480/how-can-i-incorporate-the-current-input-filename-into-my-pig-latin-script 4) Hive has a hidden column for the filename so you could use that to compute a date column https://cwiki.apache.org/confluence/display/Hive/LanguageManual+VirtualColumns ... View more

Ah just read that its an export. But again I don't think there is any automated way to do that. So I suppose your only choice is to make a custom shell/ssh action that runs a script ( shell, python ... ) that lists the files from the directory and then executes sqoop jobs for them. Would be my approach. ... View more

And shameless plug: https://community.hortonworks.com/content/kbentry/25726/spark-streaming-explained-kafka-to-phoenix.html You can have a look at the parser class I wrote. You would need to write something similar that parses your JSON object and returns a Java/Scala object that you can then use in your analytics ... View more

What do you mean with clusters? A Datamining clustering or segmentation algorithm? In this case you have different options but Spark ML is definitely a strong contender. http://spark.apache.org/docs/latest/ml-clustering.html ... View more