Member since
11-20-2015
8
Posts
3
Kudos Received
0
Solutions
06-02-2016
07:59 AM
2 Kudos
Etc. Normally you have more mappers for two reasons: a) in most analytical tasks you can filter out a huge percentage of the data at the source b) If you can choose where to compute things its better to do it in the mapper. Therefore you would want more reducers for any task where you do heavy tasks after a group by/join and you cannot filter out data in the mapper. Things I could think of: Running DataMining inside MapReduce to for example create one forecast model per product. In that case reading the data in the mapper is trivial but the modelling step running in the reducer is heavy so you would want more reducers than mappers. Inserting data into a ( partitioned ) ORC Hive table: Creating ORC files is pretty heavy and you want one reducer per partition and potentially a couple files for each. While reading a delimited file is very lightweight, so here you also want more reducers than mappers. ...
... View more
02-22-2016
06:16 PM
1 Kudo
You should be able to see the query in the HiveServer log or a Hive-related UI, like Hive View for Ambari or Hue (there should be a query history). The Resourcemanager does not show the full query, because the job is only named after a partial of the query. Why only a partial? Some queries can be quite large and Job Name is limited in regards to the allowed #characters.
... View more