question Re: Identify number of Mappers & Reducers launched in a Pig script in Archives of Support Questions (Read Only)

Identify number of Mappers & Reducers launched in a Pig script

dhiwatdg — Fri, 11 Nov 2016 07:59:28 GMT

I'm using Hortonworks' sandbox VM to run Hadoop services. I executed a Pig filter script in Tez mode. Unlike Hive, the Pig log (console) doesn't show any information about the number of mappers & reducers being executed. Am I looking at a wrong place?

Re: Identify number of Mappers & Reducers launched in a Pig script

srai1 — Fri, 11 Nov 2016 08:05:48 GMT

@Dhiwa TdG

If you verified your output for Hive via Hive/Beeline shell, then its a different story, you are actually seeing the output on STDOUT. With Pig, you can try using pig view instead.

Ambari -> admin (drop down) -> Manage Ambari -> Views -> PIG -> Create Instance (If you don't have a PIG view already)

Re: Identify number of Mappers & Reducers launched in a Pig script

dhiwatdg — Fri, 11 Nov 2016 08:13:23 GMT

Thank your for your response but I had done that already. I executed the Pig script both via terminal (pig -x tez script.pig)& from Ambari/Pig view. Also when I expanded the logs section in Pig view it was very similar too the one which I had in terminal (console). I don't see any mappers nor reducer counts. 😞

Re: Identify number of Mappers & Reducers launched in a Pig script

gkeys — Sun, 18 Aug 2019 10:33:11 GMT

For tez "tasks" represent map operations or reduce operations. A DAG is a full workflow (job) of vertices (processing of tasks) and edges (data movement between vertices).

See these links for a more detailed discussion:

http://hortonworks.com/blog/expressing-data-processing-in-apache-tez/

https://community.hortonworks.com/questions/32164/question-on-tez-dag-task-and-pig-on-tez.html

https://cwiki.apache.org/confluence/display/TEZ/How+initial+task+parallelism+works

You can see number of tasks on the console output:

You can also see this in Ambari Tez view (and drill down for greater details)

See this for understanding Ambari Tez view: https://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.0/bk_ambari_views_guide/content/section_using_tez_view.html