Support Questions
Find answers, ask questions, and share your expertise

Is there a way of seeing/reporting progress/time remaining on a HiveQL query through the beeline command line?

Contributor

..title says it all really. Would like some indication on time remaining of queries run through beeline.

Thanks,

Mike

1 REPLY 1

Rising Star

Hi @mike harding

I don't know the way of find time remaining, but you can certainly see the progress via beeline.

Let's say you are going to run 'select count(*)' from some table and explain show one mapper and one reducer:

0: jdbc:hive2://nat-d7-0629-hs2-12-rangeroff-> explain select count(*) from olympic_avro;
+-------------------------------------------------------------------------------------------------------------+--+
|                                                   Explain                                                   |
+-------------------------------------------------------------------------------------------------------------+--+
| STAGE DEPENDENCIES:                                                                                         |
|   Stage-1 is a root stage                                                                                   |
|   Stage-0 depends on stages: Stage-1                                                                        |
|                                                                                                             |
| STAGE PLANS:                                                                                                |
|   Stage: Stage-1                                                                                            |
|     Tez                                                                                                     |
|       DagId: hive_20160706123306_5e633ebb-9c9c-43df-8fb2-dc16c59ccf72:332                                   |
|       Edges:                                                                                                |
|         Reducer 2 <- Map 1 (SIMPLE_EDGE)                                                                    |
|       Vertices:                                                                                             |
|         Map 1                                                                                               |
|             Map Operator Tree:                                                                              |
|                 TableScan                                                                                   |
|                   alias: olympic_avro                                                                       |
|                   Statistics: Num rows: 1 Data size: 545870 Basic stats: COMPLETE Column stats: COMPLETE    |
|                   Select Operator                                                                           |
|                     Statistics: Num rows: 1 Data size: 545870 Basic stats: COMPLETE Column stats: COMPLETE  |
|                     Group By Operator                                                                       |
|                       aggregations: count()                                                                 |
|                       mode: hash                                                                            |
|                       outputColumnNames: _col0                                                              |
|                       Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE     |
|                       Reduce Output Operator                                                                |
|                         sort order:                                                                         |
|                         Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE   |
|                         value expressions: _col0 (type: bigint)                                             |
|         Reducer 2                                                                                           |
|             Reduce Operator Tree:                                                                           |
|               Group By Operator                                                                             |
|                 aggregations: count(VALUE._col0)                                                            |
|                 mode: mergepartial                                                                          |
|                 outputColumnNames: _col0                                                                    |
|                 Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE           |
|                 File Output Operator                                                                        |
|                   compressed: false                                                                         |
|                   Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE         |
|                   table:                                                                                    |
|                       input format: org.apache.hadoop.mapred.TextInputFormat                                |
|                       output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat             |
|                       serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe                             |
|                                                                                                             |
|   Stage: Stage-0                                                                                            |
|     Fetch Operator                                                                                          |
|       limit: -1                                                                                             |
|       Processor Tree:                                                                                       |
|         ListSink                                                                                            |
|                                                                                                             |
+-------------------------------------------------------------------------------------------------------------+--+
48 rows selected (0.262 seconds)

When you run this via beeline, you can see the progress of the mapper and reducer as shown in the below:

0: jdbc:hive2://nat-d7-0629-hs2-12-rangeroff-> select count(*) from olympic_avro;
INFO  : Session is already open
INFO  : Dag name: select count(*) from olympic_avro(Stage-1)
INFO  :
INFO  : Status: Running (Executing on YARN cluster with App id application_1467208466253_0312)
INFO  : Map 1: 0/1Reducer 2: 0/1
INFO  : Map 1: 0/1Reducer 2: 0/1
INFO  : Map 1: 0/1Reducer 2: 0/1
INFO  : Map 1: 0/1Reducer 2: 0/1
INFO  : Map 1: 0(+1)/1Reducer 2: 0/1
INFO  : Map 1: 0(+1)/1Reducer 2: 0/1
INFO  : Map 1: 1/1Reducer 2: 0/1
INFO  : Map 1: 1/1Reducer 2: 0(+1)/1
INFO  : Map 1: 1/1Reducer 2: 1/1
+-------+--+
|  _c0  |
+-------+--+
| 8618  |
+-------+--+
1 row selected (16.531 seconds)