Member since
11-01-2016
5
Posts
0
Kudos Received
0
Solutions
11-01-2016
04:53 AM
Hi Csguna, I am using hive-1.1.0 in cdh5.8.2 version. There is no issue in result by left join, btw join is not problem here. I am consirned about stages running parallelly.
... View more
11-01-2016
03:09 AM
Hello, I am trying to use hive.exec.parallel functionality by configuring set hive.exec.parallel=true; hive.exec.parallel.thread.number=8; as suggested on support portal case https://cloudera-portal.force.com/5003400000rBeMZ After configurations are made, when I run query to test I did not observe two stages running parallel and execution time is not changed compared to execution with out parallel except parallel in info for stage-5 and stage-1 ( INFO : Starting task [Stage-5:CONDITIONAL] in parallel INFO : Starting task [Stage-1:MAPRED] in parallel) below is my test case CREATE TABLE wbrp_wbrk_join_parallel_test2 TBLPROPERTIES('serialization.null.format'='') AS SELECT TWBRK.LIFRE AS VND_NBR_SAP ,TWBRP.WBELNV AS PONBR ,TWBRK.KUNRE AS STRNBR ,TWBRK.XBLNR AS INVNBR ,TWBRK.XBLNR AS SRTINVNBR ,TWBRK.BLDAT AS INVDT ,cast(TWBRK.BRTWR as decimal(21,2)) AS INV_AMT ,cast(TWBRK.MWSBK as decimal(21,2)) AS TAX_AMT ,TWBRK.NETWR AS PRODUCT ,TWBRP.ARKTX AS ARKTX FROM (select LIFRE, KUNRE, XBLNR, BLDAT, BRTWR, MWSBK, NETWR, WBELN from wbrk where condtion....) TWBRK LEFT JOIN (select WBELNV, ARKTX, WBELN from wbrp where condtion....) TWBRP ON TWBRK.WBELN = TWBRP.WBELN INFO : Query ID = hive_20161101051919_b644f24d-9a0e-4a41-a1b7-d988b992811f INFO : Total jobs = 1 INFO : Starting task [Stage-5:CONDITIONAL] in parallel INFO : Stage-1 is selected by condition resolver. INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in parallel INFO : Number of reduce tasks not specified. Estimated from input data size: 853 INFO : In order to change the average load for a reducer (in bytes): INFO : set hive.exec.reducers.bytes.per.reducer=<number> INFO : In order to limit the maximum number of reducers: INFO : set hive.exec.reducers.max=<number> INFO : In order to set a constant number of reducers: INFO : set mapreduce.job.reduces=<number> INFO : number of splits:305 INFO : Submitting tokens for job: job_1477878984962_0005 INFO : The url to track the job: ..... INFO : Starting Job = job_1477878984962_0005, Tracking URL = ..... INFO : Kill Command = /opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hadoop/bin/hadoop job -kill job_1477878984962_0005 INFO : Hadoop job information for Stage-1: number of mappers: 305; number of reducers: 853 INFO : 2016-11-01 05:19:51,525 Stage-1 map = 0%, reduce = 0% INFO : 2016-11-01 05:20:10,334 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 1202.0 sec INFO : 2016-11-01 05:20:13,526 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 1927.75 sec INFO : 2016-11-01 05:20:14,590 Stage-1 map = 4%, reduce = 0%, Cumulative CPU 2173.92 sec INFO : 2016-11-01 05:20:16,719 Stage-1 map = 6%, reduce = 0%, Cumulative CPU 2670.75 sec INFO : 2016-11-01 05:20:17,795 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 2921.09 sec INFO : 2016-11-01 05:20:19,924 Stage-1 map = 10%, reduce = 0%, Cumulative CPU 3443.04 sec INFO : 2016-11-01 05:20:20,969 Stage-1 map = 11%, reduce = 0%, Cumulative CPU 3647.82 sec INFO : 2016-11-01 05:20:22,065 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 3761.91 sec INFO : 2016-11-01 05:20:23,131 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 4175.81 sec INFO : 2016-11-01 05:20:24,170 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 4395.49 sec INFO : 2016-11-01 05:20:25,243 Stage-1 map = 16%, reduce = 0%, Cumulative CPU 4493.86 sec INFO : 2016-11-01 05:20:26,283 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 4889.57 sec INFO : 2016-11-01 05:20:27,326 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 5146.13 sec INFO : 2016-11-01 05:20:29,459 Stage-1 map = 21%, reduce = 0%, Cumulative CPU 5591.66 sec INFO : 2016-11-01 05:20:30,539 Stage-1 map = 23%, reduce = 0%, Cumulative CPU 5863.12 sec INFO : 2016-11-01 05:20:32,650 Stage-1 map = 25%, reduce = 0%, Cumulative CPU 6360.4 sec INFO : 2016-11-01 05:20:33,721 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 6608.6 sec INFO : 2016-11-01 05:20:34,791 Stage-1 map = 27%, reduce = 0%, Cumulative CPU 6739.82 sec INFO : 2016-11-01 05:20:35,863 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 7135.6 sec INFO : 2016-11-01 05:20:36,930 Stage-1 map = 30%, reduce = 0%, Cumulative CPU 7342.27 sec INFO : 2016-11-01 05:20:37,996 Stage-1 map = 31%, reduce = 0%, Cumulative CPU 7497.89 sec INFO : 2016-11-01 05:20:39,058 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 7855.56 sec INFO : 2016-11-01 05:20:40,137 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 8075.49 sec INFO : 2016-11-01 05:20:41,176 Stage-1 map = 35%, reduce = 0%, Cumulative CPU 8234.62 sec INFO : 2016-11-01 05:20:42,255 Stage-1 map = 37%, reduce = 0%, Cumulative CPU 8610.05 sec INFO : 2016-11-01 05:20:43,331 Stage-1 map = 38%, reduce = 0%, Cumulative CPU 8805.37 sec INFO : 2016-11-01 05:20:44,420 Stage-1 map = 39%, reduce = 0%, Cumulative CPU 8976.91 sec INFO : 2016-11-01 05:20:45,488 Stage-1 map = 41%, reduce = 0%, Cumulative CPU 9376.58 sec INFO : 2016-11-01 05:20:46,560 Stage-1 map = 42%, reduce = 0%, Cumulative CPU 9540.08 sec INFO : 2016-11-01 05:20:47,631 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 9724.87 sec INFO : 2016-11-01 05:20:48,698 Stage-1 map = 45%, reduce = 0%, Cumulative CPU 10102.89 sec INFO : 2016-11-01 05:20:50,794 Stage-1 map = 46%, reduce = 0%, Cumulative CPU 10465.3 sec INFO : 2016-11-01 05:20:51,875 Stage-1 map = 49%, reduce = 0%, Cumulative CPU 10855.49 sec INFO : 2016-11-01 05:20:54,011 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 11202.67 sec INFO : 2016-11-01 05:20:55,091 Stage-1 map = 51%, reduce = 0%, Cumulative CPU 11591.39 sec INFO : 2016-11-01 05:20:56,180 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 11740.18 sec INFO : 2016-11-01 05:20:58,419 Stage-1 map = 53%, reduce = 0%, Cumulative CPU 12370.26 sec INFO : 2016-11-01 05:20:59,480 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 12520.59 sec INFO : 2016-11-01 05:21:00,577 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 12810.8 sec INFO : 2016-11-01 05:21:01,658 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 13114.72 sec INFO : 2016-11-01 05:21:02,746 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 13202.98 sec INFO : 2016-11-01 05:21:03,789 Stage-1 map = 74%, reduce = 0%, Cumulative CPU 13238.86 sec INFO : 2016-11-01 05:21:04,847 Stage-1 map = 75%, reduce = 0%, Cumulative CPU 13309.41 sec INFO : 2016-11-01 05:21:05,898 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 13353.54 sec INFO : 2016-11-01 05:21:06,934 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 13390.74 sec INFO : 2016-11-01 05:21:07,985 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 13407.31 sec INFO : 2016-11-01 05:21:10,088 Stage-1 map = 80%, reduce = 0%, Cumulative CPU 13540.64 sec INFO : 2016-11-01 05:21:11,137 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 13767.89 sec INFO : 2016-11-01 05:21:12,186 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 13960.68 sec INFO : 2016-11-01 05:21:13,234 Stage-1 map = 84%, reduce = 0%, Cumulative CPU 14062.93 sec INFO : 2016-11-01 05:21:14,282 Stage-1 map = 85%, reduce = 0%, Cumulative CPU 14181.51 sec INFO : 2016-11-01 05:21:15,366 Stage-1 map = 86%, reduce = 0%, Cumulative CPU 14226.94 sec INFO : 2016-11-01 05:21:16,459 Stage-1 map = 87%, reduce = 0%, Cumulative CPU 14285.53 sec INFO : 2016-11-01 05:21:17,524 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 14350.46 sec INFO : 2016-11-01 05:21:19,739 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 14420.35 sec INFO : 2016-11-01 05:21:24,138 Stage-1 map = 90%, reduce = 0%, Cumulative CPU 14581.82 sec INFO : 2016-11-01 05:21:26,328 Stage-1 map = 91%, reduce = 0%, Cumulative CPU 14662.0 sec INFO : 2016-11-01 05:21:28,418 Stage-1 map = 91%, reduce = 2%, Cumulative CPU 14902.62 sec INFO : 2016-11-01 05:21:29,471 Stage-1 map = 92%, reduce = 4%, Cumulative CPU 15088.03 sec INFO : 2016-11-01 05:21:30,526 Stage-1 map = 92%, reduce = 5%, Cumulative CPU 15253.04 sec INFO : 2016-11-01 05:21:31,575 Stage-1 map = 92%, reduce = 6%, Cumulative CPU 15345.0 sec INFO : 2016-11-01 05:21:32,627 Stage-1 map = 93%, reduce = 6%, Cumulative CPU 15419.55 sec INFO : 2016-11-01 05:21:34,702 Stage-1 map = 94%, reduce = 7%, Cumulative CPU 15572.62 sec INFO : 2016-11-01 05:21:36,809 Stage-1 map = 95%, reduce = 7%, Cumulative CPU 15720.87 sec INFO : 2016-11-01 05:21:38,909 Stage-1 map = 96%, reduce = 7%, Cumulative CPU 15828.46 sec INFO : 2016-11-01 05:21:39,966 Stage-1 map = 97%, reduce = 7%, Cumulative CPU 15906.51 sec INFO : 2016-11-01 05:21:41,020 Stage-1 map = 98%, reduce = 7%, Cumulative CPU 15959.83 sec INFO : 2016-11-01 05:21:43,131 Stage-1 map = 99%, reduce = 7%, Cumulative CPU 16054.37 sec INFO : 2016-11-01 05:21:46,316 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 16182.34 sec INFO : 2016-11-01 05:21:48,483 Stage-1 map = 100%, reduce = 10%, Cumulative CPU 16429.34 sec INFO : 2016-11-01 05:21:49,609 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 16599.44 sec INFO : 2016-11-01 05:21:50,702 Stage-1 map = 100%, reduce = 15%, Cumulative CPU 16859.14 sec INFO : 2016-11-01 05:21:51,800 Stage-1 map = 100%, reduce = 17%, Cumulative CPU 17132.1 sec INFO : 2016-11-01 05:21:55,227 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 17848.45 sec INFO : 2016-11-01 05:22:01,829 Stage-1 map = 100%, reduce = 19%, Cumulative CPU 19385.42 sec INFO : 2016-11-01 05:22:05,149 Stage-1 map = 100%, reduce = 20%, Cumulative CPU 20165.94 sec INFO : 2016-11-01 05:22:07,333 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 20676.15 sec INFO : 2016-11-01 05:22:09,583 Stage-1 map = 100%, reduce = 22%, Cumulative CPU 21173.0 sec INFO : 2016-11-01 05:22:11,883 Stage-1 map = 100%, reduce = 23%, Cumulative CPU 21631.91 sec INFO : 2016-11-01 05:22:14,118 Stage-1 map = 100%, reduce = 24%, Cumulative CPU 22133.32 sec INFO : 2016-11-01 05:22:15,218 Stage-1 map = 100%, reduce = 25%, Cumulative CPU 22363.35 sec INFO : 2016-11-01 05:22:17,480 Stage-1 map = 100%, reduce = 26%, Cumulative CPU 22777.09 sec INFO : 2016-11-01 05:22:19,736 Stage-1 map = 100%, reduce = 27%, Cumulative CPU 23215.03 sec INFO : 2016-11-01 05:22:23,064 Stage-1 map = 100%, reduce = 28%, Cumulative CPU 23604.45 sec INFO : 2016-11-01 05:22:29,675 Stage-1 map = 100%, reduce = 29%, Cumulative CPU 23904.9 sec INFO : 2016-11-01 05:22:30,973 Stage-1 map = 100%, reduce = 30%, Cumulative CPU 23987.76 sec INFO : 2016-11-01 05:22:32,052 Stage-1 map = 100%, reduce = 31%, Cumulative CPU 24089.47 sec INFO : 2016-11-01 05:22:33,172 Stage-1 map = 100%, reduce = 32%, Cumulative CPU 24188.51 sec INFO : 2016-11-01 05:22:34,337 Stage-1 map = 100%, reduce = 33%, Cumulative CPU 24282.07 sec INFO : 2016-11-01 05:22:35,447 Stage-1 map = 100%, reduce = 35%, Cumulative CPU 24468.31 sec INFO : 2016-11-01 05:22:36,570 Stage-1 map = 100%, reduce = 37%, Cumulative CPU 24694.34 sec INFO : 2016-11-01 05:22:37,698 Stage-1 map = 100%, reduce = 38%, Cumulative CPU 24851.7 sec INFO : 2016-11-01 05:22:38,848 Stage-1 map = 100%, reduce = 40%, Cumulative CPU 25091.25 sec INFO : 2016-11-01 05:22:39,966 Stage-1 map = 100%, reduce = 41%, Cumulative CPU 25375.97 sec INFO : 2016-11-01 05:22:41,058 Stage-1 map = 100%, reduce = 42%, Cumulative CPU 25553.66 sec INFO : 2016-11-01 05:22:42,199 Stage-1 map = 100%, reduce = 43%, Cumulative CPU 25785.05 sec INFO : 2016-11-01 05:22:43,319 Stage-1 map = 100%, reduce = 44%, Cumulative CPU 26048.21 sec INFO : 2016-11-01 05:22:45,555 Stage-1 map = 100%, reduce = 45%, Cumulative CPU 26454.59 sec INFO : 2016-11-01 05:22:47,750 Stage-1 map = 100%, reduce = 46%, Cumulative CPU 26970.97 sec INFO : 2016-11-01 05:22:48,837 Stage-1 map = 100%, reduce = 47%, Cumulative CPU 27210.35 sec INFO : 2016-11-01 05:22:51,061 Stage-1 map = 100%, reduce = 48%, Cumulative CPU 27752.7 sec INFO : 2016-11-01 05:22:53,336 Stage-1 map = 100%, reduce = 49%, Cumulative CPU 28285.81 sec INFO : 2016-11-01 05:22:55,553 Stage-1 map = 100%, reduce = 50%, Cumulative CPU 28762.43 sec INFO : 2016-11-01 05:22:58,900 Stage-1 map = 100%, reduce = 51%, Cumulative CPU 29489.7 sec INFO : 2016-11-01 05:23:01,087 Stage-1 map = 100%, reduce = 52%, Cumulative CPU 29946.92 sec INFO : 2016-11-01 05:23:03,329 Stage-1 map = 100%, reduce = 53%, Cumulative CPU 30397.54 sec INFO : 2016-11-01 05:23:05,681 Stage-1 map = 100%, reduce = 54%, Cumulative CPU 30800.85 sec INFO : 2016-11-01 05:23:06,773 Stage-1 map = 100%, reduce = 55%, Cumulative CPU 31014.07 sec INFO : 2016-11-01 05:23:10,221 Stage-1 map = 100%, reduce = 56%, Cumulative CPU 31489.0 sec INFO : 2016-11-01 05:23:12,450 Stage-1 map = 100%, reduce = 57%, Cumulative CPU 31712.91 sec INFO : 2016-11-01 05:23:14,691 Stage-1 map = 100%, reduce = 58%, Cumulative CPU 31892.23 sec INFO : 2016-11-01 05:23:16,966 Stage-1 map = 100%, reduce = 59%, Cumulative CPU 32056.41 sec INFO : 2016-11-01 05:23:18,090 Stage-1 map = 100%, reduce = 60%, Cumulative CPU 32179.45 sec INFO : 2016-11-01 05:23:20,275 Stage-1 map = 100%, reduce = 61%, Cumulative CPU 32422.81 sec INFO : 2016-11-01 05:23:21,361 Stage-1 map = 100%, reduce = 62%, Cumulative CPU 32557.59 sec INFO : 2016-11-01 05:23:22,451 Stage-1 map = 100%, reduce = 63%, Cumulative CPU 32678.22 sec INFO : 2016-11-01 05:23:23,600 Stage-1 map = 100%, reduce = 64%, Cumulative CPU 32839.1 sec INFO : 2016-11-01 05:23:24,723 Stage-1 map = 100%, reduce = 65%, Cumulative CPU 33012.26 sec INFO : 2016-11-01 05:23:25,901 Stage-1 map = 100%, reduce = 66%, Cumulative CPU 33185.33 sec INFO : 2016-11-01 05:23:27,033 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 33413.55 sec INFO : 2016-11-01 05:23:28,129 Stage-1 map = 100%, reduce = 69%, Cumulative CPU 33630.55 sec INFO : 2016-11-01 05:23:29,201 Stage-1 map = 100%, reduce = 70%, Cumulative CPU 33851.67 sec INFO : 2016-11-01 05:23:30,303 Stage-1 map = 100%, reduce = 71%, Cumulative CPU 34033.29 sec INFO : 2016-11-01 05:23:31,410 Stage-1 map = 100%, reduce = 72%, Cumulative CPU 34296.61 sec INFO : 2016-11-01 05:23:32,535 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 34534.44 sec INFO : 2016-11-01 05:23:33,679 Stage-1 map = 100%, reduce = 74%, Cumulative CPU 34737.27 sec INFO : 2016-11-01 05:23:34,752 Stage-1 map = 100%, reduce = 75%, Cumulative CPU 34979.43 sec INFO : 2016-11-01 05:23:37,021 Stage-1 map = 100%, reduce = 76%, Cumulative CPU 35511.73 sec INFO : 2016-11-01 05:23:40,367 Stage-1 map = 100%, reduce = 77%, Cumulative CPU 36188.62 sec INFO : 2016-11-01 05:23:41,748 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 36470.54 sec INFO : 2016-11-01 05:23:45,101 Stage-1 map = 100%, reduce = 79%, Cumulative CPU 37183.85 sec INFO : 2016-11-01 05:23:47,375 Stage-1 map = 100%, reduce = 80%, Cumulative CPU 37647.46 sec INFO : 2016-11-01 05:23:49,703 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 38113.9 sec INFO : 2016-11-01 05:23:50,865 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 38352.23 sec INFO : 2016-11-01 05:23:53,078 Stage-1 map = 100%, reduce = 83%, Cumulative CPU 38739.22 sec INFO : 2016-11-01 05:23:54,232 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 38969.44 sec INFO : 2016-11-01 05:23:56,427 Stage-1 map = 100%, reduce = 85%, Cumulative CPU 39301.84 sec INFO : 2016-11-01 05:23:59,912 Stage-1 map = 100%, reduce = 86%, Cumulative CPU 39776.8 sec INFO : 2016-11-01 05:24:01,000 Stage-1 map = 100%, reduce = 87%, Cumulative CPU 39895.9 sec INFO : 2016-11-01 05:24:03,206 Stage-1 map = 100%, reduce = 88%, Cumulative CPU 40158.17 sec INFO : 2016-11-01 05:24:05,403 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 40369.82 sec INFO : 2016-11-01 05:24:06,495 Stage-1 map = 100%, reduce = 90%, Cumulative CPU 40571.76 sec INFO : 2016-11-01 05:24:08,735 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 41004.2 sec INFO : 2016-11-01 05:24:09,854 Stage-1 map = 100%, reduce = 93%, Cumulative CPU 41252.37 sec INFO : 2016-11-01 05:24:10,999 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 41529.78 sec INFO : 2016-11-01 05:24:12,122 Stage-1 map = 100%, reduce = 95%, Cumulative CPU 41862.09 sec INFO : 2016-11-01 05:24:13,241 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 42124.96 sec INFO : 2016-11-01 05:24:15,400 Stage-1 map = 100%, reduce = 97%, Cumulative CPU 42596.22 sec INFO : 2016-11-01 05:24:18,654 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 43166.7 sec INFO : 2016-11-01 05:24:21,817 Stage-1 map = 100%, reduce = 99%, Cumulative CPU 43557.51 sec INFO : 2016-11-01 05:24:29,135 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 43925.39 sec INFO : MapReduce Total cumulative CPU time: 0 days 12 hours 12 minutes 5 seconds 390 msec INFO : Ended Job = job_1477878984962_0005 INFO : Starting task [Stage-0:MOVE] in serial mode INFO : Moving data to: ....... INFO : Starting task [Stage-6:DDL] in serial mode INFO : Starting task [Stage-2:STATS] in serial mode INFO : Table lol_ca_sap_dsd_c.tmptbl_lol_ca_sap_dsd_pr_sc2 stats: [numFiles=853, numRows=829962807, totalSize=81514377225, rawDataSize=80684414418] INFO : MapReduce Jobs Launched: INFO : Stage-Stage-1: Map: 305 Reduce: 853 Cumulative CPU: 43925.39 sec HDFS Read: 57222810740 HDFS Write: 81514472496 SUCCESS INFO : Total MapReduce CPU Time Spent: 0 days 12 hours 12 minutes 5 seconds 390 msec INFO : Completed executing command(queryId=hive_20161101051919_b644f24d-9a0e-4a41-a1b7-d988b992811f); Time taken: 310.066 seconds INFO : OK No rows affected (310.8 seconds)
... View more
Labels: