Reply
Highlighted
New Contributor
Posts: 3
Registered: ‎04-28-2016

how to set certain configurations for Hive?

[ Edited ]

Hi

I have CDH 5.4.8 running on 1 master +3 worker nodes. I am running a "black box" application which launch couple of hive queries.  After five queries the sixth one is failing with following messages

 

------------------------------

Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.4.8.jar!/hive-log4j.properties
hive.execution.engine=mr
hive.cbo.enable=true
hive.stats.fetch.partition.stats=true
hive.script.operator.truncate.env=false
hive.compute.query.using.stats=false
hive.vectorized.execution.enabled=false
hive.vectorized.execution.reduce.enabled=true
hive.stats.autogather=true
mapreduce.input.fileinputformat.split.minsize=1
mapreduce.input.fileinputformat.split.maxsize=256000000
hive.exec.reducers.bytes.per.reducer=67108864
hive.exec.reducers.max=1099
hive.exec.parallel=false
hive.exec.parallel.thread.number=8
hive.exec.compress.intermediate=false
hive.exec.compress.output=false
mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec
mapred.output.compression.codec=org.apache.hadoop.io.compress.DefaultCodec
hive.default.fileformat=TEXTFILE
hive.auto.convert.sortmerge.join=false
hive.auto.convert.sortmerge.join.noconditionaltask is undefined
hive.optimize.bucketmapjoin=false
hive.optimize.bucketmapjoin.sortedmerge=false
hive.auto.convert.join.noconditionaltask.size=10000000
hive.auto.convert.join=true
hive.optimize.mapjoin.mapreduce is undefined
hive.mapred.local.mem=0
hive.mapjoin.smalltable.filesize=25000000
hive.mapjoin.localtask.max.memory.usage=0.9
hive.optimize.skewjoin=false
hive.optimize.skewjoin.compiletime=false
hive.optimize.ppd=true
hive.optimize.ppd.storage=true
hive.ppd.recognizetransivity=true
hive.optimize.index.filter=false
hive.optimize.sampling.orderby=false
hive.optimize.sampling.orderby.number=1000
hive.optimize.sampling.orderby.percent=0.1
bigbench.hive.optimize.sampling.orderby=true
bigbench.hive.optimize.sampling.orderby.number=20000
bigbench.hive.optimize.sampling.orderby.percent=0.1
hive.groupby.skewindata=false
hive.exec.submit.local.task.via.child=true
hive.exec.compress.output=false
OK
Time taken: 0.297 seconds
OK
Time taken: 0.422 seconds
Query ID = root_20160428144040_f9a22698-8869-41ef-ad4f-9f543f6d5588
Total jobs = 1
Execution log at: /tmp/root/root_20160428144040_f9a22698-8869-41ef-ad4f-9f543f6d5588.log
2016-04-28 02:40:43 Starting to launch local task to process map join; maximum memory = 1029701632
2016-04-28 02:40:46 Processing rows: 200000 Hashtable size: 199999 Memory usage: 222600096 percentage: 0.216
2016-04-28 02:40:47 Processing rows: 300000 Hashtable size: 299999 Memory usage: 259636328 percentage: 0.252
2016-04-28 02:40:47 Processing rows: 400000 Hashtable size: 399999 Memory usage: 291381664 percentage: 0.283
2016-04-28 02:40:47 Processing rows: 500000 Hashtable size: 499999 Memory usage: 219377648 percentage: 0.213
2016-04-28 02:40:47 Processing rows: 600000 Hashtable size: 599999 Memory usage: 255117584 percentage: 0.248
2016-04-28 02:40:47 Processing rows: 700000 Hashtable size: 699999 Memory usage: 290857528 percentage: 0.282
2016-04-28 02:40:47 Processing rows: 800000 Hashtable size: 799999 Memory usage: 326597448 percentage: 0.317
2016-04-28 02:40:47 Processing rows: 900000 Hashtable size: 899999 Memory usage: 357231680 percentage: 0.347
2016-04-28 02:40:47 Processing rows: 1000000 Hashtable size: 999999 Memory usage: 392971624 percentage: 0.382
2016-04-28 02:40:48 Processing rows: 1100000 Hashtable size: 1099999 Memory usage: 437100200 percentage: 0.424
2016-04-28 02:40:48 Processing rows: 1200000 Hashtable size: 1199999 Memory usage: 467734416 percentage: 0.454
2016-04-28 02:40:48 Processing rows: 1300000 Hashtable size: 1299999 Memory usage: 480668248 percentage: 0.467
2016-04-28 02:40:48 Processing rows: 1400000 Hashtable size: 1399999 Memory usage: 516908528 percentage: 0.502
2016-04-28 02:40:48 Processing rows: 1500000 Hashtable size: 1499999 Memory usage: 550661256 percentage: 0.535
2016-04-28 02:40:48 Processing rows: 1600000 Hashtable size: 1599999 Memory usage: 586901544 percentage: 0.57
Execution failed with exit status: 3
Obtaining error information

Task failed!
Task ID:
  Stage-15

Logs:

-------------------------------

 

while scanning other logs and researching about "error" messages, I found that setting hive.auto.convert.join to false the highlighted parameter)  could potentially help me. this parameter is by default set to true

 

I tried checking and unchecking (both try failed) a box next to this configuration on  CDH admin>HIVE>configuration page.   (attched image)test.png

i restarted HIVE service as suggested. but I still see query fail and report suggest that hive.auto.convert.join is still set as "true" .

 

How do I set this parameter to False throuh HIVE>configuration page? 

Announcements