Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to set certain configurations for Hive?

Highlighted

how to set certain configurations for Hive?

New Contributor

Hi

I have CDH 5.4.8 running on 1 master +3 worker nodes. I am running a "black box" application which launch couple of hive queries.  After five queries the sixth one is failing with following messages

 

------------------------------

Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.4.8.jar!/hive-log4j.properties
hive.execution.engine=mr
hive.cbo.enable=true
hive.stats.fetch.partition.stats=true
hive.script.operator.truncate.env=false
hive.compute.query.using.stats=false
hive.vectorized.execution.enabled=false
hive.vectorized.execution.reduce.enabled=true
hive.stats.autogather=true
mapreduce.input.fileinputformat.split.minsize=1
mapreduce.input.fileinputformat.split.maxsize=256000000
hive.exec.reducers.bytes.per.reducer=67108864
hive.exec.reducers.max=1099
hive.exec.parallel=false
hive.exec.parallel.thread.number=8
hive.exec.compress.intermediate=false
hive.exec.compress.output=false
mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec
mapred.output.compression.codec=org.apache.hadoop.io.compress.DefaultCodec
hive.default.fileformat=TEXTFILE
hive.auto.convert.sortmerge.join=false
hive.auto.convert.sortmerge.join.noconditionaltask is undefined
hive.optimize.bucketmapjoin=false
hive.optimize.bucketmapjoin.sortedmerge=false
hive.auto.convert.join.noconditionaltask.size=10000000
hive.auto.convert.join=true
hive.optimize.mapjoin.mapreduce is undefined
hive.mapred.local.mem=0
hive.mapjoin.smalltable.filesize=25000000
hive.mapjoin.localtask.max.memory.usage=0.9
hive.optimize.skewjoin=false
hive.optimize.skewjoin.compiletime=false
hive.optimize.ppd=true
hive.optimize.ppd.storage=true
hive.ppd.recognizetransivity=true
hive.optimize.index.filter=false
hive.optimize.sampling.orderby=false
hive.optimize.sampling.orderby.number=1000
hive.optimize.sampling.orderby.percent=0.1
bigbench.hive.optimize.sampling.orderby=true
bigbench.hive.optimize.sampling.orderby.number=20000
bigbench.hive.optimize.sampling.orderby.percent=0.1
hive.groupby.skewindata=false
hive.exec.submit.local.task.via.child=true
hive.exec.compress.output=false
OK
Time taken: 0.297 seconds
OK
Time taken: 0.422 seconds
Query ID = root_20160428144040_f9a22698-8869-41ef-ad4f-9f543f6d5588
Total jobs = 1
Execution log at: /tmp/root/root_20160428144040_f9a22698-8869-41ef-ad4f-9f543f6d5588.log
2016-04-28 02:40:43 Starting to launch local task to process map join; maximum memory = 1029701632
2016-04-28 02:40:46 Processing rows: 200000 Hashtable size: 199999 Memory usage: 222600096 percentage: 0.216
2016-04-28 02:40:47 Processing rows: 300000 Hashtable size: 299999 Memory usage: 259636328 percentage: 0.252
2016-04-28 02:40:47 Processing rows: 400000 Hashtable size: 399999 Memory usage: 291381664 percentage: 0.283
2016-04-28 02:40:47 Processing rows: 500000 Hashtable size: 499999 Memory usage: 219377648 percentage: 0.213
2016-04-28 02:40:47 Processing rows: 600000 Hashtable size: 599999 Memory usage: 255117584 percentage: 0.248
2016-04-28 02:40:47 Processing rows: 700000 Hashtable size: 699999 Memory usage: 290857528 percentage: 0.282
2016-04-28 02:40:47 Processing rows: 800000 Hashtable size: 799999 Memory usage: 326597448 percentage: 0.317
2016-04-28 02:40:47 Processing rows: 900000 Hashtable size: 899999 Memory usage: 357231680 percentage: 0.347
2016-04-28 02:40:47 Processing rows: 1000000 Hashtable size: 999999 Memory usage: 392971624 percentage: 0.382
2016-04-28 02:40:48 Processing rows: 1100000 Hashtable size: 1099999 Memory usage: 437100200 percentage: 0.424
2016-04-28 02:40:48 Processing rows: 1200000 Hashtable size: 1199999 Memory usage: 467734416 percentage: 0.454
2016-04-28 02:40:48 Processing rows: 1300000 Hashtable size: 1299999 Memory usage: 480668248 percentage: 0.467
2016-04-28 02:40:48 Processing rows: 1400000 Hashtable size: 1399999 Memory usage: 516908528 percentage: 0.502
2016-04-28 02:40:48 Processing rows: 1500000 Hashtable size: 1499999 Memory usage: 550661256 percentage: 0.535
2016-04-28 02:40:48 Processing rows: 1600000 Hashtable size: 1599999 Memory usage: 586901544 percentage: 0.57
Execution failed with exit status: 3
Obtaining error information

Task failed!
Task ID:
  Stage-15

Logs:

-------------------------------

 

while scanning other logs and researching about "error" messages, I found that setting hive.auto.convert.join to false the highlighted parameter)  could potentially help me. this parameter is by default set to true

 

I tried checking and unchecking (both try failed) a box next to this configuration on  CDH admin>HIVE>configuration page.   (attched image)test.png

i restarted HIVE service as suggested. but I still see query fail and report suggest that hive.auto.convert.join is still set as "true" .

 

How do I set this parameter to False throuh HIVE>configuration page?