About MasterOfPuppets

MasterOfPuppets · ‎01-26-2017

I've set-up hive.prewarm.enabled=true and it did not improve the slow latency to start and initialize executors. It still takes about 15seconds to initialize things. Any idea ?

MasterOfPuppets · ‎01-25-2017

Thanks for the feedback..This is exactly what i was looking for.

MasterOfPuppets · ‎01-25-2017

Hello. Anyone knows in what file i may update hive.prewarm.enabled property ? I've checked hive-site.xml and that property does not exist. Thanks

MasterOfPuppets · ‎01-17-2017

I think my query performs well. From the Hadoop JobHistory Web Interface i see the following results for my query: Submit Time : 2017.01.17 12:01:32 EST Start Time : Tue Jan 17 12:01:35 EST 2017 Finish Time : 2017.01.17 12:01:36 EST However the Hive Client renders data in seconds... what could be the bottleneck then ?

MasterOfPuppets · ‎01-17-2017

I've improved a bit the query executiont time by setting and running : SET hive.stats.fetch.column.stats=true; SET hive.stats.fetch.partition.stats=true; SET hive.cbo.enable=true; SET hive.stats.autogather=true; CREATE INDEX table1_idx_1 ON TABLE table1 (col2) AS BITMAP; ANALYZE TABLE table1 COMPUTE STATISTICS for COLUMNS col2; SELECT col1 FROM table1 WHERE col2=123456; Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_1484355422771_0225, Tracking URL = xxxxxxxxxxx Kill Command = /opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hadoop/bin/hadoop job -kill job_1484355422771_0225 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0 2017-01-17 09:48:31,281 Stage-1 map = 0%, reduce = 0% 2017-01-17 09:48:36,413 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.75 sec MapReduce Total cumulative CPU time: 1 seconds 750 msec Ended Job = job_1484355422771_0225 MapReduce Jobs Launched: Stage-Stage-1: Map: 1 Cumulative CPU: 1.75 sec HDFS Read: 4925140 HDFS Write: 9 SUCCESS Total MapReduce CPU Time Spent: 1 seconds 750 msec OK -3769610 Time taken: 9.996 seconds, Fetched: 1 row(s) It performs in less than 10seconds however not sure now why it takes so much time to start the Query Job.

MasterOfPuppets · ‎01-17-2017

Thanks for your feedback.

MasterOfPuppets · ‎01-16-2017

Hi there. When setting Hive engine running over TEZ engine, all Queries fail. My CDH version is 5.8.2 Is it normal ? hive> select col1 from fact_table where col2=910001031; Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/tez/dag/api/SessionNotRunning at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2532) at java.lang.Class.getConstructor0(Class.java:2842) at java.lang.Class.newInstance(Class.java:345) at org.apache.hadoop.hive.ql.exec.TaskFactory.get(TaskFactory.java:136) at org.apache.hadoop.hive.ql.exec.TaskFactory.get(TaskFactory.java:150) at org.apache.hadoop.hive.ql.parse.GenTezProcContext.<init>(GenTezProcContext.java:160) at org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:325) at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:204) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10091) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9884) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:312) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1201) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1296) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1127) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1115) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:220) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:383) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:775) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:693) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:628) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.ClassNotFoundException: org.apache.tez.dag.api.SessionNotRunning at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 30 more

MasterOfPuppets · ‎01-16-2017

Hi. What conf file has hive.exec.parallel property set ? thx

MasterOfPuppets · ‎01-16-2017

Even a simple query has same velocity as a SELECT COUNT... select * from table1 d where d.col1=375;

MasterOfPuppets · ‎01-16-2017

I've made few changes and it did not really improve perf: set mapreduce.map.memory.mb=4096; set mapreduce.reduce.memory.mb=4096; set mapreduce.map.java.opts=-Xmx4g; set mapreduce.reduce.java.opts=-Xmx4g; select COUNT(*) from tabl1; Query ID = user1_20170116100000_878d81d3-6503-48c2-868f-1a9dcc949b63 Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1484355422771_0045, Tracking URL = xxxxxxxxxxxx Kill Command = /opt/cloudera/parcels/CDH-5.8.2-1.cdh5.8.2.p0.3/lib/hadoop/bin/hadoop job -kill job_1484355422771_0045 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2017-01-16 10:01:02,660 Stage-1 map = 0%, reduce = 0% 2017-01-16 10:01:07,780 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.25 sec 2017-01-16 10:01:12,900 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.76 sec MapReduce Total cumulative CPU time: 2 seconds 760 msec Ended Job = job_1484355422771_0045 MapReduce Jobs Launched: Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 2.76 sec HDFS Read: 89678 HDFS Write: 8 SUCCESS Total MapReduce CPU Time Spent: 2 seconds 760 msec OK 4019581 Time taken: 16.164 seconds, Fetched: 1 row(s)

Online	Offline
Last Visited	‎04-14-2017 09:21 AM

Member Since	‎01-13-2017 01:06 PM
Last Visited	‎04-14-2017 09:21 AM
Posts	40

Cloudera Community

Re: x sRe: Issue: Create HBASE table on Hive

Re: Impala ODBC not returning data when WITH claus...

Re: Hive hive.prewarm.enabled property

Re: Hive hive.prewarm.enabled property

Hive hive.prewarm.enabled property

Re: Hive Queries run slowly

Re: Hive Queries run slowly

Re: Hive hive.exec.parallel property

Tez Engine not working over CDH 5.8.2

Hive hive.exec.parallel property

Re: Hive Queries run slowly

Re: Hive Queries run slowly