Support Questions
Find answers, ask questions, and share your expertise

CDH supporter come here

CDH supporter come here

Expert Contributor

i have tested all CDH 5.5.1, there are some very urgent issue need to slove. i hope CDH don't upgrade so quickly, cause your product is already not suitable to production env. why?  too many bugs, performance is too bad.

 

1) hue bug (review in hue community)

2) hive performance is too bad. (review in hive community)

3) yarn server , 

........................ 

 

i don't know how to say,  in pervious version, i always suggest my friend or custom to use CDH,but now, i won't.

5 REPLIES 5

Re: CDH supporter come here

Master Collaborator

iamfromsky,

I'm sorry to hear about your experiences with CDH. Please stand by as we evaluate and triage these issues. Your feedback is appreciated!

Re: CDH supporter come here

Expert Contributor

HI,  jkestelyn

 

so sorry, i own you an apology, and own all the CDH employees an apology.   

 

as i have opened some topics in hue, hive, and here to complain CDH 5.5.1 performance is bad, especially HIVE.

it's time for me to explain all the sutation cause some performance issue is because hardware.

 

i have compained below issues:

 

1) CDH 5.5.1 HIVE is too slow comparison with CDH 5.1.0

2) some HIVE queries  run fast sometime, but slow sometime.

3) some HIVE queries run fast in HUE/HIVE cli, but it's slow in oozie

4) distinct is too slow in HIVE,

5) set parameter for hive in hue will cause global change

6) change reduce and map vcore will cause oozie log in hue disapper.

 

1,2,3 issue is becuase one nodemanager(hostname: jq-yarn02.hadoop) issue. truth be told, the nodemanager hardware is very normal, no any one find any problems, even infra, dba, developer, i have tracked HIVE performance issue for two weeks.

today one developer told me he has found if the job execute in jq-yarn02.hadoop and jq-namenode01.hadoop then the job is slow.  i told him it's impossible, because all the service in these two machines is normal. so he goes away.

 

today night, i am going to test jobs in my home, the symptom is the same, one job sometimes ifast, sometime slow.  it's so strange, thne i put the debug , explain statement in these SQL, hope can check something. but  all the plan or debug message is the same. i have no idea at that moment, then start to smoking and ssh to jq-yarn02.hadoop(i din't know why i am going to ssh this machine), i found it's a little bit slow, but i think it's normal at that time, because i connect my company through VPN, later, i ssh to another server, it's fast. i staring the window and thinking about why this machine is so fast, but jq-yarn02.hadoop it's a little bit slower? then i use top to check the jq-yarn02.hadoop load average, it's 0-1. then i use PING to test network, it's a little lag, not much. i start to doubt this machine has problem, but no eveidence. this clue is very imporant for me to find root cause..

 

i continue to test the job, symptom is the same, and i am going to check container log, i found the logs show me jq-yarn02.hadoop can't connect namenode, and retry retry.  then i am going to telnet manually, it's normal. so sucks. continue test. and check logs, i found one thing if the open proxy is jq-yarn02.hadoop , then job is slow. and developer's iead in my mind cause he has told me  jq-yarn02.hadoop and jq-namenode01.hadoop.

 

i stop the jq-yarn02.hadoop, and test again, all the job is ok now. 

 

 

but the 4,5,6 issues still happend, so i think CDH 5.5.1 distinct in HIVE is really too slow comparsion with CDH 5.1.0, and set session parameter will cause global change(you must check mapreduce.map.speculative and mapreduce.reduce.speculative), 6 i think is bug/

 

good night.

 

 

 

 

Re: CDH supporter come here

Master Collaborator

@iamfromsky, thank you for the detailed update.  That is some great troubleshooting you performed and I'm glad it lead to the root cause of some of the issues.  We have folks responding to your Hive and Hue threads and I hope you find those responses helpful.

 

Regards,

 

Clint 

Re: CDH supporter come here

Expert Contributor

one more issue for hive:

 

as we know, simply query in hive will run Fetch Tasks directly other than Mapreduce.  but if the  table is LZO, then CDH 5.5.1 will not run fetch tasks, instead it execute mapreduce task. in CDH 5.1.0, it's no this kind problems.

 

i have tried partition table but it's not LZO table, will execute FETCH TASKS.

but if the table is partition table and lzo  , then goes to Mapreduce. pls check below case:

 

explain select * from cx_user_access_split_01

 

0	STAGE DEPENDENCIES:
1	  Stage-1 is a root stage
2	  Stage-0 depends on stages: Stage-1
3	
4	STAGE PLANS:
5	  Stage: Stage-1
6	    Map Reduce
7	      Map Operator Tree:
8	          TableScan
9	            alias: cx_user_access_split_01
10	            Statistics: Num rows: 281499 Data size: 187038196 Basic stats: PARTIAL Column stats: NONE
11	            Select Operator
12	              expressions: source_type (type: string), server_addr (type: string), remote_addr (type: string), time_local (type: string), msec (type: string), url (type: string), request (type: string), status (type: string), connection (type: string), request_time (type: string), body_bytes_sent (type: string), current_url (type: string), http_user_agent (type: string), proxy_forwarded_for (type: string), cookie (type: string), city_id (type: string), ngx_uid (type: string), js_uid (type: string), year (type: string), month (type: string), day (type: string)
13	              outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20
14	              Statistics: Num rows: 281499 Data size: 187038196 Basic stats: PARTIAL Column stats: NONE
15	              File Output Operator
16	                compressed: false
17	                Statistics: Num rows: 281499 Data size: 187038196 Basic stats: COMPLETE Column stats: NONE
18	                table:
19	                    input format: org.apache.hadoop.mapred.TextInputFormat
20	                    output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
21	                    serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
22	
23	  Stage: Stage-0
24	    Fetch Operator
25	      limit: -1
26	      Processor Tree:
27	        ListSink

 

Re: CDH supporter come here

Expert Contributor

HI, 

 

hive.fetch.task.conversion.threshold=256M  explained as  Above this size, queries are converted to fetch tasks.

may i say if total size above 256M, then simply query will execute Fetch tasks ?

 

pls check out the same query ,but i have re-size the threshold to 2500000000000, it's going to Fetch tasks other than Mapreduce.

 

could you give me some explain for this ?  and i have been caugth the HIVE session set will affect all the session case again. i am so afriad to set anything in session level for hive right now.

 

set hive.fetch.task.conversion.threshold=2500000000000;

explain select * from cx_user_access_split_01;

 

 

	STAGE DEPENDENCIES:
1	  Stage-0 is a root stage
2	
3	STAGE PLANS:
4	  Stage: Stage-0
5	    Fetch Operator
6	      limit: -1
7	      Processor Tree:
8	        TableScan
9	          alias: cx_user_access_split_01
10	          Statistics: Num rows: 281499 Data size: 187038196 Basic stats: PARTIAL Column stats: NONE
11	          Select Operator
12	            expressions: source_type (type: string), server_addr (type: string), remote_addr (type: string), time_local (type: string), msec (type: string), url (type: string), request (type: string), status (type: string), connection (type: string), request_time (type: string), body_bytes_sent (type: string), current_url (type: string), http_user_agent (type: string), proxy_forwarded_for (type: string), cookie (type: string), city_id (type: string), ngx_uid (type: string), js_uid (type: string), year (type: string), month (type: string), day (type: string)
13	            outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20
14	            Statistics: Num rows: 281499 Data size: 187038196 Basic stats: PARTIAL Column stats: NONE
15	            ListSink