we are running cdh 5.9.0(impala 2.7.0, hive 1.1.0). we know that, while quering timestamp fields(parquet table generated by hive)  with impala, we may get different result(vs hive) because of the timezone reason. the startup args of our impala is: convert_legacy_hive_parquet_utc_timestamps=false use_local_tz_for_unix_timestamp_conversions .   what we confuse is that, whatever true/false we set hive.parquet.timestamp.skip.conversion while generating different parquet table in hive, we got the same timestamp result while doing query with impala from both of the generated table. what we expect is that, when the value of hive.parquet.timestamp.skip.conversion is different, the result should be different.  but it just don't perform in this way.   we are really confuse about this, any reply will be appreciate.   bellowing steps is a test: CREATE TABLE test_timestamp (ts TIMESTAMP) STORED AS TEXTFILE; CREATE TABLE test_ts_skip_conversion_true_parquet (ts TIMESTAMP) STORED AS TEXTFILE; CREATE TABLE test_ts_skip_conversion_false_parquet (ts TIMESTAMP) STORED AS TEXTFILE;   step1, load data into and query from test_timestamp    step2, select data into test_ts_skip_conversion_true_parquet  ( hive.parquet.timestamp.skip.conversion=true ) step3, select data into test_ts_skip_conversion_false_parquet  ( hive.parquet.timestamp.skip.conversion=false )   step4, while query test_ts_skip_conversion_true_parquet and test_ts_skip_conversion_false_parquet with impala, we got the same result, but we expect different result here!         ... View more

we are running cdh 5.10 cluster with about 500 nodes, we installed nn/jn/zk as bellow: host1: namenode, journal node, zookeeper host2: namenode, journal node, zookeeper host3: journal node, zookeeper (no other service is install on these three host)   namenode's fsimage & editlog storage dir is configed at "/data1/dfs/nn/" and "/data2/dfs/nn/" journal node's editlog storage dir is configed at "/data3/dfs/nn/" /data1 and /data2 is mount at individual disk driver   when we look into the namenode's log, we found that namenode takes long time to flush editlog to journalnode, and flush to local disk driver is not taking that much time.as seeing from the log, flush to journal node is about 4x~5x times longer than flush to local disk. bellow is some log snippets:   --snippet 1 2017-08-15 09:50:12,946 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 326245 Total time for transactions(ms): 3997 Number of transactions batched in Syncs: 284851 Number of syncs: 41371 SyncTimes(ms): 14194 3543 3397 2017-08-15 09:51:12,951 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 533598 Total time for transactions(ms): 6599 Number of transactions batched in Syncs: 471796 Number of syncs: 61757 SyncTimes(ms): 21695 5450 5132 2017-08-15 09:52:12,951 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 692744 Total time for transactions(ms): 11074 Number of transactions batched in Syncs: 610178 Number of syncs: 82561 SyncTimes(ms): 31668 7356 6787 2017-08-15 09:53:12,953 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 843165 Total time for transactions(ms): 14348 Number of transactions batched in Syncs: 742316 Number of syncs: 100838 SyncTimes(ms): 40082 10075 8269 2017-08-15 09:54:15,374 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 920690 Total time for transactions(ms): 15248 Number of transactions batched in Syncs: 808311 Number of syncs: 110467 SyncTimes(ms): 43884 39289 10217 2017-08-15 09:54:30,821 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 920690 Total time for transactions(ms): 15248 Number of transactions batched in Syncs: 810222 Number of syncs: 110468 SyncTimes(ms): 43910 39428 38668 2017-08-15 09:55:30,821 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 206372 Total time for transactions(ms): 2562 Number of transactions batched in Syncs: 160222 Number of syncs: 46144 SyncTimes(ms): 16389 5156 3410   --snippet 2 2017-08-15 09:59:14,716 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 175114 Total time for transactions(ms): 2762 Number of transactions batched in Syncs: 89590 Number of syncs: 85521 SyncTimes(ms): 29138 6897 5872 2017-08-15 10:00:14,716 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 221609 Total time for transactions(ms): 3499 Number of transactions batched in Syncs: 112617 Number of syncs: 108989 SyncTimes(ms): 38074 9056 7451 2017-08-15 10:01:45,172 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 251986 Total time for transactions(ms): 4000 Number of transactions batched in Syncs: 130562 Number of syncs: 121108 SyncTimes(ms): 42776 44831 14277 is this editlog sync behave normal? how can we speed up the flush operation when flush to journal node？ Any reply is appreciated. ... View more

We are currently running hive(hiveserver2) with sentry, and user impersonation is disable. When any user connect to  hiveserver2 and submit queries,  hiveserver2 will submit all the query jobs to yarn, as the same user hive, not the actual the user who connect to hiveserver2. Is there any way that can let hiveserver2 submit jobs as the actual user?   ... View more

Currently, we are running impala 2.7, and we enable impala with sentry service. One problem is that, whenever any new user want use impala, we need to create the user account and groups on the linux os across all the nodes that running impalad.   Is their any document that given guide to develop and deploy customize group mapping service plugin on impala？ All the group mapping client suppose to request the same server for user groups, which the server is stored all the user and its groups mapping. ... View more

While we doing tpc-ds testing on impala+kudu vs impala+parquet(according to https://github.com/cloudera/impala-tpcds-kit), we found that for most of the queries, impala+parquet is 2times~10times faster than impala+kudu. Is any body ever did the same testing? ps:We are running kudu 1.3.0 with cdh 5.10. ... View more

We are running kudu 1.3.0 with cdh 5.10(the kudu client version suppose to be 1.2). When we doing tpc-ds query with impala on kudu(according to https://github.com/cloudera/impala-tpcds-kit), we found that the 'query 3 between predicate' is not push down to kudu, cause kudu scan many rows return to impala. The following is what we found in impala query profile:   tpc-ds q3.sql snippets：   any reply will be appreciate.   ... View more

Hi, I'm running a cluster with CHD 5.10. I follow the document instruction to install kudu service. First download the kudu csd(KUDU-5.10.0.jar), upload it to /opt/cloudera/csd/ on the Cloudera Manager server, and then restart Cloudera Manager server. When I login to Cloudera Manager server again and click " Hosts > Parcels ", I found the showing kudu version is "1.3.0-1.cdh5.11.0.p0.12".   The CDH cluster i'm running is  5.10, but here kudu 13 is bind with cdh 5.11. Is it ok if I install kudu with version " 1.3.0-1.cdh5.11.0.p0.12 "? Will the installation affect any component(such as hadoop/hive/hbase) currently running in version 5.10 ? ... View more

Namenode HA config share edit with QJM, the active NN will write edits to QJM and local directory.   When NN starting up, it reads fsimage from local directory, and edits from QJM.   As NN is not reading edits from local directory, why it still writing edits to a local directory?   ... View more

We got two CDH cluster with the same version(CDH-5.5.2-1.cdh5.5.2.p0.4), and both the ResourceManager of each cluster with the same configuration. One of the ResourceManager is running well, and its heap memory is stay in a constant value(e.g 800mb) as the time is going on. But the other one will throw OOM exception and exit after 15 days. When we use 'jmap -F -histo' to dump its jvm heap info, we are seeing that the size of object 'char[]' is growing up as the time is moving, and it finally throw OOM. Following is key info of jvm dump result of both the good RM and OOM RM: dump cmd：jmap -F -histo pid (The heap size of both the RM are 1GB)   A）jvm dump of good RM in cluster A we are seeing that 40w+ char[] instances with 60m+ heap mem     B）jvm dump of bak RM（OOM） in cluster B we are seeing that 30w+ char[] instances but with 400m+ heap mem   Any help wil be appreciated. ... View more