Hi,  When doing a "show create table TableX", the columns are not escaped by "`" so if there is a column with a reserverd keyword ("comment" for exemple), we can't just copy past the query returned to an other database.  Btw, I was thinking about opening a jira bug for that instead of posting it here, but I wasn't sure about the preffered way for you guys... let me know what is the prefered process for this kind on problems.     thanks ... View more

Hi,  I am trying to parse a date in impala using the unix_timestamp fonction.  select unix_timestamp('Thu Dec 17 15:55:08 IST 2015', 'EEE MMM d HH:mm:ss ZZZ yyyy') But I get the follow error:  Bad date/time conversion format: EEE MMM d HH:mm:ss ZZZ yyyy The same query works in hive. Is there something that I need to change the the pattern to make it work in impala? I am currently on cdh 5.12 thanks ... View more

Hi,  I am created a view on top of a table and it give me really bad performance results. It take about 2min to "Plan" the query on the views whereas it is only taking 55ms in the raw table.  The views is really jsut a select * from table_a union select * from table_b. Where table a and b have the same definition and are both parquet tables...   Slow query:  https://gist.github.com/anonymous/0af6f862a8069ee7b933deafe9cc0880 Fast query:  https://gist.github.com/anonymous/2ef16a696527acd86795c3080edf3e91 Any idea what is happening? thanks ... View more

Hi,   I have a view defined as `select * from t1 union select * from t2`. The 2 tables have an identical schema, t2 contains no data.    Now 3 uses cases :  - query with mt_dop=10 on the view  :  query is slow ~10s https://gist.github.com/momohuri/9ad4ba8f6fbd1d180068c8c102291f69 - query with mt_dop=0 on the view :  query is fast ~1s https://gist.github.com/momohuri/c4347eb7ef70a8eec63a0a62638d1ce7 - query with mt_dop=10 not using a view/union :  query is fast  ~1sec https://gist.github.com/momohuri/62a00d9e381771aa9172b6dea09a5191 I hope those observations can help for the next releases 🙂   thanks       ... View more

Hi,  I tried to update to 5.10 yesterday, and I believe to impala 2.8 (the logs/shell still says it is impala 2.7 for cdh 5.10). And I got suprised with a big drop in performance for most of my queries. For queries with no join, using "set mt_dop=10", improved the performance by a lot.  But for all the queries with "mt_dop=0" they got way worst.  In the shell In runned a aggregation query with no join, and it "Fetched 360 row(s) in 100.28s", the summary is the following :    +---------------------+--------+----------+----------+---------+------------+-----------+---------------+---------------------------------------------------+ | Operator | #Hosts | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail | +---------------------+--------+----------+----------+---------+------------+-----------+---------------+---------------------------------------------------+ | 10:MERGING-EXCHANGE | 1 | 150.62us | 150.62us | 360 | 500 | 0 B | -1 B | UNPARTITIONED | | 05:TOP-N | 9 | 137.12us | 218.93us | 360 | 500 | 16.00 KB | 11.72 KB | | | 09:AGGREGATE | 9 | 10.72ms | 14.33ms | 360 | 6.15M | 10.89 MB | 17.20 MB | FINALIZE | | 08:EXCHANGE | 9 | 86.09us | 95.65us | 3.24K | 6.15M | 0 B | 0 B | HASH(`date (hourly)`) | | 04:AGGREGATE | 9 | 364.87ms | 384.52ms | 3.24K | 6.15M | 2.03 MB | 17.20 MB | STREAMING | | 07:AGGREGATE | 9 | 5.27s | 5.52s | 38.07M | 226.95M | 714.15 MB | 18.60 GB | FINALIZE | | 06:EXCHANGE | 9 | 608.64ms | 628.24ms | 115.78M | 226.95M | 0 B | 0 B | HASH(udid,`date (hourly)`) | | 03:AGGREGATE | 9 | 17.56s | 24.24s | 115.78M | 226.95M | 2.63 GB | 18.60 GB | STREAMING | | 00:UNION | 9 | 1.67s | 2.47s | 226.95M | 226.95M | 1.93 MB | 0 B | | | |--02:SCAN HDFS | 9 | 543.42us | 602.49us | 0 | 0 | 0 B | 0 B | pocketgems_prod.customevent_chapterview_streaming | | 01:SCAN HDFS | 9 | 291.51ms | 1.09s | 226.95M | 226.95M | 42.01 MB | 1.29 GB | pocketgems_prod.customevent_chapterview_batch | +---------------------+--------+----------+----------+---------+------------+-----------+---------------+---------------------------------------------------+ It use to take around 35 seconds. I also downloaded the profile : https://gist.github.com/momohuri/c03683cd4263f48c1de5afd314d2662f The thing that surprised me is this  RowsReturnedRate : 3.  Any clue of why is it happening? For now I went back to cdh 5.9.1   Thanks          ... View more

Hi, I am trying to compute incremental stats for one large table (~200gb).  But I have an out of memory error : Memory limit exceeded The memory limit is set too low to initialize spilling operator (id=3). The minimum required memory to spill this operator is 272.00 MB.   It is a little bit strange to see that because I have a memory limit in the daemon and in the shell set to 80gb. But anyway, I investigated a little bit more and found this in the logs :  W0124 12:13:21.800235 6746 HdfsScanNode.java:654] Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB. I got the same error if I do it for the all the table or just one partition at the time. I couldn't find the parameter to increase HdfsScanNode uper bound.  Any idea on how I could solve that? thanks ps: I am using cdh 5.9.1   ... View more

Hi, I have a table with two fields : "user","some_integer" And I was wondering if there is a way of doing something like: select count(distinct("user")), "bit" from random_table group by bit(some_number); Thanks ... View more

Hi, I rebooted the machine that host cloudera manager. But after the reboot I got a problem.  In parcels I see that cdh: 5.9. 0-1.cdh5.9.0.p0.23 is beeing distributed, but it is stuck in one machine in the activation step. I tried to restart and hard restart the agent and the server, but I always have the same problem. The machine it is stuck on the activation is the same machine as the one that have cloudera manager, so there shouldn't be any network problem.   The agent log keep looping over the same logs : [14/Nov/2016 14:30:08 +0000] 24070 MainThread client_configs ERROR Failed to deploy client config <hadoop-conf,/etc/hadoop/conf.cloudera.hdfs>: No parcel provided required tags: set([u'cdh']) Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/client_configs.py", line 769, in rectify deploy_path = self._deploy_client_config(new_ccs[key]) File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/client_configs.py", line 479, in _deploy_client_config env, parcels_in_use = self._adapt_cc_to_env(self.deploy_env, cc) File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/client_configs.py", line 670, in _adapt_cc_to_env cc["optional_tags"]) File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/parcel.py", line 383, in prepare_environment raise ParcelTagUnsatisfiedException("No parcel provided required tags: %s" % (missing_reqs,)) ParcelTagUnsatisfiedException: No parcel provided required tags: set([u'cdh']) [14/Nov/2016 14:30:08 +0000] 24070 Thread-13 https ERROR Failed to retrieve/stroe URL: http://my_server:7180/cmf/parcel/download/CDH-5.9.0-1.cdh5.9.0.p0.23-trusty.parcel.torrent -> /opt/cloudera/parcel-cache/CDH-5.9.0-1.cdh5.9.0.p0.23-trusty.parcel.torrent HTTP Error 404: Not Found Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/https.py", line 175, in fetch_to_file resp = self.open(req_url) File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/https.py", line 170, in open return self.opener(*pargs, **kwargs) File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen return _opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 410, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 523, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.7/urllib2.py", line 448, in error return self._call_chain(*args) File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain result = func(*args) File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/https.py", line 205, in http_error_default raise e HTTPError: HTTP Error 404: Not Found [14/Nov/2016 14:30:08 +0000] 24070 Thread-13 downloader ERROR Failed fetching torrent: HTTP Error 404: Not Found Traceback (most recent call last): File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/downloader.py", line 263, in download cmf.https.ssl_url_opener.fetch_to_file(torrent_url, torrent_file) File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/https.py", line 175, in fetch_to_file resp = self.open(req_url) File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/https.py", line 170, in open return self.opener(*pargs, **kwargs) File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen return _opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 410, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 523, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.7/urllib2.py", line 448, in error return self._call_chain(*args) File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain result = func(*args) File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/https.py", line 205, in http_error_default raise e HTTPError: HTTP Error 404: Not Found [14/Nov/2016 14:30:23 +0000] 24070 MainThread parcel INFO The following requested parcels are not available: {u'CDH': u'5.9.0-1.cdh5.9.0.p0.23'}  Any idea ?  thanks   ... View more

Hi, I just saw that blog post about spark 2.0 beta : http://blog.cloudera.com/blog/2016/09/apache-spark-2-0-beta-now-available-for-cdh/ And I have a quick question, once spark 2.0 is installed on the cluster, how do I pick If i want my job to go to spark2.0 or 1.6?   thanks ... View more