Member since
09-26-2014
44
Posts
10
Kudos Received
7
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
5372 | 02-19-2015 03:41 AM | |
1315 | 01-07-2015 01:16 AM | |
10379 | 12-10-2014 04:59 AM | |
5785 | 12-08-2014 01:39 PM | |
5625 | 11-20-2014 08:16 AM |
09-20-2018
09:20 PM
HI TomasTF, I tried 3 ways, 1. Data without newline : sample below {"id":"P121", "name":"Round Tee", "reseller":"Puma", "category":"Top Wear", "price":2379.06, "discount":22.57, "profit_percent":20.13},{"id":"P122", "name":"pant", "reseller":"Reebok", "category":"bottom", "price":1234.99, "discount":99.99, "profit_percent":67.12} 2. Data with new line and comma {"id":"P121", "name":"Round Tee", "reseller":"Puma", "category":"Top Wear", "price":2379.06, "discount":22.57, "profit_percent":20.13}, {"id":"P122", "name":"pant", "reseller":"Reebok", "category":"bottom", "price":1234.99, "discount":99.99, "profit_percent":67.12} 3. Data with new line and without comma {"id":"P121", "name":"Round Tee", "reseller":"Puma", "category":"Top Wear", "price":2379.06, "discount":22.57, "profit_percent":20.13} {"id":"P122", "name":"pant", "reseller":"Reebok", "category":"bottom", "price":1234.99, "discount":99.99, "profit_percent":67.12} all three failed with same error Next, I deleted the table and recreated with Double CREATE EXTERNAL TABLE IF NOT EXISTS testtable (Id string,Name string,Reseller string,Category string,Price Double,Discount Double,Profit_percent Double) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' location "/user/testuser/test"; with Data in below format (new line without comma) {"id":"P121", "name":"Round Tee", "reseller":"Puma", "category":"Top Wear", "price":2379.06, "discount":22.57, "profit_percent":20.13} {"id":"P122", "name":"pant", "reseller":"Reebok", "category":"bottom", "price":1234.99, "discount":99.99, "profit_percent":67.12} Now Select Query is not fetching result without error. However, am still now sure why Decimal was throwing error.
... View more
09-14-2018
09:25 AM
@TomasTF, (1) Service Monitor will pull query information from the Impala Daemons so first make sure that Service Monitor is not having trouble getting information after the queries complete. View: /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR-`hostname -f`.log.out (2) When you query Impala queries via Cloudera Manager, Cloudera Manager will request query information from Service Monitor. So, check Cloudera Manager and Service Monitor logs for any log messages that may be relevant when you are trying to view queries: /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR-`hostname -f`.log.out /var/log/cloudera-scm-server/cloudera-scm-server.log (3) Make sure you have enough disk space on the volume where your Service Monitor Storage Directory is located. (4) Lastly, if there are no messages that indicate problems with the above, try increasing Impala Storage from 1GiB to 2 or 3 GiB. Perhaps for some reason you were running out of space, but I don't think that is too likely. Worth a try if nothing else mentioned above gives you any clues.
... View more
03-07-2016
10:58 AM
1 Kudo
I ran into this error, and it was caused by running out of heap size for Nodemanager. I increased the heap, and Yarn came up without errors.
... View more
11-23-2015
12:59 PM
I'm running into something similar. I'm on 5.4.2 building tables with have then analyzing with Impala and I get the same warnings, although the queries execute ok. Can you please share with me what you scripted to make "when one partition is always less than 800MB I set the block size for this table to 1GB" as you mention in your post?
... View more
06-05-2015
09:33 AM
Thank you mkempanna. Performance will depend on which SQL queries you run through the Linked Server. For example, complex queries are typically faster when pushed down to the database using OPENQUERY. If you just select * from the table, both syntaxes will perform similarly.
... View more
03-02-2015
10:29 AM
test.xml or text.xml needs to be present in hdfs and that should be passed in LOAD command
... View more
02-19-2015
02:41 PM
I got this from one of our engineers:
Impala doesn't support xml natively. Instead can you convert the xml data into one of the supported formats [1] using hive and work with them from impala. You can probably use hive to create XML based tables using xml serde [2] and then use hive to convert the data to avro based table using "insert overwrite avro_table select * from xml_table". Just make sure you create the avro_table using the avro serde and hive's insert overwrite takes care of format conversion. Btw this xml serde [2] is a third party package that we didn't test with CDH. You can probably give it a try. [1] http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/impala_file_formats.html [2] https://github.com/dvasilen/Hive-XML-SerDe
... View more
02-19-2015
03:41 AM
1 Kudo
I found the piggybank.jar in /opt/cloudera/parcels/CDH/lib/pig/. The problem was in fact that when I called register piggybank, the grunt shell gave me this exception: grunt> REGISTER piggybank.jar 2015-02-19 12:38:49,841 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS 2015-02-19 12:38:49,849 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 101: file 'piggybank.jar' does not exist. after changing the directory into the lib path the register worked well.. or use: REGISTER /opt/cloudera/parcels/CDH/lib/pig/piggybank.jar Tomas
... View more
01-07-2015
01:16 AM
This issue - with reading large tables compressed by Impala - was (based on my experiences) solved in the release of Impala 2.1 (CDH 5.3.1) Cloudera did not confirm this as a bug - when I tried to arrange a conf call with cloudera support and they tried to investigate where is the problem - they were not able define what is the root cause of this bug. I assume that this changed helped to solve the problem (Impala 2.1.0 release notes): The memory requirement for querying gzip-compressed text is reduced. Now Impala decompresses the data as it is read, rather than reading the entire gzipped file and decompressing it in memory But this is not confirmed, after upgrade Impala did not crash anymore. T
... View more
01-07-2015
01:12 AM
More interestingly this differencce dissappeared after upgrading to CDH 5.3.1. T.
... View more