Hi,   we have a sentry role that have action=ALL on db=my_db   When trying to issue a CREATE TABLE statment in Impala to create a Kudu table in my_db we get the following error:     I1205 12:32:21.124711 47537 jni-util.cc:176] org.apache.impala.catalog.AuthorizationException: User 'my_user' does not have privileges to access: name_of_sentry_server A work-around is to set action=ALL on the server level to the sentry role but we don't want to give this wide permission to the role.   Do we need to set action=ALL on the server level in order to delegate the rights to our users to create Kudu tables or how could we set up Sentry in this case?   We use CDH 5.12.1 (Kudu 1.4.0)   Br, Petter   ... View more

Hi all,   I reported IMPALA-4725  last week but it seems like it has not been triaged yet. I wanted to bring some more attention to this issue (and possible suggestions for workarounds) since it has a heavy impact on us.   To summarize it seems like Impala mixes-up values in arrays of structs which to me seems like a fundamental problem in the parquet reader. Alternatively the values gets mixed-up when presented as a result.   Either way, I would very much appreciated an initiated persons view on this issue.   We are running Impala that is bundled with CDH 5.8.3   Br, Petter ... View more

Hi all,   Best Practices for Using Impala with S3 states "Set the safety valve fs.s3a.connection.maximum to 1500 for impalad ."   Can annyone clarify which safety valve field should be used and with what syntax? I'm reading somewhere that this setting belongs to core-site.xml but Impala configuration in Cloudera Manger does not seem to have a safety valve for core-site.xml. The instructions mentions safety valve for impalad but that safety valve seems to be for command line arguments to impalad.   The problem we are trying to adress is   hdfsSeek(desiredPos=503890631): FSDataInputStream#seek error: com.cloudera.com.amazonaws.AmazonClientException: Unable to execute HTTP request: Timeout waiting for connection from pool   that we keep getting when using Impala for querying data stored in S3.   We are using CDH 5.8.3   Thanks, Petter ... View more

I have described an issue with time consuming parquet file generation in the Hive forum. See this post for a description on the environment. The question is half Impala related so I would appreciated if any Impala experts here could read that post as well.   https://community.cloudera.com/t5/Batch-SQL-Apache-Hive/How-to-improve-performance-when-creating-Parquet-files-with-Hive/m-p/43804#U43804   I have some additional questions that are Impala specific. The environment currently has three Impala nodes with 5-10GB worth of data in each partition. The question is how I should generate the parquet files to achieve the most performance out of Impala.   Currently I target the parquet file size to 1 GB each. The HDFS block size is set to 256 MB for these files and I have instructed to create row groups of the same size. Surprisingly I get many more row groups. I just picked a random file and it contained 91 row groups.   Given our environment, where should we aim at file-size, number of row groups in each file and HDFS block size for the files? Also, if it would be more beneficiary to have fewer row-groups in each file, how can we instruct Hive to generate fewer row groups since Hive does not seem to respect the parquet.block.size option?   We used the Impala version bundled with CDH 5.7.1   Thanks in advance, Petter   ... View more

It does not seem like the IS NULL / IS NOT NULL operator is supported for struct data types. We are using Impala 2.3.0/CDH5.5.1. This seem like a basic and vital operator to have. Especially when using wide tables. Anybody out there that has a patch or workaround or that actually succeeded to use this operator on structs?   I have reported IMPALA-3060 on the topic. ... View more

I'm using CDH 5.5.0 with kerberos and Sentry enabled. Trying to kill a job from the job browser fails with the message "There was a problem communicating with the server: The default static user cannot carry out this operation. (error 403)"   I can kill the same job using the yarn application -kill command.   I guess this is a configuration issue. Could someone assist me in getting this right so that I can can kill jobs from the Job Browser?   Stack trace: [26/Jan/2016 10:15:30 +0100] access WARNING 10.128.42.143 di23060584 - "POST /jobbrowser/jobs/application_1453476679853_0011/kill HTTP/1.1" [26/Jan/2016 10:15:30 +0100] connectionpool INFO Resetting dropped connection: ip-10-255-2-7.eu-west-1.compute.internal [26/Jan/2016 10:15:30 +0100] kerberos_ ERROR handle_mutual_auth(): Mutual authentication unavailable on 403 response [26/Jan/2016 10:15:30 +0100] views ERROR Killing job Traceback (most recent call last): File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/apps/jobbrowser/src/jobbrowser/views.py", line 246, in kill_job job.kill() File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/apps/jobbrowser/src/jobbrowser/yarn_models.py", line 185, in kill return self.api.kill(self.id) File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/desktop/libs/hadoop/src/hadoop/yarn/mapreduce_api.py", line 117, in kill get_resource_manager(self._user).kill(app_id) # We need to call the RM File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/desktop/libs/hadoop/src/hadoop/yarn/resource_manager_api.py", line 124, in kill return self._execute(self._root.put, 'cluster/apps/%(app_id)s/state' % {'app_id': app_id}, params=params, data=json.dumps(data), contenttype=_JSON_CONTENT_TYPE) File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/desktop/libs/hadoop/src/hadoop/yarn/resource_manager_api.py", line 141, in _execute response = function(*args, **kwargs) File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/desktop/core/src/desktop/lib/rest/resource.py", line 136, in put return self.invoke("PUT", relpath, params, data, self._make_headers(contenttype)) File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/desktop/core/src/desktop/lib/rest/resource.py", line 78, in invoke urlencode=self._urlencode) File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/desktop/core/src/desktop/lib/rest/http_client.py", line 161, in execute raise self._exc_class(ex) RestException: The default static user cannot carry out this operation. (error 403) [26/Jan/2016 10:15:30 +0100] middleware INFO Processing exception: The default static user cannot carry out this operation. (error 403): Traceback (most recent call last): File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/handlers/base.py", line 112, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/db/transaction.py", line 371, in inner return func(*args, **kwargs) File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/apps/jobbrowser/src/jobbrowser/views.py", line 83, in decorate return view_func(request, *args, **kwargs) File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/apps/jobbrowser/src/jobbrowser/views.py", line 249, in kill_job raise PopupException(e) PopupException: The default static user cannot carry out this operation. (error 403)     ... View more

I have been testing CDH5.5.0 and have noted that Impala does not like reserved words as field names in complex types. This seems strange as reserved words can be used as column names for ordinary columns. Hive does not impose the same restriction. Reserved words can be back-ticked where needed. Does anybody know if this is by design or if this is an issue in Impala 2.3.0? We are using Hive to create Parquet files with complex types. Sample to reproduce issue and error message below. In the case below the word 'replace' is reserved. In Hive: CREATE EXTERNAL TABLE MyTable ( device_id STRING, added struct<name:string,version_name:string,version_code:int,`replace`:boolean> ) STORED AS PARQUET LOCATION '/tmp/impala/mytable';   In Hive: INSERT OVERWRITE TABLE MyTable SELECT device_id, payload AS added FROM Added where import_id = 106000;   In Impala: SELECT * FROM MyTable limit 10; Output: AnalysisException: Failed to load metadata for table: 'mytable' CAUSED BY: TableLoadingException: Unsupported type 'struct<name:string,version_name:string,version_code:int,replace:boolean>' in column 'added' of table 'mytable' I0107 15:56:01.251721 21006 Frontend.java:818] analyze query SELECT * FROM MyTable limit 10 E0107 15:56:01.252320 21006 Analyzer.java:2212] Failed to load metadata for table: mytable Unsupported type 'struct<name:string,version_name:string,version_code:int,replace:boolean>' in column 'added' of table 'mytable' I0107 15:56:01.252908 21006 jni-util.cc:177] com.cloudera.impala.common.AnalysisException: Failed to load metadata for table: 'MyTable' at com.cloudera.impala.analysis.TableRef.analyze(TableRef.java:180) at com.cloudera.impala.analysis.Analyzer.resolveTableRef(Analyzer.java:512) at com.cloudera.impala.analysis.SelectStmt.analyze(SelectStmt.java:155) at com.cloudera.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:342) at com.cloudera.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:317) at com.cloudera.impala.service.Frontend.analyzeStmt(Frontend.java:827) at com.cloudera.impala.service.Frontend.createExecRequest(Frontend.java:856) at com.cloudera.impala.service.JniFrontend.createExecRequest(JniFrontend.java:147) Caused by: com.cloudera.impala.catalog.TableLoadingException: Unsupported type 'struct<name:string,version_name:string,version_code:int,replace:boolean>' in column 'added' of table 'mytable' at com.cloudera.impala.catalog.IncompleteTable.loadFromThrift(IncompleteTable.java:111) at com.cloudera.impala.catalog.Table.fromThrift(Table.java:240) at com.cloudera.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:357) at com.cloudera.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:246) at com.cloudera.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:132) at com.cloudera.impala.service.Frontend.updateCatalogCache(Frontend.java:223) at com.cloudera.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:164) at ========.<Remote stack trace on catalogd>: com.cloudera.impala.catalog.TableLoadingException: Unsupported type 'struct<name:string,version_name:string,version_code:int,replace:boolean>' in column 'added' of table 'mytable' at com.cloudera.impala.catalog.Table.parseColumnType(Table.java:331) at com.cloudera.impala.catalog.HdfsTable.addColumnsFromFieldSchemas(HdfsTable.java:571) at com.cloudera.impala.catalog.HdfsTable.load(HdfsTable.java:1073) at com.cloudera.impala.catalog.TableLoader.load(TableLoader.java:84) at com.cloudera.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:232) at com.cloudera.impala.catalog.TableLoadingMgr$2.call(TableLoadingMgr.java:229) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ... View more