@JasonBourne - if you have the same issue, here's a GitHub issue discussing it and linking to a pull request to fix it: https://github.com/cloudera/thrift_sasl/issues/28 You can see in the commits (here: https://github.com/cloudera/thrift_sasl/commits/master), they are testing a new release for a fix, but it looks like it's not quite done yet. Hopefully soon. ... View more

There's a 64kb limit on strings in Kudu but otherwise you can store any binary data in them.   https://docs.cloudera.com/documentation/kudu/5-10-x/topics/kudu_known_issues.html#schema_design_limitations ... View more

Upgrading to a newer version of Impala will solve most scalability issues that you'd see on Impala 2.9, mostly because of https://blog.cloudera.com/scalability-improvement-of-apache-impala-2-12-0-in-cdh-5-15-0/. ... View more

We have some background on schema evolution in Parquet in the docs - https://docs.cloudera.com/runtime/7.2.2/impala-reference/topics/impala-parquet.html. See "Schema Evolution for Parquet Tables". Some of the details are specific to Impala but the concepts are the same across engines including Hive and Spark that use parquet tables. At a high level, you can think of the data files being immutable while the table schema evolves. If you add a new column at the end of the table, for example, that updates the table schema but leaves the parquet files unchanged. When the table is queried, the table schema and parquet file schema are reconciled and the new column's values will be all NULL.   If you want to modify the existing rows and include new non-NULL values, that would require rewriting the data, e.g. with an INSERT OVERWRITE statement for a partition or a CREATE TABLE .. AS SELECT to create an entirely new table. Keep in mind that traditional Parquet tables are not optimized for workloads with updates - Apache Kudu in particular and also transactional tables in Hive3+ have support for row-level updates that is more convenient/efficient.   We definitely don't require rewriting the whole table every time you want to add a column, that would be impractical for large tables! ... View more

We have restarted nearly every component of the affected HDFS cluster and impala performance has improved. Sadly that doesn't explain the underlying issue. ... View more

You need to run compute stats on the base tables referenced by the views - compute stats directly on a view isn't supported. ... View more

Great news!   ... View more

These are good questions that come up frequently.   https://docs.cloudera.com/runtime/7.2.2/administering-kudu/topics/kudu-security-trusted-users.html discusses the issue. In summary, Hive/Impala tables (i.e. those with entries in the Hive Metastore) are authorized in the same way, regardless of whether backing storage is HDFS, S3, Kudu, HBase, etc - the SQL interface does the authorization to confirm that the end user has access to the table, columns, etc, then the service accesses the storage as the privileged user (Impala in this case).   In this model, if you create an external Kudu table in Impala and give permissions to a user to access the table via Impala, then they will have permissions to access the data in the underlying Kudu table. The thing that closes the loophole here is that creating the external Kudu table requires very high privileges - ALL permission on SERVER - a regular user can't create an external Kudu table pointed at an arbitrary Kudu cluster or table. ... View more

You can use an expression incited of a query.    in the expression ,  your query should be something like this.  =" SELECT A.COL1 , A. COL2 FROM schema.tableName A WHERE A.COL1 = '" & Parameters!parameterName.Value & "'    "   Notice the Quotation marks besides the parameter ( " , '  ) and equal ( = ) sign at the beginning You should create fields manually( Use query designer without parameters and let SSRS do the Refresh Fields task)   ... View more

https://issues.apache.org/jira/browse/IMPALA-8454 is the apache impala jira ... View more

Tried executing commit() or setting the timeout but no effect:   import pypyodbc connection = pypyodbc.connect(DSN="", Schema="dbname", autocommit=True) cursor = connection.cursor()   query = """INSERT INTO schema.table VALUES ('val1', 'val2')""" cursor.execute(query) cursor.commit()   connection.close() ... View more

Thanks for your input.   We are running our stage cluster in "n on-production mode" using the embedded postgres. This postgres is also used by the two hive metastore servers. The postgres db is hosted on the same node as the other cloudera services. When this host now freezes the impala insert queries freeze as well.   We were surprised to see that there seems to be no timeout from the hive metastore servers and their backing db (postgres) and no error either.   This probably also happens when backed by an external postgres or mysql database, although not tested by us.   I wonder if this might be solved by a newer CDH version. We are currently looking into upgrading and would like to do so for other reasons very much so.   ... View more

Hi @PauloRC @Tim Armstrong ,   This might be a performance regression, but also in general a performance inefficiency with a specific planner data structure.   A correctness fix for IMPALA-8386 may have introduced this perf regression in 3.2.1, IMPALA-9358 may resolve this issue, but I don't think it's available in any CDH 6.3 release yet. @PauloRC one thing to try which might mitigate the issue is to run your view query with SET ENABLE_EXPR_REWRITES=false to see if that helps. ... View more

Thank you for your reply Tim.   Just to clarify, security-wise, are we better off with our current configuration (default), with sentry service disabled, or with sentry enabled in testing mode?   You mentioned that sentry in testing mode does not authenticate the clients, but in the documentation it is mentioned that testing mode uses weaker authentication mechanisms.   We need this in order to prevent our analysts from doing accidental writes, drops, etc. on the data.   Our cluster is in a secure isolated environment.     ... View more

This is definitely a bug. Thanks for the clear report and reproduction. It's not IMPALA-7957 but is somewhat related. This is new to us so I filed https://issues.apache.org/jira/browse/IMPALA-10182 to track it.   It looks like it can only happen when you have a UNION ALL, plus subqueries where the same column appears twice in the select list, plus NULL values in those  columns.   You can work around the issue by removing the duplicated entries in the subquery select list. E.g. the following query is equivalent and returns the expected results.   SELECT MIN(t_53.c_41) c_41, CAST(NULL AS DOUBLE) c_43, CAST(NULL AS BIGINT) c_44, t_53.c2 c2, t_53.c2 c3s0, t_53.c4 c4, t_53.c4 c5s0 FROM ( SELECT t.productsubcategorykey c_41, t.productline c2, t.productsubcategorykey c4 FROM as_adventure.t1 t WHERE true GROUP BY 2, 3 ) t_53 GROUP BY 4, 5, 6, 7 UNION ALL SELECT MIN(t_53.c_41) c_41, CAST(NULL AS DOUBLE) c_43, CAST(NULL AS BIGINT) c_44, t_53.c2 c2, t_53.c2 c3s0, t_53.c5s0 c4, t_53.c5s0 c5s0 FROM ( SELECT t.productsubcategorykey c_41, t.productline c2, t.productsubcategorykey c5s0 FROM as_adventure.t1 t WHERE true GROUP BY 2, 3) t_53 GROUP BY 4, 5, 6, 7;   ... View more

You need to cast one of the branches of the else to be a compatible type with the other one.   The problem is that both decimal types have the max precision (38)  and different scale and neither can be converted automatically to the other without potentially losing precision.   A lot of the decimal behaviour such as result types of expressions was changed in CDH6 (and upstream Apache Impala 3.0). https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/impala_decimal.html has a lot of related information. ... View more

I think the reality is now that both are great technologies and the overlap in use cases is pretty big - there are a lot of SQL workloads where either can work.   I just wanted to clarify a few points.   Impala does support querying complex types from Parquet - https://docs.cloudera.com/documentation/enterprise/latest/topics/impala_complex_types.html   We also are working on a transparent query retry feature in Impala that should be released soon. ... View more

Yes we should be able to prune based on range partitions. https://docs.cloudera.com/documentation/enterprise/latest/topics/impala_kudu.html#kudu_partitioning has some examples of how to set up a table with both range and hash partitions. You can specify arbitrary timestamp ranges for the partitions.   You can see in the Impala explain plan if your WHERE predicates were converted into kudu pushdown predicates (they're labelled kudu predicates). ... View more