Version: NiFi 1.11.x   Noticed a behavior where for this version of NiFi, when the remote url is a GSLB, it doesn't recognize when a host is removed from a GSLB until the processor is stopped. Any idea why? Any ideas for a solution? ... View more

Apache NiFi is currently at version 1.16.x https://nifi.apache.org/ HDF is a frozen project where NiFi is at 1.11.4 Ambari seems to be attic'd.   Is there a way to upgrade the version of NiFi packaged with HDF to a latter version of NiFi? Is there a way to combine the two standalone apache projects of NiFi and Ambari to get the functionality of Ambari's management but with a later version of NiFi? ... View more

So the title basically states it, but I'm currently running into an issue when leveraging Presto to ready from a Hive3 environment if the table is populated with ORC data by Nifi's PutHive3Streaming processor.   Presto is able to read ORC ACID tables if Hive 3 and populated via command line or other nifi processors. I attempted to write data using PutHive3Streaming from later versions of Nifi (1.11.4) to no avail.    Error: io.prestosql.spi.PrestoException: Error opening Hive split hdfs://path/to/bucket (offset=0, length=29205493): rowsInRowGroup must be greater than zero   Versions: Nifi HDF 1.9 PrestoSQL 331/332 ... View more

UPDATE: I'm wondering if this is happening on deletion, not creation. There is a describe that runs before the external table creation and it's throwing the same error. Checking binlogs in mysql for any clues.   UPDATE2: After reading the binlog, I am more convinced the DROP TABLE <TABLE> cmd for hive isn't complete correctly and isn't delete from TBLS. -------------------------------------------------------------- This is a behavior I've experienced frequently in Hive 3, that occurred almost never in Hive. Hive Version:  3.0.0.3.1 Metastore is mysql.   As part of an incremental data sqoop, I create two tables:One external, one ORC. Sometimes during the creation of these two tables, the metastore doesn't populate the the SERDES, SDS, CDS tables with the corresponding metadata and leaves the SD_ID null in the TBLS table. The table remain non-interactive (not even droppable) until I populate dummy data in the corresponding tables and update the TBLS's SD_ID field.   The statements for table creation are as such: EXTERNAL create external table if not exists <DIFF EXTERNAL TABLE NAME> like <MAIN EXTERNAL TABLE NAME>; ORC create table <DIFF ORC TABLE> stored as orc as select * from <DIFF EXTERNAL TABLE NAME>; The logs in the metastore are a bit unclear:         2019-11-13T04:08:07,696 ERROR [pool-6-thread-187]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(201)) - java.lang.NullPointerException 2019-11-13T04:08:07,696 ERROR [pool-6-thread-187]: server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error occurred during processing of message.         And the logs in the Hiveserver2 logs only highlight after the issue has happened. ... View more

We're currently looking to upgrade our production cluster to 3.1.0, but we're running into some pain points in a test environment that are stopping us from proceeding. Versions we're interacting with: HDFS 3.1.1.3.1 Hive 3.0.0.3.1   The problem: When attempting to run multiple updates on the same table, it fails due to a write conflict. This worked in previous versions, but in the new Hive 3 if the second update statement occurs during the first, it will throw this exception and fail (schema and table generalize). This is thru the new hive client (beeline).   Error: Error while processing statement: FAILED: Hive Internal Error: org.apache.hadoop.hive.ql.lockmgr.LockException(Transaction manager has aborted the transaction txnid:306237872. Reason: Aborting [txnid:306237872,306237873] due to a write conflict on <SCHEMA>/<TABLE> committed by [txnid:306237871,306237872] u/u) (state=42000,code=12)   The table properties are: TBLPROPERTIES( 'bucketing_version'='2','transactional'='true','transactional_properties'='default','transient_lastDdlTime'='1572894940')   ... View more