question Re: NIFI HiveStreaming error in Archives of Support Questions (Read Only)

NIFI HiveStreaming error

winnie_philip — Tue, 27 Feb 2018 04:03:05 GMT

I'm working on NIFI 1.5.0. I downloaded Apache NIFI and installed on one of my HDP cluster nodes. I'm running the test against a 3 nodes HDP 2.6 cluster.

I'm following a tutorial from following link:

https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html

Everything works fine, but I get error at PutHiveStreaming step. It's basically complaining about an error connecting against Hive endpoint and the table. The table in Hive is created as managed table, and not as an external table. I can telnet to Hive metastore server and port from my NIFI node. What would be root-cause of this issue?nifi-error2.png

Error attached:

nifi-error.png

Re: NIFI HiveStreaming error

JoeWitt — Tue, 27 Feb 2018 11:52:54 GMT

I believe you'll want to run the HDF build of NiFi which has libraries tailored to work with HDP Hive.

Re: NIFI HiveStreaming error

TimothySpann — Wed, 28 Feb 2018 02:14:57 GMT

Is your cluster kerberized? Can you access Hive from the command line of that machine? That article is a bit old. can you post any log messages.

The HDF version of NiFi is configured just for this and the addition of ambari makes everything easier.

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.0/bk_installing-hdf-on-hdp/content/upgrading_ambari.html

Re: NIFI HiveStreaming error

TimothySpann — Wed, 28 Feb 2018 02:46:43 GMT

The Nifi user may not have permissions to the /apps/hive/warehouse directory

What is it's permissions? What is your Hive local scheme?

Does NiFi users have HDFS read/write permissions?

Re: NIFI HiveStreaming error

TimothySpann — Wed, 28 Feb 2018 02:48:30 GMT

In order for Hive Streaming to work the following has to be in place:

Table is stored as ORC
Transactional Property is set to “True”
The Table is Bucketed

Re: NIFI HiveStreaming error

TimothySpann — Wed, 28 Feb 2018 02:59:51 GMT

The table needs to be created properly with permissions

Example Table DDL

CREATE TABLE `inception`( uuid STRING, top1pct STRING, top1 STRING, top2pct STRING, top2 STRING, top3pct STRING, top3 STRING, top4pct STRING, top4 STRING, top5pct STRING, top5 STRING, imagefilename STRING, runtime STRING) CLUSTERED BY ( top1) INTO 3 BUCKETS ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' TBLPROPERTIES ( 'transactional'='true')

Re: NIFI HiveStreaming error

winnie_philip — Wed, 28 Feb 2018 04:00:59 GMT

Thanks Timothy. This could be a reason, I don't have a NIFI user that I've created. How do I create a NIFI user?

The table(OLYMPICS) is in default schema. Here is the DDL for the Hive table.

--created hive table
CREATE TABLE
OLYMPICS(CITY STRING,EDITION INT,SPORT STRING,SUB_SPORT STRING,ATHLETE STRING,COUNTRY STRING,GENDER STRING,EVENT STRING,EVENT_GENDER STRING,MEDAL STRING)
CLUSTERED BY (EDITION)INTO 3 BUCKETS
ROW FORMAT DELIMITED
STORED AS ORC
LOCATION '/tmp/olympics'
TBLPROPERTIES('transactional'='true');

Re: NIFI HiveStreaming error

winnie_philip — Wed, 28 Feb 2018 04:28:58 GMT

Thanks Timothy. The cluster is not Kerberized. I can access Hive from command line of that machine. I will probably have to try installing HDF version of NIFI. Attached is the NIFI app log.

nifi-applog.txt

Re: NIFI HiveStreaming error

winnie_philip — Tue, 06 Mar 2018 04:03:08 GMT

The PutHiveStreaming worked w/ HDF build of NiFi version without any problem. I downloaded

nifi-1.2.0.3.0.0.0-453-bin.tar.gz from HDF, and installed it on a node as standalone application, and file streamed to hive table without any issues. Thanks for your help.

Re: NIFI HiveStreaming error

winnie_philip — Tue, 06 Mar 2018 04:04:29 GMT

Thanks Tim. NiFi Flow worked with HDF version of NiFi.