Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NIFI HiveStreaming error

avatar
Contributor

I'm working on NIFI 1.5.0. I downloaded Apache NIFI and installed on one of my HDP cluster nodes. I'm running the test against a 3 nodes HDP 2.6 cluster.

I'm following a tutorial from following link:

https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html

Everything works fine, but I get error at PutHiveStreaming step. It's basically complaining about an error connecting against Hive endpoint and the table. The table in Hive is created as managed table, and not as an external table. I can telnet to Hive metastore server and port from my NIFI node. What would be root-cause of this issue?nifi-error2.png

Error attached:

nifi-error.png

1 ACCEPTED SOLUTION

avatar

I believe you'll want to run the HDF build of NiFi which has libraries tailored to work with HDP Hive.

View solution in original post

9 REPLIES 9

avatar

I believe you'll want to run the HDF build of NiFi which has libraries tailored to work with HDP Hive.

avatar
Contributor

The PutHiveStreaming worked w/ HDF build of NiFi version without any problem. I downloaded

nifi-1.2.0.3.0.0.0-453-bin.tar.gz from HDF, and installed it on a node as standalone application, and file streamed to hive table without any issues. Thanks for your help.

.

avatar
Master Guru

Is your cluster kerberized? Can you access Hive from the command line of that machine? That article is a bit old. can you post any log messages.

The HDF version of NiFi is configured just for this and the addition of ambari makes everything easier.

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.0/bk_installing-hdf-on-hdp/content/upgrading_...

avatar
Contributor

Thanks Timothy. The cluster is not Kerberized. I can access Hive from command line of that machine. I will probably have to try installing HDF version of NIFI. Attached is the NIFI app log.

nifi-applog.txt

avatar
Contributor

Thanks Tim. NiFi Flow worked with HDF version of NiFi.

avatar
Master Guru

The Nifi user may not have permissions to the /apps/hive/warehouse directory

What is it's permissions? What is your Hive local scheme?

Does NiFi users have HDFS read/write permissions?

avatar
Contributor

Thanks Timothy. This could be a reason, I don't have a NIFI user that I've created. How do I create a NIFI user?

The table(OLYMPICS) is in default schema. Here is the DDL for the Hive table.

--created hive table
CREATE TABLE
OLYMPICS(CITY STRING,EDITION INT,SPORT STRING,SUB_SPORT STRING,ATHLETE STRING,COUNTRY STRING,GENDER STRING,EVENT STRING,EVENT_GENDER STRING,MEDAL STRING)
CLUSTERED BY (EDITION)INTO 3 BUCKETS
ROW FORMAT DELIMITED
STORED AS ORC
LOCATION '/tmp/olympics'
TBLPROPERTIES('transactional'='true');

avatar
Master Guru

In order for Hive Streaming to work the following has to be in place:

  • Table is stored as ORC
  • Transactional Property is set to “True”
  • The Table is Bucketed

avatar
Master Guru

The table needs to be created properly with permissions

Example Table DDL

CREATE TABLE `inception`( uuid STRING, top1pct STRING, top1 STRING, top2pct STRING, top2 STRING, top3pct STRING, top3 STRING, top4pct STRING, top4 STRING, top5pct STRING, top5 STRING, imagefilename STRING, runtime STRING) CLUSTERED BY ( top1) INTO 3 BUCKETS ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' TBLPROPERTIES ( 'transactional'='true')