Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi 1.4 error Failed connecting to EndPoint in puthivestreaming

Highlighted

Nifi 1.4 error Failed connecting to EndPoint in puthivestreaming

I am with nifi 1.4. I am using ConsumeKafkaRecord_0_10 and PutHiveStreaming processor. I am getting pipe delimited messages which is parsed with CSVReader and written by using AvroRecordSetWriter. But now I am getting below error.

2017-11-06 11:11:02,699 ERROR [Timer-Driven Process Thread-6]o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=819a7fe2-015f-1000-7097-cd3f8614c9c2] Error connecting to Hive endpoint: table oneviewtest3 at thrift://parent.rolta.com:9083
2017-11-06 11:11:02,699 ERROR [Timer-Driven Process Thread-6] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=819a7fe2-015f-1000-7097-cd3f8614c9c2] Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://parent.rolta.com:9083', database='default', table='oneviewtest3', partitionVals=[] }:
org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://parent.rolta.com:9083', database='default', table='oneviewtest3', partitionVals=[] }
org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://parent.rolta.com:9083', database='default', table='oneviewtest3', partitionVals=[] }

Below is my table create statement,

create table oneviewtest3  (TNT_KEY BIGINT, PLANT_KEY BIGINT, UNIT_KEY  BIGINT, TAG_DATA_SRC STRING, TAG_DATA_RLVNC STRING, TAG_TYPE_ID STRING, DB_RQST_ID STRING, BATCH_KEY BIGINT, OPC_TAG_ID  STRING, OPC_TAG_VAL_TS STRING, OPC_SRC_STM_CODE BIGINT, OPC_TAG_VAL STRING, OPC_QLTY_VAL BIGINT, REF_ID STRING, REF_TYPE STRING, ERROR_CODE STRING, OPC_TAG_VAL_AGG_TYPE STRING, OPC_TAG_RSMPL_INTRVL_VAL BIGINT, OPC_TAG_RSMPL_INTRVL_TYPE STRING, OPC_TAG_VAL_TS_PRC  STRING, FLEX_FIELD1 STRING, FLEX_FIELD2 STRING, FLEX_FIELD3 STRING, FLEX_FIELD4 STRING, FLEX_FIELD5 STRING, ETL_BTCH_ID  BIGINT) CLUSTERED BY (OPC_TAG_ID) into 4 buckets stored as orc tblproperties ("transactional"="true","orc.compress"="NONE");

I am already following below in table create statement,

1.ORC format for table. So table has "stored as orc"

2.transactional = "true" is set in the table create statement

3.Bucketed but not sorted. So table has "clustered by (colName) into (n) buckets"

And also done following on hive,

  1. hive.support.concurrency = true
  2. hive.enforce.bucketing = true
  3. hive.exec.dynamic.partition.mode = nonstrict
  4. hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager
  5. hive.compactor.initiator.on = true
  6. hive.compactor.worker.threads = 2

I am using HDP 2.5.3 and nifi-hive-nar-1.4.0.nar. In hive-metastore I see error as,

2017-11-06 12:02:53,436 ERROR [pool-8-thread-192]: server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error occurred during processing of message. java.lang.IllegalStateException: Unexpected DataOperationType: UNSET agentInfo=Unknown txnid:424

So can anybody please help me on it.

2 REPLIES 2
Highlighted

Re: Nifi 1.4 error Failed connecting to EndPoint in puthivestreaming

New Contributor

Is this issue resolved?

,

Is this issue resolved

Highlighted

Re: Nifi 1.4 error Failed connecting to EndPoint in puthivestreaming

HDP and NiFi does not seem compliant. I used HDF.

Don't have an account?
Coming from Hortonworks? Activate your account here