Member since
12-08-2017
4
Posts
1
Kudos Received
0
Solutions
12-12-2017
09:24 AM
I am using SelectHiveQL to query hive table and getting following error
SelectHiveQL[id=495eb94a-0160-1000-0000-000067e12ae0] Unable to execute HiveQL select query select * from DPI due to org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://172.25.58.130:10000/default: java.net.ConnectException: Connection refused). No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://172.25.58.130:10000/default: java.net.ConnectException: Connection refused)
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
12-11-2017
12:03 PM
I am new to nifi and trying to load data in hive table using PutHiveStreaming and getting following error. Hive=1.2.1000 NIFI=1.0.0.2.0 Auto create partitions= False 2017-12-11 16:07:24,661 INFO [Timer-Driven Process Thread-10] hive.metastore Trying to connect to metastore with URI thrift://172.25.58.130:9083 2017-12-11 16:07:24,664 INFO [Timer-Driven Process Thread-10] hive.metastore Connected to metastore. 2017-12-11 16:07:24,668 ERROR [Timer-Driven Process Thread-10] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=2fda6ba8-0160-1000-0000-00007804959f] Failed to create HiveWriter for endpoint: {metaStoreUri='thrift://172.25.58.130:9083', database='Default', table='dim_prod2', partitionVals=[] } 2017-12-11 16:07:24,675 ERROR [Timer-Driven Process Thread-10] o.a.n.processors.hive.PutHiveStreaming org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://172.25.58.130:9083', database='Default', table='dim_prod2', partitionVals=[] } at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:80) ~[nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java:45) ~[nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] at org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java:827) [nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] at org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java:738) [nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$4(PutHiveStreaming.java:462) [nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1880) ~[na:na] at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1851) ~[na:na] at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:389) [nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064) ~[na:na] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) ~[na:na] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) ~[na:na] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) ~[na:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_77] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_77] Caused by: org.apache.nifi.util.hive.HiveWriter$TxnBatchFailure: Failed acquiring Transaction Batch from EndPoint: {metaStoreUri='thrift://172.25.58.130:9083', database='Default', table='dim_prod2', partitionVals=[] } at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:255) ~[nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:74) ~[nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] ... 19 common frames omitted Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable to acquire lock on {metaStoreUri='thrift://172.25.58.130:9083', database='Default', table='dim_prod2', partitionVals=[] } at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:578) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1] at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:547) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1] at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:252) ~[nifi-hive-processors-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579] ... 20 common frames omitted Caused by: org.apache.thrift.transport.TTransportException: null at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hive-exec-1.2.1.jar:1.2.1] at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-1.2.1.jar:1.2.1] at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[hive-exec-1.2.1.jar:1.2.1] at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[hive-exec-1.2.1.jar:1.2.1] at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[hive-exec-1.2.1.jar:1.2.1] at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) ~[hive-exec-1.2.1.jar:1.2.1] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3906) ~[hive-metastore-1.2.1.jar:1.2.1] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3893) ~[hive-metastore-1.2.1.jar:1.2.1] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1863) ~[hive-metastore-1.2.1.jar:1.2.1] at sun.reflect.GeneratedMethodAccessor449.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_77] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_77] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152) ~[hive-metastore-1.2.1.jar:1.2.1] at com.sun.proxy.$Proxy142.lock(Unknown Source) ~[na:na] at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:573) ~[hive-hcatalog-streaming-1.2.1.jar:1.2.1] ... 22 common frames omitted
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
12-11-2017
05:55 AM
1 Kudo
@Shu thanks, I was missing "/database" in connection URL.
... View more
12-08-2017
07:12 AM
I am new to NIFI, and getting following error while I am trying to access Teradata database(production environment) using querydatabasetable processor from Nifi. I have tried telnet command to teradata port 1025 from command line and its connecting. 2017-12-07 17:00:11,753 ERROR [StandardProcessScheduler Thread-4] o.a.n.p.standard.QueryDatabaseTable QueryDatabaseTable[id=2f70c1e5-0160-1000-0000-0000077bf1a3] QueryDatabaseTable[id=2f70c1e5-0160-1000-0000-0000077bf1a3] failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.; processor will not be scheduled to run for 30 seconds: java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task. 2017-12-07 17:00:11,761 ERROR [StandardProcessScheduler Thread-8] org.apache.nifi.engine.FlowEngine A flow controller task execution stopped abnormally java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_77] at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_77] at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100) ~[na:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor558.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_77] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_77] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) ~[na:na] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) ~[na:na] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) ~[na:na] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) ~[na:na] at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1244) ~[na:na] at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1240) ~[na:na] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_77] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] ... 2 common frames omitted Caused by: org.apache.nifi.processor.exception.ProcessException: Unable to communicate with database in order to determine column types at org.apache.nifi.processors.standard.AbstractDatabaseFetchProcessor.setup(AbstractDatabaseFetchProcessor.java:189) ~[na:na] at org.apache.nifi.processors.standard.QueryDatabaseTable.setup(QueryDatabaseTable.java:164) ~[na:na] ... 15 common frames omitted Caused by: java.sql.SQLException: [Teradata Database] [TeraJDBC 16.00.00.23] [Error 3807] [SQLState 42S02] Object 'DIM_PROD' does not exist. at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:309) ~[terajdbc4.jar:16.00.00.23] at com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:103) ~[terajdbc4.jar:16.00.00.23] at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:311) ~[terajdbc4.jar:16.00.00.23] at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:200) ~[terajdbc4.jar:16.00.00.23] at com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:137) ~[terajdbc4.jar:16.00.00.23] at com.teradata.jdbc.jdbc_4.statemachine.StatementController.run(StatementController.java:128) ~[terajdbc4.jar:16.00.00.23] at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:389) ~[terajdbc4.jar:16.00.00.23] at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:331) ~[terajdbc4.jar:16.00.00.23] at com.teradata.jdbc.jdbc_4.TDStatement.doNonPrepExecuteQuery(TDStatement.java:319) ~[terajdbc4.jar:16.00.00.23] at com.teradata.jdbc.jdbc_4.TDStatement.executeQuery(TDStatement.java:1121) ~[terajdbc4.jar:16.00.00.23] at org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208) ~[na:na] at org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208) ~[na:na] at org.apache.nifi.processors.standard.AbstractDatabaseFetchProcessor.setup(AbstractDatabaseFetchProcessor.java:173) ~[na:na] ... 16 common frames omitted
... View more
Labels:
- Labels:
-
Apache NiFi