Hello All, I am moving data from one Oracle database to another. My flow is like this: ExecuteSQL -> SplitAvro -> ConvertRecord ( to generate SQL statement) -> ConvertCharacterSet ( UTF-8 to ISO-8859-1) -> PutSQL Source oracle DB has character set AL32UTF8 Target oracle DB has character set WE8ISO8859P1 But I see lots of ?? in the target database. Any clues on what I am doing wrong here? P.S : My flow runs slow as I am not using prepared statement. I will work on this once this character set issue is resolved. Thanks in advance !! ... View more

Hello All, My process flow is like this: Process A: Read data from system A ( ExecuteSQL) and load into staging table A ( PutSQL) Process B: Read data from System B ( ExecuteSQL) and load into staging table B ( PutSQL) Process C: Join data between tables A and B( ExecuteSQL)and generate a file I want Process C to run only after Process A and Process B are done. How can I achieve this in NiFi. I looked into Wait/Notify and MergeContent processors and nothing seems to be working. Any ideas? ... View more

Hello All, I just finished setting up Ambari 2.5.1. I am using Standalone PostgreSQL 9.6 Setup finished successfully but I am getting error while starting Ambari server. This is what I see in Linux command line Ambari Server 'setup' completed successfully. [kfkadm@******didkfkw ~]$ ambari-server start Using python /usr/bin/python Starting ambari-server Organizing resource files at /var/lib/ambari-server/resources... Unable to check firewall status when starting without root privileges. Please do not forget to disable or adjust firewall if needed Ambari database consistency check started... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log /usr/bin/sh: line 0: ulimit: open files: cannot modify limit: Operation not permitted Waiting for server start.........Unable to determine server PID. Retrying... ......Unable to determine server PID. Retrying... ......Unable to determine server PID. Retrying... ERROR: Exiting with exit code -1. REASON: Ambari Server java process died with exitcode 1. Check /var/log/ambari-server/ambari-server.out for more information. This is what I see in ambari-server.out file Error injecting constructor, java.lang.RuntimeException: Error while creating database accessor at org.apache.ambari.server.orm.DBAccessorImpl.<init>(DBAccessorImpl.java:85) at org.apache.ambari.server.orm.DBAccessorImpl.class(DBAccessorImpl.java:73) while locating org.apache.ambari.server.orm.DBAccessorImpl while locating org.apache.ambari.server.orm.DBAccessor for field at org.apache.ambari.server.orm.dao.DaoUtils.dbAccessor(DaoUtils.java:36) at org.apache.ambari.server.orm.dao.DaoUtils.class(DaoUtils.java:36) while locating org.apache.ambari.server.orm.dao.DaoUtils for field at org.apache.ambari.server.orm.dao.UserDAO.daoUtils(UserDAO.java:45) at org.apache.ambari.server.orm.dao.UserDAO.class(UserDAO.java:45) while locating org.apache.ambari.server.orm.dao.UserDAO for field at org.apache.ambari.server.controller.internal.ActiveWidgetLayoutResourceProvider.userDAO(ActiveWidgetLayoutResourceProvider.java:61) Caused by: java.lang.RuntimeException: Error while creating database accessor at org.apache.ambari.server.orm.DBAccessorImpl.<init>(DBAccessorImpl.java:118) at org.apache.ambari.server.orm.DBAccessorImpl$$FastClassByGuice$$86dbc63e.newInstance(<generated>) at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.Scopes$1$1.get(Scopes.java:65) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:94) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.Scopes$1$1.get(Scopes.java:65) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:110) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:94) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.Scopes$1$1.get(Scopes.java:65) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:53) at com.google.inject.internal.InjectionRequestProcessor$StaticInjection$1.call(InjectionRequestProcessor.java:116) at com.google.inject.internal.InjectionRequestProcessor$StaticInjection$1.call(InjectionRequestProcessor.java:110) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) at com.google.inject.internal.InjectionRequestProcessor$StaticInjection.injectMembers(InjectionRequestProcessor.java:110) at com.google.inject.internal.InjectionRequestProcessor.injectMembers(InjectionRequestProcessor.java:78) at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:170) at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109) at com.google.inject.Guice.createInjector(Guice.java:95) at com.google.inject.Guice.createInjector(Guice.java:72) at com.google.inject.Guice.createInjector(Guice.java:62) at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:992) Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:207) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:138) at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) at org.postgresql.Driver.makeConnection(Driver.java:410) at org.postgresql.Driver.connect(Driver.java:280) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:247) at org.apache.ambari.server.orm.DBAccessorImpl.<init>(DBAccessorImpl.java:91) ... 41 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at java.net.Socket.connect(Socket.java:538) at org.postgresql.core.PGStream.<init>(PGStream.java:60) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:101) ... 52 more How do I resolve this error? ... View more

Hello All, I am passing sql.args.1.type = 2005 and sql.args.1.value as a CLOB value to PutSQL and it is throwing below error. How do I resolve this error? 2017-09-04 10:17:24,924 INFO [StandardProcessScheduler Thread-5] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled PutSQL[id=69534f73-821e-1782-ffff-fffffabb94fb] to run with 1 threads 2017-09-04 10:17:25,009 ERROR [Timer-Driven Process Thread-4] o.apache.nifi.processors.standard.PutSQL PutSQL[id=69534f73-821e-1782-ffff-fffffabb94fb] org.apache.nifi.processors.standard.PutSQL$$Lambda$653/696434096@771333e failed to process due to org.apache.nifi.processor.exception.ProcessException: Failed to process StandardFlowFileRecord[uuid=14895cdd-3eac-4f2a-adae-f014526e24b1,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1502123289026-125, container=default, section=125], offset=71441, length=39],offset=0,name=5936554321553480,size=39] due to java.lang.ClassCastException: java.lang.String cannot be cast to oracle.sql.CLOB; rolling back session: {} org.apache.nifi.processor.exception.ProcessException: Failed to process StandardFlowFileRecord[uuid=14895cdd-3eac-4f2a-adae-f014526e24b1,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1502123289026-125, container=default, section=125], offset=71441, length=39],offset=0,name=5936554321553480,size=39] due to java.lang.ClassCastException: java.lang.String cannot be cast to oracle.sql.CLOB at org.apache.nifi.processor.util.pattern.ExceptionHandler.lambda$createOnGroupError$14(ExceptionHandler.java:226) at org.apache.nifi.processor.util.pattern.ExceptionHandler.lambda$createOnError$13(ExceptionHandler.java:179) at org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$11(ExceptionHandler.java:54) at org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$11(ExceptionHandler.java:54) at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:148) at org.apache.nifi.processors.standard.PutSQL.lambda$new$72(PutSQL.java:283) at org.apache.nifi.processors.standard.PutSQL.lambda$new$75(PutSQL.java:324) at org.apache.nifi.processor.util.pattern.PutGroup.putFlowFiles(PutGroup.java:91) at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:101) at org.apache.nifi.processors.standard.PutSQL.lambda$onTrigger$86(PutSQL.java:544) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114) at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184) at org.apache.nifi.processors.standard.PutSQL.onTrigger(PutSQL.java:544) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to oracle.sql.CLOB at oracle.jdbc.driver.OraclePreparedStatement.setObjectCritical(OraclePreparedStatement.java:8874) at oracle.jdbc.driver.OraclePreparedStatement.setObjectInternal(OraclePreparedStatement.java:8396) at oracle.jdbc.driver.OraclePreparedStatement.setObject(OraclePreparedStatement.java:8980) at oracle.jdbc.driver.OraclePreparedStatementWrapper.setObject(OraclePreparedStatementWrapper.java:242) at org.apache.commons.dbcp.DelegatingPreparedStatement.setObject(DelegatingPreparedStatement.java:166) at org.apache.commons.dbcp.DelegatingPreparedStatement.setObject(DelegatingPreparedStatement.java:166) at org.apache.nifi.processors.standard.PutSQL.setParameter(PutSQL.java:888) at org.apache.nifi.processors.standard.PutSQL.setParameters(PutSQL.java:677) at org.apache.nifi.processors.standard.PutSQL.lambda$null$71(PutSQL.java:285) at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127) ... 19 common frames omitted ... View more

Hello All, Can I pass a CLOB as sql.args.N.value? Are their any size limitations for attributes? Thanks!! ... View more

Hello All, I want to run below query in PutSQL. This query is calling an Oracle function and accepts a CLOB value as parameter. selectobds_loader.getmain(:clob)from dual Does PutSQL support SELECT queries? Can I pass a CLOB as attribute in NiFi? If Yes, then what would be the value for sql.args.n.type for CLOB? Thanks in advance ... View more

Hello, Data Provenance UI allows replay of data. Is their any way to fix "bad data" in NiFi before replay. For ex - Let's say data record is missing a value in mandatory field. Instead of fixing it in source and re sending data, is it possible to modify data by adding value to mandatory field in NiFi and replay it? Thanks!! ... View more

Hello , We do lots of dataflows moving data in and out of Confluent Kafka using NiFi. Can Atlas be used to track end to end lineage in such flows ( NiFi --> Confluent Kafka --> NiFi) ? Thanks!! ... View more

Hello All, I have an Avro message coming out of SplitAvro processor. This message contains a field and its value VAL. I want to run an update SQL against a database table as follows: Update table X set Status='Processed' where ID = VAL; How can I achieve this in NiFi? Thanks!! ... View more