Member since
07-03-2017
48
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
823 | 07-17-2017 11:34 PM |
11-29-2018
03:29 PM
Hello All, I am moving data from one Oracle database to another. My flow is like this: ExecuteSQL -> SplitAvro -> ConvertRecord ( to generate SQL statement) -> ConvertCharacterSet ( UTF-8 to ISO-8859-1) -> PutSQL Source oracle DB has character set AL32UTF8 Target oracle DB has character set WE8ISO8859P1 But I see lots of ?? in the target database. Any clues on what I am doing wrong here? P.S : My flow runs slow as I am not using prepared statement. I will work on this once this character set issue is resolved. Thanks in advance !!
... View more
Labels:
- Labels:
-
Apache NiFi
03-20-2018
04:52 PM
Hello Rahul, Thanks for prompt response. Process A and Process B run like this: ExecuteSQL -> SplitAvro --> ConvertRecord --> PutSQL. So I end up with one flow file per record. Should I use PutDatabaseRecord instead of PutSQL here? Will PutDatabaseRecord give me one flow file as output upon successful completion of all INSERTs? Thanks !!
... View more
03-20-2018
04:14 PM
Hello All, My process flow is like this: Process A: Read data from system A ( ExecuteSQL) and load into staging table A ( PutSQL) Process B: Read data from System B ( ExecuteSQL) and load into staging table B ( PutSQL) Process C: Join data between tables A and B( ExecuteSQL)and generate a file I want Process C to run only after Process A and Process B are done. How can I achieve this in NiFi. I looked into Wait/Notify and MergeContent processors and nothing seems to be working. Any ideas?
... View more
Labels:
- Labels:
-
Apache NiFi
11-21-2017
06:24 PM
Thanks this solution worked.
... View more
11-17-2017
05:01 PM
Hello All,I am getting this error in
PublishKafka_0_10 processor for a message of size 2.08 MB. I have
updated Max Request Size to 10 MB in processor properties and
max.request.size to 10 MB in Kafka's server.properties. After reboot
Kafka Broker I can see that max.request.size = 10 MB in Kafka logs but I
am still getting below error. What am I missing here? 2017-11-17 11:07:47,966 ERROR [Timer-Driven Process Thread-4] o.a.n.p.kafka.pubsub.PublishKafka_0_10 PublishKafka_0_10[id=e6d932d9-97ae-1647-aa8f-86d07791ce25] Failed to send all message for StandardFlowFileRecord[uuid=fa2399e5-bea5-4113-b58b-6cdef228733c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1510934860019-132, container=default, section=132], offset=0, length=2160613],offset=0,name=12337127439954063,size=2160613] to Kafka; routing to failure due to org.apache.nifi.stream.io.exception.TokenTooLargeException: A message in the stream exceeds the maximum allowed message size of 1048576 bytes.: {} org.apache.nifi.stream.io.exception.TokenTooLargeException: A message in the stream exceeds the maximum allowed message size of 1048576 bytes. at org.apache.nifi.stream.io.util.AbstractDemarcator.extractDataToken(AbstractDemarcator.java:157) at org.apache.nifi.stream.io.util.StreamDemarcator.nextToken(StreamDemarcator.java:129) at org.apache.nifi.processors.kafka.pubsub.PublisherLease.publish(PublisherLease.java:78) at org.apache.nifi.processors.kafka.pubsub.PublishKafka_0_10$1.process(PublishKafka_0_10.java:334) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2136) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2106) at org.apache.nifi.processors.kafka.pubsub.PublishKafka_0_10.onTrigger(PublishKafka_0_10.java:330) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)Thanks and Regards
... View more
Labels:
- Labels:
-
Apache NiFi
10-18-2017
04:22 PM
Thanks Matt.
... View more
10-12-2017
06:20 PM
Hello All,Can someone please help in understanding what I am doing wrong here?I am passing following values to PutSQL processor sql.args.1.type 2005 sql.args.1.value V99|1.0|
E99|BILLING|1234567|11049532|350|201706|201709|
R03|350|11049532|4| |KK|174|A|07/27/1948| |322| |0|F| | |201712|201709|0|168900.56|0|0|0|0|40134|201712|0|0|316| |200402|0| | | | | | | | | |POPR_CUPDT| | | | | | | | | | | |0|0|2|200402|0| | | |
P03|0|1|0|0|199404|317|201711|08/17/2017| |0|201712|08/22/2017|322|0|
K06|201706|111.53|111.53|14.8|96.73|111.53|15||0|0|
K06|201707|117.48|117.48|15.85|101.63|117.48|15||0|0|
K06|201708|227.98|227.98|44.34|183.64|227.98|30||0|0|
SQL query is: select obds_loader.getmain(?) from dual PutSQL processor is throwing below error. 2017-10-12 13:56:39,129 ERROR [Timer-Driven Process Thread-1] o.apache.nifi.processors.standard.PutSQL PutSQL[id=0beb1a31-980d-38f9-fc0e-6c0ac7a68984]
Failed to update database due to a failed batch update,
java.sql.BatchUpdateException: invalid batch command. There were a total
of 1 FlowFiles that failed, 0 that succeeded, and 0 that were not
execute and will be routed to retry; : java.sql.BatchUpdateException:
invalid batch command java.sql.BatchUpdateException: invalid batch command at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:10358) at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:230) at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297) at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297) at org.apache.nifi.processors.standard.PutSQL.lambda$null$10(PutSQL.java:348) at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127) at org.apache.nifi.processors.standard.PutSQL.lambda$new$12(PutSQL.java:346) at org.apache.nifi.processor.util.pattern.PutGroup.putFlowFiles(PutGroup.java:94) at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:101) at org.apache.nifi.processors.standard.PutSQL.lambda$onTrigger$20(PutSQL.java:554) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114) at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184) at org.apache.nifi.processors.standard.PutSQL.onTrigger(PutSQL.java:554) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Thanks and Regards
... View more
Labels:
- Labels:
-
Apache NiFi
09-16-2017
12:00 AM
I am able to connect to ambari DB sudo su postgres bash-4.2$ psql -U ambari -d ambari could not change directory to "/home/kfkadm": Permission denied Password for user ambari: psql (9.6.5) Type "help" for help. ambari=> ambari.properties are mostly same except for the ones in bold below: server.jdbc.connection-pool=internal ( This one is extra in my setup) server.jdbc.database=postgres server.jdbc.database_name=ambari server.jdbc.driver=org.postgresql.Driver server.jdbc.hostname=localhost ( This one is pointing to local host instead of server name) server.jdbc.port=5432 server.jdbc.postgres.schema=ambarischema server.jdbc.rca.driver=org.postgresql.Driver server.jdbc.rca.url=jdbc:postgresql://*******:5432/ambari server.jdbc.rca.user.name=ambari server.jdbc.rca.user.passwd=/etc/ambari-server/conf/password.dat server.jdbc.url=jdbc:postgresql://**********:5432/ambari server.jdbc.user.name=ambari server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat I can see correct password at /etc/ambari-server/conf/password.dat
... View more
09-15-2017
11:38 AM
@Sridhar Reddy Thanks for replying. This is what I see [kfkadm@*********** ~]$ service postgresql status Redirecting to /bin/systemctl status postgresql.service ● postgresql.service Loaded: not-found (Reason: No such file or directory) Active: inactive (dead) [kfkadm@********** ~]$ sudo systemctl status postgresql-9.6 [sudo] password for kfkadm: ● postgresql-9.6.service - PostgreSQL 9.6 database server Loaded: loaded (/usr/lib/systemd/system/postgresql-9.6.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2017-09-14 15:46:05 EDT; 15h ago Main PID: 7428 (postmaster) CGroup: /system.slice/postgresql-9.6.service ├─7428 /usr/pgsql-9.6/bin/postmaster -D /var/lib/pgsql/9.6/data/ ├─7429 postgres: logger process ├─7431 postgres: checkpointer process ├─7432 postgres: writer process ├─7433 postgres: wal writer process ├─7434 postgres: autovacuum launcher process └─7435 postgres: stats collector process Sep 14 15:46:05 ******* systemd[1]: Starting PostgreSQL 9.6 database server... Sep 14 15:46:05 ********** postmaster[7428]: < 2017-09-14 15:46:05.333 EDT > LOG: redirecting log output to loggi...ocess Sep 14 15:46:05 ************* postmaster[7428]: < 2017-09-14 15:46:05.333 EDT > HINT: Future log output will appear ...log". Sep 14 15:46:05 ************* systemd[1]: Started PostgreSQL 9.6 database server. Hint: Some lines were ellipsized, use -l to show in full. [kfkadm@********** ~]$ netstat -ntpl | grep 5432 (No info could be read for "-p": geteuid()=30009 but you should be root.) tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN - tcp6 0 0 ::1:5432 :::* LISTEN -
... View more