Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.
mayank_rathi
Contributor
My Accepted Solutions
Show All

Re: convertcharacterset not working

by Contributor in Support Questions
‎12-10-2018 12:40 PM
‎12-10-2018 12:40 PM
Can anyone please help me with this issue? ... View more

convertcharacterset not working

by Contributor in Support Questions
‎11-29-2018 03:29 PM
‎11-29-2018 03:29 PM
Hello All, I am moving data from one Oracle database to another. My flow is like this: ExecuteSQL -> SplitAvro -> ConvertRecord ( to generate SQL statement) -> ConvertCharacterSet ( UTF-8 to ISO-8859-1) -> PutSQL Source oracle DB has character set AL32UTF8 Target oracle DB has character set WE8ISO8859P1 But I see lots of ?? in the target database. Any clues on what I am doing wrong here? P.S : My flow runs slow as I am not using prepared statement. I will work on this once this character set issue is resolved. Thanks in advance !! ... View more
Labels:

Re: How to setup depedency in NiFi

by Contributor in Support Questions
‎03-20-2018 04:52 PM
‎03-20-2018 04:52 PM
Hello Rahul, Thanks for prompt response. Process A and Process B run like this: ExecuteSQL -> SplitAvro --> ConvertRecord --> PutSQL. So I end up with one flow file per record. Should I use PutDatabaseRecord instead of PutSQL here? Will PutDatabaseRecord give me one flow file as output upon successful completion of all INSERTs? Thanks !! ... View more

How to setup depedency in NiFi

by Contributor in Support Questions
‎03-20-2018 04:14 PM
‎03-20-2018 04:14 PM
Hello All, My process flow is like this: Process A: Read data from system A ( ExecuteSQL) and load into staging table A ( PutSQL) Process B: Read data from System B ( ExecuteSQL) and load into staging table B ( PutSQL) Process C: Join data between tables A and B( ExecuteSQL)and generate a file I want Process C to run only after Process A and Process B are done. How can I achieve this in NiFi. I looked into Wait/Notify and MergeContent processors and nothing seems to be working. Any ideas? ... View more
Labels:

Re: NiFi 1.3 - PublishKafka_0_10 - A message in th...

by Contributor in Support Questions
‎11-21-2017 06:24 PM
‎11-21-2017 06:24 PM
Thanks this solution worked. ... View more

NiFi 1.3 - PublishKafka_0_10 - A message in the st...

by Contributor in Support Questions
‎11-17-2017 05:01 PM
‎11-17-2017 05:01 PM
Hello All,I am getting this error in PublishKafka_0_10 processor for a message of size 2.08 MB. I have updated Max Request Size to 10 MB in processor properties and max.request.size to 10 MB in Kafka's server.properties. After reboot Kafka Broker I can see that max.request.size = 10 MB in Kafka logs but I am still getting below error. What am I missing here? 2017-11-17 11:07:47,966 ERROR [Timer-Driven Process Thread-4] o.a.n.p.kafka.pubsub.PublishKafka_0_10 PublishKafka_0_10[id=e6d932d9-97ae-1647-aa8f-86d07791ce25] Failed to send all message for StandardFlowFileRecord[uuid=fa2399e5-bea5-4113-b58b-6cdef228733c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1510934860019-132, container=default, section=132], offset=0, length=2160613],offset=0,name=12337127439954063,size=2160613] to Kafka; routing to failure due to org.apache.nifi.stream.io.exception.TokenTooLargeException: A message in the stream exceeds the maximum allowed message size of 1048576 bytes.: {} org.apache.nifi.stream.io.exception.TokenTooLargeException: A message in the stream exceeds the maximum allowed message size of 1048576 bytes. at org.apache.nifi.stream.io.util.AbstractDemarcator.extractDataToken(AbstractDemarcator.java:157) at org.apache.nifi.stream.io.util.StreamDemarcator.nextToken(StreamDemarcator.java:129) at org.apache.nifi.processors.kafka.pubsub.PublisherLease.publish(PublisherLease.java:78) at org.apache.nifi.processors.kafka.pubsub.PublishKafka_0_10$1.process(PublishKafka_0_10.java:334) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2136) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2106) at org.apache.nifi.processors.kafka.pubsub.PublishKafka_0_10.onTrigger(PublishKafka_0_10.java:330) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)Thanks and Regards ... View more
Labels:

Re: PutSQL: java.sql.BatchUpdateException: invalid...

by Contributor in Support Questions
‎10-18-2017 04:22 PM
‎10-18-2017 04:22 PM
Thanks Matt. ... View more

PutSQL: java.sql.BatchUpdateException: invalid bat...

by Contributor in Support Questions
‎10-12-2017 06:20 PM
‎10-12-2017 06:20 PM
Hello All,Can someone please help in understanding what I am doing wrong here?I am passing following values to PutSQL processor sql.args.1.type 2005 sql.args.1.value V99|1.0| E99|BILLING|1234567|11049532|350|201706|201709| R03|350|11049532|4| |KK|174|A|07/27/1948| |322| |0|F| | |201712|201709|0|168900.56|0|0|0|0|40134|201712|0|0|316| |200402|0| | | | | | | | | |POPR_CUPDT| | | | | | | | | | | |0|0|2|200402|0| | | | P03|0|1|0|0|199404|317|201711|08/17/2017| |0|201712|08/22/2017|322|0| K06|201706|111.53|111.53|14.8|96.73|111.53|15||0|0| K06|201707|117.48|117.48|15.85|101.63|117.48|15||0|0| K06|201708|227.98|227.98|44.34|183.64|227.98|30||0|0| SQL query is: select obds_loader.getmain(?) from dual PutSQL processor is throwing below error. 2017-10-12 13:56:39,129 ERROR [Timer-Driven Process Thread-1] o.apache.nifi.processors.standard.PutSQL PutSQL[id=0beb1a31-980d-38f9-fc0e-6c0ac7a68984] Failed to update database due to a failed batch update, java.sql.BatchUpdateException: invalid batch command. There were a total of 1 FlowFiles that failed, 0 that succeeded, and 0 that were not execute and will be routed to retry; : java.sql.BatchUpdateException: invalid batch command java.sql.BatchUpdateException: invalid batch command at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:10358) at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:230) at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297) at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297) at org.apache.nifi.processors.standard.PutSQL.lambda$null$10(PutSQL.java:348) at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127) at org.apache.nifi.processors.standard.PutSQL.lambda$new$12(PutSQL.java:346) at org.apache.nifi.processor.util.pattern.PutGroup.putFlowFiles(PutGroup.java:94) at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:101) at org.apache.nifi.processors.standard.PutSQL.lambda$onTrigger$20(PutSQL.java:554) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114) at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184) at org.apache.nifi.processors.standard.PutSQL.onTrigger(PutSQL.java:554) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Thanks and Regards ... View more
Labels:

Re: Ambari: Error while creating database accessor

by Contributor in Support Questions
‎09-16-2017 12:00 AM
‎09-16-2017 12:00 AM
I am able to connect to ambari DB sudo su postgres bash-4.2$ psql -U ambari -d ambari could not change directory to "/home/kfkadm": Permission denied Password for user ambari: psql (9.6.5) Type "help" for help. ambari=> ambari.properties are mostly same except for the ones in bold below: server.jdbc.connection-pool=internal ( This one is extra in my setup) server.jdbc.database=postgres server.jdbc.database_name=ambari server.jdbc.driver=org.postgresql.Driver server.jdbc.hostname=localhost ( This one is pointing to local host instead of server name) server.jdbc.port=5432 server.jdbc.postgres.schema=ambarischema server.jdbc.rca.driver=org.postgresql.Driver server.jdbc.rca.url=jdbc:postgresql://*******:5432/ambari server.jdbc.rca.user.name=ambari server.jdbc.rca.user.passwd=/etc/ambari-server/conf/password.dat server.jdbc.url=jdbc:postgresql://**********:5432/ambari server.jdbc.user.name=ambari server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat I can see correct password at /etc/ambari-server/conf/password.dat ... View more

Re: Ambari: Error while creating database accessor

by Contributor in Support Questions
‎09-15-2017 11:38 AM
‎09-15-2017 11:38 AM
@Sridhar Reddy Thanks for replying. This is what I see [kfkadm@*********** ~]$ service postgresql status Redirecting to /bin/systemctl status postgresql.service ● postgresql.service Loaded: not-found (Reason: No such file or directory) Active: inactive (dead) [kfkadm@********** ~]$ sudo systemctl status postgresql-9.6 [sudo] password for kfkadm: ● postgresql-9.6.service - PostgreSQL 9.6 database server Loaded: loaded (/usr/lib/systemd/system/postgresql-9.6.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2017-09-14 15:46:05 EDT; 15h ago Main PID: 7428 (postmaster) CGroup: /system.slice/postgresql-9.6.service ├─7428 /usr/pgsql-9.6/bin/postmaster -D /var/lib/pgsql/9.6/data/ ├─7429 postgres: logger process ├─7431 postgres: checkpointer process ├─7432 postgres: writer process ├─7433 postgres: wal writer process ├─7434 postgres: autovacuum launcher process └─7435 postgres: stats collector process Sep 14 15:46:05 ******* systemd[1]: Starting PostgreSQL 9.6 database server... Sep 14 15:46:05 ********** postmaster[7428]: < 2017-09-14 15:46:05.333 EDT > LOG: redirecting log output to loggi...ocess Sep 14 15:46:05 ************* postmaster[7428]: < 2017-09-14 15:46:05.333 EDT > HINT: Future log output will appear ...log". Sep 14 15:46:05 ************* systemd[1]: Started PostgreSQL 9.6 database server. Hint: Some lines were ellipsized, use -l to show in full. [kfkadm@********** ~]$ netstat -ntpl | grep 5432 (No info could be read for "-p": geteuid()=30009 but you should be root.) tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN - tcp6 0 0 ::1:5432 :::* LISTEN - ... View more
Contact Me
Online
Offline
Last Visited
‎12-10-2018 12:40 PM
Public Statistics
Date Registered ‎07-03-2017 12:08 AM
Last Visited ‎12-10-2018 12:40 PM
Total Messages Posted 48
Total Kudos Received 3