Member since
06-11-2019
13
Posts
0
Kudos Received
0
Solutions
08-27-2019
03:02 AM
Hi Team,
I would like to do few types of API (HTTPS) calls that takes inputs like Authorization header (or) Username/Password for authentication and gives a json body as output.
Can you please suggest me the correct processor and steps to make this call and generate a flow that will hav the json body as content.
... View more
Labels:
- Labels:
-
Apache NiFi
07-09-2019
09:54 AM
Hi I'm using query database on my mysql database table. I'm able to perform a simple query like "select * from table_name" successfully. I want to fetch records on incremental basis, i.e i want to pull the newest record every time the processor runs. But the processor doesn't store any state from "view state". Also i want to mention that my table's first column is a alphanumeric code eg. AL0031. Can you please suggest me a way to fetch records incrementally from my mysql DB table.
... View more
Labels:
- Labels:
-
Apache NiFi
06-21-2019
05:54 PM
Hi , I keep on gettng this error while using PUTSQL processor: ************************ Batch entry 0 INSERT was aborted: ERROR: syntax error at end of input Position: 7 Call getNextException to see other errors in the batch.. There were a total of 1 FlowFiles that failed, 0 that succeeded, and 0 that were not execute and will be routed to retry; : java.sql.BatchUpdateException: Batch entry 0 INSERT was aborted: ERROR: syntax error at end of input Position: 7 Call getNextException to see other errors in the batch. java.sql.BatchUpdateException: Batch entry 0 INSERT was aborted: ERROR: syntax error at end of input Position: 7 Call getNextException to see other errors in the batch. ***************************** I used ReplaceText processor prior to this, and the output file content of that processor is : ******************************* INSERT INTO vzcheckin_visits (visit_data) VALUES ('{ "visits": [ { "buildingCode": "AZ0014", "buildingName": "6955 W Morelos Pl, Chandler, AZ 85226, USA", "entrance": "Front Desk", "createdDate": "2019-05-08 16:56:40.0", "cancelled": false }, { "buildingCode": "AZ0014", "buildingName": "6955 W Morelos Pl, Chandler, AZ 85226, USA", "cancelled": false }] } ') ****************************** The above query runs perfect in command line manually except for the ";" taht i have to give at the end. can you please assist me to solve this error. @Matt Burgess
... View more
Labels:
- Labels:
-
Apache NiFi
06-18-2019
07:48 AM
Hi flow seems to progress now after clearing the state. I actually want to pull a specific file json file from S3 and push its content to a DB. ListS3Object and FetchS3Object are fetching all file of the latest time stamp and later that. Though in fetchS3object i specify particular "Object Key" with my specific file name, all the files get fetched in the queue. Can you please suggest me a way to pull a specific json file.
... View more
06-15-2019
08:04 AM
My nifi flow is not progressing nor showing any error: Flow is for :fetch a test.json fiel from s3 and populate its content in a table in postgresql db. lists3object-> fetchs3object->JoltTrasformationJSON -> ........ ->PutSQL. The lists3object alone lists key as my test.json in "view status" I'm not able to paste screenshots here. Though all processors got started, none of them are doing any transactions (in & out ) both are 0 bytes and i dont see any errors also. Last message logged in nifi-app.log is: 2019-06-14 07:26:48,864 INFO [NiFi Web Server-92] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.aws.s3.ListS3 2019-06-14 07:26:48,864 INFO [Timer-Driven Process Thread-4] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling ListS3[id=016b1032-cd19-10b1-6d77-5a63389cd903] to run 2019-06-14 07:26:49,166 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@5aaa6545 // Another save pending = false 2019-06-14 07:26:51,166 INFO [NiFi Web Server-20] o.a.n.c.s.StandardProcessScheduler Stopping FetchS3Object[id=016b102b-cd19-10b1-385a-bb202d1cebc4] 2019-06-14 07:26:51,166 INFO [NiFi Web Server-20] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.aws.s3.FetchS3Object 2019-06-14 07:26:51,179 INFO [Timer-Driven Process Thread-4] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling FetchS3Object[id=016b102b-cd19-10b1-385a-bb202d1cebc4] to run 2019-06-14 07:26:51,185 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@5aaa6545 // Another save pending = false 2019-06-14 07:26:56,833 INFO [NiFi Web Server-92] o.a.n.c.s.StandardProcessScheduler Starting ListS3[id=016b1032-cd19-10b1-6d77-5a63389cd903] 2019-06-14 07:26:56,836 INFO [Timer-Driven Process Thread-8] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled ListS3[id=016b1032-cd19-10b1-6d77-5a63389cd903] to run with 1 threads 2019-06-14 07:26:57,202 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@5aaa6545 // Another save pending = false The flow was working partially earlier with some error message at Splitjson processor. But now i'm not able to proceed as i don understand where it is stuck. I tried restarting nifi, but no luck. Can you please suggest where could be the issue.
... View more
Labels:
- Labels:
-
Apache NiFi
06-15-2019
08:04 AM
My nifi flow is not progressing nor showing any error: I'm not able to paste screenshots here. Though all processors got started, none of them are doing any transactions (in & out ) both are 0 bytes and i dont see any errors also. Last message logged in nifi-app.log is: 2019-06-14 07:26:48,864 INFO [NiFi Web Server-92] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.aws.s3.ListS3 2019-06-14 07:26:48,864 INFO [Timer-Driven Process Thread-4] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling ListS3[id=016b1032-cd19-10b1-6d77-5a63389cd903] to run 2019-06-14 07:26:49,166 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@5aaa6545 // Another save pending = false 2019-06-14 07:26:51,166 INFO [NiFi Web Server-20] o.a.n.c.s.StandardProcessScheduler Stopping FetchS3Object[id=016b102b-cd19-10b1-385a-bb202d1cebc4] 2019-06-14 07:26:51,166 INFO [NiFi Web Server-20] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.aws.s3.FetchS3Object 2019-06-14 07:26:51,179 INFO [Timer-Driven Process Thread-4] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling FetchS3Object[id=016b102b-cd19-10b1-385a-bb202d1cebc4] to run 2019-06-14 07:26:51,185 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@5aaa6545 // Another save pending = false 2019-06-14 07:26:56,833 INFO [NiFi Web Server-92] o.a.n.c.s.StandardProcessScheduler Starting ListS3[id=016b1032-cd19-10b1-6d77-5a63389cd903] 2019-06-14 07:26:56,836 INFO [Timer-Driven Process Thread-8] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled ListS3[id=016b1032-cd19-10b1-6d77-5a63389cd903] to run with 1 threads 2019-06-14 07:26:57,202 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@5aaa6545 // Another save pending = false The flow was working partially earlier with some error message at Splitjson processor. But now i'm not able to proceed as i don understand where it is stuck. I tried restarting nifi, but no luck. Can you please suggest where could be the issue.
... View more
06-12-2019
11:31 AM
Hi, I want to insert a simple nifi flow where i'm fetching a json file(test.json) from s3 and want to insert the contents of the test.json file in my postgreql table. Can you please guide me with the flow. Currently i have: lists3object -> fetchs3object -> PUTSQL First 2 steps works for me, i dunno how to inser the content of the test.json on my postgresql table.
... View more
Labels:
- Labels:
-
Apache NiFi
06-11-2019
03:44 PM
Sorry I missed to read your complete reply. As i mentioned, i tried with jar " postgresql-42.2.5.jar " which is JDBC 4 postgres driver . But i get the same error again . I'm not sure why its reffereing to jdbc3g.Jdbc3gConnection in the error though its JDBC 4 driver. I tried giving the absolute path too in the Driver location.
... View more
06-11-2019
03:21 PM
Hi Jay, I'm not sure which jar i have to download for postgresql DB driver. I tried with below 2 jars and placed them in nifi_home/lib folder and pointed the location in "Database Driver location". jdbc-postgresql.jar postgresql-42.2.5.jar I get the same error that I pasted above for both the jars. Can you let me know which jar/link to dowload the jar that I have to use for postgres DB driver.
... View more
06-11-2019
11:45 AM
Hi, My nifi flow is a simple one where i fetchs3object and put it into a AWS Postgresql DB using PUTSQL method. Below is my Connection set up: Database Connection URL jdbc:postgresql://xxxxxxxx.amazonaws.com:5432/dhwdh Database Driver Class Name org.postgresql.Driver Database Driver Location(s) /xx/xx/xx/nifi-1.9.2/lib/jdbc-postgresql.jar Kerberos Credentials Service No value set Database User dev Password Sensitive value set Max Wait Time 500 millis I'm getting this error in PUTSQL processor: 2019-06-11 06:55:56,426 ERROR [Timer-Driven Process Thread-8] o.apache.nifi.processors.standard.PutSQL PutSQL[id=016b102c-cd19-10b1-b7d8-960311cdf697] PutSQL[id=016b102c-cd19-10b1-b7d8-960311cdf697] failed to process session due to java.lang.AbstractMethodError: org.postgresql.jdbc3g.Jdbc3gConnection.isValid(I)Z; Processor Administratively Yielded for 1 sec: java.lang.AbstractMethodError: org.postgresql.jdbc3g.Jdbc3gConnection.isValid(I)Z java.lang.AbstractMethodError: org.postgresql.jdbc3g.Jdbc3gConnection.isValid(I)Z . . . . . 2019-06-11 06:55:56,426 WARN [Timer-Driven Process Thread-8] o.a.n.controller.tasks.ConnectableTask Administratively Yielding PutSQL[id=016b102c-cd19-10b1-b7d8-960311cdf697] due to uncaught Exception: java.lang.AbstractMethodError: org.postgresql.jdbc3g.Jdbc3gConnection.isValid(I)Z java.lang.AbstractMethodError: org.postgresql.jdbc3g.Jdbc3gConnection.isValid(I)Z can some one please suggest me how to resolve this.
... View more
Labels:
- Labels:
-
Apache NiFi