Member since
11-14-2019
51
Posts
6
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1178 | 05-26-2023 08:40 AM | |
1297 | 05-22-2023 02:38 AM | |
1198 | 05-15-2023 11:25 AM |
05-15-2023
08:48 AM
Hi All, I am trying to send slack alerts from Nifi whenever there is a failure in flow. I have some 10 different processor flows, I want to use a single putslack for all those flows but the message in slack is dynamic.. Example: record failed for processor1 record failed for processor2 meaning just a variable changes in slack message. and this variable i should send from processor flows. How can putslack processor know from which previous processor it got the request ? But I don't know how to make it possible. I kept a output port that collects all flowfiles of failures. Now i created a putslack processor that connects to this output port.. but i dont see a option on how can i send variables dynamically. Can someone please guide me with suggestions, or correct me if this ideation is wrong..
... View more
Labels:
- Labels:
-
Apache NiFi
05-10-2023
06:42 AM
Thanks for pointing out. @steven-matison Are you trying to mention that if the source DB change some column formats, add/delete new/old columns ? (I am taking care of these, in the sense, whenever source changes structure, destination is already mimicking it outside Nifi.) So ideally i was thinking this will work indefinitely and they will be in sync. and both of the DBs are mysql. If my thinking is wrong, can you please enlighten me on why i should not relay on this kind of syncing ? What kind of issues are you anticipating in long run ?
... View more
05-10-2023
06:36 AM
Hey @cotopaul Thanks for your reply. ohh disappointing from Nifi.. 😞 But I am not sure why this limitation for this processor. Out of your 2 options i am interested in 1st one. May be I can also query the API for Processors status ( "/processors/run-status-details/queries Submits a query to retrieve the run status details of all processors that are in the given list of Processor IDs") instead of bulletin board. Can you please share the Nifi flow template for option 1 ? Thanks for your hacky ways. 🙂
... View more
05-09-2023
10:23 AM
My Nifi Flow starts with capturechangeMysql processor. I am trying to enhance my flows by adding email alerts whenever a processor fails. I understand I can use PutEmail for this purpose. my flow fails most of the time at the capturechangeMysql because of Binlogs or connections or some other random issues. My thinking is to create a link out of failure relationship of this processor to PutEmail, But there is no failure relationship for this processor. (It only has success relation.) In this scenario how can i alert when there is a failure in capturechangeMysql processor ? Please share your thoughts or if i am missing something or other alternatives.
... View more
Labels:
- Labels:
-
Apache NiFi
04-26-2023
09:05 AM
What is your run schedule time for the processor ? if its too low (say 1 sec or 30 sec), gtry increasing it to reasonable timeline like a minute based on your refresh requirements. @Gagan1707
... View more
04-26-2023
07:40 AM
I am looking for some suggestions and knowledge on streaming job failures and restarts. I created a Nifi flow that captures changes in Source DB and push them to destination DB (a DWH) the changes were captured using capturechangeMysql processor.. which runs 24/7 , so i call it similar to a streaming service.. During development whenever the job fails for what ever reasons, I stop the job, clear its state and fixes issues and reruns job again. But when it goes live, whenever there is a failure of job, between the time i fix and rerun the job, all the changes that were happeneing to source were missed or lost.. imagine if the complex issue took a day to fix, all the data for 24 hours would be lost at destination. I would like to know about different options on how to handle these situations and nullify data loss ? I tried the option of "Retrieve All Records" in the CaptureChangeMySQL 1.18.0 processor. but it is filling up entire queues and taking forver to process latest records. Can some please throw some light here..
... View more
Labels:
- Labels:
-
Apache NiFi
04-14-2023
08:29 AM
Hi All, Looking for some help with JOLT transformatio, which i am completely new. my input JSON looks like this {
"type": "delete",
"timestamp": 1681484884000,
"binlog_filename": "bin.000003",
"binlog_position": 6378700,
"database": "db1",
"table_name": "records",
"table_id": 108,
"columns": [{
"id": 1,
"name": "id",
"column_type": 4,
"value": 259
}, {
"id": 2,
"name": "index_id",
"column_type": 4,
"value": 1
}, {
"id": 3,
"name": "state_id",
"column_type": 4,
"value": 1
}]
} This is from mysql binlogs. I want to transform it to have a single field.. [{
"id": 259
}] from one of the tutorials here in cloudera comminity, i saw this JOLT transformation. [
{
"operation": "shift",
"spec": {
"columns": {
"*": {
"@(value)": "[#1].@(1,name)"
}
}
}
}
] But it gives all fileds.. but I just need one filed which is the ID. I need to drop rest all fields.. Can someone please help in this regards.
... View more
Labels:
- Labels:
-
Apache NiFi
03-28-2023
11:12 AM
I have a mysql destination table with column type datetime. my nifi flow is like convertJsontoSQL -> putsql (into destination table) The action is delete for putsql. after conversion of jsontosql i am getting a value '2023-03-28 15:54:28' which is a valid timestamp. but the flow is failing with below error: duetoThevalue of the sqlargs. 14.value is '2023-03-28 15:54:28', which cannot be converted to a timestamp; routing to failure: java.sql.SQLDataException: The value of the sql.args. 14.value is '2023-03-28 15:54:28', which cannot be converted to a timestamp - Caused by: java. text. ParseException: Unparseable date: "2023-03-28 15:54:28" I can able to use the same "2023-03-28 15:54:28" for manually inserting into a sample datetime column. and can able to delete a record with where clause as this value. I could not understand why Nifi is still not able to convert it to timestamp. Any suggestions are appreciated.
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
03-22-2023
03:54 AM
@mburgess @SamCloud convert jsontosql is giving SQL but with '?' marks. example: "DELETE FROM tab1 WHERE id = ? AND feature_name = ? AND state = ? AND tenant_id = ?" When i print via putfile, the json has valid values.. [{"id":1,"feature_name":"fet1","state":16,"tenant_id":"abc123"}] Want to understand where I am doing it wrong.. Please help
... View more