Member since
09-27-2018
138
Posts
23
Kudos Received
10
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
9446 | 02-28-2021 10:23 PM | |
1892 | 02-08-2021 11:53 PM | |
31172 | 12-16-2020 11:31 PM | |
7442 | 12-14-2020 11:02 PM | |
4752 | 12-14-2020 12:18 AM |
02-17-2021
05:35 AM
@justenji Thanks a lot! The suggestion you proposed works great! However, the source of the problem was not a lack of schema definition. In fact, I managed also to have a working flow with the option Schema Access Strategy = Use String Fields From Header in the CSVReader processor. The warning "None of the fields in the record map to the columns defined by the tablename table." is also obtained when the processor is not able to find the table and this can happen also when the table name is correctly configured in PutDatabaseRecord but there is some issue with user access rights (which ended up to be the actual cause of my error ...).
... View more
02-08-2021
11:53 PM
Hi @Jarinek I think you have to set -pv in front of the new value: nifi set-param -u http://localhost:8088 -pcid 85b8b5bf-0177-1000-3881-8ddd8d31fed8 -pn sample.URL -pv newvalue This one worked on my local NiFi. Remark: Haven't seen that you found it yourself. Great!
... View more
02-08-2021
07:30 AM
Hi @ashwinrk maybe this information will help you further: https://community.cloudera.com/t5/Support-Questions/NiFi-InvokeHTTP-POST-JSON/m-p/307990#M223438 It depends on the concrete target whether the "Authorization" has to be base64encoded or you can use basic authentication.
... View more
02-01-2021
01:20 AM
1 Kudo
@believe7 One possibility is shown here. Maybe there is a more comfortable way, but this would be my solution. Flow Details processors Result response If you have further questions please let me know.
... View more
12-24-2020
07:35 AM
@adhishankarit There is nothing you can pull form the NiFi Rest-API that is going to tell you about successful outcomes form processor execution on a FlowFile. Depending on data volumes, this also sounds like a resource expensive endeavor. That being said, NiFi does have a Site-To-SIte (S2S) Bulletin reporting task. When a processor throws an error it will produce a bulletin and this reporting task can capture those bulletins and send them via NiFi's S2S protocol to another NiFi instance directly into a dataflow were you can handle via dataflow design however you like. Only way you can get INFO level logs in to bulletins is by setting the bulletin level to INFO on all you processors. This only works if you have also configured your NiFI logback.xml so that all NiFi components log at the INFO level as well. Downsides to this: 1. Every processor would display the red bulletin square in upper right corner of processor, This makes using this to find components that are having issues difficult 2. This results in a lot of INFO level logging to the nifi-app.log. You mention edge nodes. You could setup a tailFile processor that tails the nifi-app.log and then send via a dataflow that log data via FlowFiles to some monitor NiFi cluster where another dataflows parses those records via a partitionRecord processor by log_level and then routes based on that log_level for additional handling/notification processing. Downside here: 1. Since you what to track success, you still need INFO level logging enabled for all components. This means even this log collection flow is producing log output. So large logs and logs being written to even when actual data being processed in other flows is not happening. NiFi does have a master bulletin board which you could hit via the rest-api, but this does not get you past the massive logging you may be producing to monitor success. https://nifi.apache.org/docs/nifi-docs/rest-api/index.html Hope this gives you some ideas, Matt
... View more
12-18-2020
05:42 AM
1 Kudo
Excellent news. Can you accept the first 2 responses to close this solution?
... View more
12-17-2020
01:24 AM
Hi @justenji Same for me, I've tried to use avro schema generator, including the inferschema from Nifi, but no luck.
... View more
12-15-2020
12:44 AM
@justenji Yes, exactly. All "main" elements are part of an array in the input.
... View more
12-14-2020
01:12 PM
@jainN If you are looking to route flowfiles that end in json versus those that are not, check out RouteOnAttribute with something similar to json => ${filename:endsWith('.json')}. You would use this after using your method of choice to list/fetch the files which would then provide a $filename for every flowfile. With this json property added to RouteOnAttribute you can drag the json route to a triggering flow, and send everything else (not json: unmatched) to a holding flow. NiFi Wait/Notify should be able to provide the trigger logic, but there are many other ways to do it with out wait/notify by using another datastore, map cache, etc. For example, your non json flow could simply write to a new location and finish. Then your json flow can process that new location some known amount of time later. The logic there is your use case ofc ourse, the point is to use RouteOnAttribute to split your flow. If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven
... View more
12-14-2020
07:23 AM
Hello @justenji Thanks for the detailed explanation, this is also what I have tested on my side and in my opinion this does the job being "record oriented" 🙂 Have a nice day
... View more