Member since
02-24-2018
20
Posts
10
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2534 | 04-03-2018 07:29 PM |
01-25-2024
07:01 AM
if Google sends you here, most probably you're looking for https://community.cloudera.com/t5/Support-Questions/PUTSQL-which-cannot-be-converted-to-a-timestamp/m-p/292827
... View more
10-21-2019
04:22 AM
Hi, Did adding Nifi hostnames to the load balancer certificate's SAN help?
... View more
03-29-2018
08:50 PM
1 Kudo
When NiFi refactored it's security model between 0.x and 1.x lines, templates were moved to be associated with the process group where you uploaded the template. This was done so that template was protected by the same security policies as the process group where it was uploaded. Unfortunately the "View Templates" capability is still from the global menu, but should really be from the context palette on the left based on the process group you are in.
... View more
04-01-2018
06:57 PM
It did indeed help. I found the following StackOverflow answer to help too: https://stackoverflow.com/questions/49467969/python-script-using-executestreamcommand Especially: Command Arguments: any flags or args, delimited by ; (i.e. /path/to/my_script.py)
Command Path: /path/to/python3 Note the Command Path that you did not specify in the processor. This also allows the use of for example a predefined Anaconda environment. Anyhow, thank you for the help!
... View more
03-19-2018
07:58 AM
Hey Vincent, Your script should be the responsible of building those events 4KB size. You could build a buffer 4KB size and add your event inside of it. Definitely this is far from optimal as: (1) if the event is small you're wasting space and (2) you can't have events bigger than 4KB as it would need a more complex logic (split before sending and joining afterwards). I said that batch generation (managed by your Python script) and batch ingestion (handled by ExecuteProcess processor in NiFi) synchronisation is tricky, because each one belongs to different processes with their own internal timers triggering those time batches... processes synchronisation needs additional artefacts that will make things even more complicated. Definitely, don't go this way... I just mention it as an argument to justify applying your "workaround" which, to me, it's the way to go. Regarding the size of data, NiFi is optimised to handle any kind of data (varying from small to large pieces of data) as outlined in the documentation. I highly recommend you to go over our documentation to understand how data is managed internally, and you'll see there are smart design decisions made when it comes to dealing with content and the metadata linked to it 🙂 Kafka makes sense when you want to avoid overkilling a destination system which doesn't process data as fast as it is generated (for example). If you think Postgres won't cope with ingestion pace (probably you've already realised that NiFi is very performant), then add Kafka in the equation. Any way, keep in mind that queues and back pressure capabilities in NiFi could be just enough for your project. Best 😉
... View more
02-25-2018
09:31 PM
Were you ever able to fix this? Currently trying to do the same thing using Nifi v1.5.
... View more
03-13-2018
10:13 PM
1 Kudo
Is there a way that one can inspect the FlowFile once it has already passed through the queues? I'm trying your recommended method of inspecting the attributes of a FlowFile but the queue finished too quick and thus end up with "The queue has no FlowFiles" instead of the output that you have displayed.
... View more