Member since
04-11-2016
471
Posts
325
Kudos Received
118
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2070 | 03-09-2018 05:31 PM | |
2631 | 03-07-2018 09:45 AM | |
2529 | 03-07-2018 09:31 AM | |
4388 | 03-03-2018 01:37 PM | |
2468 | 10-17-2017 02:15 PM |
09-16-2021
02:40 AM
1 Kudo
With the release of CDP 7.2.11, it is now super easy to deploy your custom components on your Flow Management DataHub clusters by dropping your components in a bucket of your cloud provider. Until now, when building custom components in NiFi, you had to SSH to all of your NiFi nodes to deploy your components and make them available to use in your flow definitions. This was adding an operational overhead and was also causing issues when scaling up clusters. From now on, it's easy to configure your NiFi clusters to automatically fetch custom components from an external location in the object store of the cloud provider where NiFi is running. All of your nodes will fetch the components after you dropped them in the configured location. You can find more information in the documentation.
... View more
09-16-2021
02:31 AM
1 Kudo
With the release of CDP 7.2.11, you now have the possibility to scale up and down both your light duty and heavy duty Flow Management clusters on all cloud providers. You can find more information in the documentation.
... View more
02-09-2021
10:41 AM
Here is a cool NiFi websocket app https://www.datainmotion.dev/2020/12/ingesting-websocket-data-for-live-stock.html Hosting web apps in NiFi https://www.datainmotion.dev/2020/11/flank-smart-weather-websocket.html https://www.datainmotion.dev/2020/12/ingesting-websocket-data-for-live-stock.html
... View more
01-28-2021
03:26 AM
1 Kudo
Hi @pvillard, I have some doubts about what kind of keytab file to create. I don't know much about Kerberos and do not find much information about the kind of Keytab file I should use to connect my Nifi Cluster to a kerberized Cloudera Cluster. Can you help me?
... View more
09-21-2020
05:53 AM
1 Kudo
@dvmishra It always best to start a new thread/question rather than adding a new question to an existing thread that already has an excepted answer. As far as being able to reset the offset for a specific consumer group from within NiFi itself, this is not something that can be done via the ConsumeKafka processors. The offset is not stored by NiFi. Offsets for each consumer group are stored in Kafka. Would not make make much sense to build such an option in to a NiFi processor if it was possible. Every time the processor executes it would reset in that case which is probably not the desired outcome. There are numerous threads online around reseting the offset in Kafka you may want to explore. Here are a couple: https://gist.github.com/marwei/cd40657c481f94ebe273ecc16601674b https://gist.github.com/mduhan/0e0a4b08694f50d8a646d2adf02542fc If you can figure out how to accomplish reset via a custom script of external command, NiFi does offer several script execution and command line execution processors. You may be able to use these processors to execute your script to rest the offset in Kafka. Aside from above, you can change the "group id" (new consumer group) and change the "offset reset" to "earliest". Then restart processor to start consuming topic form beginning again as a different consumer group. Hope this helps, Matt
... View more
03-31-2020
07:16 AM
@pvillard How does this work exactly? Im having issues segmenting large files as well. When i split them do i do it multiple times or just once and then I can recombine them successively. Thanks for you help!
... View more
02-17-2020
03:18 AM
Hi, can you explain how did you solve this problem?
... View more
01-31-2020
11:10 AM
Try using SELECTHIVEQL processor for your hive connection and Hive query instead of ExecuteSQL or QueryDatabase
... View more
01-28-2020
08:14 PM
Hi @pvillard, Would you happen to have a sample template for this process? Thanks Tim
... View more
12-23-2019
09:55 PM
@MattWho , Thank you for the details. I can understand your point that same file cannot be accessed across cluster nodes. But, I am using Nifi is single node (without cluster) and was thinking that this should work. Yes, i do use "Update Attribute" with the below file name conventions. This generates separate flow file for every message. I am trying to have this as one single file per node. ${filename:replace(${filename},"FileName_"):append(${now():format("yyyy-MM-dd-HH-mm-ss")})}.Json Thank you
... View more