Member since
01-27-2023
126
Posts
31
Kudos Received
23
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
46 | 05-31-2023 03:01 AM | |
143 | 05-22-2023 06:55 AM | |
146 | 05-15-2023 05:33 AM | |
285 | 05-10-2023 01:57 AM | |
97 | 05-09-2023 11:40 PM |
06-02-2023
03:45 PM
@Dracile, I do not think that SplitJson is the correct processor for you. What you are trying to achieve might be possible using some JOLT transformations. Unfortunately, I am not near a computer to test a correct transformation but I know that @SAMSAL has plenty of experience in using jolts and he might be able to further assist you.
... View more
05-31-2023
03:12 AM
@Fredi, I am not an expert when it comes to using ExecuteScript in NiFi, as I mostly go for ExecuteStreamCommand, but I really recommend your to have a look on the following two links (until somebody with far more experience provides you with an answer), as they explain everything you need to know when it comes to executing a Jython script in NiFi: See Part 2 for I/O on FlowFiles: https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-2/ta-p/249018 See Part 1 for FlowFile Creation: https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-1/ta-p/248922 If I were to look at your code and the examples provided by @mburgess, you are missing some important steps and that might be the cause of your error.
... View more
05-31-2023
03:01 AM
Well I am not expert in migrating from a version to another so my answer might not be good enough for you :(. Besides that, I had no time to read the release notes for 1.21.0 and I am not quite sure if anything changed in terms of config files. Assuming that you will keep the hostname and the port for each nifi node and you are using the embedded zookeeper, you should: 1. Stop the current NiFi instance. 2. Copy the authorizations.xml, authorizers.xml, bootstrap.conf, flow.json.gz, flow.xml.gz, logback.xml, login-identity-providers.xml, nifi.properties, stateless.properties, state-management.xml, users.xml and zookeeper.properties into the conf folder from within your new nifi instance. 3. Make sure that in nifi.properties, on your new instance, you are pointing to the same content repository, database repostiory (and all the other repositories) as in the previous instance --> assuming that you followed the best practices and had all those repositories moved on separate disks. Otherwise, make sure that you process all your data on your old instance and start fresh on your new instance. 4. Start the new NiFi Instance. If you are just interested in migrating just the flows from the canvas, you can add all your flows into a template and save them on your local machine. Afterwards, you can open your new nifi instance, upload your template and add it to your canvas.
... View more
05-30-2023
07:05 AM
1 Kudo
@ranaIrfan, I am not quite sure which version you are using but at least since 1.19, you have the ListGoogleDrive and FetchGoogleDrive Processors, which should be used in order to extract data out of your Google Drive Folder. FetchGoogleDrive --> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.19.0/org.apache.nifi.processors.gcp.drive.FetchGoogleDrive/index.html ListGoogleDrive --> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.19.0/org.apache.nifi.processors.gcp.drive.ListGoogleDrive/
... View more
05-25-2023
03:05 AM
As much as I would like to help, the provided error does not really help 😞 More than certain there is much more in your logs, either on your NiFi instance (see all the logs) either on your Registry. You should try and remember everything which has been performed since the last time you know it worked and try to revert those actions. Otherwise, based on the provided error message, you are out of luck 😞
... View more
05-25-2023
12:59 AM
everybody is entitled to an opinion but may I ask why are you saying this? 🙂 As you are on a NiFi post, I assume that you are referring to the Cloudera NiFi documentation? I find it very helpful, especially combined with the NiFi's original documentation. It even has some additional thins compared to the original documentation. No matter the feedback, positive or negative, it is good when you know what to do with it. In your case, if you would provide a better and more structured feedback, maybe somebody from Cloudera would understand your point of view and he/she could modify the documentation 🙂
... View more
05-23-2023
11:35 PM
@Adhitya, I am not quite sure what you mean with response topic, but as far as I can tell, you need to use ConsumeMQTT in order to "ingest" data out of the MQTT brokers/topics. If you want to publish within the brokers/topics, you will have to use PublishMQTT. In both cases, you will have to choose the MQTT Specification Version, where you will have to select something from the following options: v3 Auto, v3.1.1, v3.1.0 and v5.0. See: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-mqtt-nar/1.21.0/org.apache.nifi.processors.mqtt.ConsumeMQTT/
... View more
05-23-2023
01:21 AM
@mks27, To be really honest, when it comes to login and certificates, I am in no position to provide an input, as I never got to fully understand how they work 😞 While you are waiting for a better answer, from somebody with far more experience and knowledge as me, I would try the following: - Assuming that you configured LDAP authentication, I assume that you have the User and Policies Menu in your NiFi Menu (top Right). - Now, based on your error, I see that you are using the user mohit.kumar and you have no privileges to do anything. - What I would suggest is to login with the user which was provided as Initial Admin Identity and provide your user (mohit.kumar) with all the necessary roles to perform the action you are trying to perform. See: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#initial-admin-identity Have a look here as well: https://pierrevillard.com/2017/01/24/integration-of-nifi-with-ldap/comment-page-1/
... View more
05-22-2023
06:55 AM
In this case, you will need some extra steps. You could try something like: You have an ExecuteSQLRecord or ExecuteSQL, configured with the Database Connection Pooling Service and the Record Writer (if using ExecuteSQLRecord). Next, I would link the success queue to an ExtractText Processor. In this Processor I would add a new property called number_of_rows having the value " .* ". This will extract the value returned by count(*) and save it as an attribute in your FlowFile. Once you did this, you can link the success queue to an RouteOnAttribute Processor. Here, you will define a new property named bigger_as_zero and use the NiFi's EL to validate if the attribute is bigger than zero or not. If bigger, you will send that message to the processor in which you have linked this queue. Otherwise, meaning that the value is zero, you can route the unmatched queue to another ExecuteSQL/ExecuteSQLRecord. You could also apply NiFi's EL to check if the value is exactly zero, if that is what you are looking for and route the file accordingly. NiFi's EL: https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html Everything mentioned above has to be done, in addition to what @SAMSAL already said, with the fact that you need to modify you running schedule from 0 seconds to a greater value. 0 means that you will constantly execute this SQL Record. Here you will have to modify the value based on your project's requirements.
... View more
05-15-2023
05:33 AM
1 Kudo
@DrManu, I do not think that you will find a processor in NiFi which will extract only the folder name out of your location 😞 You will either have to write your own processor or use a combination of several others, already part of NiFi. How I would honestly try the mentioned scenario: - An ExecuteStreamCommand Processor in which you have defined a custom made script which will read your folder structure and generate a JSON File, where each row is basically a complete path to a specific Folder. - Afterwards, you could use an SplitJson to generate a single FlowFile for each Folder and send it down your stream for further processing.
... View more