Member since
07-30-2019
2878
Posts
1419
Kudos Received
832
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
78 | 03-27-2024 12:49 PM | |
45 | 03-27-2024 12:30 PM | |
82 | 03-20-2024 11:55 AM | |
280 | 03-15-2024 06:57 AM | |
255 | 03-07-2024 08:25 AM |
07-14-2022
06:59 AM
@Meeran The users.xml is created/managed by the File-User-Group-Provider in the Authorizers.xml file. The authorizations.xml is created/managed by the File-Access-Policy-provider in the Authoirizers.xml file. Neither of these providers support using a database for persisting the users, groups, and authorizations. For more information on the authorization providers, follow the below link: https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#authorization NiFi-Registry supports using an embedded H2 DB (default) or an external DB (Postgres or MySQL) for storing knowledge of which buckets exist, which versioned items belong to which buckets, as well as the version history for each item. The actual version controlled flows are not stored in the DB https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#metadata-database The actual Version Controlled dataflow data is stored via the configured Persistence provider. The default provider is FileSystemFlowPersistenceProvider which writes this data to a directory locally on the NiFi-Registry host. The other available option is the GitFlowPersistenceProvider which commits this data to a remote Git repository when configured correctly. For more detail follow below link: https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#persistence-providers If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
07-14-2022
06:44 AM
@Hashem1980 Neither the ConsumeIMAP or ConsumePOP3 NiFi processors support Oauth2 for authentication. NiFi does have an StandardOauth2AccessTokenProvider controller service. I would recommend opening an Apache NiFi Jira asking to have Oauth2 via this existing controller service integrated in to these processors. As an open source project contribution are always welcome. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
07-14-2022
06:31 AM
1 Kudo
@shashikumar NiFi only requires the the Java Runtime Environment (JRE). Latest versions of NiFi support both Java 8 or Java 11: system_requirements The JDK contains, in addition to the JRE, additional development tools like compilers, debuggers, etc. These may be useful to you depending on whether or not you intend to compile java code within your dataflow (not a requirement of any components like processors, controller services, reporting tasks, etc included with NiFi) or create your own custom NiFi components. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
07-14-2022
06:23 AM
@sayak17 Agree with @ckumar's response. It would be difficult to provide more detail without understanding the actual structure needed for the follow-on rest-api calls. What info is expected in headers versus raw data. Typically the access token is sent in a header and is often needed in all subsequent rest-api calls. Do you need to create unique FlowFile content to send as data? The ExtractText can be used to extract the token from the response written to the FlowFiles content and added as an attribute on the FlowFile before next invokeHTTP processor. Since I do not know much about Sharepoint or the format of the responses received, I can only give an example based on a returned token from NiFi's rest-api: The entire response body would be placed in a FlowFile attribute named "token". In the case of the NiFi token, I must then build a proper "Authorization" header with that token before being able to make additional rest-api calls using it. To do that, you can use an UpdateAttribute processor: So here I added "Bearer " to the token and write it to a new FlowFile attribute "Authorization"; although, you could also simply update the existing "token" attribute instead. Then in the next InvokeHTTP processor you can pass this "Authorization" FlowFile attribute value in a header by adding a dynamic property: Beyond this, you can do similar with your follow on responses assuming each step simply requires you to use the response values you extracted as headers or custom rest-api remote URLs. If you found any of the response received help you address your query, please take a moment to login and then click "Accept as Solution" on every response that helped. Hopefully this helps you reach your end goal here, Matt
... View more
07-11-2022
09:41 AM
1 Kudo
@Chakkara Updating a parameter context is a multi command process. 1. Fist you must GET the current parameter context configuration: curl -k -X GET https://<hostname>:<port>/nifi-api/parameter-contexts/<parameter context UUID> --compressed - This will return a potentially large json which will include the current revision, all the parameter contexts from that parameter context and all the referencing components. Assuming your NIFi is secured, your'll also need to include a method of authentication (client certificate, bearer token) in the above request. 2. Next you will POST your update-request using detail from above json response in step 1. curl -k 'https://<hostname>:<port>/nifi-api/parameter-contexts/<PARAMETR Context UUID>/update-requests' --data-raw $'{"revision":{"version":<num from step 1>},"id":"<PARAMETR Context UUID>","uri":"https://<hostname>:<port>/nifi-api/parameter-contexts/<PARAMETR Context UUID>","permissions":{"canRead":true,"canWrite":true},"component":{"name":"param-test","description":"","parameters":[{"parameter":{"name":"<Parameter name>","sensitive":false,"value":"<new parameter value>","description":""}}],"boundProcessGroups":[{"revision":{"clientId":"edcad62f-0181-1000-2fc6-de0c25282a3b","version":<num from step 1>},"id":"70b33f0a-e362-1c0b-8147-8fada18d56bd","position":{"x":136,"y":-2456},"permissions":{"canRead":true,"canWrite":true},"component":{"id":"70b33f0a-e362-1c0b-8147-8fada18d56bd","parentGroupId":"7930f131-017b-1000-0000-00007ece6b9d","position":{"x":136,"y":-2456},"name":"Param2","comments":"","variables":{},"parameterContext":{"id":"<PARAMETR Context UUID>","permissions":{"canRead":true,"canWrite":true},"component":{"id":"<PARAMETR Context UUID>","name":"param-test"}},"flowfileConcurrency":"UNBOUNDED","flowfileOutboundPolicy":"STREAM_WHEN_AVAILABLE","runningCount":2,"stoppedCount":0,"invalidCount":0,"disabledCount":0,"activeRemotePortCount":0,"inactiveRemotePortCount":0,"upToDateCount":0,"locallyModifiedCount":0,"staleCount":0,"locallyModifiedAndStaleCount":0,"syncFailureCount":0,"localInputPortCount":0,"localOutputPortCount":0,"publicInputPortCount":0,"publicOutputPortCount":0,"inputPortCount":0,"outputPortCount":0},"runningCount":2,"stoppedCount":0,"invalidCount":0,"disabledCount":0,"activeRemotePortCount":0,"inactiveRemotePortCount":0,"upToDateCount":0,"locallyModifiedCount":0,"staleCount":0,"locallyModifiedAndStaleCount":0,"syncFailureCount":0,"localInputPortCount":0,"localOutputPortCount":0,"publicInputPortCount":0,"publicOutputPortCount":0,"parameterContext":{"id":"994938a5-0180-1000-0000-00007cbba46b","permissions":{"canRead":true,"canWrite":true},"component":{"id":"<PARAMETR Context UUID>","name":"param-test"}},"inputPortCount":0,"outputPortCount":0},{"revision":{"clientId":"edcad62f-0181-1000-2fc6-de0c25282a3b","version":2},"id":"edcb3ff7-0181-1000-0000-00004abfd1ed","position":{"x":-323.1760214048029,"y":-2454.121877860687},"permissions":{"canRead":true,"canWrite":true},"component":{"id":"edcb3ff7-0181-1000-0000-00004abfd1ed","parentGroupId":"7930f131-017b-1000-0000-00007ece6b9d","position":{"x":-323.1760214048029,"y":-2454.121877860687},"name":"param1","comments":"","variables":{},"parameterContext":{"id":"994938a5-0180-1000-0000-00007cbba46b","permissions":{"canRead":true,"canWrite":true},"component":{"id":"<PARAMETR Context UUID>","name":"param-test"}},"flowfileConcurrency":"UNBOUNDED","flowfileOutboundPolicy":"STREAM_WHEN_AVAILABLE","runningCount":2,"stoppedCount":0,"invalidCount":0,"disabledCount":0,"activeRemotePortCount":0,"inactiveRemotePortCount":0,"upToDateCount":0,"locallyModifiedCount":0,"staleCount":0,"locallyModifiedAndStaleCount":0,"syncFailureCount":0,"localInputPortCount":0,"localOutputPortCount":0,"publicInputPortCount":0,"publicOutputPortCount":0,"inputPortCount":0,"outputPortCount":0},"runningCount":2,"stoppedCount":0,"invalidCount":0,"disabledCount":0,"activeRemotePortCount":0,"inactiveRemotePortCount":0,"upToDateCount":0,"locallyModifiedCount":0,"staleCount":0,"locallyModifiedAndStaleCount":0,"syncFailureCount":0,"localInputPortCount":0,"localOutputPortCount":0,"publicInputPortCount":0,"publicOutputPortCount":0,"parameterContext":{"id":"994938a5-0180-1000-0000-00007cbba46b","permissions":{"canRead":true,"canWrite":true},"component":{"id":"<PARAMETR Context UUID>","name":"param-test"}},"inputPortCount":0,"outputPortCount":0},{"revision":{"version":0},"id":"7930f131-017b-1000-0000-00007ece6b9d","position":{"x":328,"y":400},"permissions":{"canRead":true,"canWrite":true},"component":{"id":"7930f131-017b-1000-0000-00007ece6b9d","versionedComponentId":"136b68f9-1621-366e-8e8a-1db651ad8808","parentGroupId":"e377355c-017a-1000-2658-01d1a778332f","position":{"x":328,"y":400},"name":"Matt\'s PG","comments":"","variables":{},"versionControlInformation":{"groupId":"7930f131-017b-1000-0000-00007ece6b9d","registryId":"e97b2503-017a-1000-0000-00000c0a8abc","registryName":"nifi-sme-75.novalocal","bucketId":"19f7633f-5c1d-4d80-9d0d-d95c7e4885f8","bucketName":"new-bucket","flowId":"6add76b8-e56b-4d36-99e2-8aba4f5a9c33","flowName":"Matts-PG","flowDescription":"","version":2,"state":"LOCALLY_MODIFIED","stateExplanation":"Local changes have been made"},"parameterContext":{"id":"<PARAMETR Context UUID>","permissions":{"canRead":true,"canWrite":true},"component":{"id":"994938a5-0180-1000-0000-00007cbba46b","name":"param-test"}},"flowfileConcurrency":"UNBOUNDED","flowfileOutboundPolicy":"STREAM_WHEN_AVAILABLE","runningCount":39,"stoppedCount":109,"invalidCount":74,"disabledCount":7,"activeRemotePortCount":0,"inactiveRemotePortCount":2,"upToDateCount":0,"locallyModifiedCount":0,"staleCount":0,"locallyModifiedAndStaleCount":0,"syncFailureCount":0,"localInputPortCount":0,"localOutputPortCount":0,"publicInputPortCount":0,"publicOutputPortCount":0,"inputPortCount":0,"outputPortCount":0},"runningCount":39,"stoppedCount":109,"invalidCount":74,"disabledCount":7,"activeRemotePortCount":0,"inactiveRemotePortCount":2,"versionedFlowState":"LOCALLY_MODIFIED","upToDateCount":0,"locallyModifiedCount":0,"staleCount":0,"locallyModifiedAndStaleCount":0,"syncFailureCount":0,"localInputPortCount":0,"localOutputPortCount":0,"publicInputPortCount":0,"publicOutputPortCount":0,"parameterContext":{"id":"<PARAMETR Context UUID>","permissions":{"canRead":true,"canWrite":true},"component":{"id":"994938a5-0180-1000-0000-00007cbba46b","name":"param-test"}},"inputPortCount":0,"outputPortCount":0},{"revision":{"clientId":"edcad62f-0181-1000-2fc6-de0c25282a3b","version":2},"id":"373a39ad-0443-10ec-9e85-69445fefe75e","position":{"x":-320,"y":-2216},"permissions":{"canRead":true,"canWrite":true},"component":{"id":"373a39ad-0443-10ec-9e85-69445fefe75e","parentGroupId":"7930f131-017b-1000-0000-00007ece6b9d","position":{"x":-320,"y":-2216},"name":"param3","comments":"","variables":{},"parameterContext":{"id":"994938a5-0180-1000-0000-00007cbba46b","permissions":{"canRead":true,"canWrite":true},"component":{"id":"<PARAMETR Context UUID>","name":"param-test"}},"flowfileConcurrency":"UNBOUNDED","flowfileOutboundPolicy":"STREAM_WHEN_AVAILABLE","runningCount":2,"stoppedCount":0,"invalidCount":0,"disabledCount":0,"activeRemotePortCount":0,"inactiveRemotePortCount":0,"upToDateCount":0,"locallyModifiedCount":0,"staleCount":0,"locallyModifiedAndStaleCount":0,"syncFailureCount":0,"localInputPortCount":0,"localOutputPortCount":0,"publicInputPortCount":0,"publicOutputPortCount":0,"inputPortCount":0,"outputPortCount":0},"runningCount":2,"stoppedCount":0,"invalidCount":0,"disabledCount":0,"activeRemotePortCount":0,"inactiveRemotePortCount":0,"upToDateCount":0,"locallyModifiedCount":0,"staleCount":0,"locallyModifiedAndStaleCount":0,"syncFailureCount":0,"localInputPortCount":0,"localOutputPortCount":0,"publicInputPortCount":0,"publicOutputPortCount":0,"parameterContext":{"id":"994938a5-0180-1000-0000-00007cbba46b","permissions":{"canRead":true,"canWrite":true},"component":{"id":"<PARAMETR Context UUID>","name":"param-test"}},"inputPortCount":0,"outputPortCount":0}],"id":"<PARAMETR Context UUID>"}}' --compressed - This will return the uri for this new update request. A background process will take care of stopping changing, validating and starting all referenced components for the update parameter context. Assuming your NIFi is secured, your'll also need to include a method of authentication (client certificate, bearer token) in the above request. 3. Third you can execute a GET using the uri returned in step 2 to check progress of the request. curl 'https://<hostname>:<port>/nifi-api/parameter-contexts/<PARAMETER CONTEXT UUID>/update-requests/<UPDATE REQUEST UUID form step 2 response>' -k --compressed - This will return a json that will include within it the "percentCompleted". You'll need to check this endpoint until it reports 100% complete. Assuming your NIFi is secured, your'll also need to include a method of authentication (client certificate, bearer token) in the above request. 4. Final step is a DELETE to remove the update request that is now completed. Failing to do this leaves this request in NiFi Heap memory until a NiFi restart. curl -X DELETE 'https://<hostname>:<port>/nifi-api/parameter-contexts/<PARAMETER CONTEXT UUID>/update-requests/<UPDATE_REQUEST UUID from step 2 response>' -k --compressed - Assuming your NIFi is secured, your'll also need to include a method of authentication (client certificate, bearer token) in the above request. An easy way to help understand the --data-raw input needed for your specific update-request is to use the developer tolls found in most browsers to inspect/capture that actual rest-api calls made when you execute the actual action via the NiFi UI. The browser developer tools will also allow you to right click on the triggering action and copy as curl. You can then paste that action as a curl command in a terminal window and execute it yourself. Reference: https://nifi.apache.org/docs/nifi-docs/rest-api/index.html - specifically looking at the section for "Parameter Contexts" If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
07-08-2022
12:36 PM
@Meeran Your issue in both exceptions is with the mutual TLS handshake and not really a configuration issue in NiFi or NiFi-Registry. And may not even be an issue with your LBs. Your first exception shared is telling you that the Registry-Client hostname configured in your NiFi was not found as a SAN in the serverAuth certificate returned by the target server. In order for the mutual TLS handshake to succeed there would need to be a matching hostname SAN entry in the certificate. The Second exception you shared is telling you that the truststore.jks does not contain the required complete trust chain needed to trust the private certificates DN. The privateKey used by the Server and Client will have an owner DN and Issuer DN. The Issuer DN is the authority that signed that certificate and must be found in the truststore. Some issuers are intermediate authorities meaning that also have an owner DN signed by another issuer authority. This is a trust chain. So your truststore would also need to to contain the trustedCertEntry for that intermediate's issuer as well. Once you reach the issuers certificate where both the owner an issuer DN match, you have reached the root authority of that trust chain. This is all part of the TLS spec and not specific to anything NiFi is enforcing itself. you can use openssl in most but not all to get the public certs for the authorities: openssl s_client -connect <hostname>:<port> -showcerts If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
07-08-2022
12:20 PM
1 Kudo
@rafy The PutFile processor is used to write the content of FlowFiles to the local file system. It can not be used to write files tp a remote system. The reason you are seeing no error is because the PutFile processor is creating that directory path using the ip address as a folder name on your local machine where NiFi is running and writing your FlowFile's content there. If you want to write a remote server, you would need to use a processor like the putSFTP processor assuming you have an SFTP server running on that remote server that your can successfully connect with. Another option would be to install NiFi on both servers and use NiFi's Site-to-SIte capability to send FlowFiles from one NiFi to another and then on the receiving NiFi use the putFile processor to write your FlowFile content to disk. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
07-08-2022
11:47 AM
@Luwi An "active content claim" would be any content claim where a FlowFile exist still referencing bytes of content in that claim. A NiFi content claim file can contain the content for 1 too many FlowFiles. So all it takes is one small FlowFile still queued in some connection anywhere on your NiFi canvas to prevent a content claim from being eligible to be moved to archive. This is why the total reported content queued on yoru canvas will never match the disk usage in your content_repository. This article is useful in understanding this process more: https://community.cloudera.com/t5/Community-Articles/Understanding-how-NiFi-s-Content-Repository-Archiving-works/ta-p/249418 Thank you, Matt
... View more
07-08-2022
11:39 AM
@mbraunerde Assuming you do not want to lose the original content of all these files, so you have numerous challenges here. 1. You don't have a known number of files. So when collecting a single list of all Filenames, how do you know that all have been received in the queue? Since NiFi is designed as a data in motion service. 2. Preserving the original FlowFiles content. Sounds like you are maybe trying to produce a new FlowFile with content containing just the filenames of all the files received by your NiFi dataflow, but still allow original FlowFiles along with the original content still get processed separately? Overcoming these challenges a depends on some unknowns at this point. 1. How is this data ingested to your NiFi? Is it a constant stream of data? Is it burst of data once a day? If you can control the ingest and there is a known gap between stream of data, you maybe over to overcome challenge 1 above. 2. Overcoming challenge 2 can be done via cloning of the FlowFiles. Every NiFi processor has outbound relationships that can be added to NiFi connections or auto-terminated within a processor's configuration. So some point in your flow you would simply add the "success" relationship of a processor to two different connections (essentially one connection will have original FlowFile and other would have the clone. Down one dataflow path you continue to handle the FlowFiles with their original content. The other dataflow path you can use a ReplaceText processor to do a literal replace replacement strategy and set the Replacement Value to ${filename}. What this will do is replace all the content of that FlowFile with just the filename of that FlowFile. Then as @SAMSAL suggested use a MergeContent processor to merge all your FlowFiles so you have one new FlowFile containing all the Filenames. Since you are dealing with an unknown number of files, you could configure the MergeContent with an arbitrarily large Minimum Number of Entries (some value larger than you would expect to receive in a single batch. You would also need to set Maximum Number of Entries to a value equal to or larger then the min. This will cause FlowFile to continue to get added to a bin for merge without actually being merged. Then you set Max Bin Age to a value high enough that all batch FlowFile would have been processed. Max Bin Age serve as a method to force a bin to merge even if min values have not been reached after a configured amount of time. So you are building in a delay in this flow to allow for the data in motion nature of NiFi. Finally sent that merged FlowFile to your putEmail processor. Or maybe we are not understanding the use case completely. Are you looking for what is actually in a given queue and positional order? Keep in mind that NiFi is a data in motion service meaning that it should not be used to hold data in queues. Which in turn means that the queued FlowFiles in a connection are typically constantly changing. But if this is what you are looking for, you could use the invokeHTTP processor to obtain the listing of FlowFiles in a queue. This would require a series of rest-api calls. First invokeHTTP would make a request via a POST get generate a queue listing result set for a connection from all nodes. The response to that post would be the url of the result set which you would use in a second invokeHTTP to GET that result set. Finally you would need a third invokeHTTP to DELETE the result set so it is not left hanging around in NiFi heap memory. Even then you have a large json which contains a lot more than just position, filename, and NiFi cluster host names. So you would use additional processor to parse the desired information from that json record. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
07-07-2022
08:51 AM
2 Kudos
@sayak17 If you are simply looking to GET from a REST API endpoint and take the response and write to a local file on the server where your NiFi service is running, you'll want to use the InvokeHTTP processor and feed that to your putFile processor. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more