Member since
07-30-2019
2757
Posts
1363
Kudos Received
797
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
105 | 11-30-2023 07:00 AM | |
138 | 11-02-2023 10:20 AM | |
288 | 11-02-2023 09:54 AM | |
310 | 11-01-2023 06:17 AM | |
224 | 10-31-2023 06:30 AM |
11-02-2023
09:54 AM
@jai1gupta What makes you think your NiFi is not secured over https? You did not share your nifi.properties web properties. If you have set the following properties: nifi.web.https.host=<hostname>
nifi.web.https.port=<port> and have configured the NiFi keystore and truststore properties (which you did share), the your NiFi would have started at logged url being available over HTTPS://<hostname>:<port>/nifi My guess is your issue probably extends from the use of "localhost" instead of an actual resolvable hostname. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
11-02-2023
09:50 AM
@samrathal Apache NiFi has hardcoded return size to 100: https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/dao/impl/StandardConnectionDAO.java#L361 I am sure this was originally done for performance and NiFi JVM heap usage reasons. The first 100 returned should be the oldest 100 in queue (keeping in mind that a connection will also show count of FlowFiles pending processing by downstream processor and count of those currently allocated to a downstream component process. The listing only returns those pending FlowFiles and not those already owned by downstream component).What is the use case for needing to list more? Ideally what is found in a queue should be changing rapidly, so expectation is that each listing request would be different. Listing a queue does not stop NiFi processing. The intent is not for NiFi to ever hold FlowFiles in any connection. So using API to poll connection for FlowFile listings seems odd to me. What is returned by that listing could be inaccurate milliseconds later. Also be careful with your API requests. When a listing is performed through the browser three different request are made. 1. First listing-request is made and replicated to all nodes to get result sets. 2. Return from step 1 request gives the ID for the generated listing request being held in heap memory. That ID is used to fetch the results in that specific listing ID 3. A DELETE request is made to remove the listing with that ID from NiFi. *** When using API, If steps 1 and 2 are all that are being executed, the various listing request(s) will stay in heap memory. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
11-01-2023
06:17 AM
@Knowledgeknow Does your NiFi Keystore meet the following requirements? MUST contain only one PriavteKey Entry PrivateKey Entry must include ExtenedKeyUsage(EKU) for both clientAuth and serverAuth. PrivateKey Entry DistinquishedName (DN) does not contain wildcards PrivateKey Entry contains one of more SubjectAlternativeName (SAN) entries that includes the hostname and IP for the NiFi server on which it will be used. 2. Sounds like you have configured nifi.properties: nifi.remote.input.host=0.0.0.0 While this will work for a http setup, it will cause issue with the TLS handshake. Instead this should set the hostname of the NiFi server and the hostname must be a SAN entry. If you are trying to have your NIFi bind to multiple Network Interface Cards (NICs) then you should use the "nifi.web.https.network.interface*" properties in web properties section of the nifi.properties file. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
10-31-2023
06:30 AM
@plapla The DistributedMapCacheClientService is used to connect to the DistributedMapCacheServer. You have not shared how you have the DistributedMapCacheServer controller service configured. The DistributedMapCacheServer configuration determines whether the DistributedMapCacheClientService needs tobe configured with a StandardRestrictedSSLContext Service. Essentially if you configure a StandardRestrictedSSLContext service on the DistributedMapCacheServer, then it must also be configured on the DistributedMapCacheClientService. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
10-31-2023
06:19 AM
@jai1gupta The exception: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target This indicates a trust chain issue in your TLS exchange with accounts.google.com. A complete trust chain requires all know public certs between certificate that signed for accounts.google.com --> intermediate CA(s) --> rootCA (owner and issuer same DN). Your configuration shows that the oidc authentication client configuration is set to: nifi.security.user.oidc.truststore.strategy=JDK HTTPS Certificate Trust Store Strategy defines the source of certificate authorities that NiFi uses when communicating with the OpenID Connect Provider. The value of JDK uses the Java platform default configuration stored in cacerts under the Java Home directory. The value of NIFI enables using the trust store configured in the nifi.security.truststore property. The default value is JDK This means that the Java version you have installed that NiFi is using is missing some trustedCertEntries from the trust chain for accounts.google.com. Google makes all its public root and intermediate certificates available for download here: https://pki.goog/repository/#:~:text=Download%20CA%20certificates You'll want to download all these (pem files) and add any that are missing from your Java's cacerts truststore file. While you can use the following openssl command to get all the public certs in the chain, you may find at times you get redirected to a different accounts.google.com server with a different trust chain. So I recommend downloading all instead of just those returned by the openssl command: openssl s_client -connect accounts.google.com:443 -showcerts Restart your NiFi and this trust issue should be gone. I am confused why your google console setup is using http instead of https urls for your NiFi? NiFi will not support authentication and authorization unless it is secured over https. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
10-30-2023
10:52 AM
@Madhav_VD Apache NiFi is data agnostic which makes it possible to ingest data of any format. It does this by wrapping the bytes of data in to a NiFi FlowFile. The only thing time the content (data bytes) are read from the FlowFile would be by a NiFi processor component that would need to do so. And only a processor specifically designed to handle the specific content type of the data would be able to do anything with the FlowFile content. That being said, NiFi had no Content conversion processor capable of reading FFmpeg content format and writing out TIFF content format. So the question here is how woudl you accomplish the format conversion outside of NiFi? Perhaps this thread would help you there: https://superuser.com/questions/881783/convert-from-avi-to-uncompressed-tiff-using-ffmpeg NiFi in this case could still be used to automate the ingestion and conversion of your FFmpeg files by utilizing perhaps one the NiFi Scripting processors like ExecuteGroovyScript or ExecuteScript (where you can create a custom script that has some dependency on client libraries capable of doing the conversion) or maybe the ExecuteStreamCommand or ExecuteProcess processor that could call a local command line command that you pass the FlowFile content to and returns a content stream with the new Tiff format? If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
10-30-2023
06:27 AM
@jai1gupta Yes NiFi out-of-the-box in the latest releases is secured to run over HTTPS. However, that out of the box default setup is very basic (non production setup) to allow users to work within a secured NiFi. Its default setup utilizes the Single-User-Provider authentication login provider (configured in login-identity-providers.xml) and single-user-authorizer configured in the authorizers.xml. This out-of the-box provider does not support multi-user capability. For multi-user production ready deployments with ability to define very granular access controls, you'll need to use a different authentication provider (Most commonly used is probably the ldap-provider) and different authorize (most commonly used is the StandardManagedAuthorizer configured to use the file-user-group-provider and file-access-policy-provider). NiFi does not offer any built in multi-user provider that allows you to create local to NiFi users with passwords. keeping above in mind, you are trying to use the OpenID authentication method. This still means you must be using a different Authorizer. The single-user-authorizer can only be used with the Single-user-provider. More detail around your setup, OpenID exceptions, etc is needed to help here. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
10-30-2023
06:12 AM
@AhmedParvez Of course digging in to why the connections are failing and addressing that is always going to be the best option rather then building work arounds like this. MY suggestions should be used a method to sustain operations while working on the correct solution. Perhaps that solutions means contributing to improvements to these MQTT processors or identifying deficiencies in these Processors and then raising detailed Apache NiFi jira tickets in the community to improve them.
... View more
10-30-2023
06:08 AM
@ranveera95 It has been +9 years since Apache NiFi was open sourced to the world. I would not say NiFi is rather new to large scale implementations. It has been used with very large scale deployments handling massive amounts of data. Often built using both Apache MiNiFi agents and large scale Apache NiFi clusters to facilitate the collection of data from small too large scale edged devices to central cluster data management and routing NiFi clusters.
... View more
10-30-2023
05:56 AM
@plapla I see that you have configured your DistributedMapCacheClient controller service with a StandardSSLContextService. You did not share your configuration for the needed DistributedMapCachServer. I must assume that the server was also configured with a StandardRestrictedSSLContextService. So few things to comment on here related to DistributedMapCache: 1. You configured the client with "localhost". Does localhost exist as a SAN entry in the PrivateKeyEntry found in the Keystore used in the DistributedMapCache server's configured keystore within the StandardRestrictedSSLContextService? 2. Does the keystore PrivateKeyEntry used in the StandardSSLContext services used by the DistributedMapCacheClient service include ClientAuth as an Extended Key Usage (EKU)? 3. Using "localhost means that all nodes will be using different map caches. The DistributedMapCacheServer starts up a separate Distributed Map Cache server on each host in the NiFi cluster. They do not communicate with one another to share cached data. In a cluster using this specific embedded cache server it is not common to use "localhost", but rather configured the hostname of one specific host in your NiFi cluster. 4. Once you had your DistributedMapCache server setup and enabled, Did you then start your DistributedMapCacheClient service and click the validation test? 5. When you started the DistributedMapCacheServer, did any of the NiFi cluster nodes throw any exceptions in the nifi-app.log related to possible port 4557 already in use? Without the verbose output of your Keystore and truststore files (all 4 ) used by both the DistributedMapCache server and client, it would be impossible for me to tell you if they are the issue here or not. But... I suspect you may have a MutualTLS exchange issue here. History: The DistributedMapCache (Server and Client) controller services have exists for almost as long as NiFi has existed. They were originally implemented before NiFi offered its zero master cluster capability. Very old versions o NiFi had a NiFi Cluster Manager (NCM) and then worker nodes that connected to that NCM. At that time the DistributedMapCacheServer would only be installed and run on that NCM. So DMCacheClient would always be configured to the one NCM host. Now with Zero Master clustering there is not NCM and thus the server start a map cache server on all nodes. It is for this reason that more and newer DistributedMapCache client options are available that offer HA which is not offered through the DistributedMapCache. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more