@MukaAddA  Sorry, writing such script is not a strong area for me.  I just happened to notice you were doing a session.create instead of a session.get.  You may get better help by raising a new question on how to create a script to be executed by the ExecuteScript processor to accomplish your use case and provide details on that use case.  I am sure there are others in the community that are good at writing such scripts. Matt ... View more

@mslnrd  This is likely caused by LDAP on 636 uses referrals that can your initial query can be referred to across the entire domain tree across multiple LDAP servers. So somewhere within that referral your issues arrises in the hostname verification. Switching to the global catalog port 3269 and there are no referrals.  I can't speak to the issues within your ldaps servers causing the issue within the referrals, but makes sense why switching to the secure global catalog port resolved your issue.   Hope this clarifies why the change in port resolved your issue. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@kothari  It is not Ranger's job to inform the client applications using Ranger what users belong to what group.  Each client application is responsible for determining which groups the user authenticated into that service belong to. The policies generated by Ranger are downloaded by the client applications.  Within that downloaded policy json will be a resource identifier(s), list if user identities authorized (read, write, and/or delete) , and list of group identities authorized (read, write, or delete) against each resource identifier.  So when client checks the downloaded policies from Ranger it is looking for the user identity being authorized and if client is aware of the group(s) that user belongs to, will also check authorization for that group identity.   so in your case, it i s most likely that your client service/application has not been configured with the same user and group association setup in your Ranger service.   If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@pashtet04  There is a significant jump in version from NiFi 1.11.4 to 1.23.0 As with any upgrade/migration, you should NOT be just copying core configuration files from old version to new version.  You should instead use the old configuration files to update the new configuration files as those new configuration files would have introduced new properties that would then be missing if you simply copied old to new.   NiFi 1.11.4 loaded a flow.xml.gz file on startup.  With release of NiFi 1.16, NiFi introduced a new flow storage format using flow.json.gz (which also introduced a new property in the nifi.properties file: "nifi.flow.configuration.json.file").  1.16 and later version will still load from the flow.xml.gz file if a flow.json.gz is not present and then generate the flow.json.gz file which will be used for all subsequent launches starting with 1.18+ version.   I am guessing your exception about flow configuration file [NULL] comes from missing configuration property in the nifi.properties you copied over.  This may have also resulted in a failure to convert the flow.xml.gz???  I'd recommend re-copying over your original flow.xml.gz, utilize old config files to populate new config files to avoid missing properties, start NiFi 1.23 first time using original sensitive props key and sensitive props algorithm so that the flow.json.gz gets generated.  Then use the nifi.sh commands to change the sensitive props key and sensitive key algorithm. If you don't know your original sensitive props key, you could use the Encrypt-config toolkit to change it as an alternative to the nifi.sh commands. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@mslnrd  TLS/SSL is nothing specific to NiFi or LDAP.  There are certain requirements in the TLS exchange specification. This applies to any application that would use TLS.  Based on your exception: [Root exception is javax.naming.CommunicationException: my.network.com:636 [Root exception is javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching my.network.com found.]] The issue happens during the TLS exchange.  NiFi's ldap-user-group-provider within the authorizers.xml file is attempting to execute during NiFi startup in order to sync users and groups from your target LDAPS server. At a very high basic level here, you have configured NiFi to connect to your LDAPS host "my.network.com".  NiFi is acting as the client in the TLS exchange and the LDAPS as the server side.  Your LDAPS is returning a serverAuth certificate that is missing a Subject Alternative Name (SAN) entry matching the hostname (my.network.com) the client is trying to connect with.  Within the TLS spec, the client is then rejecting this connection as insecure since it can not verify the authenticity of the server that is responding.  It assumes a man in the middle type issue where client is trying to establish a secure connection with server "my.network.com"; however, certificate returned does not indicate it belongs to that server via its list of SAN entries.  So the issue here is with the certificate being used by your LDAPS server missing the expected SAN entry.  NiFi does not provide an option to force allow an insecure connection. Bottom line is the NiFi Keystore needs to meet following requirements: 1. The Keystore can contain only one PriavteKeyEntry 2. The PrivateKeyEntry must have both clientAuth and ServerAuth Extended Key Usage (EKUs) 3. The PrivateKeyEnrty must have a SAN entry that matches the hostname of the server on which NiFi is installled. The NiFi Truststore needs to meet the following: 1. The truststore can contain one too many TrustedCert entries.  2. The complete trust chain for any server certificates NiFi will need to establish a TLS connection with will need to be present in the truststore.   When a certificate is created it has an owner and a signer/issuer.  The signer of the certificate might be self-signed (meaning owner DN and signer/issuer DN are the same), it might be an intermediate CA (Owner DN is different the signer/issuer DN), and a Root CA (also has matching owner DN and signer/issuer DN).    A complete trust chain means all trusted certificates (public certs) from client certificate to root CA need to be present in the truststore. This could encompass the server public cert, one or more intermediate CA public certs, and the CA root public cert.    Often, but not always you can get all the public certificates using openssl. openssl s_client -connect <target server hostname>:<port> -showcerts Above would initiate a TLS handshake with target server and will return within the response the public certs from the target server.  It may or may not have complete trustchain.  You can then load these public certificates into your truststore.   In your case however, it does not look like a trust chain issue at this point.  As i mentioned above, your current issue is that your serverAuth certificate used by your target LDAP is missing the TLS spec required SAN entry.   If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@sahil0915  What you are proposing would require you to ingest into NiFi all ~100 million records from DC2, hash that record, write all ~100 million hashes to a map cache like Redis or HBase (which you would also need to install somewhere) using DistributedMapCache processor, then ingest all 100 million records from DC1, hash those records and finally compare the hash of those 100 million record with the hashes you added to the Distributed map cache using DetectDuplicate.    Any records routed to non-duplicate would represent what is not in DC2.    Then you would have to flush your Distributed Map Cache and repeat process except this time writing the hashes from DC3 to the Distributed Map Cache. I suspect this is going to perform poorly.  You would have NiFi ingesting ~300 million records just to create hash for a one time comparison.       If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@sahil0915  I don't know that this is a good use case for NiFi.  NiFi at the most basic level is designed to automate the movement of data between systems.  In between the ingest of data and egress of data, NiFi provides a variety of components for route, enhancing, modify, etc that data. So trying to use NiFi to compare the data existing in multiple data centers is not a good fit.   If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more