@Shelton Thank you for the detailed answer, much appreciated ! ... View more

@MattWho @mburgess Hi! Do you have any insights here? Thanks! ... View more

@sydney-  The SSL handshake error you're encountering is a common issue when connecting NiFi instances to NiFi Registry in secure environments it indicates that your NiFi instances cannot verify the SSL certificate presented by the NiFi Registry server. javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider. certpath.SunCertPathBuilder Exception: unable to find valid certification path to requested target Based on your description, there are several areas to address. The certificate used by NiFi Registry is self-signed or not issued by a trusted Certificate Authority (CA) The certificate chain is incomplete The truststore configuration is incorrect 1. Certificate Trust Configuration Verify Certificate Chain: # Check if certificate is in NiFi truststore (repeat for each instance) keytool -list -v -keystore /path/to/nifi/truststore.jks -storepass [password] # Check if certificate is in Registry truststore keytool -list -v -keystore /path/to/registry/truststore.jks -storepass [password] # Verify the Registry's certificate chain openssl s_client -connect nifi-registry.example.com:443 -showcerts Ensure Complete Certificate Chain: Add the Registry's complete certificate chain (including intermediate CAs) to NiFi's truststore Add NiFi's complete certificate chain to the Registry's truststore # Add Registry certificate to NiFi truststore keytool -import -alias nifi-registry -file registry-cert.pem -keystore /path/to/nifi/conf/truststore.jks -storepass [password] # Add NiFi certificate to Registry truststore keytool -import -alias nifi-prod -file nifi-cert.pem -keystore /path/to/registry/conf/truststore.jks -storepass [password] 2. Proper Certificate Exchange Ensure you've exchanged certificates correctly export NiFi Registry's public certificate keytool -exportcert -alias nifi-registry -keystore /path/to/registry/keystore.jks -file registry.crt -storepass [password] Import this certificate into each NiFi instance's truststore keytool -importcert -alias nifi-registry -keystore /path/to/nifi/truststore.jks -file registry.crt -storepass [password] -noprompt 3. NiFi Registry Connection Configuration In your NiFi instance (nifi.properties), verify # Registry client properties nifi.registry.client.name=NiFi Registry nifi.registry.client.url=https://nifi-registry.example.com/nifi-registry nifi.registry.client.timeout.connect=30 secs nifi.registry.client.timeout.read=30 secs Verify these configuration files in NiFi (production/development) # nifi.properties: nifi.registry.client.ssl.protocol=TLS nifi.registry.client.truststore.path=/path/to/truststore.jks nifi.registry.client.truststore.password=[password] nifi.registry.client.truststore.type=JKS In NiFi Registry # nifi-registry.properties: nifi.registry.security.truststore.path=/path/to/truststore.jks nifi.registry.security.truststore.password=[password] nifi.registry.security.truststore.type=JKS 4. LDAP Configuration For your LDAP integration issues in authorizers.xml ensure you have <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.registry.security.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">ldap-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">cn=admin-user,ou=users,dc=example,dc=com</property> <property name="NiFi Identity 1">cn=dev-nifi,ou=servers,dc=example,dc=com</property> </accessPolicyProvider> In the authorizations.xml add appropriate policies for the dev-nifi identity <policy identifier="some-uuid" resource="/buckets" action="READ"> <user identifier="dev-nifi-uuid"/> </policy> 5. Proxy Configuration For proxy user requests, add in nifi.properties nifi.registry.client.proxy.identity=cn=dev-nifi,ou=servers,dc=example,dc=com 6. Restart Order After making changes restart the Nifi instance in the below order NiFi Registry first Then restart all NiFi instances Happy hadoping ... View more

@Shelton We just followed Steps 1,3 4 and 5 to generate the automated report to Elasticsearch. It was pretty straight forward. Only things is we had to do was enable firewall in our Docker container and update Input Port's Access Policies. Thanks   ... View more

@Shelton Please read my previous answer carefully. None of the properties provided by you are in hbase codebase ... View more

@Jay2021, Welcome to our community! As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. ... View more

Thanks for the help and sorry for late reply @Shelton  I am getting the output here but the values for parent class are not getting populated, they are displayed as NULL   ... View more

Thanks @Shelton for details!!  I will try Option 3 in our pipeline shell script.   Will let you know if any further issues. ... View more

Thank you so much for helping me out. ... View more

@spserd  Looking at your issue with Spark on Kubernetes, I see a clear difference between the client and cluster deployment modes that's causing the "system" authentication problem. the issue is when running in client mode with spark-shell, you're encountering an authorization issue where Spark is trying to create executor pods as "system" instead of using your service account "spark-sa", despite providing the token. Possible Solution For client mode, you need to add a specific configuration to tell Spark to use the token for executor pod creation --conf spark.kubernetes.authenticate.executor.serviceAccountName=spark-sa So your updated command should look like this  ./bin/spark-shell \ --master k8s://https://my-k8s-cluster:6443 \ --deploy-mode client \ --name spark-shell-poc \ --conf spark.executor.instances=1 \ --conf spark.kubernetes.container.image=my-docker-hub/spark_poc:v1.4 \ --conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \ --conf spark.kubernetes.namespace=dynx-center-resources \ --conf spark.driver.pod.name=dynx-spark-driver \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa \ --conf spark.kubernetes.authenticate.executor.serviceAccountName=spark-sa \ --conf spark.kubernetes.authenticate.submission.oauthToken=$K8S_TOKEN The key is that in client mode, you need to explicitly configure the executor authentication because the driver is running outside the cluster and needs to delegate this permission. If this still doesn't work, ensure your service account has appropriate ClusterRole bindings that allow it to create and manage pods in the specified namespace. Happy hadooping  ... View more