Member since
01-10-2020
52
Posts
1
Kudos Received
3
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1787 | 08-25-2020 09:30 AM | |
| 2725 | 08-21-2020 08:31 PM | |
| 2031 | 05-28-2020 03:53 PM |
11-03-2020
01:02 PM
thanks for the response @TimothySpann i am using openjdk version "1.8.0_272" NIFI 1.9 HDF 3.2, suspecting java version is the issue but not sure my other working env with the same config have java openjdk version "1.8.0_262" verified certs on both servers, it got only one DNS, host entry. i am assuming the JAVA version is the issue here. NODE 1: keytool -v -list -keystore keystore.jks Enter keystore password: ***************** WARNING WARNING WARNING ***************** * The integrity of the information stored in your keystore * * has NOT been verified! In order to verify its integrity, * * you must provide your keystore password. * ***************** WARNING WARNING WARNING ***************** Keystore type: jks Keystore provider: SUN Your keystore contains 1 entry Alias name: nifi-key Creation date: Oct 30, 2020 Entry type: PrivateKeyEntry Certificate chain length: 2 Certificate[1]: Owner: CN=its-nifi-node-dev-nifisbox-01.its-streaming-dev.com, OU=NIFI Issuer: CN=localhost, OU=NIFI Serial number: 1757acc190400000000 Valid from: Fri Oct 30 18:36:37 UTC 2020 until: Mon Oct 30 18:36:37 UTC 2023 Certificate fingerprints: MD5: 3A:0A:3D:04:3B:2E:C5:19:4F:EE:93:15:B9:CB:A3:E3 SHA1: C7:E4:F9:A0:F6:71:0A:C5:73:D7:35:23:3B:94:2E:C3:5C:70:A8:AF SHA256: C2:11:4E:76:FE:EE:60:D4:7C:11:0C:0E:42:04:04:28:AC:95:98:74:A6:19:69:1F:64:B9:D2:24:DA:3A:2F:CA Signature algorithm name: SHA256withRSA Subject Public Key Algorithm: 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.35 Criticality=false AuthorityKeyIdentifier [ KeyIdentifier [ 0000: 66 5D B4 0E 26 CE CE 8C F7 C9 3C 78 DA 77 CB 02 f]..&.....<x.w.. 0010: F7 12 78 95 ..x. ] ] #2: ObjectId: 2.5.29.19 Criticality=false BasicConstraints:[ CA:false PathLen: undefined ] #3: ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ clientAuth serverAuth ] #4: ObjectId: 2.5.29.15 Criticality=true KeyUsage [ DigitalSignature Non_repudiation Key_Encipherment Data_Encipherment Key_Agreement ] #5: ObjectId: 2.5.29.17 Criticality=false SubjectAlternativeName [ DNSName: its-nifi-node-dev-nifisbox-01.its-streaming-dev.com ] #6: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: D1 21 C4 9E AD 9C EF 3F DF A5 6B 17 CB F1 DE 34 .!.....?..k....4 0010: B7 EA C2 87 .... ] ] Certificate[2]: Owner: CN=localhost, OU=NIFI Issuer: CN=localhost, OU=NIFI Serial number: 1757acc17a600000000 Valid from: Fri Oct 30 18:36:37 UTC 2020 until: Mon Oct 30 18:36:37 UTC 2023 Certificate fingerprints: MD5: 7D:3C:9A:4B:55:5B:6D:08:18:9C:88:B0:C1:D1:95:82 SHA1: E8:B4:06:D0:37:4B:CB:16:FA:01:FB:79:1D:28:98:60:97:D6:75:A1 SHA256: 59:5C:26:D2:2E:DD:D0:5D:8C:90:63:7D:D9:8F:A3:8E:EE:B2:E1:85:44:9A:86:9D:F8:CF:78:A2:C4:F7:D7:1C Signature algorithm name: SHA256withRSA Subject Public Key Algorithm: 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.35 Criticality=false AuthorityKeyIdentifier [ KeyIdentifier [ 0000: 66 5D B4 0E 26 CE CE 8C F7 C9 3C 78 DA 77 CB 02 f]..&.....<x.w.. 0010: F7 12 78 95 ..x. ] ] #2: ObjectId: 2.5.29.19 Criticality=false BasicConstraints:[ CA:true PathLen:2147483647 ] #3: ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ clientAuth serverAuth ] #4: ObjectId: 2.5.29.15 Criticality=true KeyUsage [ DigitalSignature Non_repudiation Key_Encipherment Data_Encipherment Key_Agreement Key_CertSign Crl_Sign ] #5: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: 66 5D B4 0E 26 CE CE 8C F7 C9 3C 78 DA 77 CB 02 f]..&.....<x.w.. 0010: F7 12 78 95 ..x. ] ] ******************************************* ******************************************* Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.jks -deststoretype pkcs12". NODE2: keytool -v -list -keystore keystore.jks Enter keystore password: ***************** WARNING WARNING WARNING ***************** * The integrity of the information stored in your keystore * * has NOT been verified! In order to verify its integrity, * * you must provide your keystore password. * ***************** WARNING WARNING WARNING ***************** Keystore type: jks Keystore provider: SUN Your keystore contains 1 entry Alias name: nifi-key Creation date: Oct 30, 2020 Entry type: PrivateKeyEntry Certificate chain length: 2 Certificate[1]: Owner: CN=its-nifi-node-dev-nifisbox-01.its-streaming-dev.com, OU=NIFI Issuer: CN=localhost, OU=NIFI Serial number: 1757acc190400000000 Valid from: Fri Oct 30 18:36:37 UTC 2020 until: Mon Oct 30 18:36:37 UTC 2023 Certificate fingerprints: MD5: 3A:0A:3D:04:3B:2E:C5:19:4F:EE:93:15:B9:CB:A3:E3 SHA1: C7:E4:F9:A0:F6:71:0A:C5:73:D7:35:23:3B:94:2E:C3:5C:70:A8:AF SHA256: C2:11:4E:76:FE:EE:60:D4:7C:11:0C:0E:42:04:04:28:AC:95:98:74:A6:19:69:1F:64:B9:D2:24:DA:3A:2F:CA Signature algorithm name: SHA256withRSA Subject Public Key Algorithm: 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.35 Criticality=false AuthorityKeyIdentifier [ KeyIdentifier [ 0000: 66 5D B4 0E 26 CE CE 8C F7 C9 3C 78 DA 77 CB 02 f]..&.....<x.w.. 0010: F7 12 78 95 ..x. ] ] #2: ObjectId: 2.5.29.19 Criticality=false BasicConstraints:[ CA:false PathLen: undefined ] #3: ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ clientAuth serverAuth ] #4: ObjectId: 2.5.29.15 Criticality=true KeyUsage [ DigitalSignature Non_repudiation Key_Encipherment Data_Encipherment Key_Agreement ] #5: ObjectId: 2.5.29.17 Criticality=false SubjectAlternativeName [ DNSName: its-nifi-node-dev-nifisbox-02.its-streaming-dev.com ] #6: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: D1 21 C4 9E AD 9C EF 3F DF A5 6B 17 CB F1 DE 34 .!.....?..k....4 0010: B7 EA C2 87 .... ] ] Certificate[2]: Owner: CN=localhost, OU=NIFI Issuer: CN=localhost, OU=NIFI Serial number: 1757acc17a600000000 Valid from: Fri Oct 30 18:36:37 UTC 2020 until: Mon Oct 30 18:36:37 UTC 2023 Certificate fingerprints: MD5: 7D:3C:9A:4B:55:5B:6D:08:18:9C:88:B0:C1:D1:95:82 SHA1: E8:B4:06:D0:37:4B:CB:16:FA:01:FB:79:1D:28:98:60:97:D6:75:A1 SHA256: 59:5C:26:D2:2E:DD:D0:5D:8C:90:63:7D:D9:8F:A3:8E:EE:B2:E1:85:44:9A:86:9D:F8:CF:78:A2:C4:F7:D7:1C Signature algorithm name: SHA256withRSA Subject Public Key Algorithm: 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.35 Criticality=false AuthorityKeyIdentifier [ KeyIdentifier [ 0000: 66 5D B4 0E 26 CE CE 8C F7 C9 3C 78 DA 77 CB 02 f]..&.....<x.w.. 0010: F7 12 78 95 ..x. ] ] #2: ObjectId: 2.5.29.19 Criticality=false BasicConstraints:[ CA:true PathLen:2147483647 ] #3: ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ clientAuth serverAuth ] #4: ObjectId: 2.5.29.15 Criticality=true KeyUsage [ DigitalSignature Non_repudiation Key_Encipherment Data_Encipherment Key_Agreement Key_CertSign Crl_Sign ] #5: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: 66 5D B4 0E 26 CE CE 8C F7 C9 3C 78 DA 77 CB 02 f]..&.....<x.w.. 0010: F7 12 78 95 ..x. ] ] ******************************************* ******************************************* Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.jks -deststoretype pkcs12". LOGIN-identity: <loginIdentityProviders> <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Identity Strategy">USE_USERNAME</property> <property name="Authentication Strategy">START_TLS</property> <property name="Manager DN">XXXX</property> <property name="Manager Password">XXXX</property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://XX.ad.XX.XX:389</property> <property name="User Search Base">dc=AD,dc=XX,dc=XX</property> <property name="User Search Filter">sAMAccountName={0}</property> <property name="Identity Strategy">USE_USERNAME</property> <property name="Authentication Expiration">12 hours</property> </provider> {% if not security_enabled %} {% endif %} </loginIdentityProviders>
... View more
11-03-2020
12:38 PM
2020-11-03 19:35:27,605 ERROR [NiFi Web Server-17] o.a.n.w.a.c.AdministrationExceptionMapper org.apache.nifi.admin.service.AdministrationException: Unable to validate the supplied credentials. Please contact the system administrator.. Returning Internal Server Error response. org.apache.nifi.admin.service.AdministrationException: Unable to validate the supplied credentials. Please contact the system administrator. at org.apache.nifi.web.api.AccessResource.createAccessToken(AccessResource.java:736) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.apache.nifi.web.filter.ExceptionFilter.doFilter(ExceptionFilter.java:46) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634) at org.apache.nifi.web.security.headers.StrictTransportSecurityFilter.doFilter(StrictTransportSecurityFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.apache.nifi.web.security.headers.XSSProtectionFilter.doFilter(XSSProtectionFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.apache.nifi.web.security.headers.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:47) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.apache.nifi.web.security.headers.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:531) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291) at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.nifi.authentication.exception.IdentityAccessException: Unable to validate the supplied credentials. Please contact the system administrator. at org.apache.nifi.ldap.LdapProvider.authenticate(LdapProvider.java:340) at org.apache.nifi.web.security.spring.LoginIdentityProviderFactoryBean$1.authenticate(LoginIdentityProviderFactoryBean.java:315) at org.apache.nifi.web.api.AccessResource.createAccessToken(AccessResource.java:728) ... 83 common frames omitted Caused by: org.springframework.security.authentication.InternalAuthenticationServiceException: Failed to negotiate TLS session; nested exception is javax.net.ssl.SSLPeerUnverifiedException: hostname of the server '' does not match the hostname in the server's certificate. at org.springframework.security.ldap.authentication.LdapAuthenticationProvider.doAuthentication(LdapAuthenticationProvider.java:206) at org.springframework.security.ldap.authentication.AbstractLdapAuthenticationProvider.authenticate(AbstractLdapAuthenticationProvider.java:85) at org.apache.nifi.ldap.LdapProvider.authenticate(LdapProvider.java:310) ... 85 common frames omitted Caused by: org.springframework.ldap.UncategorizedLdapException: Failed to negotiate TLS session; nested exception is javax.net.ssl.SSLPeerUnverifiedException: hostname of the server '' does not match the hostname in the server's certificate. at org.springframework.ldap.core.support.AbstractTlsDirContextAuthenticationStrategy.processContextAfterCreation(AbstractTlsDirContextAuthenticationStrategy.java:153) at org.springframework.ldap.core.support.AbstractContextSource.doGetContext(AbstractContextSource.java:142) at org.springframework.ldap.core.support.AbstractContextSource.getReadOnlyContext(AbstractContextSource.java:158) at org.springframework.ldap.core.LdapTemplate.executeReadOnly(LdapTemplate.java:802) at org.springframework.security.ldap.SpringSecurityLdapTemplate.searchForSingleEntry(SpringSecurityLdapTemplate.java:316) at org.springframework.security.ldap.search.FilterBasedLdapUserSearch.searchForUser(FilterBasedLdapUserSearch.java:127) at org.springframework.security.ldap.authentication.BindAuthenticator.authenticate(BindAuthenticator.java:95) at org.springframework.security.ldap.authentication.LdapAuthenticationProvider.doAuthentication(LdapAuthenticationProvider.java:187) ... 87 common frames omitted Caused by: javax.net.ssl.SSLPeerUnverifiedException: hostname of the server '' does not match the hostname in the server's certificate. at com.sun.jndi.ldap.ext.StartTlsResponseImpl.verify(StartTlsResponseImpl.java:447) at com.sun.jndi.ldap.ext.StartTlsResponseImpl.negotiate(StartTlsResponseImpl.java:225) at org.springframework.ldap.core.support.AbstractTlsDirContextAuthenticationStrategy.processContextAfterCreation(AbstractTlsDirContextAuthenticationStrategy.java:136) ... 94 common frames omitted Caused by: java.security.cert.CertificateException: Illegal given domain name: at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:207) at sun.security.util.HostnameChecker.match(HostnameChecker.java:102) at sun.security.util.HostnameChecker.match(HostnameChecker.java:108) at com.sun.jndi.ldap.ext.StartTlsResponseImpl.verify(StartTlsResponseImpl.java:426) ... 96 common frames omitted Caused by: java.lang.IllegalArgumentException: Server name value of host_name cannot be empty at javax.net.ssl.SNIHostName.checkHostName(SNIHostName.java:314) at javax.net.ssl.SNIHostName.<init>(SNIHostName.java:108) at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:205) ... 99 common frames omitted @MattWho @TimothySpann please advice able to login with initial admin cert whereas developers unable to login with AD user id.
... View more
Labels:
- Labels:
-
Apache NiFi
09-15-2020
08:22 PM
@TimothySpann thanks for your response. - i assume ConsumeKafkaRecord_2_0 1.9 will consume messages from the Kafka partitions based on "max poll records".in my case I have set it to 100000. please let me know if my assumption is wrong. could you also confirm if there are any limitations to pull bulk messages like at least 10K messages in a single poll? with 1.9 nifi message size is 2 MB, Avro format and 3 topics have 2 million messages each
... View more
09-15-2020
01:03 PM
@Wynner @pvillard @bbende @MattWho @TimothySpann I have single node nifi, where the Kafka topic has 3 partitions but ConsumeKafkaRecord_2_0 though configured with 3 threads on single node nifi, it's pulling only 50 messages or even less. i am assuming though nifi is a single node, as I have configured to use 3 threads based on the partitions on topic, it should consume data. in my case, it's pulling only 30-50 messages per second any pointers would be greatly helpful. thanks! HDF 3.4 nifi 1.9
... View more
09-14-2020
10:48 AM
NIFI 1.9
HDF 3.4.1
nifi ConsumeKafkaRecord_2_0 1.9.0.3.4.1.1-4 - slow consuming from kafka topic which got 8 million messages
ConsumeKafkaRecord_2_0 1.9.0.3.4.1.1-4 - reading only 100 messages at a time
nifi singlenode 32core * 64 GB memory
@MattWho, please advise
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
08-25-2020
09:30 AM
@BGabor thanks for your response I'm trying to understand the difference between two blocks given in the document #BLOCK1 activeDirectoryRealm = org.apache.zeppelin.realm.ActiveDirectoryGroupRealm
activeDirectoryRealm.systemUsername = userNameA
activeDirectoryRealm.systemPassword = passwordA
activeDirectoryRealm.searchBase = CN=Users,DC=SOME_GROUP,DC=COMPANY,DC=COM
activeDirectoryRealm.url = ldap://ldap.test.com:389
activeDirectoryRealm.groupRolesMap = "CN=aGroupName,OU=groups,DC=SOME_GROUP,DC=COMPANY,DC=COM":"group1"
activeDirectoryRealm.authorizationCachingEnabled = false
activeDirectoryRealm.principalSuffix = @corp.company.net
##BLOCK2
ldapRealm = org.apache.zeppelin.realm.LdapGroupRealm
# search base for ldap groups (only relevant for LdapGroupRealm):
ldapRealm.contextFactory.environment[ldap.searchBase] = dc=COMPANY,dc=COM
ldapRealm.contextFactory.url = ldap://ldap.test.com:389
ldapRealm.userDnTemplate = uid={0},ou=Users,dc=COMPANY,dc=COM
ldapRealm.contextFactory.authenticationMechanism = simple also define roles/groups that you want to have in the system, like below; [roles]
admin = *
hr = *
finance = *
group1 = * . could you help me understand? i want to extract only one group and users from AD, which block should I use. thanks
... View more
08-24-2020
03:16 PM
please suggest if anyone has successfully integrated with AD. I'm trying to use below configs and it's not working ldapRealm= org.apache.zeppelin.realm.LdapRealm ldapRealm.contextFactory.url = ldap://ad.abc.com:389 ldapRealm.contextFactory.authenticationMechanism = simple ldapRealm.contextFactory.systemUsername = svc_abc ldapRealm.contextFactory.systemPassword = passwdddd #ldapRealm.searchBase = DC=abc,DC=abc,DC=COM ldapRealm.userSearchBase = OU=IDM,DC=abc,DC=abc,DC=com ldapRealm.userSearchScope = subtree ldapRealm.userSearchAttributeName = sAMAccountName ldapRealm.userSearchFilter = (&(objectclass=person)(sAMAccountName={0})) ldapRealm.authorizationEnabled = true ldapRealm.groupSearchBase = OU=abcGroups,DC=abc,DC=abc,DC=com ldapRealm.groupObjectClass = group ldapRealm.memberAttribute= member ldapRealm.groupSearchScope = subtree ldapRealm.groupSearchFilter = (&(objectclass=group)(member={0})) ldapRealm.memberAttributeValueTemplate= $CN=g_app_zep,OU=abcGroups,DC=abc,DC=abc,DC=com ldapRealm.groupSearchEnableMatchingRuleInChain = true ldapRealm.rolesByGroup = Zeppelin_Admin: admin_role ldapRealm.allowedRolesForAuthentication = admin_role,user_role ERROR: WARN [2020-08-24 16:31:14,497] ({main} WebAppContext.java[doStart]:554) - Failed startup of context o.e.j.w.WebAppContext@22635ba0{zeppelin-web,/,file:///opt/zepplin/zeppelin-0.8.2-bin-all/webapps/webapp/,UNAVAILABLE}{/opt/zepplin/zeppelin-0.8.2-bin-all/zeppelin-web-0.8.2.war} org.apache.shiro.config.UnresolveableReferenceException: The object with id [CN=g_app_zep,OU=abcGroups,DC=abc,DC=abc,DC=com] has not yet been defined and therefore cannot be referenced. Please ensure objects are defined in the order in which they should be created and made available for future reference. please share if there is any working configs
... View more
Labels:
- Labels:
-
Apache Hadoop
08-21-2020
08:31 PM
1 Kudo
figured out the issue, related to jira https://issues.apache.org/jira/browse/NIFI-6999 flow.xml.gz is 963 mb, where toolkit mem isn't enough to restart from ambari GUI. hence started through CLI restarted the nifi from CLI rather than AMBARI , came up flow.xml.gz file size is 963MB instance : 120GB memory NIFI allocated mem: 45GB toolkit 15GB mem error: from ambari agent logs ERROR:020/08/21 19:01:30 ERROR [main] org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain: java.lang.OutOfMemoryError: Requested array size exceeds VM limit at java.lang.StringCoding.encode(StringCoding.java:350) at java.lang.String.getBytes(String.java:941) at org.apache.commons.io.IOUtils.write(IOUtils.java:2025) at org.apache.commons.io.IOUtils$write$0.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141) at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure6$_closure30.doCall(ConfigEncryptionTool.groovy:870) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at groovy.lang.Closure.call(Closure.java:414) at groovy.lang.Closure.call(Closure.java:430) at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622) at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1759) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54) at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274) at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) at org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure6.doCall(ConfigEncryptionTool.groovy:869) Requested array size exceeds VM limit
... View more
08-21-2020
06:02 PM
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 304, in <module>
Master().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 143, in start
self.configure(env, is_starting = True)
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 111, in configure
self.write_configurations(params, is_starting)
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 247, in write_configurations
support_encrypt_authorizers=params.stack_support_encrypt_authorizers
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_toolkit_util_common.py", line 574, in encrypt_sensitive_properties
Execute(encrypt_config_command, user=nifi_user, logoutput=False, environment=environment)
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/hdf/current/nifi-toolkit/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /grid/var/lib/nifi/conf/flow.xml.gz -s '[PROTECTED]' -l /usr/hdf/current/nifi/conf/login-identity-providers.xml -a /usr/hdf/current/nifi/conf/authorizers.xml -m -e '[PROTECTED]' -p '[PROTECTED]'' returned 255. 2020/08/21 18:00:37 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of nifi.properties
2020/08/21 18:00:37 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source nifi.properties and destination nifi.properties are identical [/usr/hdf/current/nifi/conf/nifi.properties] so the original will be overwritten
2020/08/21 18:00:37 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of login-identity-providers.xml
2020/08/21 18:00:37 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source login-identity-providers.xml and destination login-identity-providers.xml are identical [/usr/hdf/current/nifi/conf/login-identity-providers.xml] so the original will be overwritten
2020/08/21 18:00:37 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of authorizers.xml
2020/08/21 18:00:37 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source authorizers.xml and destination authorizers.xml are identical [/usr/hdf/current/nifi/conf/authorizers.xml] so the original will be overwritten
2020/08/21 18:00:37 INFO [main] org.apache.nifi.properties.ConfigEncryptionTool: Handling encryption of flow.xml.gz
2020/08/21 18:00:37 WARN [main] org.apache.nifi.properties.ConfigEncryptionTool: The source flow.xml.gz and destination flow.xml.gz are identical [/grid/var/lib/nifi/conf/flow.xml.gz] so the original will be overwritten
... View more