Hi everyone, I've been trying to find out if we can give to LDAP groups some roles ( admin and user ) but without succes. I'm puting down below my shiro config. It's working but when I'm logged in zeppelin as member of an admin group, the web UI is telling me that i don't have permission on the interpreter, credential, etc ... page. Anyone have a clue ? [users] # List of users with their password allowed to access Zeppelin. # To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISections # To enable admin user, uncomment the following line and set an appropriate password. #admin = password1, admin #user1 = password2, role1, role2 #user2 = password3, role3 #user3 = password4, role2 [main] ldapRealm=org.apache.zeppelin.realm.LdapRealm ldapRealm.contextFactory.systemUsername=cn=UserBind,ou=Services,dc=domain,dc=company ldapRealm.contextFactory.systemPassword=PassWord ldapRealm.contextFactory.authenticationMechanism=simple ldapRealm.contextFactory.url=ldap://ldap-server:389 ldapRealm.userDnTemplate =uid={0},ou=Users,dc=domain,dc=company # Ability to set ldap paging Size if needed; default is 100 ldapRealm.pagingSize=200 ldapRealm.authorizationEnabled=true ldapRealm.searchBase=dc=domain,dc=company ldapRealm.userSearchBase=dc=domain,dc=company ldapRealm.groupSearchBase=ou=Groups,dc=domain,dc=company ldapRealm.userObjectClass=posixAccount ldapRealm.groupObjectClass=posixGroup ldapRealm.userSearchFilter = (&(objectclass=posixAccount)(uid={0})) ldapRealm.memberAttribute=memberOf ldapRealm.userSearchAttributeName = uid ldapRealm.groupIdAttribute=cn ldapRealm.userLowerCase = true # ability set searchScopes subtree (default), one, base ldapRealm.userSearchScope = subtree; ldapRealm.groupSearchScope = subtree; ldapRealm.groupSearchFilter = (&(objectclass=posixGroup)(memberOf={0})) # Format to parse & search group member values in 'memberAttribute' ldapRealm.memberAttributeValueTemplate=uid={0},ou=Groups,dc=domain,dc=company # Map from physical AD groups to logical application roles #ldapRealm.allowedRolesForAuthentication = admin_role,user_role ldapRealm.rolesByGroup = "cn=Group-admin,dc=domain,dc=company":"admin_role", "cn=Group-user,dc=domain,dc=company":"user_role" ldapRealm.permissionsByRole= admin_role:"*", user_role:"*:ToDoItemsJdo:*:*, *:ToDoItem:*:*" ldapRealm.groupSearchEnableMatchingRuleInChain = true # Force usernames returned from ldap to lowercase, useful for AD #ldapRealm.userLowerCase = true sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager securityManager.sessionMode=native securityManager.sessionManager.globalSessionTimeout = 86400000 securityManager.sessionManager = $sessionManager ########################################## ### A sample PAM configuration #pamRealm=org.apache.zeppelin.realm.PamRealm #pamRealm.service=sshd ########################################## sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager ### If caching of user is required then uncomment below lines #cacheManager = org.apache.shiro.cache.MemoryConstrainedCacheManager #securityManager.cacheManager = $cacheManager ### Enables 'HttpOnly' flag in Zeppelin cookies cookie = org.apache.shiro.web.servlet.SimpleCookie cookie.name = JSESSIONID cookie.httpOnly = true ### Uncomment the below line only when Zeppelin is running over HTTPS #cookie.secure = true sessionManager.sessionIdCookie = $cookie #securityManager.sessionManager = $sessionManager # 86,400,000 milliseconds = 24 hour #securityManager.sessionManager.globalSessionTimeout = 86400000 shiro.loginUrl = /api/login [roles] #admin_role = * #user_role = *:ToDoItemsJdo:*:*,*:ToDoItem:*:* #role1 = * #role2 = * #role3 = * #admin = * [urls] # This section is used for url-based security. For details see the shiro.ini documentation. # # You can secure interpreter, configuration and credential information by urls. # Comment or uncomment the below urls that you want to hide: # anon means the access is anonymous. # authc means form based auth Security. # # IMPORTANT: Order matters: URL path expressions are evaluated against an incoming request # in the order they are defined and the FIRST MATCH WINS. # # To allow anonymous access to all but the stated urls, # uncomment the line second last line (/** = anon) and comment the last line (/** = authc) # #/api/version = anon # Allow all authenticated users to restart interpreters on a notebook page. # Comment out the following line if you would like to authorize only admin users to restart interpreters. #/api/interpreter/setting/restart/** = authc /api/interpreter/** = authc, roles[admin_role,user_role] /api/configurations/** = authc, roles[admin_role] /api/credential/** = authc, roles[admin_role] #/** = anon /** = authc this is the log i get when i'm logging in, as you can see the roles part is blank /api/security/ticket} SecurityRestApi.java[ticket]:88) - {"status":"OK","message":"","body":{"principal":"user","ticket":"22faec60-d905-4757-899e-2a62b6ec7463","roles":"[]"} Best regards, Toky ... View more

Hi Everyone, I'm trying to Upgrade my HDP plateforme HDP-2.4.2.0 to HDP-2.5.3.0 but i get an ERROR which i'm not familiar with. My ambari version is ambari-2.6.1.0 because my HDP upgrade target is HDP-2.6.4.0-91, can anyone help me ? Thanks, Toky ERROR [ambari-client-thread-2923] BaseManagementHandler:61 - Caught a system exception while attempting to create a resource: Repositories for os type redhat7 are not defined for version 2.5.3.0-37 of Stack HDP-2.5. org.apache.ambari.server.controller.spi.SystemException: Repositories for os type redhat7 are not defined for version 2.5.3.0-37 of Stack HDP-2.5. at org.apache.ambari.server.controller.internal.ClusterStackVersionResourceProvider.getHostVersionInstallCommand(ClusterStackVersionResourceProvider.java:718) at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:118) at org.apache.ambari.server.controller.internal.ClusterStackVersionResourceProvider.createOrchestration(ClusterStackVersionResourceProvider.java:646) at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:118) at org.apache.ambari.server.controller.internal.ClusterStackVersionResourceProvider.createOrUpdateHostVersions(ClusterStackVersionResourceProvider.java:541) at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:128) at org.apache.ambari.server.controller.internal.ClusterStackVersionResourceProvider.createResourcesAuthorized(ClusterStackVersionResourceProvider.java:483) at org.apache.ambari.server.controller.internal.AbstractAuthorizedResourceProvider.createResources(AbstractAuthorizedResourceProvider.java:220) at org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:298) at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:97) at org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:37) at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:73) at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:144) at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:126) at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:90) at org.apache.ambari.server.api.services.ClusterStackVersionService.createRequests(ClusterStackVersionService.java:121) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl. at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl. at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSe at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(Filter at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAu at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTransl at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagement at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(Anon at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilte at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCache at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr at org.apache.ambari.server.security.authentication.AmbariDelegatingAuthenticationFilter.doFil at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(Amba at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(Security at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:19 at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.j at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:16 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHea at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.jav at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.jav at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:139) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:4 at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:984) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnecti at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:236) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:748) ... View more

Hi everyone , Right now i'm trying to run my Nifi cluster using LDAP for many users. For each of my nodes have certificat and i have another one for my browser for admin stuff , which i have created with nfi-toolkit. Actually I'm able to connect to nifi with my admin account(I can add users, policies..).The problem accurs when a users tries to connect via LDAP. He gets this error: this are the log : 2017-05-16 16:48:13,550 ERROR [NiFi Web Server-126] o.a.n.w.a.c.AdministrationExceptionMapper org.apache.nifi.admin.service.AdministrationException: Unable to validate the supplied credentials. Please contact the system administrator.. Returning Internal Server Error response. org.apache.nifi.admin.service.AdministrationException: Unable to validate the supplied credentials. Please contact the system administrator. at org.apache.nifi.web.api.AccessResource.createAccessToken(AccessResource.java:445) ~[classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_112] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_112] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_112] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_112] at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.jar:1.19] at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.jar:1.19] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.jar:1.19] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.jar:1.19] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66) [classes/:na] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:207) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51) [classes/:na] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1174) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1106) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:459) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.Server.handle(Server.java:524) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] Caused by: org.apache.nifi.authentication.exception.IdentityAccessException: Unable to validate the supplied credentials. Please contact the system administrator. at org.apache.nifi.ldap.LdapProvider.authenticate(LdapProvider.java:340) ~[na:na] at org.apache.nifi.web.security.spring.LoginIdentityProviderFactoryBean$1.authenticate(LoginIdentityProviderFactoryBean.java:301) ~[nifi-web-security-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10] at org.apache.nifi.web.api.AccessResource.createAccessToken(AccessResource.java:437) ~[classes/:na] ... 58 common frames omitted Caused by: org.springframework.security.authentication.InternalAuthenticationServiceException: hostname:636; nested exception is javax.naming.CommunicationException: hostame:636 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target] at org.springframework.security.ldap.authentication.LdapAuthenticationProvider.doAuthentication(LdapAuthenticationProvider.java:208) ~[na:na] at org.springframework.security.ldap.authentication.AbstractLdapAuthenticationProvider.authenticate(AbstractLdapAuthenticationProvider.java:82) ~[na:na] at org.apache.nifi.ldap.LdapProvider.authenticate(LdapProvider.java:310) ~[na:na] ... 60 common frames omitted Caused by: org.springframework.ldap.CommunicationException: hostname:636; nested exception is javax.naming.CommunicationException: hostname:636 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target] at org.springframework.ldap.support.LdapUtils.convertLdapException(LdapUtils.java:108) ~[na:na] at org.springframework.ldap.core.support.AbstractContextSource.createContext(AbstractContextSource.java:356) ~[na:na] at org.springframework.ldap.core.support.AbstractContextSource.doGetContext(AbstractContextSource.java:140) ~[na:na] at org.springframework.ldap.core.support.AbstractContextSource.getReadOnlyContext(AbstractContextSource.java:159) ~[na:na] at org.springframework.ldap.core.LdapTemplate.executeReadOnly(LdapTemplate.java:802) ~[na:na] at org.springframework.security.ldap.SpringSecurityLdapTemplate.searchForSingleEntry(SpringSecurityLdapTemplate.java:316) ~[na:na] at org.springframework.security.ldap.search.FilterBasedLdapUserSearch.searchForUser(FilterBasedLdapUserSearch.java:126) ~[na:na] at org.springframework.security.ldap.authentication.BindAuthenticator.authenticate(BindAuthenticator.java:94) ~[na:na] at org.springframework.security.ldap.authentication.LdapAuthenticationProvider.doAuthentication(LdapAuthenticationProvider.java:189) ~[na:na] ... 62 common frames omitted Caused by: javax.naming.CommunicationException: hostname:636 at com.sun.jndi.ldap.Connection.<init>(Connection.java:226) ~[na:1.8.0_112] at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:137) ~[na:1.8.0_112] at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1614) ~[na:1.8.0_112] at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2746) ~[na:1.8.0_112] at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:319) ~[na:1.8.0_112] at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:192) ~[na:1.8.0_112] at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:210) ~[na:1.8.0_112] at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:153) ~[na:1.8.0_112] at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:83) ~[na:1.8.0_112] at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684) ~[na:1.8.0_112] at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313) ~[na:1.8.0_112] at javax.naming.InitialContext.init(InitialContext.java:244) ~[na:1.8.0_112] at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154) ~[na:1.8.0_112] at org.springframework.ldap.core.support.LdapContextSource.getDirContextInstance(LdapContextSource.java:42) ~[na:na] at org.springframework.ldap.core.support.AbstractContextSource.createContext(AbstractContextSource.java:344) ~[na:na] ... 69 common frames omitted Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.8.0_112] at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) ~[na:1.8.0_112] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) ~[na:1.8.0_112] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[na:1.8.0_112] at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509) ~[na:1.8.0_112] at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[na:1.8.0_112] at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[na:1.8.0_112] at sun.security.ssl.Handshaker.process_record(Handshaker.java:914) ~[na:1.8.0_112] at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) ~[na:1.8.0_112] at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) ~[na:1.8.0_112] at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) ~[na:1.8.0_112] at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) ~[na:1.8.0_112] at com.sun.jndi.ldap.Connection.createSocket(Connection.java:376) ~[na:1.8.0_112] at com.sun.jndi.ldap.Connection.<init>(Connection.java:203) ~[na:1.8.0_112] ... 83 common frames omitted Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387) ~[na:1.8.0_112] at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) ~[na:1.8.0_112] at sun.security.validator.Validator.validate(Validator.java:260) ~[na:1.8.0_112] at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[na:1.8.0_112] at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[na:1.8.0_112] at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[na:1.8.0_112] at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491) ~[na:1.8.0_112] ... 92 common frames omitted Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[na:1.8.0_112] at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[na:1.8.0_112] at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[na:1.8.0_112] at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382) ~[na:1.8.0_112] ... 98 common frames omitted And my login-identity-providers.xml <loginIdentityProviders> <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Authentication Strategy">START_TLS</property> <property name="Manager DN">cn=bindnifi,ou=Services,dc=mycompany</property> <property name="Manager Password">MOT DE PASSE</property> <property name="TLS - Keystore">/HDF/nifi-1.1.0.2.1.2.0-10/conf/certificat/keystore.jks</property> <property name="TLS - Keystore Password">MOT DE PASSE</property> <property name="TLS - Keystore Type">JKS</property> <property name="TLS - Truststore">/HDF/nifi-1.1.0.2.1.2.0-10/conf/certificat/truststore.jks</property> <property name="TLS - Truststore Password">MOT DE PASSE</property> <property name="TLS - Truststore Type">JKS</property> <property name="TLS - Client Auth">NONE</property> <property name="TLS - Protocol">TLS</property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">IGNORE</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://hostname:636</property> <property name="User Search Base">dc=mycompany</property> <property name="User Search Filter">uid={0}</property> <property name="Identity Strategy">USE_USERNAME</property> <property name="Authentication Expiration">12 hours</property> </provider> </loginIdentityProviders> I have already put the public key of my sercured LDAP server into my nifi keystore, i've tried many things that many post suggest and Each time I made changes and restarted NiFi, I deleted users.xml and authorizations.xml files before starting up NiFi again Thanks Toky ... View more

Hi, I have a three node HDF 2.0 unsecured cluster , and i would like to get some log file in another system. I originally intend to test this : SYSLOG ==> KAFKA ==> HDF ==> HDFS But i have read this : https://community.hortonworks.com/articles/56341/getting-started-with-minifi.html https://pierrevillard.com/tag/apache-minifi/ https://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.1/bk_minifi-quick-start/content/index.html And i was wondering maybe I can do this : ( MINIFI:HDF ) ==> KAFKA ==> HDF ==> HDFS My question are: IS MY USE CASE POSSIBLE WITH THE SECOND SCHEMAS ? AND IF IT IS HOW ? I have tried the Getting started with MiNiFi in the first link but when i replace the config.yml with my HDF template ( transform with minifi toolkit and renamed config.yml ). i get this from logs/minifi-app.log : ERROR [Thread-1] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi INFO [Thread-1] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise) I'm missing many things here and need some help. best regards, Toky ... View more

Hello everyone thanks for reading me, i'm very new to nifi as you can see in some of my previous question 🙂 I have an unsecure NIFI 1.0 cluster, i think i have configured the nifi.properties and the zookeeper.properties right but when i'm trying to use a REMOTE PROCESS GROUP , i always end up with this " NiFi Flow does not have any input ports " This is my nifi.properties for one of my three servers # Site to Site properties nifi.remote.input.host=server1 nifi.remote.input.secure=false nifi.remote.input.socket.port=8022 nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec # web properties nifi.web.war.directory=./lib nifi.web.http.host=server1 nifi.web.http.port=8070 nifi.web.https.host= nifi.web.https.port= nifi.web.jetty.working.directory=./work/jetty nifi.web.jetty.threads=200 # cluster node properties (only configure for cluster nodes) # nifi.cluster.is.node=true nifi.cluster.node.address=server1 nifi.cluster.node.protocol.port=8021 nifi.cluster.node.protocol.threads=10 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=5 sec nifi.cluster.node.read.timeout=5 sec nifi.cluster.firewall.file= nifi.cluster.flow.election.max.wait.time=5 mins nifi.cluster.flow.election.max.candidates=3 I have tried all configuration possible of the RPG with all the port here but I can't connect the RPG to a get file processor, I have read this https://community.hortonworks.com/articles/16461/nifi-understanding-how-to-use-process-groups-and-r.html and this https://community.hortonworks.com/questions/47516/nifi-rpg-and-clustering.html but those are for nifi inferiror version , can anyone explain me how to use the RPG without a NCM ? What i want to do is to get some file localy on one of my Nifi node by s2s but i think i'm missing something here Thanks for your time ... View more

Hi everyone, I'm trying to change this kind of timestamp: "2016-07-06T15:04:33.332+00:00" to this "2016/07/06" with apache Nifi, i have tried many things but failed miserably. This is my configuration of the ReplaceText processor Search Value = ${dbbf.timestamp_da} Replacement Value = ${bddf.timestamp_da:toDate("yyyy-MM-dd'T'HH:mm:ss.SSSZ"):format('yyyy/MM/dd')} Character Set = UTF-8 Maximum Buffer Size = 1 MB Replacement Strategy = Literal Replace Evaluation Mode = Entire text If Anyone can help i would really appreciate, Thanks in advance 🙂 ... View more

Hello everyone, I've start working with nifi last week so sorry if my question is an obvious one. I'm currentlty taking data in a Kafka Topic, those data are in Json format. I'm scaning every Json document and depending of the value of some fields, i would like to sort them in a directory with the name of the value. for example if i have : {"timestamp_da":"2016-07-06T15:04:33.332+00:00","type":"question","subject":"nifi"} {"timestamp_da":"2016-07-06T15:04:33.332+00:00","type":"question","subject":"Kafka"} ==> I would like to extract the timestamp value and convert it into "20160706" and then create a directory with it. After that every Json with the same timestamp will be route there. Is it possible ? or I'm I dreaming ? I initially was doing this with spark but i think it's a waste to use spark just for this. If my question wasn't clear enought i can give more details. Thanks in advance ... View more