Member since
06-29-2016
25
Posts
1
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
465 | 02-02-2018 12:31 PM | |
2959 | 05-18-2017 02:19 PM | |
1185 | 07-25-2016 07:41 AM |
11-29-2018
02:19 PM
Hi @scharan, thanks for your help, i really appreciate it. I've tried this two configuration the first one : [main]
anyofroles = org.apache.zeppelin.utils.AnyOfRolesAuthorizationFilter
ldapRealm.rolesByGroup = "cn=Group-admin,dc=domain,dc=company":"admin_role", "cn=Group-user,dc=domain,dc=company":"user_role"
[urls]
/api/interpreter/** = authc, anyofroles[admin_role, user_role] the second one: [main]
anyofrolesuser = org.apache.zeppelin.utils.AnyOfRolesUserAuthorizationFilter
ldapRealm.rolesByGroup = "cn=Group-admin,dc=domain,dc=company":"admin_role", "cn=Group-user,dc=domain,dc=company":"user_role"
[urls]
/api/interpreter/** = authc, anyofrolesuser[admin_role, user_role] but none of them changed anything, i still don't have any permission and any roles . /api/security/ticket} SecurityRestApi.java[ticket]:88) - {"status":"OK","message":"","body":{"principal":"user","ticket":"39f0b948-0f6d-4556-b447-c021ca34c8f6","roles":"[]"}} I've added my user on both LDAP group but didn't change anything either. But i think i'm close 🙂 Best reagrds, Toky
... View more
11-28-2018
03:02 PM
Hi everyone, I've been trying to find out if we can give to LDAP groups some roles ( admin and user ) but without succes. I'm puting down below my shiro config. It's working but when I'm logged in zeppelin as member of an admin group, the web UI is telling me that i don't have permission on the interpreter, credential, etc ... page. Anyone have a clue ? [users]
# List of users with their password allowed to access Zeppelin.
# To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISections
# To enable admin user, uncomment the following line and set an appropriate password.
#admin = password1, admin
#user1 = password2, role1, role2
#user2 = password3, role3
#user3 = password4, role2
[main]
ldapRealm=org.apache.zeppelin.realm.LdapRealm
ldapRealm.contextFactory.systemUsername=cn=UserBind,ou=Services,dc=domain,dc=company
ldapRealm.contextFactory.systemPassword=PassWord
ldapRealm.contextFactory.authenticationMechanism=simple
ldapRealm.contextFactory.url=ldap://ldap-server:389
ldapRealm.userDnTemplate =uid={0},ou=Users,dc=domain,dc=company
# Ability to set ldap paging Size if needed; default is 100
ldapRealm.pagingSize=200
ldapRealm.authorizationEnabled=true
ldapRealm.searchBase=dc=domain,dc=company
ldapRealm.userSearchBase=dc=domain,dc=company
ldapRealm.groupSearchBase=ou=Groups,dc=domain,dc=company
ldapRealm.userObjectClass=posixAccount
ldapRealm.groupObjectClass=posixGroup
ldapRealm.userSearchFilter = (&(objectclass=posixAccount)(uid={0}))
ldapRealm.memberAttribute=memberOf
ldapRealm.userSearchAttributeName = uid
ldapRealm.groupIdAttribute=cn
ldapRealm.userLowerCase = true
# ability set searchScopes subtree (default), one, base
ldapRealm.userSearchScope = subtree;
ldapRealm.groupSearchScope = subtree;
ldapRealm.groupSearchFilter = (&(objectclass=posixGroup)(memberOf={0}))
# Format to parse & search group member values in 'memberAttribute'
ldapRealm.memberAttributeValueTemplate=uid={0},ou=Groups,dc=domain,dc=company
# Map from physical AD groups to logical application roles
#ldapRealm.allowedRolesForAuthentication = admin_role,user_role
ldapRealm.rolesByGroup = "cn=Group-admin,dc=domain,dc=company":"admin_role", "cn=Group-user,dc=domain,dc=company":"user_role"
ldapRealm.permissionsByRole= admin_role:"*", user_role:"*:ToDoItemsJdo:*:*, *:ToDoItem:*:*"
ldapRealm.groupSearchEnableMatchingRuleInChain = true
# Force usernames returned from ldap to lowercase, useful for AD
#ldapRealm.userLowerCase = true
sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager
securityManager.sessionMode=native
securityManager.sessionManager.globalSessionTimeout = 86400000
securityManager.sessionManager = $sessionManager
##########################################
### A sample PAM configuration
#pamRealm=org.apache.zeppelin.realm.PamRealm
#pamRealm.service=sshd
##########################################
sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager
### If caching of user is required then uncomment below lines
#cacheManager = org.apache.shiro.cache.MemoryConstrainedCacheManager
#securityManager.cacheManager = $cacheManager
### Enables 'HttpOnly' flag in Zeppelin cookies
cookie = org.apache.shiro.web.servlet.SimpleCookie
cookie.name = JSESSIONID
cookie.httpOnly = true
### Uncomment the below line only when Zeppelin is running over HTTPS
#cookie.secure = true
sessionManager.sessionIdCookie = $cookie
#securityManager.sessionManager = $sessionManager
# 86,400,000 milliseconds = 24 hour
#securityManager.sessionManager.globalSessionTimeout = 86400000
shiro.loginUrl = /api/login
[roles]
#admin_role = *
#user_role = *:ToDoItemsJdo:*:*,*:ToDoItem:*:*
#role1 = *
#role2 = *
#role3 = *
#admin = *
[urls]
# This section is used for url-based security. For details see the shiro.ini documentation.
#
# You can secure interpreter, configuration and credential information by urls.
# Comment or uncomment the below urls that you want to hide:
# anon means the access is anonymous.
# authc means form based auth Security.
#
# IMPORTANT: Order matters: URL path expressions are evaluated against an incoming request
# in the order they are defined and the FIRST MATCH WINS.
#
# To allow anonymous access to all but the stated urls,
# uncomment the line second last line (/** = anon) and comment the last line (/** = authc)
#
#/api/version = anon
# Allow all authenticated users to restart interpreters on a notebook page.
# Comment out the following line if you would like to authorize only admin users to restart interpreters.
#/api/interpreter/setting/restart/** = authc
/api/interpreter/** = authc, roles[admin_role,user_role]
/api/configurations/** = authc, roles[admin_role]
/api/credential/** = authc, roles[admin_role]
#/** = anon
/** = authc
this is the log i get when i'm logging in, as you can see the roles part is blank /api/security/ticket} SecurityRestApi.java[ticket]:88) - {"status":"OK","message":"","body":{"principal":"user","ticket":"22faec60-d905-4757-899e-2a62b6ec7463","roles":"[]"} Best regards,
Toky
... View more
Labels:
- Labels:
-
Apache Zeppelin
02-02-2018
12:54 PM
Hi, Has anyone encounter this kind of error before ? Cluster Firelord downgrade is in progress. Cannot install packages. I found out the request_id is : 6703 , the URL of this request give me this information: "cluster_name" : "Firelord",
"completed_task_count" : 0,
"create_time" : 1517303769181,
"end_time" : 1517249417289,
"exclusive" : false,
"failed_task_count" : 0,
"id" : 6703,
"inputs" : null,
"operation_level" : null,
"progress_percent" : 100.0,
"queued_task_count" : 0,
"request_context" : "Downgrading from 2.5.3.0-37",
"request_schedule" : null,
"request_status" : "COMPLETED",
"resource_filters" : [ ],
"start_time" : 1517303770262,
"task_count" : 0,
"timed_out_task_count" : 0,
"type" : "INTERNAL_REQUEST" On my ambari database i executed this command :
DELETE FROM execution_command WHERE task_id IN (SELECT task_id FROM host_role_command WHERE request_id = 6703)
DELETE FROM host_role_command WHERE request_id = 6703 ; Base on this post : https://issues.apache.org/jira/browse/AMBARI-19055 but nothing changed. Can anyone give me a hand here ? Thanks, Toky
... View more
Labels:
02-02-2018
12:31 PM
For those who are interest in this matter, i checked my yum history and i had yum task that didn't go well. In this script : /var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py i changed one parameter Before:
# check package manager non-completed transactions
if self.pkg_provider.check_uncompleted_transactions():
self.pkg_provider.print_uncompleted_transaction_hint()
num_errors += 1
After
# check package manager non-completed transactions
if self.pkg_provider.check_uncompleted_transactions():
self.pkg_provider.print_uncompleted_transaction_hint()
num_errors += 0
then my packages smoothly installed, after that i changed it back .
... View more
01-25-2018
02:11 PM
Hi Everyone, I'm trying to Upgrade my HDP plateforme HDP-2.4.2.0 to HDP-2.5.3.0 but i get an ERROR which i'm not familiar with. My ambari version is ambari-2.6.1.0 because my HDP upgrade target is HDP-2.6.4.0-91, can anyone help me ? Thanks, Toky ERROR [ambari-client-thread-2923] BaseManagementHandler:61 - Caught a system exception while attempting to create a resource: Repositories for os type redhat7 are not defined for version 2.5.3.0-37 of Stack HDP-2.5.
org.apache.ambari.server.controller.spi.SystemException: Repositories for os type redhat7 are not defined for version 2.5.3.0-37 of Stack HDP-2.5.
at org.apache.ambari.server.controller.internal.ClusterStackVersionResourceProvider.getHostVersionInstallCommand(ClusterStackVersionResourceProvider.java:718)
at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:118)
at org.apache.ambari.server.controller.internal.ClusterStackVersionResourceProvider.createOrchestration(ClusterStackVersionResourceProvider.java:646)
at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:118)
at org.apache.ambari.server.controller.internal.ClusterStackVersionResourceProvider.createOrUpdateHostVersions(ClusterStackVersionResourceProvider.java:541)
at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:128)
at org.apache.ambari.server.controller.internal.ClusterStackVersionResourceProvider.createResourcesAuthorized(ClusterStackVersionResourceProvider.java:483)
at org.apache.ambari.server.controller.internal.AbstractAuthorizedResourceProvider.createResources(AbstractAuthorizedResourceProvider.java:220)
at org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:298)
at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:97)
at org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:37)
at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:73)
at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:144)
at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:126)
at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:90)
at org.apache.ambari.server.api.services.ClusterStackVersionService.createRequests(ClusterStackVersionService.java:121)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSe
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(Filter
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr
at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAu
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTransl
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagement
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(Anon
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilte
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCache
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr
at org.apache.ambari.server.security.authentication.AmbariDelegatingAuthenticationFilter.doFil
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr
at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(Amba
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(Security
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainPr
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:19
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.j
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:16
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHea
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.jav
at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.jav
at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:139)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:4
at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:984)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnecti
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:236)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:748)
... View more
Labels:
05-18-2017
02:19 PM
@Matt Clarke We find out, what was the problem it was an issue about the LDAP certtificate we generate. We changed it and now it work just fine anyway thanks for your time and your help 🙂 Toky
... View more
05-18-2017
09:09 AM
Hi @Matt Clarke <JAVA_HOME>/bin/java -Djavax.net.ssl.trustStore=/<path to NiFi truststore>/truststore SSLPoke hostname:663
i get this error Error: Could not find or load main class SSLPoke
In my LDAPS keystore there no sign of nifi public keys but in nifi truststore i can see the LDAPS server Key in nifi trustore I' don't think it' relevant but in my server JAVA isn't installed, i just do export JAVA HOME= /path to Java8. Thanks Toky
... View more
05-17-2017
05:14 PM
Thanks @Matt Clarke for your feedback. We use nifi-1.1.0.2.1.2.0-10 and i think it supports Authentication strategy "LDAPS".
In fact, about the url of LDAP, it was just a mistake in the post and not in the xml file.
Since the CA is one of nifi's node, we added the public key of LDAP server to the trustedCertEntry of all Nifi nodes but not the other way.
Do you think we really need to add all nifi nodes certificate into the LDAP server truststore ?
When i'm tailing LDAP log i don't see anything and whatever we change in the login-provider the ERROR log doesn't change when we try to log in. Thank
Toky
... View more
05-17-2017
10:25 AM
Hi everyone , Right now i'm trying to run my Nifi cluster using LDAP for many users. For each of my nodes have certificat and i have another one for my browser for admin stuff , which i have created with nfi-toolkit. Actually I'm able to connect to nifi with my admin account(I can add users, policies..).The problem accurs when a users tries to connect via LDAP. He gets this error: this are the log :
2017-05-16 16:48:13,550 ERROR [NiFi Web Server-126] o.a.n.w.a.c.AdministrationExceptionMapper org.apache.nifi.admin.service.AdministrationException: Unable to validate the supplied credentials. Please contact the system administrator.. Returning Internal Server Error response.
org.apache.nifi.admin.service.AdministrationException: Unable to validate the supplied credentials. Please contact the system administrator.
at org.apache.nifi.web.api.AccessResource.createAccessToken(AccessResource.java:445) ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_112]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_112]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_112]
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.jar:1.19]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66) [classes/:na]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:207) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE]
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE]
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51) [classes/:na]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1174) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1106) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:459) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.server.Server.handle(Server.java:524) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
Caused by: org.apache.nifi.authentication.exception.IdentityAccessException: Unable to validate the supplied credentials. Please contact the system administrator.
at org.apache.nifi.ldap.LdapProvider.authenticate(LdapProvider.java:340) ~[na:na]
at org.apache.nifi.web.security.spring.LoginIdentityProviderFactoryBean$1.authenticate(LoginIdentityProviderFactoryBean.java:301) ~[nifi-web-security-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
at org.apache.nifi.web.api.AccessResource.createAccessToken(AccessResource.java:437) ~[classes/:na]
... 58 common frames omitted
Caused by: org.springframework.security.authentication.InternalAuthenticationServiceException: hostname:636; nested exception is javax.naming.CommunicationException: hostame:636 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]
at org.springframework.security.ldap.authentication.LdapAuthenticationProvider.doAuthentication(LdapAuthenticationProvider.java:208) ~[na:na]
at org.springframework.security.ldap.authentication.AbstractLdapAuthenticationProvider.authenticate(AbstractLdapAuthenticationProvider.java:82) ~[na:na]
at org.apache.nifi.ldap.LdapProvider.authenticate(LdapProvider.java:310) ~[na:na]
... 60 common frames omitted
Caused by: org.springframework.ldap.CommunicationException: hostname:636; nested exception is javax.naming.CommunicationException: hostname:636 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]
at org.springframework.ldap.support.LdapUtils.convertLdapException(LdapUtils.java:108) ~[na:na]
at org.springframework.ldap.core.support.AbstractContextSource.createContext(AbstractContextSource.java:356) ~[na:na]
at org.springframework.ldap.core.support.AbstractContextSource.doGetContext(AbstractContextSource.java:140) ~[na:na]
at org.springframework.ldap.core.support.AbstractContextSource.getReadOnlyContext(AbstractContextSource.java:159) ~[na:na]
at org.springframework.ldap.core.LdapTemplate.executeReadOnly(LdapTemplate.java:802) ~[na:na]
at org.springframework.security.ldap.SpringSecurityLdapTemplate.searchForSingleEntry(SpringSecurityLdapTemplate.java:316) ~[na:na]
at org.springframework.security.ldap.search.FilterBasedLdapUserSearch.searchForUser(FilterBasedLdapUserSearch.java:126) ~[na:na]
at org.springframework.security.ldap.authentication.BindAuthenticator.authenticate(BindAuthenticator.java:94) ~[na:na]
at org.springframework.security.ldap.authentication.LdapAuthenticationProvider.doAuthentication(LdapAuthenticationProvider.java:189) ~[na:na]
... 62 common frames omitted
Caused by: javax.naming.CommunicationException: hostname:636
at com.sun.jndi.ldap.Connection.<init>(Connection.java:226) ~[na:1.8.0_112]
at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:137) ~[na:1.8.0_112]
at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1614) ~[na:1.8.0_112]
at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2746) ~[na:1.8.0_112]
at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:319) ~[na:1.8.0_112]
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:192) ~[na:1.8.0_112]
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:210) ~[na:1.8.0_112]
at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:153) ~[na:1.8.0_112]
at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:83) ~[na:1.8.0_112]
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684) ~[na:1.8.0_112]
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313) ~[na:1.8.0_112]
at javax.naming.InitialContext.init(InitialContext.java:244) ~[na:1.8.0_112]
at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154) ~[na:1.8.0_112]
at org.springframework.ldap.core.support.LdapContextSource.getDirContextInstance(LdapContextSource.java:42) ~[na:na]
at org.springframework.ldap.core.support.AbstractContextSource.createContext(AbstractContextSource.java:344) ~[na:na]
... 69 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.8.0_112]
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) ~[na:1.8.0_112]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) ~[na:1.8.0_112]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[na:1.8.0_112]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509) ~[na:1.8.0_112]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[na:1.8.0_112]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[na:1.8.0_112]
at sun.security.ssl.Handshaker.process_record(Handshaker.java:914) ~[na:1.8.0_112]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) ~[na:1.8.0_112]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) ~[na:1.8.0_112]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) ~[na:1.8.0_112]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) ~[na:1.8.0_112]
at com.sun.jndi.ldap.Connection.createSocket(Connection.java:376) ~[na:1.8.0_112]
at com.sun.jndi.ldap.Connection.<init>(Connection.java:203) ~[na:1.8.0_112]
... 83 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387) ~[na:1.8.0_112]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292) ~[na:1.8.0_112]
at sun.security.validator.Validator.validate(Validator.java:260) ~[na:1.8.0_112]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[na:1.8.0_112]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[na:1.8.0_112]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[na:1.8.0_112]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491) ~[na:1.8.0_112]
... 92 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[na:1.8.0_112]
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[na:1.8.0_112]
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[na:1.8.0_112]
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382) ~[na:1.8.0_112]
... 98 common frames omitted
And my login-identity-providers.xml <loginIdentityProviders>
<provider>
<identifier>ldap-provider</identifier>
<class>org.apache.nifi.ldap.LdapProvider</class>
<property name="Authentication Strategy">START_TLS</property>
<property name="Manager DN">cn=bindnifi,ou=Services,dc=mycompany</property>
<property name="Manager Password">MOT DE PASSE</property>
<property name="TLS - Keystore">/HDF/nifi-1.1.0.2.1.2.0-10/conf/certificat/keystore.jks</property>
<property name="TLS - Keystore Password">MOT DE PASSE</property>
<property name="TLS - Keystore Type">JKS</property>
<property name="TLS - Truststore">/HDF/nifi-1.1.0.2.1.2.0-10/conf/certificat/truststore.jks</property>
<property name="TLS - Truststore Password">MOT DE PASSE</property>
<property name="TLS - Truststore Type">JKS</property>
<property name="TLS - Client Auth">NONE</property>
<property name="TLS - Protocol">TLS</property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">IGNORE</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://hostname:636</property>
<property name="User Search Base">dc=mycompany</property>
<property name="User Search Filter">uid={0}</property>
<property name="Identity Strategy">USE_USERNAME</property>
<property name="Authentication Expiration">12 hours</property>
</provider>
</loginIdentityProviders>
I have already put the public key of my sercured LDAP server into my nifi keystore, i've tried many things that many post suggest and Each time I made changes and restarted NiFi, I deleted users.xml and authorizations.xml files before starting up NiFi again Thanks Toky
... View more
Labels:
- Labels:
-
Apache NiFi
02-02-2017
10:03 AM
Hi @Vineet, Yes this happened and keep happening after the install. I've found this : on this file /home/gpadmin/hawqAdminLogs/hawq_init_20170125.log. Port 5432 doesn't seems used And i have this installed on my node When i tried to initiate the standby master i have "hawq command not found" Toky
... View more
02-01-2017
02:41 PM
Hello, I have some problem to run my hawq cluster and i would like some help or advice right now I have this : I have kerberos on hadoop plateform ( just for information ) , when i try to launch my stanby master i get this : and when i tried to use my hawq cluster as gpadmin i get this : I thank you in adnvance for your time Toky
... View more
Labels:
- Labels:
-
Apache Hadoop
11-23-2016
10:26 AM
No i would like to take some file log with (Minifi and nifi ) then writte the raw file to kafka
... View more
11-22-2016
03:59 PM
Hi @Bryan Bende Tanks for the answer, I'll look into those link * this are my minifi extra log : INFO [main] org.apache.nifi.minifi.MiNiFi Controller initialization took 798231885 nanoseconds.
INFO [pool-1-thread-2] org.apache.nifi.BootstrapListener Received SHUTDOWN request from Bootstrap
INFO [pool-1-thread-2] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server...
INFO [pool-1-thread-2] o.apache.nifi.controller.FlowController Initiated graceful shutdown of flow controller...waiting up to 10 seconds
INFO [pool-1-thread-2] o.apache.nifi.controller.FlowController Controller has been terminated successfully.
INFO [pool-1-thread-2] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap
INFO [pool-1-thread-2] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise).
INFO [Thread-1] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server...
[Thread-1] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi
[Thread-1] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise). * in my template i have Getfile ==> Updateattibute ==> RPG and those attribute are available in MINIFI. * I've put KAFKA into the schemas just in case for bacjk up. best regards, Toky
... View more
11-22-2016
11:19 AM
Hello @Matt Just one more question, i have taileFile ==> RPG, the taileFiile is working but all the data are queued and doesn't pass through the RPG, do you know why ? Thanks Toky
... View more
11-22-2016
10:01 AM
Hi, I have a three node HDF 2.0 unsecured cluster , and i would like to get some log file in another system. I originally intend to test this : SYSLOG ==> KAFKA ==> HDF ==> HDFS But i have read this : https://community.hortonworks.com/articles/56341/getting-started-with-minifi.html https://pierrevillard.com/tag/apache-minifi/ https://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.1/bk_minifi-quick-start/content/index.html And i was wondering maybe I can do this : ( MINIFI:HDF ) ==> KAFKA ==> HDF ==> HDFS My question are: IS MY USE CASE POSSIBLE WITH THE SECOND SCHEMAS ? AND IF IT IS HOW ? I have tried the Getting started with MiNiFi in the first link but when i replace the config.yml with my HDF template ( transform with minifi toolkit and renamed config.yml ). i get this from logs/minifi-app.log :
ERROR [Thread-1] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi INFO [Thread-1] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise) I'm missing many things here and need some help. best regards, Toky
... View more
Labels:
- Labels:
-
Apache MiNiFi
-
Apache NiFi
11-21-2016
08:58 AM
Hello @Matt Thanks for your precious help, it's working just fine 🙂
... View more
11-18-2016
08:49 PM
Thanks Matt for your help sorry i' can't test it right now ( it's a bit late where i am) but when i''ve done it i'will inform you right Away . Just To be clear i get it right, in the option URL i put the URL of the target and set an input ports on The Ui of the target and then this input Will appear on the RPG (remote ports) If i choose raw it will be on port 8022 and 8070 if i choose http. Feel free to correct me if i made mistake. Toky
... View more
11-18-2016
06:25 PM
Question 1 : yes this configs are from the system i'm trying to send data ( for the record all my system with nifi have this configs) Question 2 : i don't undrstand your question Question 3 : you are correct i was able to enable the transmision but when i go to remote ports idon't find any Toky
... View more
11-18-2016
05:35 PM
Hello everyone thanks for reading me, i'm very new to nifi as you can see in some of my previous question 🙂 I have an unsecure NIFI 1.0 cluster, i think i have configured the nifi.properties and the zookeeper.properties right but when i'm trying to use a REMOTE PROCESS GROUP , i always end up with this " NiFi Flow does not have any input ports " This is my nifi.properties for one of my three servers # Site to Site properties
nifi.remote.input.host=server1
nifi.remote.input.secure=false
nifi.remote.input.socket.port=8022
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec # web properties
nifi.web.war.directory=./lib
nifi.web.http.host=server1
nifi.web.http.port=8070
nifi.web.https.host=
nifi.web.https.port=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200 # cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=server1
nifi.cluster.node.protocol.port=8021
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=5 mins
nifi.cluster.flow.election.max.candidates=3 I have tried all configuration possible of the RPG with all the port here but I can't connect the RPG to a get file processor, I have read this https://community.hortonworks.com/articles/16461/nifi-understanding-how-to-use-process-groups-and-r.html and this https://community.hortonworks.com/questions/47516/nifi-rpg-and-clustering.html but those are for nifi inferiror version , can anyone explain me how to use the RPG without a NCM ? What i want to do is to get some file localy on one of my Nifi node by s2s but i think i'm missing something here Thanks for your time
... View more
Labels:
- Labels:
-
Apache NiFi
07-25-2016
07:41 AM
1 Kudo
I appreciate the help and thank you for that. I have found the format of the timestamp which is "yyyy-MM-dd'T'HH:mm:ss.SSSXXX" it was a lot easier after that.
... View more
07-11-2016
10:04 AM
Hi everyone, I'm trying to change this kind of timestamp:
"2016-07-06T15:04:33.332+00:00" to this "2016/07/06" with apache Nifi, i have tried
many things but failed miserably. This is my configuration of the ReplaceText processor Search Value = ${dbbf.timestamp_da}
Replacement Value = ${bddf.timestamp_da:toDate("yyyy-MM-dd'T'HH:mm:ss.SSSZ"):format('yyyy/MM/dd')} Character Set = UTF-8
Maximum Buffer Size = 1 MB
Replacement Strategy = Literal Replace
Evaluation Mode = Entire text If Anyone can help i would really appreciate, Thanks in advance 🙂
... View more
Labels:
- Labels:
-
Apache NiFi
07-08-2016
03:58 PM
Thank you very much for your Help sir and have a good day or night depend on where you are 🙂
... View more
07-08-2016
02:33 PM
Hello everyone, I've start working with nifi last week so sorry if my question is an obvious one. I'm currentlty taking data in a Kafka Topic, those data are in Json format.
I'm scaning every Json document and depending of the value of some fields, i would like
to sort them in a directory with the name of the value. for example if i have : {"timestamp_da":"2016-07-06T15:04:33.332+00:00","type":"question","subject":"nifi"} {"timestamp_da":"2016-07-06T15:04:33.332+00:00","type":"question","subject":"Kafka"} ==> I would like to extract the timestamp value and convert it into "20160706" and then create a directory with it. After that every Json with the same timestamp will be route there.
Is it possible ? or I'm I dreaming ? I initially was doing this with spark but i think it's a waste to use spark just for this.
If my question wasn't clear enought i can give more details. Thanks in advance
... View more
Labels:
- Labels:
-
Apache NiFi