Member since
01-29-2018
6
Posts
0
Kudos Received
0
Solutions
06-20-2019
09:28 AM
Getting below error while Start making an entry for Version Control in NiFi Registry: Version Details: HDF - 3.3.0 NiFi - 1.8.0 NiFi Registry - 0.3.0 NiFi Console Log: Unable to obtain listing of buckets: org.apache.nifi.registry.client.NiFiRegistryException: Error retrieving all buckets: Untrusted proxy [CN=node1.domain.com, OU=IT, O=COMPANY, L=Area, ST=State, C=US]. Contact the system administrator. NiFi Registry Log: 2019-06-19 06:04:03,248 INFO [NiFi Registry Web Server-18] o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos service ticket login not supported by this NiFi Registry. Returning Conflict response.
2019-06-19 06:04:03,620 INFO [NiFi Registry Web Server-17] o.a.n.r.w.s.NiFiRegistrySecurityConfig Client could not be authenticated due to: org.springframework.security.authentication.AuthenticationCredentialsNotFoundException: An Authentication object was not found in the SecurityContext Returning 401 response.
2019-06-19 06:22:49,013 INFO [NiFi Registry Web Server-63] o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos service ticket login not supported by this NiFi Registry. Returning Conflict response.
2019-06-19 06:22:49,263 INFO [NiFi Registry Web Server-17] o.a.n.r.w.s.NiFiRegistrySecurityConfig Client could not be authenticated due to: org.springframework.security.authentication.AuthenticationCredentialsNotFoundException: An Authentication object was not found in the SecurityContext Returning 401 response.
2019-06-19 06:27:01,720 INFO [NiFi Registry Web Server-68] o.a.n.r.w.s.NiFiRegistrySecurityConfig Identity in proxy chain not trusted to act as a proxy: org.apache.nifi.registry.web.security.authentication.exception.UntrustedProxyException: Untrusted proxy [CN=node1.domain.com, OU=IT, O=COMPANY, L=Area, ST=State, C=US]. Returning 403 response.
2019-06-19 06:28:10,402 INFO [NiFi Registry Web Server-68] o.a.n.r.w.s.NiFiRegistrySecurityConfig Identity in proxy chain not trusted to act as a proxy: org.apache.nifi.registry.web.security.authentication.exception.UntrustedProxyException: Untrusted proxy [CN=node1.domain.com, OU=IT, O=COMPANY, L=Area, ST=State, C=US]. Returning 403 response. Configuration: SSL - For both NiFi and NiFi Registry, we are using a host-specific self-signed certificate. And both services refer to the same Java Keystores. Authentication Used - LDAP Kerberos - No Ranger Plugin to NiFi - Yes NiFi Service: Authorization - Ranger Initial Admin Identity - admin nifi.security.user.login.identity.provider - ldap-provider Node Identities <property name="Node Identity 1">CN=node1.domain.com, OU=IT, O=COMPANY, L=Area, ST=State, C=US</property>
<property name="Node Identity 2">CN=node2.domain.com, OU=IT, O=COMPANY, L=Area, ST=State, C=US</property> NiFi Registry Service: Initial Admin Identity CN=admin, OU=Hadoop, OU=Accounts-Service, DC=Domain, DC=com nifi.registry.security.identity.provider=ldap-identity-provider NiFi Identities <property name="NiFi Identity 1">CN=node1.domain.com, OU=IT, O=COMPANY, L=Area, ST=State, C=US</property>
<property name="NiFi Identity 2">CN=node2.domain.com, OU=IT, O=COMPANY, L=Area, ST=State, C=US</property> authorizers.xml <userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
<property name="Users File">{{nifi_registry_internal_config_dir}}/users.xml</property>
<property name="Initial User Identity 1">CN=node1.domain.com, OU=IT, O=COMPANY, L=Area, ST=State, C=US</property>
<property name="Initial User Identity 2">CN=node2.domain.com, OU=IT, O=COMPANY, L=Area, ST=State, C=US</property>
<property name="Initial User Identity 3">CN=admin, OU=Hadoop, OU=Accounts-Service, DC=Domain, DC=com</property>
{{nifi_registry_ssl_config_content | replace("NiFi","Initial User")}}
</userGroupProvider> <accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">{{nifi_registry_internal_config_dir}}/authorizations.xml</property>
<property name="Initial Admin Identity">CN=admin, OU=Hadoop, OU=Accounts-Service, DC=Domain, DC=com</property>
{{nifi_registry_ssl_config_content}}
</accessPolicyProvider> Note: I am able to login to both the services using LDAP user credentials. Also, created Ranger Policy: /proxy - Read & Write - CN=node1.domain.com, OU=IT, O=COMPANY, L=Area, ST=State, C=US Read & Write - CN=node2.domain.com, OU=IT, O=COMPANY, L=Area, ST=State, C=US Kindly help me to resolve this issue.
... View more
Labels:
- Labels:
-
Apache NiFi
08-06-2018
11:22 AM
We need to take backup all the topics in Kafka to the file named in respective topic names and need to restore the topic as per user requirement. Note: This script needs to be run in the Kerberized environment. kafkabackup.sh Making required directories monyear=`date | awk '{print $2$6}'`
dat=`date| awk '{print $2$3$6}'`
export BACKUPDIR=/root/backup/$monyear
mkdir -p $BACKUPDIR
mkdir -p $BACKUPDIR/$dat
cd $BACKUPSDIR
BKDIR=$BACKUPDIR/$dat
Log into Kafka Get topics from Kafka Broker kinit -kt /etc/security/keytabs/kafka.service.keytab kafka/node1.localdomaino@domain.co
cd /usr/hdp/current/kafka-broker/bin/
export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=/etc/kafka/conf/kafka_client_jaas.conf"
./kafka-topics.sh --zookeeper adminnode.localdomain:2181 --list > $BKDIR/listtopics.txt
Remove if any mark of deletion topics exists sed -i.bak '/deletion/d' $BKDIR/listtopics.txt Starting kill script in parallel bash checkandkill.sh& Reading the file contents for topics for line in $(cat $BKDIR/listtopics.txt)
do
echo $line
./test.sh --bootstrap-server node1.localdomain:6668 --topic $line --consumer.config /home/kafka/conf.properties --from-beginning --security-protocol SASL_SSL > $BKDIR/$line
done
Delete empty files /usr/bin/find . -size 0 -delete Killing checkandkill daemon and exit ps -ef |grep -i checkandkill.sh| grep -v grep | awk '{print $2}' | xargs kill
exit
When consumer runs, it constantly waits for messages to receive. We need to kill the process. checkandkill.sh sleep 0.5m
for line in $(cat /root/backup/listtopics.txt)
do
echo $line
sleep 1m
ps -ef |grep -i $line| grep -v grep | awk '{print $2}' | xargs kill
done Need your help to complete restoration script.
... View more
Labels:
- Labels:
-
Apache Kafka
01-29-2018
02:32 PM
When I save the workflow, it shows the below error.
Error occurred while saving workflow
Access Error to file due to access control
org.apache.oozie.ambari.view.exception.WfmWebException at org.apache.oozie.ambari.view.OozieProxyImpersonator.saveWorkflow(OozieProxyImpersonator.java:257) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:287) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authentication.AmbariDelegatingAuthenticationFilter.doFilter(AmbariDelegatingAuthenticationFilter.java:132) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.view.AmbariViewsMDCLoggingFilter.doFilter(AmbariViewsMDCLoggingFilter.java:54) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.view.ViewThrottleFilter.doFilter(ViewThrottleFilter.java:161) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:125) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:125) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:212) at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:201) at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:150) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:984) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1045) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:236) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.security.authorize.AuthorizationException: Unauthorized connection for super-user: root from IP 192.168.0.56 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:519) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:497) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:114) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathOutputStreamRunner$1.close(WebHdfsFileSystem.java:950) at org.apache.oozie.ambari.view.HDFSFileUtils.writeToFile(HDFSFileUtils.java:89) at org.apache.oozie.ambari.view.WorkflowFilesService.createFile(WorkflowFilesService.java:42) at org.apache.oozie.ambari.view.OozieProxyImpersonator.saveWorkflowXml(OozieProxyImpersonator.java:274) at org.apache.oozie.ambari.view.OozieProxyImpersonator.saveWorkflow(OozieProxyImpersonator.java:244) ... 97 more Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Unauthorized connection for super-user: root from IP 192.168.0.56 at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:119) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:480) ... 103 more<br>
... View more
Labels:
- Labels:
-
Apache Oozie