Problem When trying to import larger notebooks into Zeppelin, the import does nothing. Checking the /var/log/zeppelin/ logs, it is apparent that this is a jetty websocket text size issue. WARN [2017-03-13 14:22:30,427] ({qtp110945054-15} Parser.java[notifyWebSocketException]:235) - org.eclipse.jetty.websocket.api.MessageTooLargeException: Text message size [2408484] exceeds maximum size [1024000] at org.eclipse.jetty.websocket.api.WebSocketPolicy.assertValidTextMessageSize(WebSocketPolicy.java:140) at org.eclipse.jetty.websocket.common.Parser.assertSanePayloadLength(Parser.java:127) at org.eclipse.jetty.websocket.common.Parser.parseFrame(Parser.java:482) at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:254) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) INFO [2017-03-13 14:22:30,428] ({qtp110945054-15} NotebookServer.java[onClose]:217) - Closed connection to 84.20.132.89 : 24324. (1009) Text message size [2408484] exceeds maximum size [1024000] Solution Steps to resolve the issue - go to Ambari UI -> Zeppelin -> Configs -> Advanced zeppelin-config - increase the value of zeppelin.websocket.max.text.message.size to i.e. 3072000 - save the changes - restart required services ... View more

Problem When I run the below code on Pig View (HDP 2.5.3 and Ambari 2.4.2) in Kerberos enabled REGISTER '/usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-pig-adapter.jar'; a = LOAD 'test123.foo' USING org.apache.hive.hcatalog.pig.HCatLoader(); dump a; I get the following errors - PIG log ... 07 Mar 2017 12:22:32,971 ERROR [ambari-client-thread-28] [PIG 1.0.0 Pig] JobService:232 - Exception occurred : java.io.FileNotFoundException: File /user/<userID>/pig/jobs/pighivetest_07-03-2017-12-22-15/stderr not found. ... Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File /user/<userID>/pig/jobs/pighivetest_07-03-2017-12-22-15/stderr not found. ... - Application log ... 2017-03-07 12:22:45,223 [main] ERROR org.apache.thrift.transport.TSaslTransport - SASL negotiation failure javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) ... Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) ... Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: GSS initiate failed ... 2017-03-07 12:24:45,466 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2245: Cannot get schema from loadFunc org.apache.hive.hcatalog.pig.HCatLoader Details at logfile: /hadoop/1/hadoop/yarn/local/usercache/<userID>/appcache/application_1488824278698_2495/container_e187_1488824278698_2495_01_000002/pig_1488889361907.log ... Interestingly, the same job runs perfectly fine while running from pig command line. Solution In Pig View, under the script section, there is Arguments one to add pig properties. To fix the problem, add "-useHCatalog" like presented below. ... View more

Problem Here is my user I run %livy.sparkr with %sh whoami admin Permissions set to directory and file: %sh ls -lrt /home/admin total 12 -rwxr-xr-x 1 admin admin 3950 Mar 9 13:12 common.r drwxrwxr-x 2 admin admin 4096 Mar 10 15:25 analysis drwxrwxr-x 3 admin admin 4096 Mar 13 12:49 motorAnalysis %sh ls -lrt /home/admin/motorAnalysis total 8 -rwxrwxr-x 1 admin admin 3950 Mar 10 16:49 common.r drwxrwxr-x 5 admin admin 4096 Mar 10 16:49 data Running %livy.sparkr I am getting %livy.sparkr ... source("/home/admin/motorAnalysis/common.r") null device 1 Error in file(filename, "r", encoding = encoding) : cannot open the connection Calls: source -> file In addition: Warning message: In file(filename, "r", encoding = encoding) : cannot open file '/home/admin/motorAnalysis/common.r': No such file or directory Solution Restart livy interpreter from Zeppelin UI fixes the problem ... View more

Problem I have my shiro configured as [users] # List of users with their password allowed to access Zeppelin. # To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISections #admin = password1 admin = change #user1 = password2, role1, role2 #user2 = password3, role3 #user3 = password4, role2 # Sample LDAP configuration, for user Authentication, currently tested for single Realm [main] ldapRealm = org.apache.zeppelin.server.LdapGroupRealm ldapRealm.contextFactory.environment[ldap.searchBase] = ou=my,ou=company,o=com ldapRealm.userDnTemplate = uid={0},ou=my,ou=company,o=com ldapRealm.contextFactory.url = ldap://<ldap_host>:389 ldapRealm.contextFactory.authenticationMechanism = SIMPLE # doc horton ldapRealm.contextFactory.systemUsername = <system_username_bind> ldapRealm.contextFactory.systemPassword = <system_username_password> #sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager #securityManager.sessionManager = $sessionManager # 86,400,000 milliseconds = 24 hour #securityManager.sessionManager.globalSessionTimeout = 86400000 shiro.loginUrl = /api/login [urls] # anon means the access is anonymous. # authcBasic means Basic Auth Security # To enfore security, comment the line below and uncomment the next one /api/version = anon #/** = anon /** = authc After logging into zeppelin UI and running scripts from Notebook I can see the following error in zeppelin log WARN [2017-02-28 09:20:09,062] ({qtp1918627686-14} ServletHandler.java[doHandle]:620) - javax.servlet.ServletException: Filtered request failed. at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:384) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.apache.zeppelin.server.CorsFilter.doFilter(CorsFilter.java:72) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NoSuchMethodError: javax.ws.rs.ClientErrorException.validate(Ljavax/ws/rs/core/Response;Ljavax/ws/rs/core/Response$Status$Family;)Ljavax/ws/rs/core/Response; at javax.ws.rs.ClientErrorException.<init>(ClientErrorException.java:88) at org.apache.cxf.jaxrs.utils.JAXRSUtils.findTargetMethod(JAXRSUtils.java:503) at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:207) at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.handleMessage(JAXRSInInterceptor.java:103) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272) at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239) at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:248) at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:222) at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:153) at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:167) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:211) at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) ... 22 more Solution To solve the problem enable sessionManager, cacheManager and securityManager attributes in shiro config so it looks like [users] # List of users with their password allowed to access Zeppelin. # To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISections #admin = password1 admin = change #user1 = password2, role1, role2 #user2 = password3, role3 #user3 = password4, role2 # Sample LDAP configuration, for user Authentication, currently tested for single Realm [main] ldapRealm = org.apache.zeppelin.server.LdapGroupRealm ldapRealm.contextFactory.environment[ldap.searchBase] = ou=my,ou=company,o=com ldapRealm.userDnTemplate = uid={0},ou=my,ou=company,o=com ldapRealm.contextFactory.url = ldap://<ldap_host>:389 ldapRealm.contextFactory.authenticationMechanism = SIMPLE # doc horton ldapRealm.contextFactory.systemUsername = <system_username_bind> ldapRealm.contextFactory.systemPassword = <system_username_password> sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager cacheManager = org.apache.shiro.cache.MemoryConstrainedCacheManager securityManager.realm = $ldapRealm securityManager.cacheManager = $cacheManager securityManager.sessionManager = $sessionManager securityManager.sessionManager.globalSessionTimeout = 86400000 shiro.loginUrl = /api/login [urls] # anon means the access is anonymous. # authcBasic means Basic Auth Security # To enfore security, comment the line below and uncomment the next one /api/version = anon #/** = anon /** = authc ... View more

Problem When running $ ambari-server restart I can see "tones" of these messages in ambari-server.log 27 Feb 2017 11:38:00,942 WARN [main] AbstractLifeCycle:204 - FAILED SelectChannelConnector@0.0.0.0:8080: java.net.BindException: Address already in use java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:463) at sun.nio.ch.Net.bind(Net.java:455) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316) at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.eclipse.jetty.server.Server.doStart(Server.java:293) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:616) at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:925) 27 Feb 2017 11:38:00,942 WARN [main] AbstractLifeCycle:204 - FAILED SelectChannelConnector@0.0.0.0:8080: java.net.BindException: Address already in use java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:463) at sun.nio.ch.Net.bind(Net.java:455) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316) at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.eclipse.jetty.server.Server.doStart(Server.java:293) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:616) at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:925) 27 Feb 2017 11:38:00,944 WARN [main] AbstractLifeCycle:204 - FAILED org.eclipse.jetty.server.Server@5c0ecfc4: java.net.BindException: Address already in use java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:463) at sun.nio.ch.Net.bind(Net.java:455) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316) at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) Solution On AWS Cloudbreak starts the instances with Amazon Linux. On Amazon-linux Upstart is used for system init. To restart ambari server with upstart you have to use "initctl" command like: $ initctl restart ambari-server If you have ambari server already running, do the following after logging in to ambari server node as root: a) $ ambari-server stop b) $ ps -ef | grep ambari-server c) kill all the outstanding ambari-server's processes d) backup and remove all logs from /var/log/ambari-server e) $ initctl start ambari-server f) tail -f /var/log/ambari-server/ambari-server.log Now, there is no more "Address already in use" errors in ambari-server.log. ... View more

Problem Interpreters do not work through zeppelin with no internet access. Checking out /usr/hdp/current/zeppelin-server/local-repo - the folder is empty or contains “org” folder with no jars Running hive through Zeppelin returns %jdbc(hive) show tables; ... org.apache.hive.jdbc.HiveDriver class java.lang.ClassNotFoundException ... Solution The permanent solution is currently planned to be delivered in HDP 2.6. Here is a workaround to follow: a) I have tarred /usr/hdp/current/zeppelin-server/local-repo into zeppelin-local-repo.tar.gz and located on https://drive.google.com/drive/folders/0B-YVWxQz56HubWhUdEdVWGZ1Mms?usp=sharing. As this is my google drive I can allow you to access it after receiving the request from you b) download the file into /tmp/zeppelin folder c) extract it - this should created local-repo folder. So, you will get /tmp/zeppelin/local-repo with all the subfolders in it d) copy the content of /tmp/zeppelin/local-repo into /usr/hdp/current/zeppelin-server/local-repo e) change the owner of local-repo and all its folders/file to zeppelin:hadoop (or zeppelin:hdfs <- whatever the GROUP you have) f) change the permissions local-repo and all its folders/file to 755 g) restart Zeppelin service NOTE: step a) is something you can do yourself. Just install the environment on a temporary machine with internet access and get the content of /usr/hdp/current/zeppelin-server/local-repo from there. ... View more

Problem I log in to Ambari as numeric username (or partially numeric) i.e. 123user, 12345678. When accessing Hive View the following comes up Here is the full stack of the error: Service 'userhome' check failed: java.lang.IllegalArgumentException: Invalid value: "12345678" does not belong to the domain ^[A-Za-z_][A-Za-z0-9._-]*[$]?$ at org.apache.hadoop.hdfs.web.resources.StringParam$Domain.parse(StringParam.java:53) at org.apache.hadoop.hdfs.web.resources.StringParam.<init>(StringParam.java:25) at org.apache.hadoop.hdfs.web.resources.UserParam.<init>(UserParam.java:68) at org.apache.hadoop.hdfs.web.resources.UserParam.<init>(UserParam.java:75) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHomeDirectory(WebHdfsFileSystem.java:387) at org.apache.ambari.view.utils.hdfs.HdfsApi$7.run(HdfsApi.java:187) at org.apache.ambari.view.utils.hdfs.HdfsApi$7.run(HdfsApi.java:185) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.ambari.view.utils.hdfs.HdfsApi.execute(HdfsApi.java:397) at org.apache.ambari.view.utils.hdfs.HdfsApi.getHomeDir(HdfsApi.java:185) at org.apache.ambari.view.commons.hdfs.UserService.homeDir(UserService.java:57) at org.apache.ambari.view.hive2.resources.files.FileService.userhomeSmokeTest(FileService.java:245) at org.apache.ambari.view.hive2.HelpService.userhomeStatus(HelpService.java:89) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:257) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.jwt.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:96) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) at org.apache.ambari.server.security.authentication.AmbariAuthenticationFilter.doFilter(AmbariAuthenticationFilter.java:88) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.view.AmbariViewsMDCLoggingFilter.doFilter(AmbariViewsMDCLoggingFilter.java:54) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.view.ViewThrottleFilter.doFilter(ViewThrottleFilter.java:161) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:109) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:109) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:212) at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:201) at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:150) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:641) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:231) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Solution This is a known issue. And the permanent solution for this problem has been scheduled for delivery in Ambari 2.5 Workaround As for the workaround, please follow up the below steps: NOTE: these are the exact steps I did follow on my local environment (HDP 2.5.0.0-1245, Ambari 2.4.0.1). a) Copy /var/lib/ambari-server/resources/views/work/HIVE\{1.5.0\}/WEB-INF/lib/hadoop-common-2.7.3.2.5.0.0-1245.jar into temp dir: /tmp b) unzip /tmp/hadoop-common-2.7.3.2.5.0.0-1245.jar c) Extract the core-default.xml file from the above jar file and add the following entry in it: <property>   <name>dfs.webhdfs.user.provider.user.pattern</name>   <value>^[A-Za-z0-9_][A-Za-z0-9._-]*[$]?$</value>   <description>     Valid pattern for user and group names for webhdfs, it must be a valid java regex.   </description> </property> d) Now create a "classes" directory in /var/lib/ambariserver/resources/views/work/HIVE\{1.5.0\}/WEB-INF and move the edited core-default.xml inside it. Example: $ cd /var/lib/ambari-server/resources/views/work/HIVE{1.5.0}/WEB-INF $ mkdir classes $ cp -f /tmp/core-default.xml /var/lib/ambari-server/resources/views/work/HIVE\{1.5.0\}/WEB-INF/classes/ e) Then restart the ambari-server. $ ambari-server restart Now, I can see it is working fine for user 12345678 ... View more

Environment HDP 2.5.x Ambari 2.4.x Problem I need to use anaconda for %livy.pyspark. Now, it is using the default python2.6 %livy.pyspark import sys print(sys.path) ----------------------------------------- ['/var/hadoop/b/yarn/local/usercache/<user>/appcache/application_1483612761447_100542/container_e11_1483612761447_100542_01_000001/tmp', u'/var/hadoop/b/yarn/local/usercache/<user>/appcache/application_1483612761447_100542/spark-0e25c417-e8c0-4b60-b167-789dc3293bd7/userFiles-a1d6eced-0de1-4e3d-9bc1-f5aea925915d/py4j-0.9-src.zip', u'/var/hadoop/b/yarn/local/usercache/<user>/appcache/application_1483612761447_100542/spark-0e25c417-e8c0-4b60-b167-789dc3293bd7/userFiles-a1d6eced-0de1-4e3d-9bc1-f5aea925915d/pyspark.zip', u'/var/hadoop/b/yarn/local/usercache/<user>/appcache/application_1483612761447_100542/spark-0e25c417-e8c0-4b60-b167-789dc3293bd7/userFiles-a1d6eced-0de1-4e3d-9bc1-f5aea925915d', '/var/hadoop/b/yarn/local/usercache/<user>/appcache/application_1483612761447_100542/container_e11_1483612761447_100542_01_000001/pyspark.zip', '/var/hadoop/b/yarn/local/usercache/<user>/appcache/application_1483612761447_100542/container_e11_1483612761447_100542_01_000001/py4j-0.9-src.zip', '/usr/lib64/python26.zip', '/usr/lib64/python2.6', '/usr/lib64/python2.6/plat-linux2', '/usr/lib64/python2.6/lib-tk', '/usr/lib64/python2.6/lib-old', '/usr/lib64/python2.6/lib-dynload', '/usr/lib64/python2.6/site-packages', '/usr/lib64/python2.6/site-packages/gtk-2.0', '/usr/lib/python2.6/site-packages'] How can I get this working with python-3.5? Solution Based on https://issues.apache.org/jira/browse/ZEPPELIN-1609, there is a new pyspark implemented within livy - %livy.pyspark3. This is delivered in Zeppelin 0.7 that comes with HDP 2.6. For now, do the following: > go to Ambari UI -> Spark -> Config -> Advanced livy-env -> content -> set: export PYSPARK_PYTHON=/opt/anaconda/bin/python - path to new python version -> set: export PYSPARK_DRIVER_PYTHON=/opt/anaconda/bin/python - path to new python version > save the changes > Restart all required services ... View more

Overview R is currently supported through livy interpreter by running %livy.sparkr Environment I did test the below solution with HDP 2.5.0.0-1245 and Ambari 2.4.0.1 HDP 2.5.3.0-37 and Ambari 2.4.2.0 Steps To Follow 1. Install R on all YARN worker nodes $ yum install R-devel libcurl-devel openssl-devel 2. Confirm R works fine [root@dkhdp251 zeppelin]# R -e "print(1+1)" R version 3.3.2 (2016-10-31) -- "Sincere Pumpkin Patch" Copyright (C) 2016 The R Foundation for Statistical Computing Platform: x86_64-redhat-linux-gnu (64-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under certain conditions. Type 'license()' or 'licence()' for distribution details. Natural language support but running in an English locale R is a collaborative project with many contributors. Type 'contributors()' for more information and 'citation()' on how to cite R or R packages in publications. Type 'demo()' for some demos, 'help()' for on-line help, or 'help.start()' for an HTML browser interface to help. Type 'q()' to quit R. > print(1+1) [1] 2 3. Confirm R works fine from Zeppelin 4. Install the following packages from Zeppelin: devtools, data.table, base64env and knitr %livy.sparkr install.packages('data.table', repos = 'http://cran.us.r-project.org') install.packages('base64enc', repos = 'http://cran.us.r-project.org') install.packages('knitr', repos = 'http://cran.us.r-project.org') install.packages('ggplot2', repos = 'http://cran.us.r-project.org') 5. Confirm the packages have been installed %livy.sparkr library(data.table) library(base64enc) library(knitr) library(ggplot2) The above should run and finish returning nothing in output. 6. Run the following code to build the graphics %livy.sparkr library(data.table) library(ggplot2) library(knitr) set.seed(42) # generate sample data dat <- rbind(data.table(gender="female",value=rnorm(1e4)), data.table(gender="male",value=rnorm(1e4,2,1)) ) # plot p1 <- ggplot(dat,aes(x=value,color=gender)) + geom_density() # save to tmp file ggsave(filename="/tmp/myplot.png", plot=p1) # get base64 of the image for display in html printImageURI<-function(file){ uri=image_uri(file) file.remove(file) cat(sprintf("%%html <img width='700' src=\"%s\" />\n", uri)) } printImageURI("/tmp/myplot.png") This will generate the following graphics ... View more