Member since
09-18-2015
216
Posts
208
Kudos Received
49
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1303 | 09-13-2017 06:04 AM | |
| 2536 | 06-27-2017 06:31 PM | |
| 2452 | 06-27-2017 06:27 PM | |
| 10123 | 11-04-2016 08:02 PM | |
| 9782 | 05-25-2016 03:42 PM |
02-12-2016
03:54 AM
1 Kudo
It's mandatory to restart system after you disable THP and then you can start services on the machine. Start from Ambari Server and do it on all other nodes one by one. Plan for maintenance downtime for Non HA components i.e. MapReduce2.. And while you are doing it, it might be good idea to revisit other tunings as well so that you have everything done in a go i.e. Swappiness, IPv6 etc.
... View more
02-09-2016
10:34 PM
1 Kudo
@Neeraj Sabharwal Pig View works if I point webhdfs specifically to Active Namenode URI but it fails when I put Nameservice Uri for Namenode HA which is hdpqa in my case.
... View more
02-09-2016
01:59 PM
Stack Trace
java.net.UnknownHostException: hdpqa
java.net.UnknownHostException: hdpqa
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1168)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1104)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:998)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:932)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.connect(WebHdfsFileSystem.java:513)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.connect(WebHdfsFileSystem.java:506)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:595)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:530)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$OffsetUrlOpener.connect(WebHdfsFileSystem.java:839)
at org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:119)
at org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103)
at org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:187)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.ambari.view.pig.utils.FilePaginator.readPage(FilePaginator.java:88)
at org.apache.ambari.view.pig.resources.files.FileService.getFile(FileService.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:182)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:209)
at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:198)
at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:145)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
-
Apache Pig
-
HDFS
-
Security
01-13-2016
04:27 PM
2 Kudos
I have restarted Ambari Server and all agents along with complete HDP stack multiple times in past 5 days for different activities but these alerts don't go away.
... View more
Labels:
- Labels:
-
Apache Ambari
01-13-2016
04:26 AM
2 Kudos
Got the solution after working with support. This is BUG in current Ambari and HDP versions. https://hortonworks.jira.com/browse/BUG-38390 Below is workaround: Change Log directory for Ranger Admin logs On Ranger Admin nodes:
ADMIN LOGS (Change the symbolic link for in ews directory):
mkdir /opt/hadoop/log/ranger/admin
chmod 775 /opt/hadoop/log/ranger/admin
chown ranger_qa:ranger_qa_grp /opt/hadoop/log/ranger/admin
cd /usr/hdp/current/ranger-admin/ews
unlink logs
ln -s /opt/hadoop/log/ranger/admin logs
From Ambari Web UI --> Change Log directory for Ranger Admin and restart Ranger. Note: here ranger_qa and ranger_qa_grp are ranger user and group respectively which are customized in my case. Change log directory for USERSYNC LOGS
On Ranger Usersync node: mkdir /opt/hadoop/log/ranger/usersync
chmod 775 /opt/hadoop/log/ranger/usersync
chown ranger_qa:ranger_qa_grp /opt/hadoop/log/ranger/usersync Note: here ranger_qa and ranger_qa_grp are ranger user and group respectively which are customized in my case.
1. Make a back of file /usr/hdp/current/ranger-usersync/ranger-usersync-services.sh
2. Change the file to the path to your new log location.
logdir=/var/log/ranger/usersync cd /usr/hdp/current/ranger-usersync/
cp ranger-usersync-services.sh{,.backup01122015}
vi ranger-usersync-services.sh
#logdir=/var/log/ranger/usersync
logdir=/opt/hadoop/log/ranger/usersync
From Ambari Web UI --> Change Log directory for Ranger Usersync and restart Ranger
... View more
01-13-2016
03:39 AM
1 Kudo
While Ambari UI provides feature to change log directories from UI, but it still is not being taken by Ranger. Looks like Ranger is overriding what I am setting in Ambari.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Ranger
01-13-2016
03:27 AM
1 Kudo
@Hefei Li It is strongly recommended not to change hostname of nodes. This would need changes in Ambari database to reflect the change in Ambari. I would suggest reaching out to Hortonworks support to get engg. help based on specific Ambari and HDP version (Don't recommend trying it yourself to avoid DB inconsistencies) if you have HRWX support in place.
... View more
01-13-2016
03:16 AM
@Mehdi TAZI Lily is not an HDP product, it was contributed by Ngdata. You should reach out to their mailing list for any HA advice? You may have to come up with your own HA for it, then you should certainly share your docs here for the best of the community! As far as HBase, make sure you run multiple master servers. Here is how you can setup HBase Master HA. http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_Ambari_Users_Guide/content/_hbase_high_availability.html
... View more
01-13-2016
02:57 AM
This issue was resolved. Root cause of the issue was that customer had symlinked /usr/hdp to /opt/hadoop/usr/hdp and as per discussion with Ranger Engg., that apparently is not supported currently. Once, I removed symlink and reverted back to /usr/hdp, issue got fixed. unlink /usr/hdp
cp -rp /opt/hadoop/usr/* /usr/
rm -rf /opt/hadoop/usr
... View more
01-06-2016
07:43 PM
2 Kudos
To do that, one way would be to identify current Active Namenode on both clusters and then use that. Here is an example for distcp problem. hdfs distcp hdfs://active1:8020/path hdfs://active2:8020/path Alternatively, configure client with both NameService Ids and make it identify those. Our own @hkropp has blogged this here. http://henning.kropponline.de/2015/03/15/distcp-tw...
... View more