Non of the above commands give anything suspicious... these are the log using last command.. anything suspious..? 2016-04-10 20:09:01,856 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:start(192)) - NameNode metrics system started 2016-04-10 20:09:01,858 INFO namenode.NameNode (NameNode.java:setClientNamenodeAddress(424)) - fs.defaultFS is hdfs://ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal:8020 2016-04-10 20:09:01,858 INFO namenode.NameNode (NameNode.java:setClientNamenodeAddress(444)) - Clients are to use ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal:8020 to access this namenode/service. 2016-04-10 20:09:02,025 INFO hdfs.DFSUtil (DFSUtil.java:httpServerTemplateForNNAndJN(1726)) - Starting Web-server for hdfs at: http://ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal:50070 2016-04-10 20:09:02,072 INFO mortbay.log (Slf4jLog.java:info(67)) - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2016-04-10 20:09:02,081 INFO server.AuthenticationFilter (AuthenticationFilter.java:constructSecretProvider(294)) - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2016-04-10 20:09:02,086 INFO http.HttpRequestLog (HttpRequestLog.java:getRequestLog(80)) - Http request log for http.requests.namenode is not defined 2016-04-10 20:09:02,091 INFO http.HttpServer2 (HttpServer2.java:addGlobalFilter(710)) - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2016-04-10 20:09:02,093 INFO http.HttpServer2 (HttpServer2.java:addFilter(685)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context hdfs 2016-04-10 20:09:02,093 INFO http.HttpServer2 (HttpServer2.java:addFilter(693)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2016-04-10 20:09:02,093 INFO http.HttpServer2 (HttpServer2.java:addFilter(693)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2016-04-10 20:09:02,114 INFO http.HttpServer2 (NameNodeHttpServer.java:initWebHdfs(86)) - Added filter 'org.apache.hadoop.hdfs.web.AuthFilter' (class=org.apache.hadoop.hdfs.web.AuthFilter) 2016-04-10 20:09:02,116 INFO http.HttpServer2 (HttpServer2.java:addJerseyResourcePackage(609)) - addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources, pathSpec=/webhdfs/v1/* 2016-04-10 20:09:02,127 INFO http.HttpServer2 (HttpServer2.java:openListeners(915)) - Jetty bound to port 50070 2016-04-10 20:09:02,128 INFO mortbay.log (Slf4jLog.java:info(67)) - jetty-6.1.26.hwx 2016-04-10 20:09:02,381 INFO mortbay.log (Slf4jLog.java:info(67)) - Started HttpServer2$SelectChannelConnectorWithSafeStartup@ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal:50070 2016-04-10 20:09:02,404 WARN common.Util (Util.java:stringAsURI(56)) - Path /hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration. 2016-04-10 20:09:02,404 WARN common.Util (Util.java:stringAsURI(56)) - Path /hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration. 2016-04-10 20:09:02,404 WARN namenode.FSNamesystem (FSNamesystem.java:checkConfiguration(654)) - Only one image storage directory (dfs.namenode.name.dir) configured. Beware of data loss due to lack of redundant storage directories! 2016-04-10 20:09:02,405 WARN namenode.FSNamesystem (FSNamesystem.java:checkConfiguration(659)) - Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of data loss due to lack of redundant storage directories! 2016-04-10 20:09:02,410 WARN common.Util (Util.java:stringAsURI(56)) - Path /hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration. ... View more

Looks like I have disabled both of the things firewall and selinux ...still no luck. [root@ip-xxxxxxxx ec2-user]# systemctl status firewalld.service ● firewalld.service Loaded: not-found (Reason: No such file or directory) Active: inactive (dead) [root@ip-xx-xx-xx-xx ec2-user]# sestatus SELinux status: disabled is there any logs or something which can help? ... View more

Commands doesn't seems to give answer. ... View more

It is not working... looks like all of my commands are failing.. non of them are able to perform task here is the attached output of commnds I ran. errors.txt ... View more

Hi, Can anyone help me please? ... View more

Hi @Sunile Manjee It ws working finr that day. But today when i started my service, data node is not starting up. Can that be related? see the link below for my problem. https://community.hortonworks.com/questions/26802/data-node-process-not-starting-up.html ... View more

how to do that can you please help me. ... View more

Sorry for late reply.. yes it resolved the issue.. Thank you very much ... View more

[hdfs@ bin]$ ls -lrt total 32 -rwxr-xr-x. 1 root root 1857 Dec 15 22:25 rcc -rwxr-xr-x. 1 root root 6594 Dec 15 22:25 hadoop.distro -rwxr-xr-x. 1 root root 775 Dec 15 22:26 yarn -rwxr-xr-x. 1 root root 782 Dec 15 22:26 mapred -rwxr-xr-x. 1 root root 775 Dec 15 22:26 hdfs -rwxr-xr-x. 1 root root 807 Dec 15 22:26 hadoop-fuse-dfs -rwxr-xr-x. 1 root root 772 Apr 4 14:21 hadoop [hdfs@ bin]$ cat hadoop #!/bin/bash # Autodetect JAVA_HOME if not defined if [ -e /usr/libexec/bigtop-detect-javahome ]; then . /usr/libexec/bigtop-detect-javahome elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then . /usr/lib/bigtop-utils/bigtop-detect-javahome fi export HADOOP_HOME=${HADOOP_HOME:-/usr/hdp/2.3.4.0-3485/hadoop} export HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:-/usr/hdp/2.3.4.0-3485/hadoop-mapreduce} export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-/usr/hdp/2.3.4.0-3485/hadoop-yarn} export HADOOP_LIBEXEC_DIR=${HADOOP_HOME}/libexec export HDP_VERSION=${HDP_VERSION:-2.3.4.0-3485} export HADOOP_OPTS="${HADOOP_OPTS} -Dhdp.version=${HDP_VERSION}" export YARN_OPTS="${YARN_OPTS} -Dhdp.version=${HDP_VERSION}" exec /usr/hdp/2.3.4.0-3485//hadoop/bin/hadoop.distro "$@" [hdfs@ip- bin]$ echo $HADOOP_HOME [hdfs@ip-bin]$ pwd /usr/hdp/current/hadoop-client/bin [hdfs@ip-bin]$ ... View more

Can you or someone help me with the main question, I am not able to figure out the reason for job failure. after googling it i think it may be related to the variable not set ..I am adding more details about how the variables are showing null... Could you please help.. ... View more

Thanks this is what i was looking for.... Although this logs does not give much informaiton. Do you know if there is way to make log more readable.? or probably i'll have to ask another question. ... View more

yes, you are right. It is creating logs into the the same file but it is when a pig script failing. It is not creating log where map reduce is failing. I need to analyse map reduce logs in order to resolve the issue. ... View more

Thank you very much for your help.. It is working now.. Will you be able to answer my question? you mentioned to add property hadoop.proxyuser.hive.hosts=* but hontor document says to add hadoop.proxyuser.root.hosts=* I have added both but not sure why and which one is working. ... View more

I already have these property added. Let me give you the brief This were running fine until i restarted my aws server and its dns chnaged and I see these errors. Write now I am getting below errors : org.apache.ambari.view.hive.client.HiveClientException: H060 Unable to open Hive session: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out ... View more