Re: Data node process not starting up

amit_29sharmait · ‎04-12-2016

Thank you all... I have got it resolved. I did 2 things.. 1. for name node : I simply changed the port to some other value 2. For data-node : I found an error message in logs saying that "/" has permission 777 , cannot start data-node. I remember I had changed it manually for some other problem. Reverting it resolved the issue.

amit_29sharmait · ‎04-12-2016

Non of the above commands give anything suspicious... these are the log using last command.. anything suspious..? 2016-04-10 20:09:01,856 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:start(192)) - NameNode metrics system started 2016-04-10 20:09:01,858 INFO namenode.NameNode (NameNode.java:setClientNamenodeAddress(424)) - fs.defaultFS is hdfs://ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal:8020 2016-04-10 20:09:01,858 INFO namenode.NameNode (NameNode.java:setClientNamenodeAddress(444)) - Clients are to use ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal:8020 to access this namenode/service. 2016-04-10 20:09:02,025 INFO hdfs.DFSUtil (DFSUtil.java:httpServerTemplateForNNAndJN(1726)) - Starting Web-server for hdfs at: http://ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal:50070 2016-04-10 20:09:02,072 INFO mortbay.log (Slf4jLog.java:info(67)) - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2016-04-10 20:09:02,081 INFO server.AuthenticationFilter (AuthenticationFilter.java:constructSecretProvider(294)) - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2016-04-10 20:09:02,086 INFO http.HttpRequestLog (HttpRequestLog.java:getRequestLog(80)) - Http request log for http.requests.namenode is not defined 2016-04-10 20:09:02,091 INFO http.HttpServer2 (HttpServer2.java:addGlobalFilter(710)) - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2016-04-10 20:09:02,093 INFO http.HttpServer2 (HttpServer2.java:addFilter(685)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context hdfs 2016-04-10 20:09:02,093 INFO http.HttpServer2 (HttpServer2.java:addFilter(693)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2016-04-10 20:09:02,093 INFO http.HttpServer2 (HttpServer2.java:addFilter(693)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2016-04-10 20:09:02,114 INFO http.HttpServer2 (NameNodeHttpServer.java:initWebHdfs(86)) - Added filter 'org.apache.hadoop.hdfs.web.AuthFilter' (class=org.apache.hadoop.hdfs.web.AuthFilter) 2016-04-10 20:09:02,116 INFO http.HttpServer2 (HttpServer2.java:addJerseyResourcePackage(609)) - addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources, pathSpec=/webhdfs/v1/* 2016-04-10 20:09:02,127 INFO http.HttpServer2 (HttpServer2.java:openListeners(915)) - Jetty bound to port 50070 2016-04-10 20:09:02,128 INFO mortbay.log (Slf4jLog.java:info(67)) - jetty-6.1.26.hwx 2016-04-10 20:09:02,381 INFO mortbay.log (Slf4jLog.java:info(67)) - Started HttpServer2$SelectChannelConnectorWithSafeStartup@ip-xxx-xxx-xxx-xxx.us-west-2.compute.internal:50070 2016-04-10 20:09:02,404 WARN common.Util (Util.java:stringAsURI(56)) - Path /hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration. 2016-04-10 20:09:02,404 WARN common.Util (Util.java:stringAsURI(56)) - Path /hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration. 2016-04-10 20:09:02,404 WARN namenode.FSNamesystem (FSNamesystem.java:checkConfiguration(654)) - Only one image storage directory (dfs.namenode.name.dir) configured. Beware of data loss due to lack of redundant storage directories! 2016-04-10 20:09:02,405 WARN namenode.FSNamesystem (FSNamesystem.java:checkConfiguration(659)) - Only one namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of data loss due to lack of redundant storage directories! 2016-04-10 20:09:02,410 WARN common.Util (Util.java:stringAsURI(56)) - Path /hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration.

amit_29sharmait · ‎04-11-2016

Looks like I have disabled both of the things firewall and selinux ...still no luck. [root@ip-xxxxxxxx ec2-user]# systemctl status firewalld.service ● firewalld.service Loaded: not-found (Reason: No such file or directory) Active: inactive (dead) [root@ip-xx-xx-xx-xx ec2-user]# sestatus SELinux status: disabled is there any logs or something which can help?

amit_29sharmait · ‎04-10-2016

Commands doesn't seems to give answer.

amit_29sharmait · ‎04-10-2016

It is not working... looks like all of my commands are failing.. non of them are able to perform task here is the attached output of commnds I ran. errors.txt

amit_29sharmait · ‎04-09-2016

Hi, Can anyone help me please?

amit_29sharmait · ‎04-09-2016

Hi @Sunile Manjee It ws working finr that day. But today when i started my service, data node is not starting up. Can that be related? see the link below for my problem. https://community.hortonworks.com/questions/26802/data-node-process-not-starting-up.html

amit_29sharmait · ‎04-09-2016

how to do that can you please help me.

amit_29sharmait · ‎04-09-2016

Even I have right permission my data node not starting up. Gives following error :(Connection failed: [Errno 111] Connection refused to 0.0.0.0:50010) [root@ip pig]# cd /hadoop/hdfs/data [root@ip hdfs]# ls -lrt total 0 drwxr-x---. 3 hdfs hadoop 20 Apr 5 10:54 data drwxr-xr-x. 4 hdfs hadoop 63 Apr 9 14:08 namenode drwxr-xr-x. 3 hdfs hadoop 38 Apr 9 14:16 namesecondary Just want to add .. last time when the everything was running i did floowing.. added the mapred user to HDFS group and given him rwx permission on / directory... can this be a reason of failure? Just to add Not only datanode everything else is not running now.. is there any way to know what may have gone wrong?

amit_29sharmait · ‎04-09-2016

Sorry for late reply.. yes it resolved the issue.. Thank you very much

amit_29sharmait · ‎04-04-2016

I tried to enable history server using the link , I could do it only till hdfs dis -mkdir -p /app-logs as this was failing and i could not proceed. Now when I run the pig script which is creating map reduce it is failing.with below error. nay idea? Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=mapred, access=READ, inode="/mr-history/tmp/hdfs/job_1459806783854_0001-1459807556718-hdfs-PigLatin%3ADefaultJobName-1459807582179-1-1-SUCCEEDED-default-1459807564263.jhist":hdfs:hdfs:-rwxrwx--- at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)

amit_29sharmait · ‎04-04-2016

[hdfs@ bin]$ ls -lrt total 32 -rwxr-xr-x. 1 root root 1857 Dec 15 22:25 rcc -rwxr-xr-x. 1 root root 6594 Dec 15 22:25 hadoop.distro -rwxr-xr-x. 1 root root 775 Dec 15 22:26 yarn -rwxr-xr-x. 1 root root 782 Dec 15 22:26 mapred -rwxr-xr-x. 1 root root 775 Dec 15 22:26 hdfs -rwxr-xr-x. 1 root root 807 Dec 15 22:26 hadoop-fuse-dfs -rwxr-xr-x. 1 root root 772 Apr 4 14:21 hadoop [hdfs@ bin]$ cat hadoop #!/bin/bash # Autodetect JAVA_HOME if not defined if [ -e /usr/libexec/bigtop-detect-javahome ]; then . /usr/libexec/bigtop-detect-javahome elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then . /usr/lib/bigtop-utils/bigtop-detect-javahome fi export HADOOP_HOME=${HADOOP_HOME:-/usr/hdp/2.3.4.0-3485/hadoop} export HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:-/usr/hdp/2.3.4.0-3485/hadoop-mapreduce} export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-/usr/hdp/2.3.4.0-3485/hadoop-yarn} export HADOOP_LIBEXEC_DIR=${HADOOP_HOME}/libexec export HDP_VERSION=${HDP_VERSION:-2.3.4.0-3485} export HADOOP_OPTS="${HADOOP_OPTS} -Dhdp.version=${HDP_VERSION}" export YARN_OPTS="${YARN_OPTS} -Dhdp.version=${HDP_VERSION}" exec /usr/hdp/2.3.4.0-3485//hadoop/bin/hadoop.distro "$@" [hdfs@ip- bin]$ echo $HADOOP_HOME [hdfs@ip-bin]$ pwd /usr/hdp/current/hadoop-client/bin [hdfs@ip-bin]$

amit_29sharmait · ‎04-04-2016

Can you or someone help me with the main question, I am not able to figure out the reason for job failure. after googling it i think it may be related to the variable not set ..I am adding more details about how the variables are showing null... Could you please help..

amit_29sharmait · ‎04-03-2016

Thanks this is what i was looking for.... Although this logs does not give much informaiton. Do you know if there is way to make log more readable.? or probably i'll have to ask another question.

amit_29sharmait · ‎04-03-2016

yes, you are right. It is creating logs into the the same file but it is when a pig script failing. It is not creating log where map reduce is failing. I need to analyse map reduce logs in order to resolve the issue.

amit_29sharmait · ‎04-03-2016

My pig script failed when I am trying to do order by. I want to check the logs but I can't see the location of the logs. I got to know from the post that it should be set in the file (/etc/hadoop/hadoop-env.sh) but the file has everything commented as shown below. Does that mean it is not being written? how can I set those values to look into the logs as my scripts is failing again and again and I want to check the logs to troubleshoot it. also my file ( hadoop-env.sh)seems to be at different path(/etc/hadoop/conf.install) is there anything wrong ?? # On secure datanodes, user to run the datanode as after dropping privileges. # This **MUST** be uncommented to enable secure HDFS if using privileged ports # to provide authentication of data transfer protocol. This **MUST NOT** be # defined if SASL is configured for authentication of data transfer protocol # using non-privileged ports. #export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER} # Where log files are stored. $HADOOP_HOME/logs by default. #export HADOOP_LOG_DIR=${HADOOP_LOG_DIR}/$USER

amit_29sharmait · ‎04-01-2016

Hi, I have uploaded both the file structure and the log of the file i have. i don't see the file the path you mentioned. hive.jpglog4j.txt

amit_29sharmait · ‎04-01-2016

sorry, it was a typo. Please find below I tries with full path. I have updated the question as well [hive@ip-hive]$ ls -lrt total 8 -rwxrwxrwx. 1 root root 13 Apr 1 15:50 hive1.sql -rwxrwxrwx. 1 root root 17 Apr 1 15:53 hive.hive [hive@ip-hive]$ hive -f /home/ec2-user/scripts/hive/hive1.sql WARNING: Use "yarn jar" to launch YARN applications. Logging initialized using configuration in file:/etc/hive/2.3.4.0-3485/0/hive-log4j.properties Could not open input file for reading. (File file:/home/ec2-user/scripts/hive/hive1.sql does not exist)

amit_29sharmait · ‎04-01-2016

I am getting following error even though I have the script located at the location. [hive@ip-hive]$ pwd /home/ec2-user/scripts/hive [hive@ip-hive]$ ls -lrt total 8 -rwxrwxrwx. 1 root root 13 Apr 1 15:50 hive1.sql -rwxrwxrwx. 1 root root 17 Apr 1 15:53 hive.hive [hive@ip-hive]$ hive -f /home/ec2-user/scripts/hive/hive1.sql WARNING: Use "yarn jar" to launch YARN applications. Logging initialized using configuration in file:/etc/hive/2.3.4.0-3485/0/hive-log4j.properties Could not open input file for reading. (File file:/home/ec2-user/scripts/hive/hive1.sql does not exist)

amit_29sharmait · ‎03-29-2016

Is it something which i need to do manually ? as I have hortonworks distribution installed on a cloud server... and I have pig view as well.

amit_29sharmait · ‎03-29-2016

there is no pig folder inside /var/log 😞

amit_29sharmait · ‎03-29-2016

Hi, I can see the job in failed status on 8088 port. But when I click on the logs it gets timeout.. Do you know is there any reason to access the log... Any help will be appreciated. Thanks

amit_29sharmait · ‎03-29-2016

I think I am running into a real problem now and need some expert advice. I am running below scripts on grunt shell and everything is running fine except dump d; I am getting below error. 2016-03-28 20:08:30,686 [main] ERROR org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce job(s) failed! script running : a = LOAD '/user/admin/pig/scripts/pig1.csv' USING PigStorage(',') AS (pid chararray,id:int,city:chararray); describe a ; b = LIMIT a 100; C = FOREACH a generate pid, id; dump C; d = group C by pid; dump d;

amit_29sharmait · ‎03-28-2016

Hi , I tried following exact same procedure from the link bleow. http://hortonworks.com/hadoop-tutorial/hello-world-an-introduction-to-hadoop-hcatalog-hive-and-pig/#section_5 but it gives me same error: File does not exist: /user/admin/pig/jobs/riskfactorpig_28-03-2016-13-44-12/stdout at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1828) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1799) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1712) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:652) My script location is showing as /user/admin/pig/scripts/riskfactorpig-2016-03-28_01-43.pig but not sure why it is looking into(fro error) /user/admin/pig/jobs/riskfactorpig_28-03-2016-13-44-12/stdout at

amit_29sharmait · ‎03-28-2016

Hi , I tried following exact same procedure from the link bleow. http://hortonworks.com/hadoop-tutorial/hello-world-an-introduction-to-hadoop-hcatalog-hive-and-pig/#section_5 but it gives me same error: File does not exist: /user/admin/pig/jobs/riskfactorpig_28-03-2016-13-44-12/stdout at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1828) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1799) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1712) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:652) My script location is showing as /user/admin/pig/scripts/riskfactorpig-2016-03-28_01-43.pig but not sure why it is looking into(fro error) /user/admin/pig/jobs/riskfactorpig_28-03-2016-13-44-12/stdout at

amit_29sharmait · ‎03-28-2016

I am runnign my first pig script ,it is failing with below error. any idea? File does not exist: /user/admin/pig/jobs/explain_p1_28-03-2016-00-58-46/stderr at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1828) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1799) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1712) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:652) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)

amit_29sharmait · ‎03-15-2016

I have running a query in mr engine it is failing after 10 minutes but the same query is running fine in tez in 2 minutes. query: select byear, count(*) from master_partiton group by byear (---byear is partitioned column) INFO : Number of reduce tasks not specified. Estimated from input data size: 1 INFO : In order to change the average load for a reducer (in bytes): INFO : set hive.exec.reducers.bytes.per.reducer=<number> INFO : In order to limit the maximum number of reducers: INFO : set hive.exec.reducers.max=<number> INFO : In order to set a constant number of reducers: INFO : set mapreduce.job.reduces=<number> INFO : number of splits:1 INFO : Submitting tokens for job: job_1457889258827_0016 INFO : The url to track the job: http://ip-xx-xx-xx-xx.us-west-2.compute.internal:8088/proxy/application_1457889258827_0016/ INFO : Starting Job = job_1457889258827_0016, Tracking URL = http://ip-xx-xx-xx-xx.us-west-2.compute.internal:8088/proxy/application_1457889258827_0016/ INFO : Kill Command = /usr/hdp/2.3.4.0-3485/hadoop/bin/hadoop job -kill job_1457889258827_0016 INFO : Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 INFO : 2016-03-15 00:35:36,399 Stage-1 map = 0%, reduce = 0% INFO : 2016-03-15 00:35:59,157 Stage-1 map = 100%, reduce = 100% ERROR : Ended Job = job_1457889258827_0016 with errors

amit_29sharmait · ‎03-15-2016

I am trying ti access the command line interface from server where I have hortonworks install but I am getting below error.Can I not access the command line? [root@ip-xxx-xx-xx ec2-user]# hive WARNING: Use "yarn jar" to launch YARN applications. Logging initialized using configuration in file:/etc/hive/2.3.4.0-3485/0/hive-log4j.properties Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user/root":hdfs:hdfs:drwxr-xr-x

amit_29sharmait · ‎03-15-2016

Thanks... The answer does not say what may be the reason. Do you have any idea?

amit_29sharmait · ‎03-15-2016

I am running a explain plan on a query It says "Plan not optimized by CBO" When I checked the logs it says using 2016-03-14 00:37:12,082 INFO [HiveServer2-Handler-Pool: Thread-41]: parse.BaseSemanticAnalyzer (CalcitePlanner.java:canCBOHandleAst(392)) - Not invoking CBO because the statement has too few joins Any idea what does this mean?

Online	Offline
Last Visited	‎06-22-2016 06:49 PM

Date Registered	‎02-21-2016 09:11 PM
Last Visited	‎06-22-2016 06:49 PM
Total Messages Posted	35
Total Kudos Received	19

Re: Data node process not starting up

Re: Data node process not starting up

Re: Data node process not starting up

Re: Data node process not starting up

Re: Data node process not starting up

Re: Data node process not starting up

Re: Data node process not starting up

Re: Pig Map reduce permission denied after history...

Re: Data node process not starting up

Data node process not starting up

Re: Pig Map reduce permission denied after history...

Pig Map reduce permission denied after history ser...

Re: location of hadoop and pig logs

Re: location of hadoop and pig logs

Re: location of hadoop and pig logs

Re: location of hadoop and pig logs

location of hadoop and pig logs

Re: Hive script error : script does not exists err...

Re: Hive script error : script does not exists err...

Hive script error : script does not exists error

Re: pig map reduce is failing

Re: pig map reduce is failing

Re: pig map reduce is failing

pig map reduce is failing

Re: First pig script failing .

Re: First pig script failing .

First pig script failing .

Query failing with mr but running in tez

Error in accessing command line interface from hor...

Re: Hive : Query not optimized by CBO

Hive : Query not optimized by CBO