File does not exist: /tmp/.pigjobs/t3_20-04-2016-12-38-13/stderr at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1821) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1792) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1705) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:588) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2137) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2133) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2131)
players = LOAD '/tmp/FIT5148/basketball_players.csv' using PigStorage(',');
Does the user running the scripy have access to read the contents of the file? I suggest you try placing the file in /user/masha first, /tmp is not a good place.
Thank you very much for the answer. Yes, user has access to read, write etc. I placed my script to another directory /user/admin, but it doesn't help. I received the same error. It looks like that for some reason it can't create output file. Do you have any ideas?
Assuming user running script is root
sudo -u hdfs hdfs dfs -mkdir /user/root sudo -u hdfs hdfs dfs -chown -R root:hdfs /user/root hdfs dfs -put basketball_players.csv /user/root
then change your script to
players = LOAD '/user/root/basketball_players.csv' using PigStorage(','); DUMP players;
this will only work if you are logged in as root. For /user/admin, you need to be logged in as admin. Same is true in Ambari Pig view, the user logged in is the user who owns /user/username
Please make sure your user, probably t3, has permissions to create the directory and that it exisits:
hdfs dfs -chown/-chmod
to change ownership or permissions.
Thank you for the answer. User has all permissions. My understanding was that /tmp/.pigjobs has to be created automatically for the output of the script, when I run the script. For example I didn't create /tmp/.pigscript or /tmp/.pigstore, I only created script with name t2. " /tmp/.pigscript " and "/tmp/.pigstore" appeared in "tmp" automatically.
BTW I created /tmp/.pigjobs manually, but it didn't help../tmp/.pigjobs
Do you have any other ideas? Thank you
I would avoid creating files that start with a dot. Those are temperature files created as part of your script execution.
@Masha Golod yes, it's temporary file for job output. Seems weird. What is sometime confusing is that the file could also be created locally (on the machine/OS itself or in HDFS). Could you check, if you have issues with your local filesystem?