Pig script error: ​File does not exist: /tmp/.pigjobs

[#1]

File does not exist: /tmp/.pigjobs/t3_20-04-2016-12-38-13/stderr at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1821) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1792) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1705) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:588) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2137) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2133) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2131)

Script contents:

players = LOAD '/tmp/FIT5148/basketball_players.csv' using

PigStorage(',');

DUMP players;

[#2]

Does the user running the scripy have access to read the contents of the file? I suggest you try placing the file in /user/masha first, /tmp is not a good place.

[#3]

Thank you very much for the answer. Yes, user has access to read, write etc. I placed my script to another directory /user/admin, but it doesn't help. I received the same error. It looks like that for some reason it can't create output file. Do you have any ideas?

[#4]

@Masha Golod

Assuming user running script is root

sudo -u hdfs hdfs dfs -mkdir /user/root
sudo -u hdfs hdfs dfs -chown -R root:hdfs /user/root
hdfs dfs -put basketball_players.csv /user/root

then change your script to

players = LOAD '/user/root/basketball_players.csv' using PigStorage(',');
DUMP players;

this will only work if you are logged in as root. For /user/admin, you need to be logged in as admin. Same is true in Ambari Pig view, the user logged in is the user who owns /user/username

[#5]

Please make sure your user, probably t3, has permissions to create the directory and that it exisits:

/tmp/.pigjobs/

Use

hdfs dfs -chown/-chmod

to change ownership or permissions.

[#6]

Thank you for the answer. User has all permissions. My understanding was that /tmp/.pigjobs has to be created automatically for the output of the script, when I run the script. For example I didn't create /tmp/.pigscript or /tmp/.pigstore, I only created script with name t2. " /tmp/.pigscript " and "/tmp/.pigstore" appeared in "tmp" automatically.

BTW I created /tmp/.pigjobs manually, but it didn't help../tmp/.pigjobs

Do you have any other ideas? Thank you

[#7]

I would avoid creating files that start with a dot. Those are temperature files created as part of your script execution.

[#8]

@Masha Golod yes, it's temporary file for job output. Seems weird. What is sometime confusing is that the file could also be created locally (on the machine/OS itself or in HDFS). Could you check, if you have issues with your local filesystem?

[#9]

@Masha Golod did you see @Manish Gupta answer? Following his advise could you make sure the user you are executing the pig job with does have a /user/<username> folder in HDFS? The user should own that folder.

[#10]

As much as I know, I don't have any issues with local filesystem, but how I can check it?