Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (2)
avatar

Problem

Here is my user I run %livy.sparkr with

%sh 
whoami 

admin 

Permissions set to directory and file:

%sh
ls -lrt /home/admin

total 12
-rwxr-xr-x 1 admin admin 3950 Mar 9 13:12 common.r
drwxrwxr-x 2 admin admin 4096 Mar 10 15:25 analysis
drwxrwxr-x 3 admin admin 4096 Mar 13 12:49 motorAnalysis

%sh
ls -lrt /home/admin/motorAnalysis


total 8
-rwxrwxr-x 1 admin admin 3950 Mar 10 16:49 common.r
drwxrwxr-x 5 admin admin 4096 Mar 10 16:49 data

Running %livy.sparkr I am getting

%livy.sparkr 
...
source("/home/admin/motorAnalysis/common.r") 

null device 
1 Error in file(filename, "r", encoding = encoding) : 
cannot open the connection 
Calls: source -> file 
In addition: Warning message: 
In file(filename, "r", encoding = encoding) : 
cannot open file '/home/admin/motorAnalysis/common.r': No such file or directory 

Solution

Restart livy interpreter from Zeppelin UI fixes the problem

1,209 Views
0 Kudos
Comments

Daniel Kozlowski, Any explanation on why was this error happening on 1st attempt ? And how did it get resolved on interpreter restart ?

@yvora

This problem happened at the customer's site on HDP 2.5.0. The customer had already spark job running on RM UI. Then created a folder, permissions, etc using sh interpreter or command line. Running livy.sparkr using source function displayed "No such file or directory" even though the file was there. Restarting livy interpreter fixed the issue.

BTW: I have tested it out on my local HDP 2.5.3 and all works fine - no livy interpreter needs to be restarted.