Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to upload .jar file to hdfs and execute the jar using HDP-Oracle Virtual VM

avatar

I have created a simple java+spark project to read and perform a calculation on JavaRDD. I have setup HDP-Oracle VM on my machine.My question is where to upload .jar(ex: Aambari file view) and execute the jar

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Amithesh Merugu

It seems there is a confusion here is the F: Linux directory? . The designation looks a window directory if so then copy using winscp /FileZilla to you /tmp on the on the sandbox or linux box first.

As root switch to hdfs user

# su - hdfs

Make sure the permission are correct on the /user/maria_dev check the owner

$hdfs dfs -ls /user/maria_dev

The output should give the owner to spark eg maria_dev

drwxr-xr-x - maria_dev hdfs 0 2017-08-08 23:55 /user/maria_dev

Now copy the file to the hdfs directory as the hdfs user

$ hdfs dfs -CopyFromLocal /tmp/sparktest.jar /user/toto

Now the file should be available in hdfs you can list the file:

$ hdfs dfs -ls /user/maria_dev

Now you can run your spark job as see the progress in the YARN UI choose running on the left pane

http://ambari_host:8088/cluster

Hope that helps

View solution in original post

6 REPLIES 6

avatar
Master Mentor

@Amithesh Merugu

It can be local or you can upload to hdfs but to do that you need maybe to create your home directory in /user

As root switch to hdfs user

# su - hdfs

check existing directories

$ hdfs dfs -ls / 

Make a home directory for your user (toto)

$ hdfs dfs -mkdir /user/toto 

Change ownership

$ hdfs dfs -chown toto:hdfs /user/toto 

Copy your jar to hdfs imagining the jars are in your local home directory /home/toto/test.jar

As hdfs user while in your

$ hdfs dfs -CopyFromLocal test.jar /user/toto

Now you can execute it from hdfs by passing the paths to the input and output directories in HDFS.

Hope that helps

avatar

@Geoffrey Shelton Okot

Thanks for quick response

I have jar in my local directory at F:/dev/jars/sparktest.jar and I would like to move to maria_dev directory which is present in hdfs.

$hdfs dfs F:/dev/jars sparktest.jar /user/maria_dev

I tried the above command it is not working

avatar
Master Mentor

@Amithesh Merugu

If you want to copy a File "F:/dev/jars sparktest.jar" from your local file system to the HDFS location "/user/maria_dev" then you will need the following:

1. Switch to "hdfs" user ( OR else you will have to login to the shell as "maria_dev" to wrtie inside the user directory Or any username who has write access insie the "/user/maria_dev" directory

# su - hdfs

2. Now you can use the "put" switch to place the file with dfs command as following:

# hdfs dfs -put F:/dev/jars sparktest.jar /user/maria_dev

3. Now you should be able to list the file:

# hdfs dfs -ls /user/maria_dev

.

avatar
Master Mentor

@Amithesh Merugu

It seems there is a confusion here is the F: Linux directory? . The designation looks a window directory if so then copy using winscp /FileZilla to you /tmp on the on the sandbox or linux box first.

As root switch to hdfs user

# su - hdfs

Make sure the permission are correct on the /user/maria_dev check the owner

$hdfs dfs -ls /user/maria_dev

The output should give the owner to spark eg maria_dev

drwxr-xr-x - maria_dev hdfs 0 2017-08-08 23:55 /user/maria_dev

Now copy the file to the hdfs directory as the hdfs user

$ hdfs dfs -CopyFromLocal /tmp/sparktest.jar /user/toto

Now the file should be available in hdfs you can list the file:

$ hdfs dfs -ls /user/maria_dev

Now you can run your spark job as see the progress in the YARN UI choose running on the left pane

http://ambari_host:8088/cluster

Hope that helps

avatar
Master Mentor

@Amithesh Merugu

Any feeback?

avatar

@Geoffrey Shelton Okot

Perfect!.It worked for me, I used WinSCP to upload the jar to tmp folder.And using -CopyFromLocal command moved the jar from tmp to the maria_dev user and able execute the jar.

Thanks a lot for quick response