Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to upload .jar file to hdfs and execute the jar using HDP-Oracle Virtual VM

avatar

I have created a simple java+spark project to read and perform a calculation on JavaRDD. I have setup HDP-Oracle VM on my machine.My question is where to upload .jar(ex: Aambari file view) and execute the jar

1 ACCEPTED SOLUTION

avatar
Master Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
6 REPLIES 6

avatar
Master Mentor

@Amithesh Merugu

It can be local or you can upload to hdfs but to do that you need maybe to create your home directory in /user

As root switch to hdfs user

# su - hdfs

check existing directories

$ hdfs dfs -ls / 

Make a home directory for your user (toto)

$ hdfs dfs -mkdir /user/toto 

Change ownership

$ hdfs dfs -chown toto:hdfs /user/toto 

Copy your jar to hdfs imagining the jars are in your local home directory /home/toto/test.jar

As hdfs user while in your

$ hdfs dfs -CopyFromLocal test.jar /user/toto

Now you can execute it from hdfs by passing the paths to the input and output directories in HDFS.

Hope that helps

avatar

@Geoffrey Shelton Okot

Thanks for quick response

I have jar in my local directory at F:/dev/jars/sparktest.jar and I would like to move to maria_dev directory which is present in hdfs.

$hdfs dfs F:/dev/jars sparktest.jar /user/maria_dev

I tried the above command it is not working

avatar
Master Mentor

@Amithesh Merugu

If you want to copy a File "F:/dev/jars sparktest.jar" from your local file system to the HDFS location "/user/maria_dev" then you will need the following:

1. Switch to "hdfs" user ( OR else you will have to login to the shell as "maria_dev" to wrtie inside the user directory Or any username who has write access insie the "/user/maria_dev" directory

# su - hdfs

2. Now you can use the "put" switch to place the file with dfs command as following:

# hdfs dfs -put F:/dev/jars sparktest.jar /user/maria_dev

3. Now you should be able to list the file:

# hdfs dfs -ls /user/maria_dev

.

avatar
Master Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Master Mentor

@Amithesh Merugu

Any feeback?

avatar

@Geoffrey Shelton Okot

Perfect!.It worked for me, I used WinSCP to upload the jar to tmp folder.And using -CopyFromLocal command moved the jar from tmp to the maria_dev user and able execute the jar.

Thanks a lot for quick response