Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

how to properly override mapper's JVM options

avatar
Expert Contributor

Hi,

I'm currently struggling with map reduce configuration...

I'm trying to implement the common "wordcount example", but I modified the implementation so that mappers calls an HTTPS web service to track overall progression (just for the sake of demonstration). I have to provide the mappers' JVM with a custom truststore that containe the certificate of the CA that issued the web server's certificate and I tried to use following syntax :

hadoop jar mycustommr.jar TestHttpsMR -Dmapreduce.map.java.opts="-Djavax.net.ssl.trustStore=/my/custom/path/cacerts -Djavax.net.ssl.trustStorePassword=mypassword" wordcount_in wordcount_out 

But I systematically hit following error :

"Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory wordcount_in already exists"

which indicates that arguments are not properly parsed : it seems that -Dmapreduce.map.java.opts="-Djavax.net.ssl.trustStore=/my/custom/path/cacerts -Djavax.net.ssl.trustStorePassword=mypassword" is interpreted as an application argument (the first one) instead of being passed to the mappers' JVM

What's wrong with this syntax ? How could I override mapreduce.map.java.opts property without disturbing application parameters ?

Thanks for your help

1 ACCEPTED SOLUTION

avatar
Expert Contributor

I figured out what was wrong :

In fact my class has to extends Configured and implements Tools in order to parse the confirguration properties from the command line. Works fine now !

I even figured out that I could set the property in ambari : label "MR Map Java Heap Size" actually maps the "mapreduce.map.java.opts" property, which is pretty confusing ...

View solution in original post

1 REPLY 1

avatar
Expert Contributor

I figured out what was wrong :

In fact my class has to extends Configured and implements Tools in order to parse the confirguration properties from the command line. Works fine now !

I even figured out that I could set the property in ambari : label "MR Map Java Heap Size" actually maps the "mapreduce.map.java.opts" property, which is pretty confusing ...