Support Questions
Find answers, ask questions, and share your expertise

how to properly override mapper's JVM options

Rising Star

Hi,

I'm currently struggling with map reduce configuration...

I'm trying to implement the common "wordcount example", but I modified the implementation so that mappers calls an HTTPS web service to track overall progression (just for the sake of demonstration). I have to provide the mappers' JVM with a custom truststore that containe the certificate of the CA that issued the web server's certificate and I tried to use following syntax :

hadoop jar mycustommr.jar TestHttpsMR -Dmapreduce.map.java.opts="-Djavax.net.ssl.trustStore=/my/custom/path/cacerts -Djavax.net.ssl.trustStorePassword=mypassword" wordcount_in wordcount_out 

But I systematically hit following error :

"Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory wordcount_in already exists"

which indicates that arguments are not properly parsed : it seems that -Dmapreduce.map.java.opts="-Djavax.net.ssl.trustStore=/my/custom/path/cacerts -Djavax.net.ssl.trustStorePassword=mypassword" is interpreted as an application argument (the first one) instead of being passed to the mappers' JVM

What's wrong with this syntax ? How could I override mapreduce.map.java.opts property without disturbing application parameters ?

Thanks for your help

1 ACCEPTED SOLUTION

Accepted Solutions

Re: how to properly override mapper's JVM options

Rising Star

I figured out what was wrong :

In fact my class has to extends Configured and implements Tools in order to parse the confirguration properties from the command line. Works fine now !

I even figured out that I could set the property in ambari : label "MR Map Java Heap Size" actually maps the "mapreduce.map.java.opts" property, which is pretty confusing ...

View solution in original post

1 REPLY 1

Re: how to properly override mapper's JVM options

Rising Star

I figured out what was wrong :

In fact my class has to extends Configured and implements Tools in order to parse the confirguration properties from the command line. Works fine now !

I even figured out that I could set the property in ambari : label "MR Map Java Heap Size" actually maps the "mapreduce.map.java.opts" property, which is pretty confusing ...

View solution in original post