how to properly override mapper's JVM options

schausson — Wed, 13 Sep 2017 21:01:16 GMT

Hi,

I'm currently struggling with map reduce configuration...

I'm trying to implement the common "wordcount example", but I modified the implementation so that mappers calls an HTTPS web service to track overall progression (just for the sake of demonstration). I have to provide the mappers' JVM with a custom truststore that containe the certificate of the CA that issued the web server's certificate and I tried to use following syntax :

hadoop jar mycustommr.jar TestHttpsMR -Dmapreduce.map.java.opts="-Djavax.net.ssl.trustStore=/my/custom/path/cacerts -Djavax.net.ssl.trustStorePassword=mypassword" wordcount_in wordcount_out

But I systematically hit following error :

"Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory wordcount_in already exists"

which indicates that arguments are not properly parsed : it seems that -Dmapreduce.map.java.opts="-Djavax.net.ssl.trustStore=/my/custom/path/cacerts -Djavax.net.ssl.trustStorePassword=mypassword" is interpreted as an application argument (the first one) instead of being passed to the mappers' JVM

What's wrong with this syntax ? How could I override mapreduce.map.java.opts property without disturbing application parameters ?

Thanks for your help

Re: how to properly override mapper's JVM options

schausson — Thu, 14 Sep 2017 17:08:16 GMT

I figured out what was wrong :

In fact my class has to extends Configured and implements Tools in order to parse the confirguration properties from the command line. Works fine now !

I even figured out that I could set the property in ambari : label "MR Map Java Heap Size" actually maps the "mapreduce.map.java.opts" property, which is pretty confusing ...

question Re: how to properly override mapper's JVM options in Archives of Support Questions (Read Only)

how to properly override mapper's JVM options

Re: how to properly override mapper's JVM options