About Eddie7

Eddie7 · ‎10-08-2018

This works indeed. With your example I recognized that my problem was not to provide the parameters to mapreduce.map.java.opts. I supplied the complete opts string with a variable: export MAPPER_OPTS="-Dmapreduce.map.java.opts='-Xss5M -agentlib:jdwp=transport=dt_socket,server=y,address=8787'" (I tried different combinations of singlequote, double quote and masking with backslash) Resolving this in my yarn command caused the problems. (Though having this in a variable because it changes often would be easier. 🙂 ) Thanks for your help, Eddie

Eddie7 · ‎10-08-2018

Putting the agentlib in YARN_OPTS instead of mapreduce.map.java.opts didn't work for me, debugging is not possible. I also have other use cases where it would be nice to be able to add more properties to mapreduce.map.java.opts. Thanks, Eddie

Eddie7 · ‎10-02-2018

I am trying to add more than one option to mapreduce.map.java.opts without success. E.g.: yarn jar My.jar Myclass.class -Dmapreduce.map.java.opts=-Xss5M -agentlib:jdwp=transport=dt_socket,server=y,address=8787 -Dmapreduce.map.memory.mb=6000 ... followed by more options and maybe more than 2 arguments for mapreduce.map.java.opts. How do I have to do this that -Xss and -agentlib in my example are both treated as arguments for -Dmapreduce.map.memory.mb and not for the yarn jar job? I tried everything I could think of - single quotes, double quotes, ... Thanks for every hint, Eddie

Eddie7 · ‎09-07-2018

I have an existing application which works quite fine. If I add a specific jar (no code changes yet) mappers don't start because there is a probem reading the configuration: 2018-09-06 15:26:12,221 [main] ERROR org.apache.hadoop.conf.Configuration - error parsing conf core-default.xml javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2570) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2532) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2424) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1238) at org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49) at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32) at org.apache.hadoop.yarn.api.records.ApplicationId.newInstance(ApplicationId.java:49) at org.apache.hadoop.yarn.api.records.ContainerId.toApplicationAttemptId(ContainerId.java:249) at org.apache.hadoop.yarn.api.records.ContainerId.toApplicationAttemptId(ContainerId.java:244) at org.apache.hadoop.yarn.api.records.ContainerId.fromString(ContainerId.java:223) at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:191) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1463) It looks as if a wrong xml parser is used here. Intererstingly, running the application local in eclipse works just fine. I tried to debug my application. In the driver I didn't see any problems, and the mapper didn't reach my first breakpoint (first line in the main method of YarnChild). Looking at the stack trace, from where is MRAppMaster called (started)? How could I get hadoop using the correct xml parser? I added javax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl all places I was thinking of: in my driver class, in mapreduce.map.java.opts, in YARN_CLIENT_OPTS, as parameter when calling "yarn jar" How could I solve this? Thanks in advance

Eddie7 · ‎09-27-2017

I have the same problem (HDP 2.6, vmware). Reading a lot of related answers I believe that the problem is port forwarding and/or docker configuration. What has to be done to enable acess to a specific port (9999 in the example above) of the sandbox? (Probably the same is done for scp - the sandbox is accessed via port 2222 though there is nothing listening at port 2222 in the sandbox.) Unfortunately the most useful commands for examining this problem are not installed in the docker machine (e.g. ifconfig, netstat, ...)

Eddie7 · ‎09-13-2017

I am not able to use shared folders with the sandbox (HDP 2.6, vmware). I added shared folders and enabled it in the settings, but I can't see the folder in /media. After some reading I thought it might be necessary to install the vmware tools, after downloading, mounting them as cdrom and invoking vmware-install.pl it ended with "WARNING: Couldn't open directory /lib/modules/4.12.4-1.el7.elrepo.x86_64:" after /usr/bin/vmware-config-tools.pl was invoked. What do I have to do to install the vmware tools? Will this help using shared folders or is there still another problem? Eddie

Eddie7 · ‎07-12-2017

The problem is virtualbox. With vmware everything is working as expected.

Eddie7 · ‎07-06-2017

I tried to start the sandbox (HDP 2.6, 8 GB assigned) on a Core i7 with 16 GB RAM. Booting is not finished after more than 30 hours (CPU is very busy). Free RAM is available, according to the task manager VirtualBox is using about 124 MB of RAM only. Is there anything I could do?

Online	Offline
Last Visited	‎10-16-2019 11:20 AM

Date Registered	‎07-06-2017 06:17 AM
Last Visited	‎10-16-2019 11:20 AM
Total Messages Posted	8

Re: How to put several options in mapreduce.map.ja...

Re: How to put several options in mapreduce.map.ja...

How to put several options in mapreduce.map.java.o...

hadop seems to use a wrong xml parser

Re: How to do remote debugging for Apache Hadoop C...

Problem using shared folders in sandbox (HDP 2.6 v...

Re: HDP 2.5 Sandbox not starting

Re: HDP 2.5 Sandbox not starting