Member since
07-06-2017
8
Posts
0
Kudos Received
0
Solutions
10-08-2018
01:05 PM
This works indeed. With your example I recognized that my problem was not to provide the parameters to mapreduce.map.java.opts. I supplied the complete opts string with a variable: export MAPPER_OPTS="-Dmapreduce.map.java.opts='-Xss5M -agentlib:jdwp=transport=dt_socket,server=y,address=8787'" (I tried different combinations of singlequote, double quote and masking with backslash) Resolving this in my yarn command caused the problems. (Though having this in a variable because it changes often would be easier. 🙂 ) Thanks for your help, Eddie
... View more
10-08-2018
08:55 AM
Putting the agentlib in YARN_OPTS instead of mapreduce.map.java.opts didn't work for me, debugging is not possible. I also have other use cases where it would be nice to be able to add more properties to mapreduce.map.java.opts. Thanks, Eddie
... View more
10-02-2018
01:59 PM
I am trying to add more than one option to mapreduce.map.java.opts without success. E.g.: yarn jar My.jar Myclass.class -Dmapreduce.map.java.opts=-Xss5M -agentlib:jdwp=transport=dt_socket,server=y,address=8787 -Dmapreduce.map.memory.mb=6000 ... followed by more options and maybe more than 2 arguments for mapreduce.map.java.opts. How do I have to do this that -Xss and -agentlib in my example are both treated as arguments for -Dmapreduce.map.memory.mb and not for the yarn jar job? I tried everything I could think of - single quotes, double quotes, ... Thanks for every hint, Eddie
... View more
Labels:
09-07-2018
01:13 PM
I have an existing application which works quite fine. If I add a specific jar (no code changes yet) mappers don't start because there is a probem reading the configuration: 2018-09-06 15:26:12,221 [main] ERROR org.apache.hadoop.conf.Configuration - error parsing conf core-default.xml
javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2570)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2532)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2424)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1238)
at org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
at org.apache.hadoop.yarn.api.records.ApplicationId.newInstance(ApplicationId.java:49)
at org.apache.hadoop.yarn.api.records.ContainerId.toApplicationAttemptId(ContainerId.java:249)
at org.apache.hadoop.yarn.api.records.ContainerId.toApplicationAttemptId(ContainerId.java:244)
at org.apache.hadoop.yarn.api.records.ContainerId.fromString(ContainerId.java:223)
at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:191)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1463)
It looks as if a wrong xml parser is used here. Intererstingly, running the application local in eclipse works just fine. I tried to debug my application. In the driver I didn't see any problems, and the mapper didn't reach my first breakpoint (first line in the main method of YarnChild). Looking at the stack trace, from where is MRAppMaster called (started)? How could I get hadoop using the correct xml parser? I added javax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl all places I was thinking of: in my driver class, in mapreduce.map.java.opts, in YARN_CLIENT_OPTS, as parameter when calling "yarn jar" How could I solve this? Thanks in advance
... View more
09-27-2017
01:47 PM
I have the same problem (HDP 2.6, vmware). Reading a lot of related answers I believe that the problem is port forwarding and/or docker configuration. What has to be done to enable acess to a specific port (9999 in the example above) of the sandbox? (Probably the same is done for scp - the sandbox is accessed via port 2222 though there is nothing listening at port 2222 in the sandbox.) Unfortunately the most useful commands for examining this problem are not installed in the docker machine (e.g. ifconfig, netstat, ...)
... View more
09-13-2017
01:48 PM
I am not able to use shared folders with the sandbox (HDP 2.6, vmware). I added shared folders and enabled it in the settings, but I can't see the folder in /media. After some reading I thought it might be necessary to install the vmware tools, after downloading, mounting them as cdrom and invoking vmware-install.pl it ended with "WARNING: Couldn't open directory /lib/modules/4.12.4-1.el7.elrepo.x86_64:" after /usr/bin/vmware-config-tools.pl was invoked. What do I have to do to install the vmware tools? Will this help using shared folders or is there still another problem? Eddie
... View more
07-12-2017
05:26 AM
The problem is virtualbox. With vmware everything is working as expected.
... View more
07-06-2017
06:22 AM
I tried to start the sandbox (HDP 2.6, 8 GB assigned) on a Core i7 with 16 GB RAM. Booting is not finished after more than 30 hours (CPU is very busy). Free RAM is available, according to the task manager VirtualBox is using about 124 MB of RAM only.
Is there anything I could do?
... View more