I'm using rest curl to extract BDP jobs status from history, and calculating the total data volume and avg replication time for each job, its talking over 9 hours to complete with huge file. Is it possible to have filter to extract last 24 hours BDP jobs only to reduce time and file size?   Thanks, Scott ... View more

Hi,   Could you please let me know how to solve this issue. I'm having the same error mentioned below.      File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/agent.py", line 710, in start     self._init_after_first_heartbeat_response(resp_data)   File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/agent.py", line 840, in _init_after_first_heartbeat_response     self.client_configs.load()   File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/client_configs.py", line 682, in load     new_deployed.update(self._lookup_alternatives(fname))   File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/client_configs.py", line 432, in _lookup_alternatives     return self._parse_alternatives(alt_name, out)   File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.2-py2.7.egg/cmf/client_configs.py", line 444, in _parse_alternatives     path, _, _, priority_str = line.rstrip().split(" ") ValueError: too many values to unpack       I tried to see the jdk versions. But same jdk is present in all other servers. Not sure why am having issue only with this server.   [root@ip-10-0-1-32 ~]# rpm -qa "*jdk*" java-1.8.0-openjdk-devel-1.8.0.144-0.b01.el7_4.x86_64 java-1.8.0-openjdk-headless-1.8.0.144-0.b01.el7_4.x86_64 jdk1.8.0_102-1.8.0_102-fcs.x86_64 copy-jdk-configs-2.2-3.el7.noarch java-1.8.0-openjdk-1.8.0.144-0.b01.el7_4.x86_64     Thanks   ... View more

The way things are implemented, a MapJoin optimization will always use local task operation.  If you would like to remove all instances of local tasks, you will have to disable MapJoins.   Please examine these two explain plans (first with MapJoin enabled, second with disabled)     | STAGE PLANS: | | Stage: Stage-5 | | Map Reduce Local Work | | Alias -> Map Local Tables: | | s07 | | Fetch Operator | | limit: -1 | | Alias -> Map Local Operator Tree: | | s07 | | TableScan | | alias: s07 | | filterExpr: code is not null (type: boolean) | | Statistics: Num rows: 225 Data size: 46055 Basic stats: COMPLETE Column stats: NONE | | Filter Operator | | predicate: code is not null (type: boolean) | | Statistics: Num rows: 113 Data size: 23129 Basic stats: COMPLETE Column stats: NONE | | HashTable Sink Operator | | keys: | | 0 code (type: string) | | 1 code (type: string) |       | STAGE PLANS: | | Stage: Stage-1 | | Map Reduce | | Map Operator Tree: | | TableScan | | alias: s07 | | filterExpr: code is not null (type: boolean) | | Statistics: Num rows: 225 Data size: 46055 Basic stats: COMPLETE Column stats: NONE | | Filter Operator | | predicate: code is not null (type: boolean) | | Statistics: Num rows: 113 Data size: 23129 Basic stats: COMPLETE Column stats: NONE | | Reduce Output Operator | | key expressions: code (type: string) | | sort order: + | | Map-reduce partition columns: code (type: string) | | Statistics: Num rows: 113 Data size: 23129 Basic stats: COMPLETE Column stats: NONE | | value expressions: description (type: string), salary (type: int) | | TableScan | | alias: s08 | | filterExpr: code is not null (type: boolean) | | Statistics: Num rows: 442 Data size: 46069 Basic stats: COMPLETE Column stats: NONE | | Filter Operator | | predicate: code is not null (type: boolean) | | Statistics: Num rows: 221 Data size: 23034 Basic stats: COMPLETE Column stats: NONE | | Reduce Output Operator | | key expressions: code (type: string) | | sort order: + | | Map-reduce partition columns: code (type: string) | | Statistics: Num rows: 221 Data size: 23034 Basic stats: COMPLETE Column stats: NONE | | value expressions: salary (type: int) |     We can see that the first one uses "Map Reduce Local Work" and the second one does not.   set hive.auto.convert.join=false;   https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties   This can be important becaue I'm seeing a case where the Local Job Runners are leaking the log file output from these Local Job Runners into the HS2's /tmp directory in the following format:   /tmp/hive_20171219184242_3ecaf468-51c7-4ced-99b3-6bd9eaaa980a.log   Disable the MapJoin optimization and these log files are not generated. ... View more

Glad to help, @parnigot! ... View more

In the logs for the ApplicationMaster/SparkDriver (which was around 4GB) I've found a StackOverflowError from Spark reporter thread: I've found this Spark issue https://issues.apache.org/jira/browse/SPARK-18750 that matches my error.   The job was launched used dynamicAllocation and requested an insane number of containers (16000 with 20GB/8cores) and apparently this can cause a SO in the Spark thread managing the executors.   An easy workaround is to disable dynamicAllocation and use a fixed number of executor. With 10 executors the job is running fine.     ... View more