About Java 8 support

[#1]

In your documentation, you mention that Java 8 is also supported. But according to hadoop documentation, they mention only Java 7, and there is a separate branch of Hadoop where they do porting to Java 8 (and hadoop-2.8 is expected to be first release with Java 8 support).

How is it possible you support Java 8 with (currently) hadoop-2.7.1? Do you backport their changes from Java8 branch?

Thanks.

[#2 · Accepted solution]

Java 6, 7, and 8 were quite FORWARD-compatible, meaning that programs that ran in earlier versions generally would run successfully in later versions. At the same time, with Enterprise software one cannot just assume such compatibility, one must test and certify. On the other hand, each new Java version has added language features that are not BACKWARD-compatible, meaning that a program that uses new Java 8 language features will not be able to run under Java 7. Thus it would require Java 8.

To date, Hadoop has only required Java 7. I looked at the Apache docs, and I think the work being done in branch-2.8 is only to make sure it is COMPATIBLE with Java 8, not REQUIRING Java 8. At some point in the future, however, there will be a branch designated to use more efficient new Java 8 language constructs, and therefore require Java 8 to be installed in the server. Once the Hadoop community accepts the requirement for Java 8, from that version forward Hadoop will no longer run successfully in Java 7. In the meantime they are making sure that from hadoop-2.8 forward, it is at least assured of being compatible with Java 8, for users who prefer that.

Looking back over the "JDK Requirements" in the install documentation for various versions of HDP:

• HDP-2.1
• HDP-2.2
• HDP-2.3
• HDP-2.4

We see that Java 6 started to be deprecated in HDP-2.1, remained usable with HDP-2.2, but was not compatible with HDP-2.3. That's because in HDP-2.3 we started using versions of the Hadoop stack components that utilized Java 7 language features and therefore were no longer compatible with Java 6.

Starting with HDP-2.3 we continued supporting Java 7, but also started certifying with Java 8. No Java 8 language features were used (so that we could still support Java 7), but we tested with both Java 7 and 8 and certified that HDP-2.3 and HDP-2.4 worked with them both. That is the situation so far today.

[#3]

following this question about jdk8 compatibility - sorry for rising things from the grave but ....

can HDP 1.1.2.21 work with jdk8 ?

i am having issue with memory on map reduce jobs since we moved to jdk8,

like :

TaskTree [pid=13508,tipID=attempt_201709180208_0675_m_000000_0] is running beyond memory-limits. Current usage : 2583085056bytes. Limit : 1610612736bytes. Killing task. TaskTree [pid=13508,tipID=attempt_201709180208_0675_m_000000_0] is running beyond memory-limits. Current usage : 2583085056bytes. Limit : 1610612736bytes. Killing task.

is something in the mapred configuration should change , and if yes - what ?