Zeppelin and Java 8 on HDP 2.5

[#1]

We could install Zeppelin through Ambari, and it runs with green light. However, when I open and try %sql (I guess Spark interpereter), it got error. When I check the zeppelin log file, I see the out of memory error with -XX:MaxPermSize=512m. I know this parameter is untill jdk 7. The jdk 8 has a different new parameter. I'm certain we need to use ZEPPELIN_MEM to set the new parameter, but I don't know how to do it correctly through Ambari.

Could anyone help on this?

[#2]

@Shigeru Takehara

There are various kind of OutOfMemory errors. (Example: OutOfMemory in Native Space, OurOfMemory in Heap Space, OutOfMemory due to GC overhead....etc)

As you said "I see the out of memory error with -XX:MaxPermSize=512m", Can you please let us know which kind of OutOfMemory error are you getting? Sharing log will be more useful.

Even if you are using Java8 and still if the "-XX:MaxPermSize=512m" is applied then java8 will simply ignore it.

.

If you want to make any changes in the Zeppelin memory settings then you can login to ambari UI and the navigate to

"Zeppelin Notebook" --> "Configs" (Tab) --> "Advanced zeppelin-env"

'and then find the "zeppelin_env_content" there you will find "export ZEPPELIN_MEM" that you can edit.

[#3]

log file says:

INFO [2017-04-07 11:53:58,682] ({pool-2-thread-4} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1491580438681 started by scheduler org.apache.zeppelin.spark.SparkInterpreter283573820 INFO [2017-04-07 11:54:02,145] ({pool-2-thread-4} Logging.scala[logInfo]:58) - Changing view acls to: zeppelin INFO [2017-04-07 11:54:02,145] ({pool-2-thread-4} Logging.scala[logInfo]:58) - Changing modify acls to: zeppelin INFO [2017-04-07 11:54:02,145] ({pool-2-thread-4} Logging.scala[logInfo]:58) - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(zeppelin); users with modify permissions: Set(zeppelin) INFO [2017-04-07 11:54:05,322] ({pool-2-thread-4} Logging.scala[logInfo]:58) - Starting HTTP Server INFO [2017-04-07 11:54:05,945] ({pool-2-thread-4} Server.java[doStart]:272) - jetty-8.y.z-SNAPSHOT INFO [2017-04-07 11:54:08,337] ({pool-2-thread-4} AbstractConnector.java[doStart]:338) - Started SocketConnector@0.0.0.0:22900 INFO [2017-04-07 11:54:08,338] ({pool-2-thread-4} Logging.scala[logInfo]:58) - Successfully started service 'HTTP class server' on port 22900. ERROR [2017-04-07 11:54:10,658] ({pool-2-thread-4} Job.java[run]:189) - Job failed java.lang.OutOfMemoryError: GC overhead limit exceeded INFO [2017-04-07 11:54:10,658] ({pool-2-thread-4} SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1491580438681 finished by scheduler org.apache.zeppelin.spark.SparkInterpreter283573820 INFO [2017-04-07 11:54:11,954] ({pool-1-thread-18} Logging.scala[logInfo]:58) - Changing view acls to: zeppelin INFO [2017-04-07 11:54:11,954] ({pool-1-thread-18} Logging.scala[logInfo]:58) - Changing modify acls to: zeppelin INFO [2017-04-07 11:54:11,955] ({pool-1-thread-18} Logging.scala[logInfo]:58) - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(zeppelin); users with modify permissions: Set(zeppelin)

"xxx.out" file says:

Exception in thread "qtp686649452-427" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1855) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2068) at org.spark-project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.spark-project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.spark-project.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) Exception in thread "pool-1-thread-4" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-1-thread-4" Exception in thread "pool-1-thread-7" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "pool-1-thread-6" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "pool-1-thread-9" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "qtp1834748429-966" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "pool-1-thread-8" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "pool-1-thread-13" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "pool-1-thread-12" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "pool-1-thread-11" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "pool-1-thread-1" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "pool-1-thread-3" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "qtp184565796-1023" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "qtp1834748429-962" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "qtp851486649-779" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "qtp1459091505-28" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "qtp1613009598-14753" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "pool-1-thread-18" java.lang.OutOfMemoryError: GC overhead limit exceeded

[#4 · Accepted solution]

@Shigeru Takehara

From your logs we see that it is OutOfMemory of type "GC overhead limit exceeded"

.

The detail message "GC overhead limit exceeded" indicates that the garbage collector is running all the time and Java program is making very slow progress. After a garbage collection, if the Java process is spending more than approximately 98% of its time doing garbage collection and if it is recovering less than 2% of the heap and has been doing so far the last 5 (compile time constant) consecutive garbage collections, then a java.lang.OutOfMemoryError is thrown.

This exception is typically thrown because the amount of live data barely fits into the Java heap having little free space for new allocations.

So initially you should try increasing the "-Xmx" (Heap Size) of Zeppelin from Ambari UI.

[#5]

Yes, that's right. On Ambari, should we add zeppelin_mem, let's say -XmXXXXm to Custom zeppelin-env?

[#6]

@Shigeru Takehara

If you want to make any changes in the Zeppelin memory settings then you can login to ambari UI and the navigate to

"Zeppelin Notebook" --> "Configs" (Tab) --> "Advanced zeppelin-env"

'and then find the "zeppelin_env_content" there you will find "export ZEPPELIN_MEM" that you can edit.

[#7]

I added:

export ZEPPELIN_MEM="-Xms1024m -Xmx2048m -XX:MaxMetaspaceSize=512m"

The out of memory issue was fixed; however,

I got another new issue related to yarn configuration. I will ask it with "new question".

Thanks!