Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Segmentation Fault in libjvm.so

Highlighted

Segmentation Fault in libjvm.so

New Contributor

We run CDH 5 with Impala 2.1.0 on a 6-node cluster. When trying to create a parquet table with the following statement we encounter a segmentation fault:

 

INSERT INTO tpch_parquet.lineitem SELECT * FROM default.lineitem;

The query is part of a longer script important the TPCH data set, which can be found here: init.sql

 

When running this query, I can see backend by backend disappearing on the web page (http://...:25000/backends). Looking in the logs of the disappearing server, no interesting messages can be found (impalad.INFO).

 

The system log shows that a segfault occured:

 

Feb 17 23:23:33 MY_HOSTNAME kernel: [1458634.269419] impalad[32664]: segfault at 7fb348c03380 ip 00007fb346a7f4fc sp 00007fb2e8002ee0 error 7 in libjvm.so[7fb346106000+b73000]

From my research most segfaults in libjvm.so occur, when using the wrong java version. We use Oracle Java 1.7.

 

What could be the problem?

4 REPLIES 4
Highlighted

Re: Segmentation Fault in libjvm.so

Cloudera Employee

This might not indicate a crash in the JVM. The embedded JVM running in the Impala process sets up a signal handler for segfaults, which causes all segfaults to go through the JVM, even ones that originated in native non-JVM code.

 

Are you sure the log file is from the crashing impalad? This log is from an impalad that just started, it hasn't even received any queries to run.

Highlighted

Re: Segmentation Fault in libjvm.so

New Contributor

I just restarted impala on all nodes and run the script (init.sql) from above again. On the impalad website I can see that 2 of 6 backends backends are gone. The query summary of the failed query says "exception" as status. When looking at the query summary, status says:

Status: Cancelled due to unreachable impalad(s): NODE5:22000, NODE3:22000

 (I obscured the real host names).

 

This is the impalad.INFO.

Highlighted

Re: Segmentation Fault in libjvm.so

New Contributor

It actually seems like the segmentation fault only occurs when I run the following query:

 

INSERT INTO tpch_parquet.lineitem 	SELECT * FROM default.lineitem;

All of the following queries work fine:

 

INSERT INTO tpch_parquet.part 		SELECT * FROM default.part;
INSERT INTO tpch_parquet.supplier 	SELECT * FROM default.supplier;
INSERT INTO tpch_parquet.partsupp 	SELECT * FROM default.partsupp;
INSERT INTO tpch_parquet.nation 	SELECT * FROM default.nation;
INSERT INTO tpch_parquet.region 	SELECT * FROM default.region;
INSERT INTO tpch_parquet.customer	SELECT * FROM default.customer;
INSERT INTO tpch_parquet.orders 	SELECT * FROM default.orders;

The table lineitem is by far the largest table with a 74.11 GB csv file in HDFS. Might that be a problem somehow?

Highlighted

Re: Segmentation Fault in libjvm.so

New Contributor

I found a workaround: When generating statistics first, I can run the problematic query. So it seems like Impala can't handle the 70GB file?

Don't have an account?
Coming from Hortonworks? Activate your account here