Support Questions
Find answers, ask questions, and share your expertise

Problem with spark atlas connector

Problem with spark atlas connector

Rising Star

Hi all,


we have a problem with spark ThriftServer when enable Spark Atlas Connector.


When activate SAC we received this error at start:

 

 

21/01/11 08:25:37 INFO ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
21/01/11 08:25:37 INFO SessionState: Created local directory: /tmp/8fb9febe-2e96-47c1-b0a6-416b46c204d7_resources
21/01/11 08:25:37 INFO SessionState: Created HDFS directory: /tmp/spark/spark/8fb9febe-2e96-47c1-b0a6-416b46c204d7
21/01/11 08:25:37 INFO SessionState: Created local directory: /tmp/spark/8fb9febe-2e96-47c1-b0a6-416b46c204d7
21/01/11 08:25:37 INFO SessionState: Created HDFS directory: /tmp/spark/spark/8fb9febe-2e96-47c1-b0a6-416b46c204d7/_tmp_space.db
21/01/11 08:25:37 INFO HiveSessionImpl: Operation log session directory is created: /tmp/spark/operation_logs/8fb9febe-2e96-47c1-b0a6-416b46c204d7
21/01/11 08:25:37 INFO StreamingQueryManager: Registered listener com.hortonworks.spark.atlas.SparkAtlasStreamingQueryEventTracker
21/01/11 08:25:38 ERROR TThreadPoolServer: Thrift error occurred during processing of message.
org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:374)
at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:451)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:433)
at org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:43)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:425)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:321)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:225)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:53)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

 

 


After this error the cycle starts again with same error, every 1 minute.
 

All works correctly (on atlas we see the aging of the data on spark), but after 24h spark going down (all time for ThriftServer, History Server sometimes stay up).

 

ThriftServer stopped after 975 attempts to registry the listener:

 

 

[root@coordinator03 spark2]# grep -o 'Registered listener com.hortonworks.spark.atlas.SparkAtlasStreamingQueryEventTracker' spark-spark-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-coordinator03.out.2 | wc -l
975
[root@coordinator03 spark2]# grep -o 'Registered listener com.hortonworks.spark.atlas.SparkAtlasStreamingQueryEventTracker' spark-spark-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-coordinator03.out.3 | wc -l
975
[root@coordinator03 spark2]# grep -o 'Registered listener com.hortonworks.spark.atlas.SparkAtlasStreamingQueryEventTracker' spark-spark-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-coordinator03.out.4 | wc -l
975
[root@coordinator03 spark2]# grep -o 'Registered listener com.hortonworks.spark.atlas.SparkAtlasStreamingQueryEventTracker' spark-spark-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-coordinator03.out.5 | wc -l
975

 

 

 

The final error before stopped service is this:

 

 

21/01/12 00:35:37 ERROR ThriftCLIService: Error starting HiveServer2: could not start ThriftBinaryCLIService
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
at org.apache.thrift.server.TThreadPoolServer.execute(TThreadPoolServer.java:192)
at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:175)
at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:99)
at java.lang.Thread.run(Thread.java:745)

 

 


Any idea for resolution?
At the moment we disable SAC.
Thanks a lot