Reply
Contributor
Posts: 66
Registered: ‎12-30-2015

What is the recommended way of restarting Impala cluster?

Hello,

 

I'd like to ask if there is any recommended way of restarting Impala cluster.

 

Here are the steps I typically use to restart Impala cluster.

- stop impala daemon ( in all nodes )

- stop catalog daemon

- stop statestore daemon

-------

- start statestore daemon

- start catalog daemon

- start impala daemon ( in all nodes )

- run queries warming tables

 

 

Today, I used a little different way to restart Impala cluster.

Shutting down steps are same.

Restarting steps are a little difrerent.

- start statestore daemon

- start catalog daemon

- start impala daemon ( only WN1 )

- run queries warming tables

- start impala daemon ( rest of WN# except WN1 )

( 90min after)

 

I tried to run a query on one of WN.

However, on many WN, I saw this kind of log or msg.

 

There was an error processing the impalad catalog update. Requesting a full topic update to recover: NegativeArraySizeException: null

 

 

Question1:

What is the recommended way of restarting Impala cluster when crashed or not crashed?

 

Question2:

What is wrong with the restarting steps I did today?

 

Thank you

Gatsby

 

Contributor
Posts: 66
Registered: ‎12-30-2015

Re: What is the recommended way of restarting Impala cluster?

any suggestion?

Explorer
Posts: 11
Registered: ‎08-13-2014

Re: What is the recommended way of restarting Impala cluster?

Generally we try to stop our automated jobs and just restart the whole service in cloudera manager and wait a bit and it comes back fine.

 

We've been observing the same NegativeArraySizeException error in our logs on impalads and usually once we see it on one node, restarting that node will not fix it and eventually other nodes hit it so we just end up restarting the whole cluster including catalogserver / statestore and it goes away from a while... until it comes back again.

 

Anyone know the best way to further debug this weird Exception?  Here's some sample logs around the exception.

 

I0310 19:23:48.008160 29162 thrift-server.cc:387] ThriftServer 'StatestoreSubscriber' started on port: 23000
I0310 19:23:48.008188 29162 statestore-subscriber.cc:203] Registering with statestore
I0310 19:23:48.009297 29162 statestore-subscriber.cc:176] Subscriber registration ID: 214328a84bdbb9ee:dc01031b408e4bbb
I0310 19:23:48.009317 29162 statestore-subscriber.cc:207] statestore registration successful
I0310 19:23:48.013540 29162 thrift-server.cc:387] ThriftServer 'beeswax-frontend' started on port: 21000
I0310 19:23:48.018061 29162 thrift-server.cc:387] ThriftServer 'hiveserver2-frontend' started on port: 21050
I0310 19:23:48.018076 29162 impalad-main.cc:89] Impala has started.
I0310 19:24:07.921928 29224 logging.cc:193] Old log file deleted during log rotation: /var/log/impalad/impalad.hdws78.houzz.com.impala.log.INFO.20160307-181535.6444
I0310 19:24:07.922178 29224 logging.cc:193] Old log file deleted during log rotation: /var/log/impalad/impalad.hdws78.houzz.com.impala.log.WARNING.20160307-175317.28369
I0310 19:24:07.922299 29224 logging.cc:193] Old log file deleted during log rotation: /var/log/impalad/impalad.hdws78.houzz.com.impala.log.ERROR.20160307-175317.28369
I0310 19:24:14.841969 29479 impala-server.cc:1176] Catalog topic entry deletion: TABLE:sumit.sessions_rnked
I0310 19:24:24.161226 29479 jni-util.cc:177] java.lang.NegativeArraySizeException
I0310 19:24:24.163894 29479 status.cc:114] NegativeArraySizeException: null
    @           0x7800f7  (unknown)
    @           0xa5d84d  (unknown)
    @           0x969a7a  (unknown)
    @           0x970261  (unknown)
    @           0x96919a  (unknown)
    @           0x997d61  (unknown)
    @           0xa32f61  (unknown)
    @           0xa380be  (unknown)
    @           0xb550e4  (unknown)
    @           0xb52bba  (unknown)
    @           0x99a606  (unknown)
    @          0x15a5f1d  (unknown)
    @           0x91be13  (unknown)
    @           0x91cf98  (unknown)
    @           0xa9e01d  (unknown)
    @           0xa9e81e  (unknown)
    @           0xca0a73  (unknown)
    @     0x7f8b05ae8e9a  start_thread
    @     0x7f8b043af38d  (unknown)
E0310 19:24:24.163928 29479 impala-server.cc:1208] There was an error processing the impalad catalog update. Requesting a full topic update to recover: NegativeArraySizeException: null
I0310 19:24:26.412088 29479 simple-scheduler.cc:338] Registering local backend with statestore
I0310 19:24:54.873220 29479 impala-server.cc:1176] Catalog topic entry deletion: TABLE:sumit.sessions_rnked
I0310 19:25:04.085335 29479 jni-util.cc:177] java.lang.NegativeArraySizeException
I0310 19:25:04.087870 29479 status.cc:114] NegativeArraySizeException: null
Highlighted
Explorer
Posts: 16
Registered: ‎11-25-2015

Re: What is the recommended way of restarting Impala cluster?

we just hit the same issue... CDH 5.4 ... is there a know fix?