storm nimbus error

[#1]

I am using Metron 0.5.0 and HDP 2.6.5 in 11 nodes.

when I check nimbus log I sea this Error:

How can I solve it?

2018-11-20 03:29:39.572 o.a.s.b.BlobStoreUtils timer [ERROR] Could not update the blob with keyyaf-3-1542565827-stormconf.ser
2018-11-20 03:29:39.579 o.a.s.b.BlobStoreUtils timer [ERROR] Could not update the blob with keyyaf-3-1542565827-stormcode.ser
2018-11-20 03:29:39.591 o.a.s.b.BlobStoreUtils timer [ERROR] Could not update the blob with keybro-2-1542565816-stormconf.ser
2018-11-20 03:29:39.598 o.a.s.b.BlobStoreUtils timer [ERROR] Could not update the blob with keybro-2-1542565816-stormcode.ser
2018-11-20 03:29:39.609 o.a.s.b.BlobStoreUtils timer [ERROR] Could not update the blob with keyenrichment-1-1542565719-stormconf.ser
2018-11-20 03:29:39.616 o.a.s.b.BlobStoreUtils timer [ERROR] Could not update the blob with keyenrichment-1-1542565719-stormcode.ser
2018-11-20 03:29:39.646 o.a.s.b.BlobStoreUtils timer [ERROR] Could not update the blob with keyprofiler-7-1542566049-stormconf.ser
2018-11-20 03:29:39.653 o.a.s.b.BlobStoreUtils timer [ERROR] Could not update the blob with keyprofiler-7-1542566049-stormcode.ser
2018-11-20 03:29:39.664 o.a.s.b.BlobStoreUtils timer [ERROR] Could not update the blob with keysnort-4-1542565838-stormconf.ser
2018-11-20 03:29:39.672 o.a.s.b.BlobStoreUtils timer [ERROR] Could not update the blob with keysnort-4-1542565838-stormcode.ser

,....

[#2]

@Jay Kumar SenSharma @Ambud Sharma @Teng Wang Could you help me for solving this error?

[#3]

@mojgan ghasemi

I am not much familiar with the mentioned error. However i will research a bit on this to be helpful on this thread.

Can you please provide a bit more context of this error? Like when exactly do you see this error and is it releating error?

Have you recently started seeing this error Or this error is there in the klogs for a long time / from beginning.

By any chance can you attach the whole log starting from Nimbus start till this error ... it might be a sideeffect of any perviously logged error in the log.

[#4]

Thank you for your answer

I sent squid into metron. I sea it in the Storm UI but I couldn't get it in the HDFS I checked Strom'log and I saw this Error it started about 4 days age

after I sea Error, I searched and restarted zookeeper but it wan't fixed.

2018-11-17 23:04:51.788 o.a.s.z.Zookeeper main-EventThread [INFO] active-topology-dependencies [] local-blobs [bro-2-1542457228-stormjar.jar,snort-4-1542457250-stormcode.ser,yaf-3-1542457239-stormconf.ser,bro-2-1542457228-stormconf.ser,yaf-3-1542457239-stormjar.jar,bro-2-1542457228-stormcode.ser,enrichment-1-1542457194-stormjar.jar,enrichment-1-1542457194-stormconf.ser,yaf-3-1542457239-stormcode.ser,snort-4-1542457250-stormjar.jar,snort-4-1542457250-stormconf.ser,enrichment-1-1542457194-stormcode.ser] diff-topology-dependencies []
2018-11-17 23:04:51.788 o.a.s.z.Zookeeper main-EventThread [INFO] Accepting leadership, all active topologies and corresponding dependencies found locally.
2018-11-17 23:04:51.947 o.a.s.s.o.a.z.ClientCnxn main-SendThread(node2.node:2181) [INFO] Opening socket connection to server node2.node/localhost:port. Will not attempt to authenticate using SASL (unknown error)
2018-11-17 23:04:51.947 o.a.s.s.o.a.z.ClientCnxn main-SendThread(node2.node:2181) [INFO] Socket connection established, initiating session, client: /10.0.13.2:58692, server: node2.node/10.0.13.2:2181
2018-11-17 23:04:51.953 o.a.s.s.o.a.z.ClientCnxn main-SendThread(node2.node:2181) [INFO] Unable to reconnect to ZooKeeper service, session 0x16720aab036000d has expired, closing socket connection
2018-11-17 23:04:51.956 o.a.s.s.o.a.c.f.s.ConnectionStateManager main-EventThread [INFO] State change: LOST
2018-11-17 23:04:51.956 o.a.s.c.zookeeper-state-factory main-EventThread [WARN] Received event :expired::none: with disconnected Writer Zookeeper.
2018-11-17 23:04:51.957 o.a.s.s.o.a.c.ConnectionState main-EventThread [WARN] Session expired event received
2018-11-17 23:04:51.957 o.a.s.s.o.a.z.ZooKeeper main-EventThread [INFO] Initiating client connection, connectString=node2.node:2181/storm sessionTimeout=30000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@5190010f
2018-11-17 23:04:51.965 o.a.s.s.o.a.z.ClientCnxn main-EventThread [INFO] EventThread shut down
2018-11-17 23:04:51.971 o.a.s.s.o.a.z.ClientCnxn main-SendThread(node2.node:2181) [INFO] Opening socket connection to server node2.node/localhost:port. Will not attempt to authenticate using SASL (unknown error)
2018-11-17 23:04:51.971 o.a.s.s.o.a.z.ClientCnxn main-SendThread(node2.node:2181) [INFO] Socket connection established, initiating session, client: /10.0.13.2:58696, server: node2.node/10.0.13.2:2181
2018-11-17 23:04:51.975 o.a.s.s.o.a.z.ClientCnxn main-SendThread(node2.node:2181) [INFO] Session establishment complete on server node2.node/localhost:port, sessionid = 0x16720aab03600c4, negotiated timeout = 30000
2018-11-17 23:04:51.976 o.a.s.s.o.a.c.f.s.ConnectionStateManager main-EventThread [INFO] State change: RECONNECTED




2018-11-17 23:04:52.579 o.a.s.b.BlobStoreUtils timer [ERROR] Could not update the blob with keyyaf-3-1542565827-stormconf.ser
2018-11-17 23:04:53.579 o.a.s.b.BlobStoreUtils timer [ERROR] Could not update the blob with keyyaf-3-1542565827-stormcode.ser
2018-11-17 23:04:54.579 o.a.s.b.BlobStoreUtils timer [ERROR] Could not update the blob with keybro-2-1542565816-stormconf.ser
2018-11-17 23:04:55.579 o.a.s.b.BlobStoreUtils timer [ERROR] Could not update the blob with keybro-2-1542565816-stormcode.ser
2018-11-17 23:04:56.579 o.a.s.b.BlobStoreUtils timer [ERROR] Could not update the blob with keyenrichment-1-1542565719-stormconf.ser