Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Metron squid demo not indexing

Metron squid demo not indexing

Explorer

Hey,

im new in the metron world and i was trying the squid demo for this link.

https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%...

everything worked as expected except for the index in the elasticsearch it doesn't show using the below link.

http://localhost:9200/_cat/indices?v

health status index   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   twitter IlyhgBkVS3ejRrMdeeNdzw   5   1          0            0       810b           810b
yellow open   .kibana w44VwW9_RyC2QvVJ9RU3GQ   1   1         46            4    126.7kb        126.7kb

i have 5 node cluster installed using Ambari.

currently the storm UI log shows no errors as follows below:

Capture.JPG

 

2020-01-29 15:33:35.231 o.a.m.p.GrokParser Thread-13-parserBolt-executor[5 5] [DEBUG] Grok parser formatting timestamp 1580109305.258
2020-01-29 15:33:35.231 o.a.m.p.GrokParser Thread-13-parserBolt-executor[5 5] [DEBUG] Grok parser parsed message: {elapsed=355, code=301, ip_dst_addr=104.108.35.126, original_string=1580109305.258    355 127.0.0.1 TCP_MISS/301 2179 GET http://www.aliexpress.com/af/shoes.html? - HIER_DIRECT/104.108.35.126 -, method=GET, bytes=2179, action=TCP_MISS, ip_src_addr=127.0.0.1, url=http://www.aliexpress.com/af/shoes.html?, timestamp=1580109305258}
2020-01-29 15:33:35.231 o.a.m.s.c.CachingStellarProcessor Thread-13-parserBolt-executor[5 5] [DEBUG] Created cache key; org.apache.metron.stellar.common.CachingStellarProcessor$Key@500373cc[expression=URL_TO_HOST(url),input={url=http://www.aliexpress.com/af/shoes.html?}]
2020-01-29 15:33:35.231 o.a.m.s.c.CachingStellarProcessor Thread-13-parserBolt-executor[5 5] [DEBUG] Created cache key; org.apache.metron.stellar.common.CachingStellarProcessor$Key@19fc94d6[expression=DOMAIN_REMOVE_SUBDOMAINS(full_hostname),input={full_hostname=www.aliexpress.com}]
2020-01-29 15:33:35.232 o.a.m.p.GrokParser Thread-13-parserBolt-executor[5 5] [DEBUG] Grok parser validating message: {full_hostname=www.aliexpress.com, code=301, method=GET, url=http://www.aliexpress.com/af/shoes.html?, source.type=squid, elapsed=355, ip_dst_addr=104.108.35.126, original_string=1580109305.258    355 127.0.0.1 TCP_MISS/301 2179 GET http://www.aliexpress.com/af/shoes.html? - HIER_DIRECT/104.108.35.126 -, bytes=2179, domain_without_subdomains=aliexpress.com, action=TCP_MISS, guid=cb522852-5d23-46a5-a39f-f2ded6dfc8a0, ip_src_addr=127.0.0.1, timestamp=1580109305258}
2020-01-29 15:33:35.232 o.a.m.p.GrokParser Thread-13-parserBolt-executor[5 5] [DEBUG] Grok parser validated message: {full_hostname=www.aliexpress.com, code=301, method=GET, url=http://www.aliexpress.com/af/shoes.html?, source.type=squid, elapsed=355, ip_dst_addr=104.108.35.126, original_string=1580109305.258    355 127.0.0.1 TCP_MISS/301 2179 GET http://www.aliexpress.com/af/shoes.html? - HIER_DIRECT/104.108.35.126 -, bytes=2179, domain_without_subdomains=aliexpress.com, action=TCP_MISS, guid=cb522852-5d23-46a5-a39f-f2ded6dfc8a0, ip_src_addr=127.0.0.1, timestamp=1580109305258}
2020-01-29 15:33:35.232 o.a.m.w.AckTuplesPolicy Thread-13-parserBolt-executor[5 5] [DEBUG] Adding tuple with messages ids: cb522852-5d23-46a5-a39f-f2ded6dfc8a0
2020-01-29 15:33:35.291 o.a.s.u.LocalState heartbeat-timer [DEBUG] New Local State for /hadoop/storm/workers/5829d9a2-da96-49fc-b594-9d5d50fe2af6/heartbeats
2020-01-29 15:33:35.410 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:35.411 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:35.612 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:35.613 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:35.813 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:35.814 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:36.015 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:36.016 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:36.217 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:36.218 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:36.293 o.a.s.u.LocalState heartbeat-timer [DEBUG] New Local State for /hadoop/storm/workers/5829d9a2-da96-49fc-b594-9d5d50fe2af6/heartbeats
2020-01-29 15:33:36.418 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:36.419 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:36.620 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:36.621 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:36.821 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:36.831 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:37.031 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:37.033 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:37.215 o.a.k.c.c.i.AbstractCoordinator Thread-15-kafkaSpout-executor[4 4] [DEBUG] Received successful heartbeat response for group squid_parser
2020-01-29 15:33:37.233 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:37.234 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:37.295 o.a.s.u.LocalState heartbeat-timer [DEBUG] New Local State for /hadoop/storm/workers/5829d9a2-da96-49fc-b594-9d5d50fe2af6/heartbeats
2020-01-29 15:33:37.324 o.a.s.s.o.a.z.ClientCnxn main-SendThread(mdr-3:2181) [DEBUG] Reading reply sessionid:0x36fc314e93528ce, packet:: clientPath:null serverPath:null finished:false header:: 353,3  replyHeader:: 353,4298258571,0  request:: '/storm/workerbeats/squid-73-1580304550/ab01252f-a649-4317-9e74-21acf9289615-6701,F  response:: s{4298257363,4298258548,1580304569310,1580304824885,85,0,0,0,550,0,4298257363} 
2020-01-29 15:33:37.327 o.a.s.s.o.a.z.ClientCnxn main-SendThread(mdr-3:2181) [DEBUG] Reading reply sessionid:0x36fc314e93528ce, packet:: clientPath:null serverPath:null finished:false header:: 354,5  replyHeader:: 354,4298258572,0  request:: '/storm/workerbeats/squid-73-1580304550/ab01252f-a649-4317-9e74-21acf9289615-6701,#1fffffff8b80000000ffffffed574b6effffffc23010ffffff9dfffffffc5a20ffffffc9ffffffa2ffffffa2ffffff87ffffffe8629ffffff884f5137ffffffad7affffff85ffffffee6b59105048ffffff9affffff9845ffffff8fffffffd0ffffffe3fffffff466ffffffbd411bfffffff2ffffff81ffffffd436ffffffce1022554544ffffff8a6211ffffffbf673cfffffff3fffffffc6662ffffff8360ffffffedffffffe86d134c3afffffff7ffffffbd4e7730fffffff27a5e7f30fffffff05cffffffd01d277e77ffffffd548ffffffa6ffffffe8dffffffd0ffffffb7f7041ffffffb3ffffffdd7814ffffffff474ffffffe0ffffff80ffffffe1ffffff80ffffff96ffffffff6effffffc6ffffffb7ffffffd5fffffff546ffffff9effffffe7ffffffb4ffffffe2ffffffa16627fffffff0621ffffffd17bffffffc4fffffffcffffffa5ffffff9dffffffd0ffffffb4959fffffffa2ccffffffc611ffffffa1ffffffb359ffffffe8ffffffcf28fffffff321ffffffbb6effffffe3ffffffdb18ffffff9e4ffffffb746ffffffc3fffffffe494ffffffcd7ffffffba587458ffffffb0fffffff4ffffffabffffff92ffffffecffffffa2ffffffe3ffffff82ffffffc1ffffffc7ffffffc5ffffffac449795cffffffaaffffffc3ffffffb3ffffffb8542728ffffffc4ffffffa522ffffff89bffffffe6452645ffffff9958177514494c30ffffff9ffffffffaffffffd938751e23751effffffa3ffffffd4794038733bffffffb5411e67ffffffe9a5affffffbaffffff8256ffffffd1ffffffdb5a6b1a467efffffff8ffffffbc5affffffb074ffffff8b4d42ffffffe878ffffffbeffffffbd556245ffffffc244ffffff9122ffffff81727162ffffffc048ffffffef42501ffffff88ffffffaa44ffffffc2443522ffffff81721562ffffffc06a6f3affffffc334ffffffabffffffbcffffffe7ffffffecffffffb22bffffff9cfffffffcffffffeffffffff8ffffff8a4ffffffffefffffff61ffffffe29ffffffb7ffffff9368ffffff82ffffffd2ffffffb0ffffffeeffffffb2060ffffffe7ffffffa6ffffffbbffffffc6ffffffceffffffde6f15ffffff85ffffffc84fffffffed3fffffffde2ffffffdaffffffcf51ffffffddffffffc3ffffffcd7effffffa5ffffff88ffffffd1ffffff906dffffffd6ffffffb91010ffffffbdffffff831a5cffffffda39ffffffa8ffffffe1ffffffa8ffffffbe41417154ffffffd770ffffff987ffffffa0ffffffb46750ffffff834b3b6351cffffffd52f2829ffffffc45a6effffffa5ffffffb5ffffffdcffffffaaffffffe73bffffffe5ffffff8a1016ffffffa8fffffffdffffffbd1c512a1696ffffffa5101e57ffffffedffffff9b236779fffffffcffffffc2ffffff8a42ffffff822855ffffff824ffffff834a3fffffff8f3bffffffe007a567ffffffe6743affffffa72fffffffebffffffd526ffffffab3ffffffd7137f4a37bffffffb6ffffff8bffffffd1ffffffd169ffffffaeffffff9dffffffb920ffffff87ffffffdaffffffb939ffffffd9ffffffd4ffffffcb7fffffffc0754a17ffffffc953ffffff8759ffffffe4ffffff97fffffffa6a672effffffa8ffffffb4766e4effffffcd75fffffff3ffffff8b4e67ffffffa64e67ffffffaa7a172effffffd142ffffffb1ffffff97ffffffbcffffffe7affffffbc64ffffff86ffffffa4ffffffa8fffffff3ffffffb314ffffffbdffffffc85ffffffffdffffffa5ffffffc4fffffffd75456fffffffc1ffffffc7ffffffd878ffffffed7e7c36ffffffb65bffffffd04cfffffff81ffffffbaffffffcffffffffe59281300,-1  response:: s{4298257363,4298258572,1580304569310,1580304827890,86,0,0,0,551,0,4298257363} 
2020-01-29 15:33:37.434 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:37.436 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:37.636 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:37.638 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:37.838 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:37.840 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:38.040 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:38.041 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:38.243 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:38.244 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:38.298 o.a.s.u.LocalState heartbeat-timer [DEBUG] New Local State for /hadoop/storm/workers/5829d9a2-da96-49fc-b594-9d5d50fe2af6/heartbeats
2020-01-29 15:33:38.444 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:38.446 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:38.646 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:38.648 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:38.766 o.a.s.s.o.a.z.ClientCnxn main-SendThread(mdr-3:2181) [DEBUG] Reading reply sessionid:0x36fc314e93528ce, packet:: clientPath:null serverPath:null finished:false header:: 355,3  replyHeader:: 355,4298258581,0  request:: '/storm/assignments/squid-73-1580304550,T  response:: s{4298257313,4298257313,1580304561404,1580304561404,0,0,0,0,204,0,4298257313} 
2020-01-29 15:33:38.769 o.a.s.s.o.a.z.ClientCnxn main-SendThread(mdr-3:2181) [DEBUG] Reading reply sessionid:0x36fc314e93528ce, packet:: clientPath:null serverPath:null finished:false header:: 356,3  replyHeader:: 356,4298258581,0  request:: '/storm/storms/squid-73-1580304550,T  response:: s{4298257312,4298257312,1580304561076,1580304561076,0,0,0,0,194,0,4298257312} 
2020-01-29 15:33:38.769 o.a.s.s.o.a.z.ClientCnxn main-SendThread(mdr-3:2181) [DEBUG] Reading reply sessionid:0x36fc314e93528ce, packet:: clientPath:null serverPath:null finished:false header:: 357,4  replyHeader:: 357,4298258581,0  request:: '/storm/storms/squid-73-1580304550,T  response:: #1fffffff8b8000000035ffffff8e3deffffffc230cffffff855dffffffa0101affffffba70ffffff904870416ffffffa64e456223ffffff8a5a53ffffffa2ffffffd2ffffffa6ffffffd8ffffffe9ffffffc0d389ffffffe724fffffffdffffffc1ffffffd2ffffffb3ffffff9ffffffff42cfffffffbffffff9310140ffffffccffffffafffffffdeffffff96216ffffffc1472ffffff96ffffffc34861254530ffffffebffffffa0ffffffa436fffffff7ffffffdaffffffe4ffffff9dffffffebfffffffd105ffffffedffffff91ffffffc851ffffff86ffffffccffffffa6ffffffc22b59ffffff8f3427426b7effffffb3ffffffc76ffffffa63a6bffffffddffffffa0275bffffffb0ffffffa34affffff99ffffffce14f54f533affffffd7ffffffa939382affffffb66dffffffadffffffd83b6a543efffffff4ffffff8b6dfffffff0695bffffffccffffffa6ffffff853cffffffc4fffffff3fffffff9ffffffa433ffffffc44827fffffff7fffffffcffffffa36cffffffb436453dfffffffd1710ffffffdfeffffff9fffffffaf1cffffffa9ffffffe3fffffff160a5bffffffb91b40242423ffffffd0f0ffffffa7ffffffa7ffffffa6fffffff3000,s{4298257312,4298257312,1580304561076,1580304561076,0,0,0,0,194,0,4298257312} 
2020-01-29 15:33:38.770 o.a.s.d.worker refresh-active-timer [DEBUG] Event debug options {}
2020-01-29 15:33:38.849 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:38.850 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:39.050 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:39.052 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:39.253 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:39.254 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:39.300 o.a.s.u.LocalState heartbeat-timer [DEBUG] New Local State for /hadoop/storm/workers/5829d9a2-da96-49fc-b594-9d5d50fe2af6/heartbeats
2020-01-29 15:33:39.436 o.a.z.ClientCnxn Thread-13-parserBolt-executor[5 5]-SendThread(mdr-3:2181) [DEBUG] Got ping response for sessionid: 0x36fc314e93528cf after 0ms
2020-01-29 15:33:39.454 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:39.456 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:39.656 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:39.658 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:39.859 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:39.860 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:40.060 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:40.062 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:40.215 o.a.k.c.c.i.AbstractCoordinator Thread-15-kafkaSpout-executor[4 4] [DEBUG] Received successful heartbeat response for group squid_parser
2020-01-29 15:33:40.262 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:40.264 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:40.303 o.a.s.u.LocalState heartbeat-timer [DEBUG] New Local State for /hadoop/storm/workers/5829d9a2-da96-49fc-b594-9d5d50fe2af6/heartbeats
2020-01-29 15:33:40.329 o.a.s.s.o.a.z.ClientCnxn main-SendThread(mdr-3:2181) [DEBUG] Reading reply sessionid:0x36fc314e93528ce, packet:: clientPath:null serverPath:null finished:false header:: 358,3  replyHeader:: 358,4298258584,0  request:: '/storm/workerbeats/squid-73-1580304550/ab01252f-a649-4317-9e74-21acf9289615-6701,F  response:: s{4298257363,4298258572,1580304569310,1580304827890,86,0,0,0,551,0,4298257363} 
2020-01-29 15:33:40.332 o.a.s.s.o.a.z.ClientCnxn main-SendThread(mdr-3:2181) [DEBUG] Reading reply sessionid:0x36fc314e93528ce, packet:: clientPath:null serverPath:null finished:false header:: 359,5  replyHeader:: 359,4298258585,0  request:: '/storm/workerbeats/squid-73-1580304550/ab01252f-a649-4317-9e74-21acf9289615-6701,#1fffffff8b80000000ffffffed574b6effffffc23010ffffff9dfffffffcfffffff8255954fffffff410ffffffdd205fffffff129ffffffeaffffffa655ffffffafffffffd07d2db242ffffff80ffffffa0ffffff8959fffffff48ffffffbd4ffffffffd61bffffffb4211f486dffffffe3c21525544ffffffa42816fffffff17bffffffc633ffffffcf6f2636680ffffffd0effffffdfffffffb6fffffff349ffffffe7ffffffbeffffffd7ffffffe9e465effffffcfffffffebf6ffffff9ebffffffbaffffffe344ffffffef6affffff8d78ffffff8affffffde07dfffffff701734ffffffdbffffff8d46ffffffd1404f6e18e68ffffffd9ffffffef66745b5d6fffffffe4794e2b1a6a76c6f1012ffffffbeffffff87ffffffcc5fffffffd9314dffffff9bffffff90ffffff95ffffffcfffffff82fffffff938247436bfffffffc19653effffffa4ffffffd76d741bffffffc3ffffffb3ffffffe0ffffffd668ffffffd83fffffff8bffffffa0fffffff940ffffff97ffffffcbeffffff9bffffffaffffffffcffffffb224fffffffbffffffe8ffffffb860fffffff071314b514216ffffff97fffffff2fffffff0342effffffe5972712949ffffffe2ffffff8279ffffff95495e26ffffffd6551d791213ffffffccffffffa77e3a4effffff9cffffffc748ffffff9cffffffc728741e10ffffffceffffffdc5e6dffffff90ffffffc559ffffffbaffffff82ffffff96ffffffacffffffa0ffffff95fffffff4ffffffb6ffffffd6ffffff866ffffffa11f3cffffffafffffff972cffffffd962ffffff93103a5effffffec6effffff9558ffffff913051ffffffa448ffffffa05cffffff9c1830ffffffd2ffffffbb10540ffffffa22affffff913051ffffff8d48ffffffa05cffffff8518ffffffb0ffffffdaffffff9b2e30ffffffcd2affffffefffffffb9ffffffb8ffffffeca27ffffffff3bffffffbaffffffa2ffffff93ffffffbf7bffffff8078ffffffc2ffffffed38ffffff9affffffa034ffffffacffffffbb340ffffffd8ffffffb9ffffffc9ffffffaeffffffb1ffffffb3f5b4521ffffffb253fffffffbffffff8fffffffb7ffffff80fffffff67354fffffff770735829643460ffffffdb4d26444ffffffefffffffa061776e6a38ffffffaa6f5050ffffff9cffffffd4351cffffffe7128ffffffec19ffffffd4ffffffe0ffffffc2ffffff8e41d47fffffff5b4aaffffffb1ffffff965b492dffffffb7ffffffaafffffff94effffffa911ffffffc2ffffffe66a7f2f4614ffffff8a45ffffff824129ffffff84ffffffc7ffffff95fffffffbffffffe6ffffffc8581effffffbfffffffb0ffffffa2ffffff9020affffff9520ffffffc1ffffffa0ffffffd2ffffffcfffffffe3ffffff8e38ffffff80ffffff9effffffd6ffffff815ffffff9d2effffffe8ffffffcb66ffffffbd4dffffffeb407dffffffe24fffffffe976ffffffc9fffffff6313a39ffffffcdffffff9533ffffffe7ffffffe450393727ffffff9b6afffffff9ffffff8fffffffb84effffffe12259ffffffea30ffffff8bfffffffc525fffffffe5ffffffcc39ffffff9556ffffffceffffffcdffffffa9ffffffb96a7effffffd1ffffffe9ffffffccffffffc4ffffffe94c55ffffffefffffffc2255a28fffffff6ffffff92fffffff75cffffff81ffffff97ffffffccffffff9014757effffff96ffffffa217fffffff9ffffffabffffffbf14ffffffbbffffffbfffffffaeffffffe82dfffffff8181bffffffafffffffddffffff8fffffffcfffffffc66eb5a1d7e036ffffffd94438281300,-1  response:: s{4298257363,4298258585,1580304569310,1580304830896,87,0,0,0,551,0,4298257363} 
2020-01-29 15:33:40.464 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:40.465 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:40.666 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:40.667 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:40.867 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:40.869 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:41.069 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:41.070 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:41.270 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:41.272 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:41.305 o.a.s.u.LocalState heartbeat-timer [DEBUG] New Local State for /hadoop/storm/workers/5829d9a2-da96-49fc-b594-9d5d50fe2af6/heartbeats
2020-01-29 15:33:41.472 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:41.473 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:41.674 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:41.675 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:41.875 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:41.877 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:42.077 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:42.078 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:42.279 o.a.s.k.s.KafkaSpout Thread-15-kafkaSpout-executor[4 4] [DEBUG] Polled [0] records from Kafka
2020-01-29 15:33:42.280 o.a.s.k.s.KafkaSpoutRetryExponentialBackoff Thread-15-kafkaSpout-executor[4 4] [DEBUG] Topic partitions with entries ready to be retried [{}] 
2020-01-29 15:33:42.307 o.a.s.u.LocalState heartbeat-timer [DEBUG] New Local State for /hadoop/storm/workers/5829d9a2-da96-49fc-b594-9d5d50fe2af6/heartbeats

following file contents as follows ${METRON_HOME}/config/zookeeper/indexing/squid.json


{
"elasticsearch": {
"index": "squid",
"batchSize": 5,
"enabled" : true
},
"hdfs":{
"index": "squid",
"batchSize": 5,
"enabled" : true
}
}

 

i really appreciate all the help 

many thanks

Don't have an account?
Coming from Hortonworks? Activate your account here