Re: Metron 4.2.0 deployment to AWS

nnat25191 · ‎06-11-2019

After more troubleshooting and by adding debug level to the storm topology, I'm seeing a continual of this message 2019-06-11 11:55:01.562 o.a.m.p.b.WriterHandler Thread-12-parserBolt-executor[5 5] [DEBUG] Flushing message queues older than their batchTimeouts by default the topology.message.timeout.sec is 30 It looks like all my messages have been flushed. Does this mean my message is older than 30s? I thought it's reasonable to have messages that are older than 30s. Please shine a light and thank you.

nnat25191 · ‎06-03-2019

Hi, I've been struggling with this issue and hoping someone can shed some light here. I'm running Metron 0.7.1 with Bro 2.5.5; Sensors yaf and snort are working and I can see the data in dashboard. Bro is running; sending logs to Kafka topic "bro"; I can see lots of logs when I run the kafka-console-consumer.sh; however, when I check bro parser topology I got nothing. I checked the storm worker log for bro and there were no error and it seems to be idle; 2019-06-03 14:27:56.199 o.a.k.c.p.ProducerConfig Thread-5-parserBolt-executor[8 8] [WARN] The configuration request.required.acks = 1 was supplied but isn't a known config. 2019-06-03 14:27:56.199 o.a.k.c.u.AppInfoParser Thread-5-parserBolt-executor[8 8] [INFO] Kafka version : 0.10.0.2.5.0.0-1245 2019-06-03 14:27:56.199 o.a.k.c.u.AppInfoParser Thread-5-parserBolt-executor[8 8] [INFO] Kafka commitId : dae559f56f07e2cd 2019-06-03 14:27:56.204 o.a.s.d.executor Thread-5-parserBolt-executor[8 8] [INFO] Prepared bolt parserBolt:(8) I used the metron/bin/load_tool.sh to check the throughput of bro and I see numbers; why my bro parser in the storm is not picking up any? Monitoring bro every 10000 ms Summarizing over the last 5 monitoring periods (50000ms) 2019/06/03 15:31:33 - 10 eps throughput measured for bro (Mean: 10, Std Dev: 0) 2019/06/03 15:31:43 - 11 eps throughput measured for bro (Mean: 10, Std Dev: 0) 2019/06/03 15:31:53 - 6 eps throughput measured for bro (Mean: 9, Std Dev: 2) 2019/06/03 15:32:03 - 18 eps throughput measured for bro (Mean: 11, Std Dev: 4) 2019/06/03 15:32:13 - 3 eps throughput measured for bro (Mean: 9, Std Dev: 5) 2019/06/03 15:32:23 - 3 eps throughput measured for bro (Mean: 8, Std Dev: 6) 2019/06/03 15:32:33 - 4 eps throughput measured for bro (Mean: 6, Std Dev: 6)

nnat25191 · ‎10-16-2018

Hi All, I'm stuck trying to build the latest Metron from the github repo. It looks like it has issue with npm; however my VM, I'm connected to the internet and tried NPM version 3.3 and tried version 6.4 but still getting the same error. Please help! [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.3:npm (npm ci) on project metron-config: Failed to run task: 'npm ci' failed. (error code 1) -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.3:npm (npm ci) on project metron-config: Failed to run task at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)Caused by: org.apache.maven.plugin.MojoFailureException: Failed to run task at com.github.eirslett.maven.plugins.frontend.mojo.AbstractFrontendMojo.execute(AbstractFrontendMojo.java:95) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)... 20 moreCaused by: com.github.eirslett.maven.plugins.frontend.lib.TaskRunnerException: 'npm ci' failed. (error code 1) at com.github.eirslett.maven.plugins.frontend.lib.NodeTaskExecutor.execute(NodeTaskExecutor.java:60) at com.github.eirslett.maven.plugins.frontend.mojo.NpmMojo.execute(NpmMojo.java:62) at com.github.eirslett.maven.plugins.frontend.mojo.AbstractFrontendMojo.execute(AbstractFrontendMojo.java:89)... 22 more

nnat25191 · ‎03-29-2018

@Rahul Soni My apology, I overlooked and didn't realize how messy the log looked. Thank you for reviewing my question. I read that exit code 143 is related to memory, and I'm running the job at Xmx5120m. I tried to modify the memory setting for HDFS, yarn, and mapreduce from Ambari and I got the HDFS corrupted. Then I fixed it by reverting configuration changes through Ambari. Within one hour, there were so many sequence files created and I can't seem to query and export the pcap for that one hour. My next resort is to look at the pcap-backend code and write a tool to export individual sequence file into pcap because I have a requirement to dump out the pcap files for our two days exercise.

nnat25191 · ‎03-23-2018

screen-shot-2018-03-23-at-52234-pm.png Hi all, I've been stuck on this problem for over a day and I can't seem to resolve it. I messed around with increasing yarn and mapreduce memory as well as breaking down my timerange for smaller window, but nothing seems to work. I'm trying to export out all my pcap data for two days and stuck on a 1hour window. I even broke it down to 30min and 15mins windows. sudo -su hdfs ./bin/pcap_query.sh fixed -df yyyyMMdd-HHmm -rpf '10000' -st 20180306-1730 -et 20180306-1800 -nr 200 The problem is my mapper tasks keep getting killed by ApplicationMaster. Each of the killed task lasted about 40s seconds and the success ones finished at about 30s second. Please help with any suggestion is appreciated. Container killed by the ApplicationMaster.Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143. 18/03/23 17:19:12 INFO mapreduce.Job: Task Id : attempt_1521838071529_0003_m_000022_2, Status : FAILED Error: java.io.EOFExceptionat java.io.DataInputStream.readFully(DataInputStream.java:197)at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:70)at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:120)at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2439)at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2571)at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:72)at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164) 18/03/23 17:19:17 INFO mapreduce.Job:map 100% reduce 100% 18/03/23 17:19:17 INFO mapreduce.Job: Job job_1521838071529_0003 failed with state FAILED due to: Task failed task_1521838071529_0003_m_000020Job failed as tasks failed. failed Maps:1 failed Reduces:0 18/03/23 17:19:17 INFO mapreduce.Job: Counters: 41 File System CountersFILE: Number of bytes read=0 FILE: Number of bytes written=25707046 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations= HDFS: Number of bytes read=4320257853 HDFS: Number of bytes written=0 HDFS: Number of read operations=32 HDFS: Number of large read operations=0 HDFS: Number of write operations=0 Job CountersFailed map tasks=12 Killed map tasks=24 Killed reduce tasks=200 Launched map tasks=26 Other local map tasks=11Data-local map tasks=13 Rack-local map tasks=2Total time spent by all maps in occupied slots (ms)=174434 Total time spent by all reduces in occupied slots (ms)=0Total time spent by all map tasks (ms)=174434Total time spent by all reduce tasks (ms)=0Total vcore-milliseconds taken by all map tasks=174434Total vcore-milliseconds taken by all reduce tasks=0 Total megabyte-milliseconds taken by all map tasks=1428963328 Total megabyte-milliseconds taken by all reduce tasks=0 Map-Reduce FrameworkMap input records=26522800Map output records=119240Map output bytes=24087217Map output materialized bytes=24454354Input split bytes=1240 Combine input records=0 Spilled Records=119240 Failed Shuffles=0 Merged Map outputs=0GC time elapsed (ms)=3410 CPU time spent (ms)=113580Physical memory (bytes) snapshot=21083152384 Virtual memory (bytes) snapshot=72407220224 Total committed heap usage (bytes)=22689087488 File Input Format CountersBytes Read=4320256613org.apache.metron.pcap.mr.PcapJob$PCAP_COUNTERMALFORMED_PACKET_COUNT=61 Exception in thread "main" java.lang.RuntimeException: Unable to complete query due to errors.Please check logs for full errors.at org.apache.metron.pcap.mr.PcapJob.query(PcapJob.java:214)at org.apache.metron.pcap.query.PcapCli.run(PcapCli.java:101)at org.apache.metron.pcap.query.PcapCli.main(PcapCli.java:53)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.hadoop.util.RunJar.run(RunJar.java:233)at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

nnat25191 · ‎02-27-2018

Figured out my issue. It was the Firewall blocking my ssh.

nnat25191 · ‎02-27-2018

Hi, I'm following the instruction to deploy a 10-node cluster on AWS. I managed to get to seeing the 10 nodes got created on my AWS console, but it failed at the step of "Wait for connectivity to the host(s)" How is it that it could create the instance but not able to ssh? Isn't it the same credential? Any suggestion is appreciated.

nnat25191 · ‎02-16-2018

I think I found the answer to my questions. I'm posting it here in case it help someone or someone can correct my understanding... 1. pycapa by default uses UTC. However, when I looked into the content of the packet and it's using my current system time zone which is EST. 2. the timestamp in the filename is nanosecond since epoch as long as my configuration for ts.granularity is set as MICROSECONDS. 3. timestamp and file last modified timestamp are in sync as long as my ts.granularity is set as MIRCOSECONDS

nnat25191 · ‎02-06-2018

Hi, I'm trying to understand these three different timestamp; they don't seem to be in synced 1. pycapa captures network traffic and use the timestamp as the key; the timestamp is in microsecond from epoch 2. the timestamp in the sequence filename is in nanosecond from epoch I try to convert these nanosecond from epoch to human readable time and they are very off. For example taking the time from below...This packet was captured on 2018-02-06 but the timestamp in the filename is way in the future. Is my formula not correct? nanosecond to second is 1000000000. Looking at the code, it looks like PcapCLI scan for the timestamp in the filename when we run our query. Then it runs query against the matching files. [root@hdp-slave-3 0.4.1]# date -d @$((5309030255444767488/1000000000)) Thu Mar 27 22:37:35 EDT 2138 3. the user input of query timestamp (pcap_query -st yyyyMMdd). The timestamp we use to query must match the timestamp in the filename and then in the packets stored in sequenceFile format. -rw-r--r-- 3 storm hdfs575 2018-02-06 13:34 /apps/metron/pcap/pcap_pcap_5309030255444767488_0_pcap-7-1517848461 Please shed some light, I'm confused and seem to make things more complicated than it should be. thank you!

nnat25191 · ‎02-06-2018

@Simon Elliston Ball Wonderful news! Thank you for the snapshot and the information, I was able to run the script, but I need to work on the query syntax. I assume the query syntax is in Stellar.

nnat25191 · ‎02-05-2018

I figured this out. If it ever help anyone; it writes to whatever the current directory you're in. It's not true that the default is /tmp directory; documentation might be outdated.

nnat25191 · ‎02-05-2018

Hi, I'm using Metron 0.4.2 to run pcap_query.sh. From reading the documents, the default output directory is /tmp if -bop is not specified. I tried using default as well as specifying -bop, but I always get Permission denied. If it's trying to write to /tmp, then any user has permission to write. I also checked hdfs /tmp and that's also a true statement that any user can write. I try to dig into the code, but it's not obvious to me where is it attempting to write the output file. I tried to run as hdfs/metron/storm user, but all ends with the same error. Thank you in advance for your help. 18/02/05 13:37:29 ERROR query.PcapCli: Unable to write filejava.io.FileNotFoundException: pcap-data-201802051835-9b5f3448c9914581a78f705e4c3b4de2+0001.pcap (Permission denied) at java.io.FileOutputStream.open0(Native Method)at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:162) at org.apache.metron.pcap.query.ResultsWriter.write(ResultsWriter.java:32) at org.apache.metron.pcap.query.PcapCli.run(PcapCli.java:167) at org.apache.metron.pcap.query.PcapCli.main(PcapCli.java:53) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

nnat25191 · ‎01-31-2018

Hi, I've noticed in the later releases of Metron, there's a new script pcap_zeppelin_run.sh. Does anyone know the use case of this script? I'm very curious in any pcap development within Metron. From reading the script, it looks like a it wraps around the pcap_query script and it's still in the testing phase? Will we be doing pcap query from zeppelin notebook? thanks

nnat25191 · ‎01-26-2018

Finally, I managed to resolved all issues with the indexing topology. The issue was that I upgraded metron via ambari mpack and it didn't copy new jars/scripts or anything to the designated Metron node. I manually created a new 0.4.2 directory, copy jars, configs, and scripts over; therefore many issues arrived with configuration and mismatched variables. Now; I'm still dealing with metron REST issues.

nnat25191 · ‎01-26-2018

@sridhar reddy Thank you for your response. I have tried upgrading via public repo as well as download the xml file to load it in. I got the files from here https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.0/bk_ambari-installation/content/hdp_26_repositories.html I also tried the file you attached although i'm running on centos7, but since hdp-2.6.4 is already in my config it won't let me register another hdp-2.6.4.

nnat25191 · ‎01-25-2018

Hi all, I'm trying to upgrade metron from 0.4.0 to 0.4.2 by using ambari mpack. Everything starts up fine except for the indexing parser topology and metron rest. I couldn't figure out how to solve this error below. That class exists in the jar included in the script that was executing. So I don't see why it's complaining. Any help is greatly appreciated. File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/metron/0.4.2/bin/zk_load_configs.sh --zk_quorum hadoop-slave-1:2181,hadoop-slave-2:2181,hadoop-master:2181 --mode PATCH --config_type GLOBAL --patch_file /tmp/metron-global-config-patch.json' returned 1. Error: Could not find or load main class org.apache.metron.common.cli.ConfigurationManager

nnat25191 · ‎01-24-2018

Hello all, I've upgraded my ambari successfully to 2.6.1. I'm trying to upgrade HDP; screen-shot-2018-01-24-at-43445-pm.png I see 2.6.4.0 available, but the option to install from the "stack and version" page has been disabled and all the components version are the same as the current version. Then on the page of selecting the versions, I can click on "Install on..." button then I chose my cluster, but it doesn't do anything. screen-shot-2018-01-24-at-43725-pm.png I attached the two screenshots. Any help is greatly appreciated

nnat25191 · ‎01-24-2018

I was able to upgrade ambari to 2.6.1 successfully. Thank you again for your help @Jonathan Hurley

nnat25191 · ‎01-24-2018

Thank you so very much @Jonathan Hurley With your help, I got my id and set my current state. I can now proceed to try again with my upgrade. ambari=> SELECT repo_version_id, version, display_name FROM repo_version ORDER BY version; repo_version_id |version | display_name -----------------+------------+-------------- 1 | 2.5.3.0-37 | HDP-2.5.3.0 (1 row) ambari=> UPDATE cluster_version SET state = 'CURRENT' WHERE repo_version_id = 1; UPDATE 1 Thank you!

nnat25191 · ‎01-24-2018

Thank you @Jonathan Hurley for your response and explanation. Pardon my very limited knowledge of ambari and postgresql. I logged on to ambari postgres and see the below relationship, but how do I find out what's my repo_version_id? ambari | repo_version| table | postgres ambari | cluster_version | table | postgres Thank you for your help.

nnat25191 · ‎01-23-2018

Hello All, I've been stuck with upgrading many versions of ambari-server to finally get to the version 2.6.1.0; but kept failing. The last stack is from ambari-server of 2.5.2 to 2.6.1 with the following error Traceback (most recent call last): File "/usr/sbin/ambari-server.py", line 952, in <module> mainBody() File "/usr/sbin/ambari-server.py", line 922, in mainBody main(options, args, parser) File "/usr/sbin/ambari-server.py", line 874, in main action_obj.execute() File "/usr/sbin/ambari-server.py", line 78, in execute self.fn(*self.args, **self.kwargs) File "/usr/lib/python2.6/site-packages/ambari_server/serverUpgrade.py", line 258, in upgrade retcode = run_schema_upgrade(args) File "/usr/lib/python2.6/site-packages/ambari_server/serverUpgrade.py", line 159, in run_schema_upgrade upgrade_response = json.loads(stdout) File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 307, in loads return _default_decoder.decode(s) File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 335, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 353, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded 23 Jan 2018 13:30:58,621 INFO [main] LockFactory:53 - Lock profiling is disabled 23 Jan 2018 13:30:58,634 INFO [main] SchemaUpgradeHelper:420 - Upgrading schema to target version = 2.6.1.0 23 Jan 2018 13:30:58,673 INFO [main] SchemaUpgradeHelper:429 - Upgrading schema from source version = 2.5.2 23 Jan 2018 13:30:58,677 INFO [main] SchemaUpgradeHelper:163 - Upgrade path: [{ upgradeCatalog: sourceVersion = 2.5.2, targetVersion = 2.6.0 }, { upgradeCatalog: sourceVersion = 2.6.0, targetVersion = 2.6.1 }, { upgradeCatalog: sourceVersion = null, targetVersion = 2.6.1 }, { upgradeCatalog: sourceVersion = null, targetVersion = 2.6.1 }] 23 Jan 2018 13:30:58,677 INFO [main] SchemaUpgradeHelper:201 - Executing DDL upgrade... 23 Jan 2018 13:30:58,678 INFO [main] DBAccessorImpl:874 - Executing query: ALTER SCHEMA ambari OWNER TO "ambari"; 23 Jan 2018 13:30:58,679 INFO [main] DBAccessorImpl:874 - Executing query: ALTER ROLE "ambari" SET search_path to 'ambari'; 23 Jan 2018 13:30:58,710 ERROR [main] SchemaUpgradeHelper:208 - Upgrade failed. org.apache.ambari.server.AmbariException: Unable to find any CURRENT repositories. at org.apache.ambari.server.upgrade.UpgradeCatalog260.getCurrentVersionID(UpgradeCatalog260.java:540) at org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:210) at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:931) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:206) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:444) 23 Jan 2018 13:30:58,713 ERROR [main] SchemaUpgradeHelper:464 - Exception occurred during upgrade, failed org.apache.ambari.server.AmbariException: Unable to find any CURRENT repositories. at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:209) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:444) Caused by: org.apache.ambari.server.AmbariException: Unable to find any CURRENT repositories. at org.apache.ambari.server.upgrade.UpgradeCatalog260.getCurrentVersionID(UpgradeCatalog260.java:540) at org.apache.ambari.server.upgrade.UpgradeCatalog260.executeDDLUpdates(UpgradeCatalog260.java:210) at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:931) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:206) ... 1 more

nnat25191 · ‎01-22-2018

I get this same issue when running "ambari-server upgrade" going from version 2.4.3 to 2.6.1. Any suggestion please?

nnat25191 · ‎01-11-2018

@Bharath Phatak Hi Bharath, So sorry I didn't see your question earlier. To modify the storm topology you can modify two files $METRON_HOME/config/<topology>.properties (topology like enrichment or elasticsearch for indexing) $METRON_HOME/flux/indexing/remote.yml With the latest release of metron 0.4.1 you can modify these setting from Ambari easily. For version 0.4.0, indexing topology you can modify it from ambari -> metron -> config -> advance tab. good luck!

nnat25191 · ‎12-07-2017

@Jasper Thank you for your response. I looked into supervisor.log and nimbus.log The odd thing is that I only have issue with the indexing topology. screen-shot-2017-12-07-at-95232-am.png I used to be able to check kafka consumer group for "indexing" like you suggested. After I increased the partitions because I thought it would help processing all the backlog indicated by LAG. After increasing my partitions, I lost my "indexing" consumer group somehow. The supervisor log keeps saying this message over and over and what does that mean? I assume the ID is referring to a host/worker? o.a.s.d.supervisor [INFO] 17112828-b2a1-4db1-8a7e-880a618477ce still hasn't started o.a.s.d.supervisor [INFO] 63c44874-7c97-4c75-905f-f8e845625700 still hasn't started The nimbus log has messages like o.a.s.d.nimbus [INFO] Executor indexing-3-1512599375:[32 32] not alive o.a.s.d.nimbus [INFO] Executor indexing-3-1512599375:[64 64] not alive o.a.s.d.nimbus [INFO] Executor indexing-3-1512599375:[56 56] not alive o.a.s.d.nimbus [INFO] Executor indexing-3-1512599375:[24 24] not alive o.a.s.d.nimbus [INFO] Executor indexing-3-1512599375:[40 40] not alive In my indexing logs, I get a lot of messages like (Re-) joining group indexing but they are not error. Also if you look at my screenshot of the indexing topology, I have a large number of FAILED in kafkaSpout but my storm bolts have none FAILED. I tried to read up to understand Storm topology Stats, but it's still not very clear to me. There is a gap between these numbers that I don't know where those tuples went. https://stackoverflow.com/questions/38891740/kafka-consumer-stuck-in-re-joining-group o.a.s.k.s.KafkaSpout [INFO] Initialization complete o.a.s.k.s.KafkaSpout [INFO] Initialization complete o.a.k.c.c.i.ConsumerCoordinator [INFO] Revoking previously assigned partitions [indexing-7, indexing-6] for group indexing o.a.s.k.s.KafkaSpout [INFO] Partitions revoked. [consumer-group=indexing, consumer=org.apache.kafka.clients.consumer.KafkaConsumer@26e55ff, topic-partitions=[indexing-7, indexing-6]] o.a.k.c.c.i.AbstractCoordinator [INFO] (Re-)joining group indexing o.a.k.c.c.i.ConsumerCoordinator [INFO] Revoking previously assigned partitions [indexing-10] for group indexing o.a.s.k.s.KafkaSpout [INFO] Partitions revoked. [consumer-group=indexing, consumer=org.apache.kafka.clients.consumer.KafkaConsumer@63e19a8d, topic-partitions=[indexing-10]] As always, thank you for your help and insights.

nnat25191 · ‎12-06-2017

I'm struggling with indexing topology. On all of my servers, I keep seeing connection refused for this topology. It looks like the topology is receiving the data from kafka, but it fails to ACK or do anything with it. Why? o.a.k.c.p.ProducerConfig [WARN] The configuration request.required.acks = 1 was supplied but isn't a known config. o.a.k.c.u.AppInfoParser [INFO] Kafka version : 0.10.0.2.5.0.0-1245 o.a.k.c.u.AppInfoParser [INFO] Kafka commitId : dae559f56f07e2cd o.a.s.d.executor [INFO] Prepared bolt indexingErrorBolt:(42) o.a.s.m.n.Client [ERROR] connection attempt 9 to Netty-Client-hadoop-slave-1/<ip>:6706 failed: java.net.ConnectException: Connection refused: hadoop-slave-1/<ip>:6706 o.a.s.m.n.Client [ERROR] connection attempt 10 to Netty-Client-hadoop-slave-1/<ip>:6706 failed: java.net.ConnectException: Connection refused: hadoop-slave-1/<ip>:6706 o.e.plugins [INFO] [Plasma] modules [], plugins [], sites [] o.a.s.m.n.Client [ERROR] connection attempt 11 to Netty-Client-hadoop-slave-1/<ip>:6706 failed: java.net.ConnectException: Connection refused: hadoop-slave-1/<ip>:6706 o.a.s.m.n.Client [ERROR] connection attempt 12 to Netty-Client-hadoop-slave-1/:6706 failed: java.net.ConnectException: Connection refused: hadoop-slave-1/:6706 o.a.s.m.n.Client [ERROR] connection attempt 13 to Netty-Client-hadoop-slave-1/:6706 failed: java.net.ConnectException: Connection refused: hadoop-slave-1/:6706 o.a.s.m.n.Client [ERROR] connection attempt 14 to Netty-Client-hadoop-slave-1/:6706 failed: java.net.ConnectException: Connection refused: hadoop-slave-1/:6706 o.a.s.m.n.Client [ERROR] connection attempt 15 to Netty-Client-hadoop-slave-1/:6706 failed: java.net.ConnectException: Connection refused: hadoop-slave-1/:6706

nnat25191 · ‎12-06-2017

Thank you @Jasper 1. I have 16 Acker, 8 workers, 12 kafkaSpout, 12 indexingBolt, 12 hdfsIndexingBolt 2. I now disabled HDFS channel and focusing only to ES. I noticed there are three places we can modify indexing topology setting; flux/indexing/remote.yaml, config/elasticsearch.properties, and ambari > metron > indexing. I assume Ambari would take preceding and overwrite the other two? However, I need to look up the configuration name in the ambari. 3. I modified a few of the data type such as bro and yaf to disable the HDFS indexing. If I disable it, I don't need to set batch size or batchTimeout I assume. My indexing topology got worse and now all failed and 0 ack; I turn on DEBUG mode and hopefully, i'll understand the issue better. Will be back for more questions and as always thank you for your help!

nnat25191 · ‎12-05-2017

Thank you very much @Jasper I was able to do that for the indexing topology, but how do you set that for parsing and enrichment topology? I still see a large number of failed under indexing topology but nothing obvious in logs. Occasionally, I see kafka coordinate mark and discover dead topic and I don't know how to fix that, but it goes away after awhile

nnat25191 · ‎12-04-2017

Thank you @Jasper I noticed I kept getting the error about fetching an offset out of range. I changed the kafka log retention rule to be shorter b/c I kept getting Out of Space because of pcap ingestion and kafka-log for pcap took all my space. Since I stop ingesting pcap, I reverted back the kafka retention rule and hopefully it won't complain about trying to read an offset that is already wiped out. If this doesn't work, I'll try the DEBUG mode that you suggested. Thank you again for your help! Fetch offset 82387394 is out of range for partition indexing-0, resetting offset

nnat25191 · ‎12-04-2017

FYI...I didn't know the name of some of my kafka consumer groups and I figured out a way to list them all and describe each of them. Hopefully, it would help someone like myself /usr/hdp/<VERSION>/kafka/bin/kafka-consumer-groups.sh --list --zookeeper <zk_host:port> | while read group; do echo $group; /usr/hdp/<VERSION>/kafka/bin/kafka-consumer-groups.sh --zookeeper <zk_host:port> --describe -group ${group}; done

nnat25191 · ‎12-04-2017

Thank you so much @Jasper This is pretty neat! It provides offset, logsize and Lag information. I'm fairly new to Kafka as well and it looks like i'm very behind on indexing. However, from Storm UI, it didn't look like I have that many incoming for indexing topic. I was reading and they say Lag should be close to 0, which would indicate that the system is caught up. How do I get Lag down to 0? Do I need more indexing storm workers? Offset: 82387393 logSize: 326704262 Lag: 244316869 As always, thank you for your time and response.

Online	Offline
Last Visited	‎06-13-2019 03:53 PM

Date Registered	‎01-08-2017 10:33 PM
Last Visited	‎06-13-2019 03:53 PM
Total Messages Posted	79
Total Kudos Received	6

Re: Metron 4.2.0 deployment to AWS

Re: Metron 0.7.1 bro_parser topology not processin...

Metron 0.7.1 bro_parser topology not processing

Metron 0.6.0 build failed

Re: pcap-query.sh mapreduce jobs failed when query...

pcap-query.sh mapreduce jobs failed when querying ...

Re: Metron 4.2.0 deployment to AWS

Metron 4.2.0 deployment to AWS

Re: Making sense of timestamp in pcap_query, times...

Making sense of timestamp in pcap_query, timestamp...

Re: metron pcap query

Re: PcapCli Unable to write

PcapCli Unable to write

metron pcap query

Re: metron 0.4.2 upgrade

Re: upgrade HDP-2.5.3.0 to 2.6.4.0

metron 0.4.2 upgrade

upgrade HDP-2.5.3.0 to 2.6.4.0

Re: upgrade ambari-server 2.5.2 to 2.6.1

Re: upgrade ambari-server 2.5.2 to 2.6.1

Re: upgrade ambari-server 2.5.2 to 2.6.1

upgrade ambari-server 2.5.2 to 2.6.1

Re: Unable to Start Ambari server after installati...

Re: Index topology stats failed to ACK

Re: Index topology stats failed to ACK

Re: Index topology stats failed to ACK

Re: Index topology stats failed to ACK

Re: How do I verify all of my data from enrichment...

Re: How do I verify all of my data from enrichment...

Re: How do I verify all of my data from enrichment...

Re: How do I verify all of my data from enrichment...