Member since
09-29-2015
44
Posts
20
Kudos Received
12
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
600 | 05-22-2017 09:38 PM | |
411 | 02-23-2017 08:59 PM | |
622 | 09-27-2016 02:00 PM | |
275 | 09-19-2016 01:01 PM | |
300 | 09-13-2016 01:10 PM |
05-22-2017
09:38 PM
@Jon Zeolla - is on the right track. Your Kafka broker seems to have died and as a result, the Metron service start process cannot contact it to create the topics it needs.
I'd recommend either using Metron's Full-Dev environment. If you're determined to get it running on the Sandbox, you'll need to do a resource sizing activity to make sure all the services can live in the amount of memory available to the container.
... View more
02-23-2017
08:59 PM
1 Kudo
It looks like Yaf is sending some ipv6 traffic. The Yaf parser is only setup for ipv4. Try adding --ip4-only to the yaf startup script (/opt/yaf/start-yaf.sh).
Please let me know if that works and we can open up a JIRA to address it.
Thanks!
-David...
... View more
01-04-2017
02:24 PM
hth, If you post a link to the doc you followed, I'd be happy to update it. Thanks! -D...
... View more
01-03-2017
03:23 PM
The quick dev environment won't currently show the Metron service in the Ambari view. It installs it without Ambari. If you navigate to node1:5000 you should see a Kibana dashboard. -D...
... View more
01-03-2017
01:33 PM
Hi @Rashmi Varma, It looks like it's looking for an hbase table that isn't present, specifically for a whois type enrichment, if I'm not mistaken. Can you supply your configurations? Thanks! -D...
... View more
01-03-2017
01:31 PM
HI @Bramantya Anggriawan, I'd recommend against using the Symantec service defination. Metron actually has an Ambari Management pack that will install service definitions for Elasticsearch, Kibana, and Metron.
You can find it here. The README has installation details. Please let me know if there's anything confusing there. Thanks! -D...
... View more
01-03-2017
01:25 PM
Hi @Maziar Tasbihi, Can you tell me why you ran run_ansbile_role manually after the vagrant up? Vagrant up should be sufficient for starting quick dev. Thanks, -D...
... View more
12-09-2016
06:29 PM
It looks as if the install of HBase Master failed during the cluster install. If you go to Ambari and click on the HBase service, you should be able to re-attempt the install.
If it passes, check the cluster state- if it is up and running, you can re-run the Ansible install, otherwise, you'll need to dig into why the Ambari install is failing.
... View more
10-31-2016
09:21 PM
Hi @Rishi, I'd run something newer than TP1. That said, I'm not sure why Maven can't find the apache-rat plugin. Please try from current master here. Thanks! -D...
... View more
09-27-2016
07:08 PM
Glad to hear it. If you haven't, would you mind accepting the answer that helped? Thanks! -D...
... View more
09-27-2016
02:00 PM
2 Kudos
Hi, It seems that the Ambari client gets unhappy when you ask it to start services and they are all started. I have opened METRON-466 to track it. In the meantime, you should be able to comment out lines 34-42 in metron-deployment/roles/ambari_config/start_hdp.yml to get you going. I hope to have a fix up shortly. Thanks! -D...
... View more
09-19-2016
01:01 PM
1 Kudo
This wiki should describe the process.
... View more
09-16-2016
12:34 PM
Glad to hear it worked. If you haven't, would you mind accepting the above answer? Thanks! -David...
... View more
09-14-2016
04:19 PM
It had over 10GB free. The webhdfs copy causes the namenode to exhaust its 1GB heap. This is a new behavior with HDP 2.4.3. 2.3.x, 2.4.0, 2.4.2 and 2.5.0 all work in this configuration with the same machines. You may notice from the stack trace, this is prior to the history server trying to actually start.
... View more
09-14-2016
03:25 PM
The reference you posted suggests 1GB should be enough for fewer than 1 million files. That is no longer accurate as the History Server will not start with a 1GB heap namenode.
... View more
09-14-2016
03:16 PM
1 Kudo
I run small clusters for development. Since 2.4.3 was released, I've noticed that I cannot bring up a cluster with 1GB of memory for the Namenode heap. The namenode runs out of memory, is terminated and the history server fails to start with the following error. Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 190, in <module>
HistoryServer().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py", line 101, in start
host_sys_prepped=params.host_sys_prepped)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py", line 257, in copy_to_hdfs
replace_existing_files=replace_existing_files,
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 459, in action_create_on_execute
self.action_delayed("create")
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 456, in action_delayed
self.get_hdfs_resource_executor().action_delayed(action_name, self)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 255, in action_delayed
self._create_resource()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 269, in _create_resource
self._create_file(self.main_resource.resource.target, source=self.main_resource.resource.source, mode=self.mode)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 322, in _create_file
self.util.run_command(target, 'CREATE', method='PUT', overwrite=True, assertable_result=False, file_to_put=source, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 179, in run_command
_, out, err = get_user_call_output(cmd, user=self.run_user, logoutput=self.logoutput, quiet=False)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/get_user_call_output.py", line 61, in get_user_call_output
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/2.4.3.0-227/hadoop/mapreduce.tar.gz 'http://ec2-52-36-201-54.us-west-2.compute.amazonaws.com:50070/webhdfs/v1/hdp/apps/2.4.3.0-227/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444' 1>/tmp/tmpEBh1rW 2>/tmp/tmpT1HZP0' returned 52. curl: (52) Empty reply from server
100
Increasing the NN Heap to 2GB works, but that takes 25% of my available memory on my dev vm. Is there any guidance for minimum NN Heap size with 2.4.3? The closest I could find was: 2.3.6. Thanks! -D...
... View more
09-13-2016
01:10 PM
1 Kudo
That error is often caused by running a version of Ansible that is not 2.0.0.2. Could you run incubator-metron/metron-deployment/scripts/platform-info.sh and verify that the versions output match Required Versions?
... View more
08-25-2016
05:25 PM
The wiki will be the most accurate and up to date.
... View more
07-28-2016
07:50 PM
1 Kudo
Hi Matthew, I think the instances are being created in the us-west-2 (Oregon) region. I've had those types of connection issues because I'm using OSX 10.10.5 and the version of OpenSSH that it uses doesn't support control_path that we use by default in Metron. If you change your control path to : [ssh_connection] control_path = ~/.ssh/ansible-ssh-%%h-%%p-%%r and re-run, it may connect.
... View more
07-27-2016
09:49 PM
If you ssh vagrant@node1, does it connect?
... View more
07-27-2016
06:26 PM
It looks like Elasticsearch is down. From node1, have you done a 'sudo service elasticsearch status' to verify?
... View more
07-27-2016
12:48 PM
You're right, I did. Must have had zookeeper stuck in my head. Thanks! Can you start it using Monit?
... View more
07-26-2016
07:13 PM
It looks like it's unhappy about your ssh control socket. Can you try changing the control_path in /Users/shishir.saxena/Downloads/incubator-metron-master/metron-deployment/vagrant/full-dev-platform/ansible.cfg to [ssh_connection] control_path = ~/.ssh/ansible-ssh-%%h-%%p-%%r
... View more
07-26-2016
07:08 PM
Hi @Baban Gaigole - can you try starting the enrichment topology using Monit at http://node1:2812? Also- today I'm working to get a more robust Squid Grok pattern, so be sure to keep checking Adding a New Telemetry Data Source. It is in the process of being updated as we find out more.
... View more
07-15-2016
08:28 PM
The above answer gives a lot of good detail, but for what you're trying to do, the WARN messages are actually okay. Though, I'll be happy when they're gone. If you run /usr/metron/0.2.0BETA/bin/zk_load_configs.sh -i /usr/metron/0.2.0BETA/config/zookeeper -m DUMP -z hostname:2181 you should see the changes you PUSHed.
... View more
06-16-2016
02:33 PM
Hi Sai, Can you tell me what version of Metron, Ansible, Python and Vagrant you are running? Thanks! -D...
... View more
05-07-2016
03:18 PM
Hi Silvio, It looks as if your roles_path didn't get set up. That is in ansible.cfg in the singlenode-vagrant folder. Could you verify that the file exists and contains a line "roles_path = ../../roles"
... View more
05-06-2016
10:33 PM
Thanks for that. It looks like there is a version mismatch in your Python environment. If you do a python --version, what do you get?
... View more
05-06-2016
09:47 PM
1 Kudo
This looks like basic ssh connectivity is having an issue. Some things to try: Verify that ssh centos@<any-ec2-host> works. If it does not, run ssh -vvv centos@<any-ec2-host> to troubleshoot the issue. If you're running a custom ~/.ssh/config, make sure that you're not specifying a non-default (id_rsa) key.
... View more
05-06-2016
03:14 PM
If what @Henry Sowell suggested doesn't help, would you do the following: Pull in current master Run PROJECTDIR/metron-deployment/scripts/platform-info.sh and attach the output You may have some versioning issues. Thanks!
... View more