The intention behind this is very much to move towards PCAP query within zeppelin. This script is effectively a backend to provide access to pcap query via a zeppelin interpreter. If you install the sample zeppelin notebooks you will find one demonstrating the PCAP capabilities. The notebook is used like this: ... View more

The pycapa probe is used to ingest PCAP, which is then pushed to Kafka. Pycapa (http://metron.apache.org/current-book/metron-sensors/pycapa/index.html) is really intended for test use and is probably good up to around 1Gbps. In a real production you want to use fastcapa (http://metron.apache.org/current-book/metron-sensors/fastcapa/index.html) which does the same job, but in an accelerated way. The PCAP metron topology then takes this and stores it on HDFS sequence files. ($METRON_HOME/bin/start_pcap_topology.sh to start this). pcap-replay is a testing tool used to feed sample pcap data to an interface which you can then listen to with pycapa or fastcapa. pcap-service was the backend for an older interface panel, which is no longer really supported. We'll take a look at get the functionality pushed in the the new metron-rest service somewhere on the roadmap, in the meantime, your best bet is to use the query and inspector tools. There are various ways of then querying the PCAP data through the cli tools documented here: http://metron.apache.org/current-book/metron-platform/metron-pcap-backend/index.html has a lot of other useful information about the way PCAP is collected in Metron. It sounds like you are using monit from your description of services. This is deprecated, please use Ambari to manage services in metron. I would also recommend using the HCP deployment if you can rather than a direct Apache build. 3 nodes is also a tiny metron cluster, so you're unlikely to be able to get the levels of performance for anything like full scape PCAP, but it should be ok for a PoC test grade environment. ... View more

Metron uses Storm, Kafka, HDFS, Spark, Zepplein, Zookeeper and HBase primarily, which are available in other distributions. However, the primary deployment method is through Ambari, so it tends to work a lot better on HDP. All Hortonworks testing of the platform is certainly done on the HDP platform, so it will certainly be a lot easier to use. For AWS type deployments, it may also be worth considering HDC, which is essentially HDP but packaged up in the same way as EMR running on Amazon. s3 could also make sense as a long term storage platform for Metron replacing the HDFS default. ... View more

Sounds like you may have some connectivity issues to the public repo. One way to solve this is to download separately and use a local repo. Checkout the docs at http://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-installation/content/using_a_local_repository.html which show you how to download all the relevant tar balls to setup a local repo server. ... View more

Metron uses a number of Hadoop ecosystem components, and so tends to require separate master nodes for these for performance, this can also be used for resilience, though this diagram does not show full master HA. To expand the abbreviations:- NN = Name Node (the Hadoop HDFS name node stores file system meta data) SN = Secondary Name Node (not very well named, but provides compaction and optimisation services for the NN) RM = Resource Manager (the container coordinator which manages YARN resources and allocates them to running jobs) ZS = Zookeeper Server (zookeeper is used extensively in Metron for storage and coordination of configuration. It is also used for similar purposes by many other Hadoop components) DN = Data Node (this is an HDFS Data Node and responsible for storing the actual blocks in HDFS) ... View more