Metron 0.3.0 is built to run against HDP 2.5. There are major differences between 2.5 and 2.4 in terms of Storm versions, this is likely what's causing your problem. ... View more

Metron 0.2.1BETA is based on HDP2.4, and storm 0.10.x For HDP 2.5 you want to be using Metron 0.3.0, which has just been released. This will be built against the storm version your specified. ... View more

There are a variety of meanings of correlation in Metron. One means is adding correlation keys in the enrichment process, which then allow you to view events together once they are indexed. So you can correlate events by adding a common search key you can then pivot on in Kibana. This is a great means of investigating correlations between alerts and events. For a more statistical approach to correlation, you will want to look into the profiler which maintains windows of data, which can then be used to correlate time series data using, for example, arima in a model managed by the model as a service infrastructure. This area of Metron is growing quite fast at the moment. I would suggest also looking at the Stellar statistics functions which can be used to build simple anomaly based models as well. It's also easy enough to add functions to Stellar if you want to extend the functionality. ... View more

That is the number of threads currently running in the processor. You sometimes see this if you have threads running for a while (e.g. long running processes like TailFile, or Listening processes). If you increase the number of concurrent tasks, you will also see this go up. ... View more

There are a number of ways you can monitor data running through the flow. The first is to take a look at the processors in the UI, which will show you the amount of data that has gone through them in the last 5 minutes. You can also right click on the processors, or on connections and select stats, which will graph these values for you and show you the 'pulse' of your data flow. These stats are also rolled up on a summary stats screen (the little table icon on the top right toolbar which lists all the processors) Another method is to use the MonitorActivity processor with something like PutEmail to alert you, if for example you haven't received any data in your flow for the last n seconds. If you're worried about data being queued up, the top of the UI has a handy indicator of how much data is currently queued. Process groups also conveniently show the totals for any queues within them. This can often indicate if there is a bottleneck in your flow somewhere, and how far the data has got through that pipeline. Another option to check that data has fully passed through your flow is to checkout the data provenance (either in the UI - right click on a processor, data provenance) or via API. This will show you everything that has happened to your data in microscopic detail. In terms of just getting a simple notification when things are done, I often use the Success relation (for example after PutHDFS) to send out a notification with something like PutEmail, or InvokeHttp to a web service) to notify that the process has completed successfully for a particular piece of data. ... View more

That sounds like it could be a visual representation of a range of different control characters in the file. One thing to do would be to use a regex like [^\d\w] to greedily replace the non-ascii and non-digit characters. ... View more

The bold number is the label from the LabelledPoint. Your map to create projected creates a copy of LabelledPoint replacing the features member with the principal components, but leaving the label untouched. Hence, you are getting the output tuple of (label, features) where features are your PCA result, and label is the original label. ... View more