Created 01-15-2016 11:22 AM
Hi,
I have a topology that receives website clicks events from a stream. it keep track of events in last 20, 30 and 40 minutes of different topics. ok now how to consume this information in my website i want to show numbers of clicks on top of page of specific topic. that means somehow i need that real time information out of running topology
my best guess is that i call a update service after say 30 seconds or produce a json file and save it to a location. but that is not real time exact information. is there a way that i call a topology service and get this realtime upto exact that time information
Regards
Shahzad Aslam
Created 01-15-2016 01:27 PM
Or...You can write HBASE bolt to store data in HBase and retrieve the information in almost near realtime.
See this example http://hortonworks.com/hadoop-tutorial/real-time-data-ingestion-hbase-hive-using-storm-bolt/
Created 01-15-2016 02:03 PM
yes that is correct but i want to find out variations in this pattern and figure out which one is the fastest
- what is usual practice in this pattern, is storm topology is programmed that it pushes latest updates after very X seconds or minutes to a external system OR is there any direct method to get that data directly from running topology ?
Created 01-15-2016 02:12 PM
@Shahzad Aslam it depends how you code your bolt, you can program it so that with every tuple you can do a post or you can implement a tick tuple and every so often, 30sec or so, issue a tick tuple and post then.
Created 03-28-2016 06:31 AM
Use websockets or and HTTP Pubsub server with Jetty or Tomcat (Cometd Ajax). Just implement a Bolt that posts to either of these instrumentations. Here is an example of a bolt that connects to an HTTP pubsub server and posts and event as well as the java script that you will need on your JSP.
Publish HTTP Bolt .. public class PublishTransaction extends BaseRichBolt { private static final long serialVersionUID = 1L; private String pubSubUrl = Constants.pubSubUrl; private String incomingTransactionsChannel = Constants.incomingTransactionsChannel; private BayeuxClient bayuexClient; private OutputCollector collector; public void execute(Tuple tuple) { EnrichedTransaction transaction = (EnrichedTransaction) tuple.getValueByField("EnrichedTransaction"); Map<String, Object> data = new HashMap<String, Object>(); data.put("transactionId", transaction.getTransactionId()); data.put("transactionTimeStamp", transaction.getTransactionTimeStamp()); bayuexClient.getChannel(incomingTransactionsChannel).publish(data); collector.emit(tuple, new Values((EnrichedTransaction)transaction)); collector.ack(tuple); } public void prepare(Map arg0, TopologyContext arg1, OutputCollector collector) { this.collector = collector; HttpClient httpClient = new HttpClient(); try { httpClient.start(); } catch (Exception e) { e.printStackTrace(); } // Prepare the transport Map<String, Object> options = new HashMap<String, Object>(); ClientTransport transport = new LongPollingTransport(options, httpClient); // Create the BayeuxClient bayuexClient = new BayeuxClient(pubSubUrl, transport); bayuexClient.handshake(); boolean handshaken = bayuexClient.waitFor(3000, BayeuxClient.State.CONNECTED); if (handshaken) { System.out.println("Connected to Cometd Http PubSub Platform"); } else{ System.out.println("Could not connect to Cometd Http PubSub Platform"); } } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("EnrichedTransaction")); }
Then some Javascript on the view (JSP or whatever tech you use on the user side
JavaScript <script src="//ajax.googleapis.com/ajax/libs/dojo/1.7.8/dojo/dojo.js"></script> <script> dojo.require("dojo.io.script"); dojo.require("dojox.cometd"); dojo.require("dojox.cometd.longPollTransport"); var table; var tableData; var pubSubUrl = "http://ip:port/cometd"; var alertChannel = "/fraudAlert"; var incomingTransactionsChannel = "/incomingTransactions"; var customerValidationChannel = "/accountStatusUpdate"; dojo.ready(connectAccountStatusUpdateTopic) function connectAccountStatusUpdateTopic(){ dojox.cometd.init(pubSubUrl); dojox.cometd.subscribe("/*", function(message){ if(message.channel == customerValidationChannel){ console.log(message) var fraudulent = message.data.fraudulent; if(fraudulent=="true"){ document.getElementById("account_container").style.backgroundColor = "#FF0000"; document.getElementById("accountStatus").innerHTML = "Suspended"; }else{ document.getElementById("account_container").style.backgroundColor = "#9CF"; document.getElementById("accountStatus").innerHTML = "Active"; location.href='CustomerOverview?requestType=customerDetails&accountNumber=' + ${accountDetails.accountNumber}; } }if(message.channel == incomingTransactionsChannel || message.channel == alertChannel){ } }); }
Created 03-28-2016 06:34 AM
Just use Cometd or Websockets. Check out the answer below for an example. Only other thing to do is load Cometd maven dep.
<dependency> <groupId>org.cometd.java</groupId> <artifactId>cometd-java-client</artifactId> <version>3.0.7</version> </dependency>
Created 03-28-2016 07:32 PM
There is always Impetus' Stream Analytics and its monitoring capabilitites.
Created 03-29-2016 03:00 AM
@Shahzad Aslam Did you review the Storm Graphite ?