Support Questions
Find answers, ask questions, and share your expertise

How to get realtime information from runing storm topology

Highlighted

How to get realtime information from runing storm topology

Explorer

Hi,

I have a topology that receives website clicks events from a stream. it keep track of events in last 20, 30 and 40 minutes of different topics. ok now how to consume this information in my website i want to show numbers of clicks on top of page of specific topic. that means somehow i need that real time information out of running topology

my best guess is that i call a update service after say 30 seconds or produce a json file and save it to a location. but that is not real time exact information. is there a way that i call a topology service and get this realtime upto exact that time information

Regards

Shahzad Aslam

7 REPLIES 7
Highlighted

Re: How to get realtime information from runing storm topology

@Shahzad Aslam

Or...You can write HBASE bolt to store data in HBase and retrieve the information in almost near realtime.

See this example http://hortonworks.com/hadoop-tutorial/real-time-data-ingestion-hbase-hive-using-storm-bolt/

Re: How to get realtime information from runing storm topology

Explorer

yes that is correct but i want to find out variations in this pattern and figure out which one is the fastest

- what is usual practice in this pattern, is storm topology is programmed that it pushes latest updates after very X seconds or minutes to a external system OR is there any direct method to get that data directly from running topology ?

Highlighted

Re: How to get realtime information from runing storm topology

Mentor

@Shahzad Aslam it depends how you code your bolt, you can program it so that with every tuple you can do a post or you can implement a tick tuple and every so often, 30sec or so, issue a tick tuple and post then.

Highlighted

Re: How to get realtime information from runing storm topology

Guru

Use websockets or and HTTP Pubsub server with Jetty or Tomcat (Cometd Ajax). Just implement a Bolt that posts to either of these instrumentations. Here is an example of a bolt that connects to an HTTP pubsub server and posts and event as well as the java script that you will need on your JSP.

Publish HTTP Bolt ..

public class PublishTransaction extends BaseRichBolt {

	private static final long serialVersionUID = 1L;

	private String pubSubUrl = Constants.pubSubUrl;

	private String incomingTransactionsChannel = Constants.incomingTransactionsChannel;

	private BayeuxClient bayuexClient;

	private OutputCollector collector;

	

	public void execute(Tuple tuple) {

EnrichedTransaction transaction = (EnrichedTransaction) tuple.getValueByField("EnrichedTransaction");

		

		Map<String, Object> data = new HashMap<String, Object>();

		data.put("transactionId", transaction.getTransactionId());

		data.put("transactionTimeStamp", transaction.getTransactionTimeStamp());

		bayuexClient.getChannel(incomingTransactionsChannel).publish(data);		

		collector.emit(tuple, new Values((EnrichedTransaction)transaction));

		collector.ack(tuple);

	}




	public void prepare(Map arg0, TopologyContext arg1, OutputCollector collector) {

		this.collector = collector;

		
		HttpClient httpClient = new HttpClient();

		try {

			httpClient.start();

		} catch (Exception e) {

			e.printStackTrace();

		}


		// Prepare the transport

		Map<String, Object> options = new HashMap<String, Object>();

		ClientTransport transport = new LongPollingTransport(options, httpClient);


		// Create the BayeuxClient

		bayuexClient = new BayeuxClient(pubSubUrl, transport);

		

		bayuexClient.handshake();

		boolean handshaken = bayuexClient.waitFor(3000, BayeuxClient.State.CONNECTED);

		if (handshaken)

		{

			System.out.println("Connected to Cometd Http PubSub Platform");

		}

		else{

			System.out.println("Could not connect to Cometd Http PubSub Platform");

		}

	}


	public void declareOutputFields(OutputFieldsDeclarer declarer) {

		declarer.declare(new Fields("EnrichedTransaction"));

	}

Then some Javascript on the view (JSP or whatever tech you use on the user side

JavaScript

<script src="//ajax.googleapis.com/ajax/libs/dojo/1.7.8/dojo/dojo.js"></script>

<script>
	dojo.require("dojo.io.script");

  	dojo.require("dojox.cometd");

  	dojo.require("dojox.cometd.longPollTransport");

  

  	var table;

  	var tableData;

  	var pubSubUrl = "http://ip:port/cometd";

  	var alertChannel = "/fraudAlert";

    var incomingTransactionsChannel = "/incomingTransactions";

  	var customerValidationChannel = "/accountStatusUpdate";
	

	dojo.ready(connectAccountStatusUpdateTopic)	

 	  
  	function connectAccountStatusUpdateTopic(){

  		dojox.cometd.init(pubSubUrl);



  		dojox.cometd.subscribe("/*", function(message){

  			if(message.channel == customerValidationChannel){

  				console.log(message)

  				var fraudulent = message.data.fraudulent;

  				if(fraudulent=="true"){

  					document.getElementById("account_container").style.backgroundColor = "#FF0000";

  					document.getElementById("accountStatus").innerHTML = "Suspended";

  				}else{

  					document.getElementById("account_container").style.backgroundColor = "#9CF";

  					document.getElementById("accountStatus").innerHTML = "Active";

  					location.href='CustomerOverview?requestType=customerDetails&accountNumber=' + ${accountDetails.accountNumber};

  				}

  			}if(message.channel == incomingTransactionsChannel || message.channel == alertChannel){

  				

  			}

  		});

  	}
Highlighted

Re: How to get realtime information from runing storm topology

Guru

Just use Cometd or Websockets. Check out the answer below for an example. Only other thing to do is load Cometd maven dep.

        <dependency>
		<groupId>org.cometd.java</groupId>
		<artifactId>cometd-java-client</artifactId>
		<version>3.0.7</version>
	</dependency>

Highlighted

Re: How to get realtime information from runing storm topology

Expert Contributor

There is always Impetus' Stream Analytics and its monitoring capabilitites.

http://streamanalytix.com/features

Highlighted

Re: How to get realtime information from runing storm topology