Member since 
    
	
		
		
		05-02-2019
	
	
	
	
	
	
	
	
	
	
	
	
	
	
			
      
                319
            
            
                Posts
            
        
                145
            
            
                Kudos Received
            
        
                59
            
            
                Solutions
            
        My Accepted Solutions
| Title | Views | Posted | 
|---|---|---|
| 8180 | 06-03-2019 09:31 PM | |
| 2189 | 05-22-2019 02:38 AM | |
| 2820 | 05-22-2019 02:21 AM | |
| 1686 | 05-04-2019 08:17 PM | |
| 2104 | 04-14-2019 12:06 AM | 
			
    
	
		
		
		05-31-2017
	
		
		03:11 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 welp!  That shows what happens when you are too "intelligent" to read through the basic stuff because you think you already know it. 😞  Thank you again @Lester Martin!! 😄 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-07-2017
	
		
		03:23 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		4 Kudos
		
	
				
		
	
		
					
							 Well, it's a good question. I will first explain what is Field Grouping and how Window Bolt works. Then I will give my suggestion on how to solve this problem.  1) Field Grouping and Window Bolt  When you use field grouping based some key, storm can guarantee the same key tuple flow into the same bolt.   But with window bolt(BaseWindowedBolt), in your case average last 3 event, no matter how many bolt you have, storm will not guarantee the 3 successive key flow into the same bolt within window.  I will explain it with the following example:  First we define a window bolt, with tumbling window using 3 tuples.  Then we connect this window bolt to upstream spout with fieldGrouping using "key".  In your case, we use some abbreviation for convenient   D stand for deviceid,   S stand for siteid,   P stand for ip,   T stand for timestamp,   V stand for val  You want the same device_site_ip(D_S_P) data flow into the same bolt within same window, we can combine D_S_P as key, use K stand for it, then we assume the following 15 test data:  1  K1, T1, V1
2  K2, T2, V2
3  K3, T3, V3
4  K1, T4, V4
5  K1, T5, V5
6  K1, T6, V6
7  K3, T7, V7
8  K2, T8, V8
9  K3, T9, V9
10 K4, T10, V10
11 K4, T11, V11
12 K1, T12, V12
13 K4, T13, V13
14 K3, T14, V14
15 K2, T15, V15  Then the core code we want to use as follow:  BaseWindowedBolt winBolt = new winBolt().withTumblingWindow(new BaseWindowedBolt.Count(3));
builder.setBolt("winBolt", winBolt, boltCount).fieldsGrouping("spout", new Fields("key"));  We assume there are 2 winBolt(Bolt1 and Bolt2), with field grouping and window bolt.  Field Grouping will calculate hash code of Key, and put the same hash code key into the same bolt.  Let say hash code for K1 is H1, hash code for K2 is H2, hash code for K3 is H3, hash code for K4 is H4.  Then all H1 and H3 tuple flow into Bolt1, and H2 and H4 tuple flow into Bolt2.  In Bolt1, we have 9 data as follow:  1  K1, T1, V1
3  K3, T3, V3
4  K1, T4, V4
5  K1, T5, V5
6  K1, T6, V6
7  K3, T7, V7
9  K3, T9, V9
12 K1, T12, V12
14 K3, T14, V14  In Bolt2, we have 6 data as follow:  2  K2, T2, V2
8  K2, T8, V8
10 K4, T10, V10
11 K4, T11, V11
13 K4, T13, V13
15 K2, T15, V15
  The above result inside each bolt is the result of Field Grouping  As for window bolt functionality, with 3 tuple Tumbling Window, you will see the following happens:  In Bolt1:  First window we got is:  1  K1, T1, V1
3  K3, T3, V3
4  K1, T4, V4  Second window we got is:  5  K1, T5, V5
6  K1, T6, V6
7  K3, T7, V7  Third window we got is:  9  K3, T9, V9
12 K1, T12, V12
14 K3, T14, V14  In Bolt2:  First window we got is:  2  K2, T2, V2
8  K2, T8, V8
10 K4, T10, V10  Second window we got is:  11 K4, T11, V11
13 K4, T13, V13
15 K2, T15, V15  Now you can see, it is explain why it's not we expectation that same key into same bolt in same window  2) My suggestion how to solve this problem  Now I have 2 method to solve this.  The first one is simpler, which only use storm. We don't need Window Bolt, and just use Field Grouping. Guarantee the same key flow into the same bolt worker. Then in the down stream bolt, we use a HashMap, where key is (D_S_P) pair and the data structure of value is a fixed size queue(you should implemented yourself). The fixed size queue memories the last N tuple of this key. Each time there is new tuple flow into this bolt worker, we use it's key to find the fixed size queue of this key and add it into the queue, and if necessary remove the oldest tuple in this queue. After update this key, then we can calculate the current average of the last N tuple, and emit it. The disadvantage of this approach the HashMap is in memory, there is no data persistence. Each time we relaunch the topology we need to construct the HashMap and accumulate the fixed size queue for each key.  The second one is to use HBase as a storage. In storm, we don't need Field Grouping and Window Bolt. Just Shuffle grouping is enough for it. For each tuple, after some parsing, we put it into HBase Table. The row key of this Table is (D_S_P) key, and we set the version number of this table as N. So this HBase Table will keep the last N tuple of this key. Then get this N tuple of the current key from HBase, and calculate the average of it, and emit it. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-04-2017
	
		
		04:38 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 It did the trick for me.  I sure hope it helps out @Joan Viladrosa, too!  Thanks, Sriharsha! 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		05-10-2017
	
		
		01:24 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Thanks! I would be interested to learn more when you are ready to announce it.  
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		03-08-2017
	
		
		03:51 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Thanks for all the great responses here and below.  Yes, indeed, the worker nodes that I have all of this running on are overloaded.  Taking into account the distribution of services and the underlying box resource footprint.  Thanks again!! 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		10-12-2017
	
		
		08:18 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Hi,  I have got 3 node cluster running kerberized hdp 2.6.2 with Ranger but without Ranger Storm plugin.  I also see the errors when I try to run command "storm list".  Storm sevice check runs fine. I get the following error when I use underprivileged user account with a valid token.  Any clues most apprecieted.   2290 [main] WARN  o.a.s.s.a.k.ClientCallbackHandler - Could not login: the client is being asked for a password, but the  client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock.
2298 [main] ERROR o.a.s.s.a.k.KerberosSaslTransportPlugin - Server failed to login in principal:javax.security.auth.login.LoginException: No password provided
javax.security.auth.login.LoginException: No password provided
        at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:919) ~[?:1.8.0_112]
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760) ~[?:1.8.0_112]
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) ~[?:1.8.0_112]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112] 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		05-01-2019
	
		
		03:50 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 It is confusing what triggers this task to run.  Do you have any additional info on that or know if there is any way to configure it more precisely? 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		04-27-2018
	
		
		08:25 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Is it removed from HDP 2.6.4 GA version too as I don't see the zeppelin view 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		01-23-2017
	
		
		01:36 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		1 Kudo
		
	
				
		
	
		
					
							 Good write-up from @Ambud Sharma plus you can visit http://storm.apache.org/releases/1.0.2/Guaranteeing-message-processing.html for info from the source.  Additionally, take a peek at the picture below I just exported from our http://hortonworks.com/training/class/hdp-developer-storm-and-trident-fundamentals/ course that might help visualize all of this information.      Good luck and happy Storming! 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		 
        













