Member since 
    
	
		
		
		05-07-2018
	
	
	
	
	
	
	
	
	
	
	
	
	
	
			
      
                331
            
            
                Posts
            
        
                45
            
            
                Kudos Received
            
        
                35
            
            
                Solutions
            
        My Accepted Solutions
| Title | Views | Posted | 
|---|---|---|
| 9601 | 09-12-2018 10:09 PM | |
| 3733 | 09-10-2018 02:07 PM | |
| 11503 | 09-08-2018 05:47 AM | |
| 4088 | 09-08-2018 12:05 AM | |
| 4928 | 08-15-2018 10:44 PM | 
			
    
	
		
		
		09-25-2018
	
		
		02:33 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Hey @Vinicius Higa Murakami  .. I now get it . The issue was with the minifi 0.3 tool kit . When i tried to create the config.yml file from minifi tool kit version 0.3 , i kept on getting this issue. As soon as I switched to the latest version for tool kit , this problem got fixed and the port id was correct. Thanks for the help. I have accepted your answer as well  
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		09-17-2018
	
		
		05:46 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							@A C Just to understand, did you run the spark submit using yarn cluster as master/deploy mode?   If so, let's try to check the job properties for the following parameter:  ${resourceManager}  Also, here it is another example regarding pyspark + oozie (using shell to submit spark).  https://github.com/hgrif/oozie-pyspark-workflow  Hope this helps 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		09-13-2018
	
		
		10:16 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Hello @Teresa Tavernelli!  Glad to know that you made it 🙂   So regarding the Mysql, you can give it a shot with the following parameters:  hostname -> your-sandbox-hostname
port -> 3306
user -> root  ps: I didn't test it myself   Also if it doesn't work, try with the hive configs for mysql (instead of using username as root, change it to hive)  Hope this helps! 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		08-29-2018
	
		
		06:01 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		2 Kudos
		
	
				
		
	
		
					
							 How to use make a simple flow using SNMPSET/SNMPGET   Append A - Troubleshooting common mistakes    Pre-requisites:   - Nifi Cluster Installed (I'm using HDF 3.1.2.0 hence with Nifi 1.5)   - Centos7     With your NIFI cluster installed, you're up to initiate this step-by-step.   First of all, we'll need to install a snmp server to retrieve/set values of/to MIB's. In my case I'm using net-snmp tool, and it comes with some MIB samples to play. Here we're going to use the SNMPv2-MIB.txt under the /usr/share/snmp/mibs path     1)Install net-snmp  yum install -y net-snmp net-snmp-utils net-snmp-libs   2)Give full access of read-write to anyone at public community (DON'T DO IT IN PROD, please)  echo > /etc/snmp/snmpd.conf 
printf "agentAddress udp:161\nrwcommunity public\nrwuser public\nrocommunity public default system\nrouser public" >> /etc/snmp/snmpd.conf   3)Start the SNMP SERVER DAEMON - port 161  service snmpd start   4)Test a simple snmpwalk to see if those changes at step 2 are working  snmpwalk -v2c -mALL -c public localhost system   5)Test a simple snmpget  snmpget -v2c -mALL -c public localhost SNMPv2-MIB::sysContact.0   6)Setting a simple value to overwrite the value above, and check again to see if the value has been replaced with the new value  snmpset -v2c -mALL -c public localhost SNMPv2-MIB::sysContact.0 = "Vinicius Higa Murakami" 
snmpget -v2c -mALL -c public localhost SNMPv2-MIB::sysContact.0   7)Login as nifi user and take the same steps as above (just to ensure that the nifi user has access to read/write)  su - nifi 
snmpset -v2c -mALL -c public localhost SNMPv2-MIB::sysContact.0 = "Nifi it's here" 
snmpget -v2c -mALL -c public localhost SNMPv2-MIB::sysContact.0   8)Now we're ready to use nifi and draw the simple flow, using SNMP processors.   Go to nifi UI and draw the following components:        GenerateFlowFile Property      SetSNMP Property      LogAttribute Property      GetSNMP Property    LogAttribute Property      9)Run the nifi flow (snmpset and snmpget) and check if your value is showing up in the nifi-app.log       Append A   To troubleshoot the snmp, you can enable the DEBUG log by adding the following line to the /etc/sysconfig/snmpd  OPTIONS="-A -p /var/run/snmpd -a -LF 7 /var/log/snmpd.log"   And then make some snmpget (step 7) to check if it's logging the connections into the /var/log/snmpd.log. Should appear 2 lines:  Connection from UDP: [127.0.0.1]:40769->[127.0.0.1]:161 
Received SNMP packet(s) from UDP: [127.0.0.1]:40769->[127.0.0.1]:161   If you're having issues with the SNMPSET like the below.      Check if your snmpset it's able to reach the snmp server and send snmp packets, to do this, you can use tail -f /var/log/snmpd.log to monitor the connections and start the flow to watch the behaviour.   Other attention point is to check if your nifi has permissions to set/get values from MIB's (make sure you did the step 7).   And lastly is to check if your snmp$oid is valid.    And here's the template used:  template-kb-snmp.xml 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
	
					
			
		
	
	
	
	
				
		
	
	
			
    
	
		
		
		08-29-2018
	
		
		11:56 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Thanks a lot!  @Vinicius Higa Murakami, I have one small question, how I can connect to livy2 server?  I need to know some parameters to connect(look at the screenshot), but I don't know where I could find them.      I'm work at SandBox 2.6.5(last version)  Please help me, if u can.  Thanks! 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		08-22-2018
	
		
		08:35 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Hi @Vinicius Higa Murakami,  That worked. After installing the hive interact module i was able to create the external table.    Thank you so much!  Sorry for the late response but i went on vacation 😛 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		08-21-2018
	
		
		10:08 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Update:  It is found that .hiverc was used in hive user for HIVE CLI so the difference was found.  hive.exec.scratchdir=/user/hive/scratch  hive.exec.stagingdir=/user/hive/staging  The issue is hdfs /user/hive directory is encrypted with Ranger and hdfs /tmp/hive directory is non-encrypted and can read/write by all user in hadoop group.  hive-site.xml       <property>         <name>hive.security.authorization.sqlstd.confwhitelist.append</name>        <value>hive\.exec\.scratchdir|hive\.exec\.stagingdir</value>        <description>append conf property in white list followed by pipeline</description>      </property>  Restart the metastore and hiveserver.  I tested with beeline with session level change . The execution is fast like HIVE CLI .  hive.exec.scratchdir=/user/hive/scratch  hive.exec.stagingdir=/user/hive/staging  I tested with HIVE CLI with session level change. The execution is slow with MAP reduce Job for moving data.  hive.exec.scratchdir=/tmp/hive/scratch  hive.exec.stagingdir=/tmp/hive/staging  So the root cause is data are encrypted in /user/hive and not encrypted in /tmp/hive.  Solution is to make ssession level change to use same encryption zone.  So below INFO log will be printed if the encryption zones are different.  metadata.Hive: Copying source hdfs://edhcluster/tmp/hive/staging_hive_2018-08-07_16- 29-12_750_8973639287951385407-1/-ext-10000/000001_0 to hdfs://edhcluster/user/hive/warehouse/temp_tro/000001_0 because HDFS encryption zones are different.  Thanks,  Manjil 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		07-29-2018
	
		
		03:15 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Good to know @Mohammad Shamim!  Please if the issue is solved, I'd kindly ask you to accept the answer. This will help the other HCC users to find the best answer faster and will encourage the other users to keep doing a good job as well 🙂  
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		07-05-2018
	
		
		05:41 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Hi @heta desai!  Yes you can 🙂   Here's a link with more details:  http://druid.io/docs/latest/ingestion/data-formats.html    Hope this helps! 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-27-2018
	
		
		03:56 AM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							@Raj ji You can use ExecuteProcess (doesn't allow any incoming connections) (or) ExecuteStreamCommand processors to trigger the shell script.  ExecuteProcess configs:      As your executable script is on Machine 4 and NiFi installed on Machine1 so create a shell script on Machine 1 which ssh into Machine 4 and trigger your Python Script.  Refer to this and this links describes how to use username/password while doing ssh to remote machine.  As you are going to store the logs into a file, so you can use Tail file processor to tail the log file and check is there any ERROR/WARN, by using RouteText Processor then trigger mail.  (or)  Fetch the application id (or) application name of the process and then use yarn rest api to get the status of the job  Please refer to how to monitor yarn applications using NiFi and Starting Spark jobs directly via YARN REST API and this link describes yarn rest api capabilities. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		 
         
					
				













