Member since 
    
	
		
		
		02-05-2018
	
	
	
	
	
	
	
	
	
	
	
	
	
	
			
      
                25
            
            
                Posts
            
        
                3
            
            
                Kudos Received
            
        
                2
            
            
                Solutions
            
        My Accepted Solutions
| Title | Views | Posted | 
|---|---|---|
| 2346 | 08-17-2018 04:00 PM | |
| 1842 | 07-31-2018 08:07 PM | 
			
    
	
		
		
		09-20-2018
	
		
		02:37 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		1 Kudo
		
	
				
		
	
		
					
							 It's totally possible with Nifi!  There are a number of ways you could go about it, and it depends on if you think the dates will have the same format, or if they will vary. (ie: mm/dd/yyyy or dd-mm-yyyy)  I found another useful conversation here that covers using the routeOnContent or routeOnText processors to evaluate a regex expression, checking if the value matches what you expect.  I would recommend looking at the documentation on those processors, as well as testing out your regex expression on a site like Regexr. Hope that gets you started! 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		09-07-2018
	
		
		03:18 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 I'm not seeing Solr as an available service to add onto my HDP 3 cluster. I checked the Ambari upgrade documents and didn't see Solr as one of the services that was being removed. Anyone have news on where Solr is at?  I read in this post that the Ambari Infra Solr is for internal use only, so that won't suit my needs. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
			
	
					
			
		
	
	
	
	
				
		
	
	
- Labels:
- 
						
							
		
			Apache Ambari
- 
						
							
		
			Apache Solr
			
    
	
		
		
		09-06-2018
	
		
		08:30 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 I am also having a similar issue with hive import on HDP3. Has anyone been successful yet? My sqoop command successfully pulls the database to HDFS, but hangs on the zookeeper connect as well. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		08-22-2018
	
		
		02:36 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Thanks for the Zeppelin articles Franklin!  Could you elaborate on the Execution mode 'master' value, and what options there are/when to use each? I'm having trouble finding a good resource on them. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		08-20-2018
	
		
		06:21 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 That solved my problem! I went to Hortonworks downloads and grabbed the correct versions of Nifi toolkit. I am now onto the next error:  resource_management.core.exceptions.ExecutionFailed: Execution of '/var/lib/ambari-agent/tmp/nifi-toolkit-1.5.0.3.1.1.0-35/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /var/lib/nifi/conf/flow.xml.gz -s '[PROTECTED]' -a /usr/hdf/current/nifi/conf/authorizers.xml -p '[PROTECTED]'' returned 1. Error: Could not find or load main class org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain
  But I'll save that for another post once I've investigated some more, thanks! 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		08-20-2018
	
		
		04:27 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Hey all, I am having issues getting Nifi and Nifi Registry to start so that they may be updated to 1.7 with the release of HDF 3.2. I'm not seeing many other topics covering this issue. I have checked that the file exists that is being referenced here, there is a directory for both Nifi 1.5 and 1.7 toolkit. I have tried moving the 1.7 directory out of the tmp folder in case that was creating confusion but that didn't work either. Any ideas?  stderr: 
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_registry.py", line 280, in <module>
    Master().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_registry.py", line 128, in start
    self.configure(env, is_starting = True)
  File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_registry.py", line 98, in configure
    self.write_configurations(params, is_starting)
  File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_registry.py", line 200, in write_configurations
    is_starting, params.toolkit_tmp_dir, params.stack_version_buildnum)
  File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_toolkit_util.py", line 342, in encrypt_sensitive_properties
    encrypt_config_script = get_toolkit_script('encrypt-config.sh',toolkit_tmp_dir, stack_version_buildnum)
  File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_toolkit_util.py", line 61, in get_toolkit_script
    raise Exception("Couldn't find nifi toolkit directory in " + scriptDir)
Exception: Couldn't find nifi toolkit directory in /var/lib/ambari-agent/tmp
 stdout:
2018-08-20 16:11:12,671 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.1.0-35 -> 3.1.1.0-35
2018-08-20 16:11:12,849 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.1.0-35 -> 3.1.1.0-35
2018-08-20 16:11:12,852 - Group['nifiregistry'] {}
2018-08-20 16:11:12,853 - Group['hadoop'] {}
2018-08-20 16:11:12,853 - Group['nifi'] {}
2018-08-20 16:11:12,854 - User['streamline'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,855 - User['logsearch'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,855 - User['registry'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,856 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,857 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,857 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,858 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,859 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,860 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,860 - User['nifiregistry'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifiregistry'], 'uid': None}
2018-08-20 16:11:12,861 - User['nifi'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifi'], 'uid': None}
2018-08-20 16:11:12,862 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-08-20 16:11:12,863 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-08-20 16:11:12,868 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2018-08-20 16:11:12,883 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2018-08-20 16:11:12,889 - Skipping Execute[('setenforce', '0')] due to not_if
2018-08-20 16:11:12,889 - Testing the JVM's JCE policy to see it if supports an unlimited key length.
2018-08-20 16:11:13,068 - The unlimited key JCE policy is required, and appears to have been installed.
2018-08-20 16:11:13,313 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.1.0-35 -> 3.1.1.0-35
2018-08-20 16:11:13,325 - Toolkit files dir is /var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/files
2018-08-20 16:11:13,325 - Toolkit tmp dir is /var/lib/ambari-agent/tmp
2018-08-20 16:11:13,376 - Directory['/var/lib/ambari-agent/tmp/nifi-toolkit-1.7.0.3.2.0.0-520'] {'group': 'nifiregistry', 'cd_access': 'a', 'create_parents': False, 'recursive_ownership': True, 'owner': 'nifiregistry', 'mode': 0755}
2018-08-20 16:11:13,378 - Changing owner for /var/lib/ambari-agent/tmp/nifi-toolkit-1.7.0.3.2.0.0-520 from 0 to nifiregistry
2018-08-20 16:11:13,378 - Changing group for /var/lib/ambari-agent/tmp/nifi-toolkit-1.7.0.3.2.0.0-520 from 0 to nifiregistry
2018-08-20 16:11:13,388 - Directory['/var/run/nifi-registry'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,388 - Directory['/var/lib/nifi-registry'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,389 - Directory['/var/lib/nifi-registry/conf'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,390 - Directory['/var/lib/nifi-registry/database'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,391 - Directory['/usr/hdf/current/nifi-registry/conf'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,391 - Directory['/usr/hdf/current/nifi-registry/bin'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,392 - Directory['/usr/hdf/current/nifi-registry/lib'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,394 - Directory['/usr/hdf/current/nifi-registry/docs'] {'owner': 'nifiregistry', 'group': 'nifiregistry', 'create_parents': True, 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,396 - PropertiesFile['/usr/hdf/current/nifi-registry/conf/nifi-registry.properties'] {'owner': 'nifiregistry', 'group': 'nifiregistry', 'mode': 0600, 'properties': ...}
2018-08-20 16:11:13,402 - Generating properties file: /usr/hdf/current/nifi-registry/conf/nifi-registry.properties
2018-08-20 16:11:13,402 - File['/usr/hdf/current/nifi-registry/conf/nifi-registry.properties'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0600, 'encoding': 'UTF-8'}
2018-08-20 16:11:13,422 - Writing File['/usr/hdf/current/nifi-registry/conf/nifi-registry.properties'] because contents don't match
2018-08-20 16:11:13,425 - File['/usr/hdf/current/nifi-registry/conf/bootstrap.conf'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0600}
2018-08-20 16:11:13,428 - File['/usr/hdf/current/nifi-registry/conf/logback.xml'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0400}
2018-08-20 16:11:13,436 - File['/usr/hdf/current/nifi-registry/conf/authorizers.xml'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0600}
2018-08-20 16:11:13,440 - File['/usr/hdf/current/nifi-registry/conf/identity-providers.xml'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0600}
2018-08-20 16:11:13,442 - File['/usr/hdf/current/nifi-registry/conf/providers.xml'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0400}
2018-08-20 16:11:13,444 - File['/usr/hdf/current/nifi-registry/bin/nifi-registry-env.sh'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0755}
2018-08-20 16:11:13,445 - Encrypting NiFi Registry sensitive configuration properties
Command failed after 1 tries
 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
		
			
				
						
							Labels:
						
						
		
			
	
					
			
		
	
	
	
	
				
		
	
	
- Labels:
- 
						
							
		
			Apache NiFi
			
    
	
		
		
		08-17-2018
	
		
		04:00 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 As I was perusing the updated documentation for HDF 3.2 today I found what the documentation was referencing. The parent to the page I was stuck on has an image with steps written out.   Here it is. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		08-01-2018
	
		
		04:32 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 No problem on the tags, I just wanted to ensure you got some traffic on this!  I had never looked into JOLT before now, this is some pretty cool stuff! I'll look into it and maybe I'll be able to come up with something for you. Sounds like you have got your minimum viable product for now though:P 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		07-31-2018
	
		
		09:34 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 I assume this is being done in Nifi? (I would add the Apache-Nifi tag if so)  If you know the structure of the JSON object that you will be receiving, you can use an evaluateJSONPath processor to pull the JSON key's value into a flowfile attribute. Then an updateAttribute processor where you use Nifi expression language's 'equals' to determine if the value is as you expected.  https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#equals  The output of updateAttribute will be true or false and you can pass that to your routeOnAttribute processor.  example.jpg  I hope this helps or gives you some ideas! 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		 
        










