Member since 
    
	
		
		
		11-16-2015
	
	
	
	
	
	
	
	
	
	
	
	
	
	
			
      
                905
            
            
                Posts
            
        
                664
            
            
                Kudos Received
            
        
                249
            
            
                Solutions
            
        My Accepted Solutions
| Title | Views | Posted | 
|---|---|---|
| 254 | 09-30-2025 05:23 AM | |
| 665 | 06-26-2025 01:21 PM | |
| 512 | 06-19-2025 02:48 PM | |
| 760 | 05-30-2025 01:53 PM | |
| 11008 | 02-22-2024 12:38 PM | 
			
    
	
		
		
		06-21-2017
	
		
		06:23 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 The documentation says "The Expression Language allows single quotes and double quotes to be used interchangeably". Try double-quotes in your EL expression. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-21-2017
	
		
		05:27 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		1 Kudo
		
	
				
		
	
		
					
							 You can accomplish this with ExecuteScript, the following example uses Groovy as the language and Jayway's JsonPath as the library for JSONPath parsing.  First I had to download 3 JARs (json-path and its required transitive dependencies) into a directory:      Then I set the Module Directory property to the path of this directory:      Note that I have also added a dynamic property, whose name will become an attribute and whose value supports Expression Language and (after evaluation) should contain a JSONPath expression used to retrieve the value from the content of the JSON flow file.  The Script Body is the following Groovy script:  import com.jayway.jsonpath.*
def flowFile = session.get()
if(!flowFile) return
def inputStream = session.read(flowFile)
def json = JsonPath.parse(inputStream)
inputStream.close()
context.properties.findAll {p,s -> p.dynamic}.each {pd, name ->
   def prop = context.getProperty(pd)
   try {
      flowFile = session.putAttribute(flowFile, pd.name, json.read(prop.evaluateAttributeExpressions(flowFile).value))
   } catch (e) {
      log.error("Error evaluating JSONPath expression in property $name: ${prop?.value} , ignoring...", e)
   }
}
session.transfer(flowFile, REL_SUCCESS)  I tested this with a GenerateFlowFile:      After the ExecuteScript transfers the flow file, it has the desired attribute name/value:      This should work with any number of attributes/JSONPaths per flow file. In addition, I have written NIFI-4100 to cover the improvement to the EvaluateJsonPath processor to support Expression Language. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-21-2017
	
		
		04:35 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Also, as of NiFi 1.3.0 / HDF 3.0.0, GenerateTableFetch accepts incoming connections/flow files, so you can use ListDatabaseTables -> GenerateTableFetch -> RPG -> Input Port -> ExecuteSQL to fully distribute the fetching of batches of rows across your NiFi cluster. The RPG -> Input Port part is optional and only used on a cluster if you want to fetch rows in parallel. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-21-2017
	
		
		04:33 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		5 Kudos
		
	
				
		
	
		
					
							 The answer in this StackOverflow post refers to documentation saying an array will be returned; this appears to be happening after the JSONPath is evaluated (which is why appending a [0] does not work). The post also implies the answer: in NiFi, you can choose "json" as the Return Type and "flowfile-attribute" as the Destination in EvaluateJsonPath (let's say your dynamic property has key "my.attr" and a value of your JSONPath expression above), then follow that processor with an UpdateAttribute processor, setting "my.attr" to the following:  ${my.attr:jsonPath('$[0]')}  This will overwrite the original "my.attr" value by hoisting the value out of the array. If you need that value back in the content, you can follow this with a ReplaceText processor to replace the Entire Text with the value of "my.attr". 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-20-2017
	
		
		07:47 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		2 Kudos
		
	
				
		
	
		
					
							 What version of NiFi/HDF are you using? As of NiFi 1.2.0 / HDF 3.0.0, PutHiveQL can accept multiple statements in one flow file, so if you are currently dealing with one INSERT statement per flow file, try MergeContent to batch them up into a single flow file. This should increase performance, but since Hive is an auto-commit database, PutHiveQL is probably not the best choice for large/fast ingest needs. You may be better off putting the data into HDFS and creating/loading a table from it.  For PutHiveStreaming, there is a known issue that can reduce performance, it also was fixed in NiFi 1.2.0 / HDF 3.0.0. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-20-2017
	
		
		07:34 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 That sounds like a different issue altogether, perhaps you'd get more/better responses if you create a new question in HCC to deal with it.  The short answer is: your ControllerService implementation can have a method annotated with @OnEnabled, looks like you're calling yours onConfigured(). That method can take a ConfigurationContext:  @OnEnabled
public void onConfigured(final ConfigurationContext context) throws InitializationException {
// Do stuff here 
}  And you can call getProperty(), getProperties(), etc. on it 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-20-2017
	
		
		07:14 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 In addition to the approach (sqlplus with ExecuteProcess / ExecuteStreamCommand) mentioned in the other post, you could also connect via a scripting language using ExecuteScript, calling Connection.prepareCall() and such as described here. I have an example on how to interact with SQL using Groovy and ExecuteScript here. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-20-2017
	
		
		05:46 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 Usually (as is the case for the nifi-hadoop-bundle), the NAR depends on the nifi-hadoop-libraries-nar, as it provides the Hadoop libraries (such as the provided dependencies you have in your processor POM like hadoop-common), and its parent NAR is nifi-standard-services-api-nar (which you have in your NAR POM). Currently, NARs can only have one parent, so you wouldn't be able to depend on both the hadoop-libraries and standard-services-api NARs at the same time. Since the former depends on the latter, this works for your processor.  Try replacing the NAR POM dependency on nifi-standard-services-api-nar to nifi-hadoop-libraries-nar, this should provide all the classes/JARs/dependencies you need. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-20-2017
	
		
		05:37 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
	
		1 Kudo
		
	
				
		
	
		
					
							 I think you're running into this issue, perhaps try an explicit module unload at the end of your script?  That will probably have a performance impact, but if it works, we can file an improvement Jira to look at adding this to the Jython support in NiFi, to unload the modules if the Module Directory property (or the files it points to) change. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		
			
    
	
		
		
		06-20-2017
	
		
		05:30 PM
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		
	
				
		
			
					
				
		
	
		
					
							 I use bytearray() in my examples, but I haven't been able to figure out when you need it and when you don't. I suspect it might be when the type is 'unicode' or 'java.lang.String' instead of Jython's 'str' type. The following two lines worked for me:  insertquery = "insert into Tweets_test values ('"+str(obj['id'])+"','"+obj['text']+"','"+str(obj['id_str'])+"');"
outputStream.write(insertquery)  This page says that a Jython String will be coerced to byte[] when necessary, and that seems to be what's going on above. 
						
					
					... View more
				
			
			
			
			
			
			
			
			
			
		 
        













