I have this entry in my schema:   { "name": "a_string_array", "type": { "type": "array", "items": "string" } }     And I have added a property to UpdateRecord:   Property Value /a_string_array "a string value"     I expected the output JSON to look something like this:   { ....., "a_string_array": ["a string value"] }     But I get the following error:   Cannot convert value ["a string value"] of type class java.lang.String to Object Array for field a_string_array     I've tried making the Property name  "/a_string_array[*]" and "/a_string_array[0]", neither of which throw errors but the output is null.  I've also tried putting brackets around the Value but that doesn't work either.   UpdateRecord is a much cleaner solution than writing a whole ExecuteScript, I'd much rather use it if I can, but I'm just not understanding how this is supposed to work for arrays.  Help appreciated! ... View more

I'm on NiFi 1.11.4 trying to write a simple groovy script which can output the results of a postgres SQL query to an attribute (something not currently supported by ExecuteSql).  I've got a postgres Driver jar (confirmed working outside of NiFi) but when I specify it in the "Modules Directory" option of ExecuteScript, I get this: classNotFoundException: org.postgresql.Driver   Here is my ExecuteScript setup:   And here is the script body:       import groovy.sql.Sql def url = 'jdbc:postgresql://mydatabaseUrl:5432/mydb' def user = 'myuser' def password = 'mypassword' def driver = 'org.postgresql.Driver' //def sql = Sql.newInstance(url, user, password, driver) Sql.withInstance(url, user, password, driver) { sql -> def first = sql.firstRow('SELECT * from myschema.mytablelimit 1;') log.error(first) // easy way to print directly to the NiFi GUI }         I'm very sure that the postgresql-42.2.5.jar I'm using is good.  I've tested it using the exact same script but invoking it with this and got a response, all worked fine.   groovy -cp /data/apps/nifi/conf/postgresql-42.2.5.jar postgrestest.groovy   All evidence seems to indicate that the 'Modules Directory' isn't properly adding the jar to my classpath.  I tried to work around this by loading the jar directly in the script using this .getClass().classLoader.rootLoader.addURL( new File ( "file.jar" ).toURL()) but it looks like "this" means something different in ExecuteScript, which makes sense but is a rabbit hole I'm not trying to go down today.  I also tried this blog post to reuse an existing Controller Service, but only got timeouts (Controller Service is tested and working on other ExecuteSql processors). I've also tested ExecuteScript Modules Directory with other jars, and those seem to work fine.  This might be a Postgres specific problem ... View more

I'm trying to use ReplaceText to remove x number of lines from the top of a flowfile based on a flowfile attribute.  I'm using the following regex but ReplaceText says its invalid:   ^(.*?\n){${skip_lines}}    It seems like I should be able to reference a flowfile attribute from regex according to this question but I just get an error.  Any idea how I should be doing this?  My full config is below:   ... View more

I'm getting the following error when using GenerateFlowFile to pass a JSON to a FlattenJson processor.  I really have no idea how to troubleshoot this, is there a modules directory that I could place the jar file that seems to be missing or something?  Full message below 14:52:37 UTC ERROR 073500a4-0174-1000-640b-2486efecdb6d   FlattenJson[id=073500a4-0174-1000-640b-2486efecdb6d] FlattenJson[id=073500a4-0174-1000-640b-2486efecdb6d] failed to process session due to Could not initialize class com.github.wnameless.json.flattener.StringEscapePolicy; Processor Administratively Yielded for 1 sec: java.lang.NoClassDefFoundError: Could not initialize class com.github.wnameless.json.flattener.StringEscapePolicy ... View more

I am creating a NiFi pipeline to send phone numbers to a phone number normalizer.  I have used EvaluateJSONPath to put all the phone numbers into NiFi attributes, then used ReplaceText to overwrite the flowfile content with the input JSON and the values of the attributes.  Here's the input JSON used in ReplaceText:   { "input": [ { "nbr": "${TelephoneNumber1}", "recordId": "TelephoneNumber1" }, { "nbr": "${TelephoneNumber2}", "recordId": "TelephoneNumber2" }, { "nbr": "${TelephoneNumber3}", "recordId": "TelephoneNumber3" }, { "nbr": "${TelephoneNumber4}", "recordId": "TelephoneNumber4" } ], "app_id": "test" }     I am trying to remove the whole entry from the "input" array if the "nbr" field is blank.  Meaning that this   { "input": [ { "nbr": "", "recordId": "TelephoneNumber1" }, { "nbr": "5551212", "recordId": "TelephoneNumber2" }, { "nbr": "5558787", "recordId": "TelephoneNumber3" }, { "nbr": "", "recordId": "TelephoneNumber4" } ], "app_id": "test" }   Would come out looking like this:   { "input": [ { "nbr": "5551212", "recordId": "TelephoneNumber2" }, { "nbr": "5558787", "recordId": "TelephoneNumber3" } ], "app_id": "test" }     I am currently able to accomplish this using regex and 2 ReplaceText processors to make the replacements and clean up dangling/missing commas and other formatting issues, but it basically just treats the JSON like a string instead of a data structure.   Is there a better way to do this that treats JSON like a data structure and doesn't risk creating invalid JSON? ... View more

I feel a little silly asking because I can't find anything about this on the internet, but was InferAvroSchema removed from NiFi 1.11?  My organization recently upgraded our NiFi version and I noticed it was missing, but figured it was something they had been messing with.   However I upgraded my home server's NiFi and I notice its missing from there too.   I'm hoping that it was replaced by another processor or something?  I really use this a lot.   ... View more

I've got a standalone HBase running on my local Windows 10 machine.  It starts up fine, I can connect to it with `hbase shell` and life is grand.   But connecting to it with Java has been a complete failure.  Here's my setup in brief: hbase-site.xml     <configuration> <property> <name>hbase.master.port</name> <value>16000</value> </property> <property> <name>hbase.master.info.bindAddress</name> <value>localhost</value> </property> <property> <name>hbase.cluster.distributed</name> <value>false</value> </property> </configuration>       Java code:     Configuration configuration = HBaseConfiguration.create(); configuration.addResource(new Path(hbaseSiteXMLPath)); UserGroupInformation.setConfiguration(configuration); Connection connection = ConnectionFactory.createConnection(configuration); Table table = connection.getTable(TableName.valueOf(myTableName)); Result result = table.get(get); table.close(); connection.close();       This seems like it should be straightforward, but hours later I'm still getting the same Exception as I have all day:     MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-secure/master     I've changed `zookeeper.znode.parent` to `/hbase` but the Exception remains the same, except `/hbase-secure` changes to `/hbase`.  I've also added ` hbase.zookeeper.quorum` to `localhost` and ` hbase.zookeeper.property.clientPort` to 2181, but that seemed to look for an existing zookeeper connection and fail when it didn't exist rather than create a new one, so I took those out.   Any help appreciated, very frustrated. ... View more

This is something that should be pretty straightforward that isn't working for some reason. I'm trying to query a table and return rows that match 2 filters. I'm getting a single result that matches one of the filters and doesn't have the other column in it. I was expecting this code to return all rows where PersonAddressHash = ba3d1874fee1d9076bde083d819c6a2165ec5790 AND PdfHash = 5151c1394a08a61cbd4d821914bbdca5fa333975dd413685bd1f6b086f555f3e, but its only matching on the PdfHash: Scan scan = new Scan(); // Create filters and add them to scan object         SingleColumnValueFilter filter1 = new SingleColumnValueFilter(             Bytes.toBytes("data"),              Bytes.toBytes("PersonAddressHash"),              CompareOperator.EQUAL,              Bytes.toBytes("ba3d1874fee1d9076bde083d819c6a2165ec5790"));         SingleColumnValueFilter filter2 = new SingleColumnValueFilter(             Bytes.toBytes("data"),              Bytes.toBytes("PdfHash"),              CompareOperator.EQUAL,              Bytes.toBytes("5151c1394a08a61cbd4d821914bbdca5fa333975dd413685bd1f6b086f555f3e")         FilterList filters = new FilterList(FilterList.Operator.MUST_PASS_ALL);         filters.addFilter(filter1);         filters.addFilter(filter2);         scan.setFilter(filters); // Read the results into a JSON array, then print it to stdout         ResultScanner scanner = table.getScanner(scan);         JSONArray resultsJSON = new JSONArray();         for (Result result : scanner) {             JSONObject swap = new JSONObject();             for (Cell cell : result.listCells()) {                 String qualifier = Bytes.toString(CellUtil.cloneQualifier(cell));                 String value = Bytes.toString(CellUtil.cloneValue(cell));                 swap.put(qualifier, value);             }             resultsJSON.put(swap);         }         System.out.print(resultsJSON.toString(4));     } I'm specifying FilterList.Operator.MUST_PASS_ALL yet its only passing one of the filters. My only guess is about the missing column perhaps? Not sure why this isn't working, help appreciated. ... View more

I have this XML file: <request>     <requestType>BULKRETRIEVE</requestType>     <requestDomainType>ROI</requestDomainType>     <systemName>SYSTEMTEST</systemName>     <location>USA</location>         <userInformation>         <userId>1313</userId>         <firstName>Some</firstName> <!-- required -->         <lastName>Guy</lastName> <!-- required -->         <email>email@address.com</email> <!-- required if phone not included -->         <phone></phone> <!-- required if email not included -->     </userInformation>       <requestObject>         <startDate>2019-01-01T00:00:00.000-05:00</startDate>         <endDate>2019-01-31T00:00:00.000-05:00</endDate>         <type>ROI</type>     </requestObject> </request> Using this schema {     "namespace": "com.organization.somethingspecific",     "name": "request",     "type": "record",     "fields": [       {"name": "requestType", "type": ["string","null"], "default": null},       {"name": "requestDomainType", "type": ["string","null"], "default": null},       {"name": "systemName", "type": ["string","null"], "default": null},       {"name": "location", "type": ["string","null"], "default": null},       {"name": "userInformation", "type": ["null", {         "name": "userInformation", "type": "array", "items": {           "name": "userInformation", "type": "record", "fields": [             {"name": "userId", "type": ["string","null"], "default": null},             {"name": "firstName", "type": ["string","null"], "default": null},             {"name": "lastName", "type": ["string","null"], "default": null},             {"name": "email", "type": ["string","null"], "default": null},             {"name": "phone", "type": ["string","null"], "default": null}           ]         }       }], "default": null},       {"name": "requestObject", "type": ["null",{         "name": "requestObject", "type": "array", "items": {           "name": "requestObject", "type": "record", "fields": [             {"name": "startDate", "type": ["string","null"], "default": null},             {"name": "endDate", "type": ["string","null"], "default": null},             {"name": "type", "type": ["string","null"], "default": null}           ]         }       }], "default": null}     ] } I am confident that the schema is correct because I am able to convert from XML to JSON no problem. However UpdateRecord is not able to alter nested fields no matter how I reference them. I have the following UpdateRecord processor: At the very least I would expect //startDate to work since that is SUPPOSED to ignore hierarchy, but I can only update top level fields such as /requestType which is the only one that actually updates of the 4 in my configuration. I am following the documentation exactly, what else could be wrong? Any help appreciated. ... View more

I am using an ExecuteScript processor to read the keys and values of all the attributes from a 0 byte empty flowFile into a JSON string, write the JSON string to the contents of the flowFile and transfer to REL_SUCCESS. This is the script in the ExecuteScriptProcessor: from org.apache.commons.io import IOUtils from java.nio.charset import StandardCharsets from org.apache.nifi.processor.io import OutputStreamCallback # Define a subclass of OutputStreamCallback for use in session.write() class PyOutputStreamCallback(OutputStreamCallback): def __init__(self): pass def process(self, outputStream): atts = {} for key, value in flowFile.getAttributes().iteritems(): atts[key] = str(value).encode('utf-8') atts_string = str(atts) outputStream.write(bytearray(atts_string.encode('utf-8'))) # end class flowFile = session.get() if (flowFile != None): flowFile = session.write(flowFile, PyOutputStreamCallback()) session.transfer(flowFile, REL_SUCCESS) # implicit return at the end I have been using GenerateFlowFile and UpdateAttributes to generate a test flowFile, the info for which is the image attached to this post. When I try to run the pipeline, I get this output: SyntaxError: mismatched input '/' expecting EOF in <script> at line number 1 at column number 0 From what I've found on Google, this error typically means an indenting or formatting issue with Python, but I've made absolutely sure that is not the case by deleting and re-tabbing each line individually since its such a short script, but the error still persists. So it seems to be NiFi related but I have no idea what could be causing it, I can't find a '/' character anywhere in the sample flowFile attributes (the 'path' attribute had a '/' but I changed it to 'test' and the issue persisted) or my code. EDIT: I just realized there is an AttributesToJson processor that I can use to keep going, but I am still curious as to why my code above isn't working, I feel like this is a use case similar to things I will want to do in the future and it makes me nervous not understanding why this won't work. ... View more

I need to replace all of the contents of my flowfile with a file on the disk. Seems like FetchFile just adds the contents to the flowfile, is there a way to drop the contents of my flowfile first? ... View more

I'm getting the error "Cluster is still in the process of voting on the appropriate Data Flow." on the sandbox HDF docker image. Not really sure how to troubleshoot this, I'm not familiar with NiFi or HDF at all. Suggestions appreciated ... View more