Thanks it worked /Simon ... View more

I don't currently see a processor that would do this for you, but if you are comfortable with a scripting language such as Groovy, Javascript, or Clojure, you could use ExecuteScript with any of these libraries to infer the character set for an incoming stream. For more information on including these library JAR(s) in your ExecuteScript configuration, see my ExecuteScript Cookbook (part 3) and/or my separate blog post. Since this seems like a good feature to have in NiFi proper, I have created a Jira (NIFI-4550) to add an InferCharacterSet processor. ... View more

@Simon Jespersen You can use UpdateAttribute and create a new attribute with the following EL : ${filename:getDelimitedField(3,'_')} ... View more

@Simon Jespersen happy to help 🙂 Thanks ... View more

Hi @Simon Jespersen, in your evalJsonpath processor you are using Path Not Found Behavior property as warn i.e it will generate a warning when a JSON path expression is not found, as in your csv file you are for some of the records wont have any data for zip. This warn message wont effect your flowfile, flowfile still routes to success relationship with all the available content will be extracted as attributes and for no content attributes values will be Empty string set. if you don't want to see those warn messages on the processor then just change Path Not Found Behavior property to ignore(default) which will ignore if the content is not found for any of the processor. Example:- i have recreated same WARN message as you are having with the below Json doc { "name" : "else", "adresse" : "route 66", "by" : "Hadoop City" <br>} with ignore property This is my json doc to evaljson processor with ignore as path not found property processor wont return any warn messages as it ignore if there is no content for the jsonpath expression. With warn property:- If you change path not found property to warn processor will return the same warn message as you are having in the question. both cases results the same output as zip attribute value is Empty string set and routes to Success relation. ... View more

@Simon Jespersen This looks like an authentication issue. For the given topic can you add ACLs for anonymous user as the protocol is PLAINTEXT? bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=<zookeeper:host> --add --allow-principal User:ANONYMOUS --operation Read --operation Write --operation Describe --topic <topic> ... View more

@nramanaiahHi, yes i found out the same,it seems to be happening when running hive through JDBC and ODBC. My problem is that i am running this in NIFI PutHiveQL processor ... View more

Hi Matt, We are running HDF-2.1.2 planning to upgrade ASAP thanks for your anwser ... View more

Hi Dennis, Even though I declared --null-string as --null-string '\\N' in the sqoop import synatax. still i am seeing "null" string value for all string fields where value is null in source schma. can you please provide any solution to it. below is my sqoop import syntax. import -Dhadoop.security.credential.provider.path=${jceks_loc} --password-alias ${passwordAlias} --username ${userName} --connect ${connection} --table ${sourceSchema}.${sourceTable} --hive-import --hive-table ${hiveTable} --target-dir ${hdfsDir} --fields-terminated-by "|" --delete-target-dir --null-string '\\N' -m 1 Thanks, Brahma ... View more

@Matt Clarke Thank you very very much, your answer was very useful for me ... View more