Member since
06-30-2016
44
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1469 | 03-16-2018 01:56 PM |
07-09-2018
02:37 PM
Hello, I'm using the processor inferAvroSchema to infer the schema table from a csv. I want to use the GetAvroMetadata processor to extract the record name from the results of inferAvroSchema but I don't know how to configure the processor. I know that I can use a groovy code to extract the record name but I think GetAvroMetadata is used for this purpose too. capture.png Can someone help me plz. Thank you
... View more
Labels:
- Labels:
-
Apache NiFi
06-29-2018
10:30 AM
Hello, My dataflow is as follow: My purpose is to create a table from a csv file and then put the data in this table. Everything is working good but I need to check if the table is created before storing the csv file. So I used the Execute SQL to run this query: select count(table_name) from information_schema.tables where table_name='warranty' But it returns only the number of rows and not the flowfile. My test is if number of row is one so I execute the rest. So I need to use dataflow which can check if the table is well created and execute the other processors if not it stop every thing. Can you help me plz I know that i can find another way but don't find ideas. Thank you for your help.
... View more
Labels:
- Labels:
-
Apache NiFi
06-27-2018
09:33 AM
@Shu Thank you for your response when I tried this regex expression : ^[!@#$%^&*(),.?":{}|<>] It changed only the first occurance of a special character. I have in input this csv file : Name,T&C_Test,Date_of_(Transformation),date_/_of _deliverance 1,1457,26/12/2016,14/12/2017 2,4789,18/04/2017,03/09/2018 I expect an output file like this one: Name,TC_Test,Date_of_Transformation,date_of _deliverance 1,1457,26/12/2016,14/12/2017 2,4789,18/04/2017,03/09/2018
... View more
06-26-2018
12:08 PM
I tried with ^ at the beginning of the expression. No results 😕
... View more
06-25-2018
04:39 PM
Hello, The header of my csv file contains some special character. Example Name,T&C_Test,Date_of_(Transformation) And I want to use the processor Replace Text to edit only the header. When I use the Append Stratesy for the entire text, it dosen't work. Help plz. Thank you
... View more
Labels:
- Labels:
-
Apache NiFi
05-30-2018
08:48 AM
@Matt Burgess I have a csv file .. I will try this approach Thank you ! I accept this answer 🙂
... View more
05-29-2018
04:33 PM
@Matt Burgess Thank you for your response I'm using hive-jdbc-2.1.0.jar. So the processor PutDatabaseRecord does not offer the insert functionality wih a hive pool connection? if it is the case ,why this connection is proposed in the processor ? Can we do an update and delete query with PutDatabaseRecord + Hive ?
... View more
05-29-2018
02:59 PM
Hello, I'm using the processor PutDatabaseRecord with a Hive connection pool to insert data in my table. But I get the following error: Error : My log file : I used the same connection pool with putHiveQL and everything is working fine. Can someone help me plz? Thank you
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
05-20-2018
10:54 AM
Hello, I have a timestamp column in my flowfile written in this way 2008:05:17 17:23:01. I want to add a quota to this value so i can insert it in postgresql table. I used replace text like this : search value : ((\d{4})([\-\/.])([0-3]?\d)\2([0-3]?\d)|([0-3]?\d)([\-\/.])([0-3]?\d)\6(\d{4}))(?:\s+([012]?\d)([:hap])([0-5]\d))([:hap])([0-5]\d) replacement value : ' ((\d{4})([\-\/.])([0-3]?\d)\2([0-3]?\d)|([0-3]?\d)([\-\/.])([0-3]?\d)\6(\d{4}))(?:\s+([012]?\d)([:hap])([0-5]\d))([:hap])([0-5]\d) ' but didn't work need ur help plz
... View more
Labels:
- Labels:
-
Apache NiFi
05-14-2018
09:28 AM
Hello @Rahul Soni, I set up a nifi registry but I could not connect it with my version nifi 1.4. The GUI does not offer me registry clients. Do you have a solution to unlock this problem ? Thank you
... View more
04-30-2018
08:41 AM
Thank you @Rahul Soni. I did not know this solution. I check and I'll let you know.
... View more
04-27-2018
02:48 PM
Hello, I inquired about the directory of templates nifi before asking the question. And I saw that all templates are stored in flow.xml.gz. My question is can I opt for a directory that stores the templates instead of this file. Is it more correct and will not cause problems. Thank you
... View more
Labels:
- Labels:
-
Apache NiFi
04-19-2018
01:31 PM
Hello, when I run a nifi stream with a misconfigured processor. At first I have this error: 502 Unexpected Response. When I do a reload of the page I have 502 Bad Gateway. Should I change something in the nifi config to avoid this problem. Thank you
... View more
Labels:
- Labels:
-
Apache NiFi
03-27-2018
09:06 AM
@Matt Burgess I tried a little test with listen http i made a ListenHttp processor as follows: Then I created a little python code: pythonreq.png I have an error: and I do not know if the configurations are correct
... View more
03-26-2018
03:47 PM
Hello, I want to be able to send automatically generated data with random from python to nifi using the requests library. on the nifi side I intend to use gethttp to retrieve the data. My question: what should I put in the URL parameter? I do not understand much t the APIs I searched on the internet since this morning but it remains a little vague someone can help me? thank you
... View more
Labels:
- Labels:
-
Apache NiFi
03-19-2018
01:06 PM
@Rahul Soni thank you for your very detailed answer. The problem is that I have to define the schema of my table in validateCSV. Firstly in my CSV file I have 15 columns and I want to automate the work so that it is not dependent on a single file and therefore I avoid entering values manually.
... View more
03-16-2018
03:20 PM
Hello, I have a CSV file that I want to insert into a table. Sometimes I have string columns to insert in long columns that pose a normal problem. I'm looking for a way to put those lines aside ( in a flowfile ) and insert the rest. I'm using the putDatabaseRecord processor to insert the csv file. Thank you
... View more
03-16-2018
01:56 PM
I used UpdateAttribut in which I get inferred.avro.schema with getAttribut and apply a 'replace' expression
... View more
03-07-2018
11:52 AM
Hi @Pierre Villard Thank you for your response. I want to automate the work as much as possible. For a user of the application I want him to be able to specify only the file to import and then nifi does the necessary. I did a function that created a sql table automatically from inferAvroSchema and I want to adapt this schema. Avro Registry works manually. For each file I have to fill it manually.
... View more
03-06-2018
04:19 PM
Hello, I use the processor inferavroschema to define the schema of my table SQL then I use PutDatabaseRecord except for some columns I have null values not detectable by inferavroschema For example for a column in my file csv it can have null values but so the schema avro they have string values. During the insertion I have errors I would like to know if there is a way to change the schema Avro without going through a manual writing of the structure of the table with Schema Registry. Thank you
... View more
Labels:
- Labels:
-
Schema Registry
03-05-2018
06:15 PM
My file extension is avro i can't use def schema = ff.read().withReader("UTF-8"){newJsonSlurper().parse(it)} I have to use each and then parse schema. Do you an example plz @Matt Burgess
... View more
03-05-2018
05:00 PM
Not for me 😞 def createTable = "create table ${schema.name} (" +
schema.fields.collect{ "\n ${it.name.padRight(39)} ${typeMap[it.type]}" }.join(',') +
"\n)" No such a no such property : name for class : java.lang.string And i Used this to escape the first error : def schema = '' session.read(ff, {inputStream ->
schema= IOUtils.toString(inputStream, StandardCharsets.UTF_8)
} as InputStreamCallback)
... View more
03-05-2018
04:26 PM
Thank you for your response. Actually it's my post. I used SQL.mydb.execute(createTable.toString()) but I got the same error which due to this instruction def schema = ff.read().withReader("UTF-8"){newJsonSlurper().parse(it). So I changed the code import groovy.json.JsonSlurper
import org.apache.commons.io.IOUtils
import java.nio.charset.StandardCharsets
def ff = session.get()
if(!ff)return
def schema = ''
session.read(ff, {inputStream ->
schema= IOUtils.toString(inputStream, StandardCharsets.UTF_8)
} as InputStreamCallback)
//define type mapping
def typeMap = [
"string" : "varchar(255)",
"long" : "numeric(10)",
[ "null", "string" ]: "varchar(255)",
[ "null", "long" ] : "numeric(10)",
]
//build create table statement
def createTable = "create table ${schema.name} (" +
schema.fields.collect{ "\n ${it.name.padRight(39)} ${typeMap[it.type]}" }.join(',') +
"\n)"
//execute statement through the custom defined property
//SQL.mydb references http://docs.groovy-lang.org/2.4.10/html/api/groovy/sql/Sql.html object
SQL.mydb.execute(createTable.toString())
//transfer flow file to success
REL_SUCCESS << ff error : groovy.lang.missingpropertyexception no such property for class java.lang.string
... View more
03-05-2018
03:09 PM
I'm using groovy in Execute processor to read an avro schema to create a sql table. My code is : import groovy.json.JsonSlurper
def ff = session.get()
if(!ff)return
def schema = ff.read().withReader("UTF-8"){ new JsonSlurper().parse(it) }
def typeMap = [
"string" : "varchar(255)",
"long" : "numeric(10)",
[ "null", "string" ]: "varchar(255)",
[ "null", "long" ] : "numeric(10)",
]
def createTable = "create table ${schema.name} (" +
schema.fields.collect{ "\n ${it.name.padRight(39)} ${typeMap[it.type]}" }.join(',') +
"\n)"
SQL.mydb.execute(createTable)
REL_SUCCESS << ff And I got this error can someone help me plz ERROR [Timer-Driven Process Thread-3] o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=e65b733e-0161-1000-45f0-3264d6fb51dd] ExecuteSc$ Possible solutions: getId(), find(), grep(), each(groovy.lang.Closure), find(groovy.lang.Closure), grep(java.lang.Object); rolling back session: {} org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingMethodException: No signature of m$ Possible solutions: getId(), find(), grep(), each(groovy.lang.Closure), find(groovy.lang.Closure), grep(java.lang.Object) at org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:230) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingMethodException: No signature of method: org.apache.nifi.controller.repositor$ Possible solutions: getId(), find(), grep(), each(groovy.lang.Closure), find(groovy.lang.Closure), grep(java.lang.Object)
... View more
Labels:
- Labels:
-
Apache NiFi
03-01-2018
03:12 PM
for my test file I have 10 columns but I want the work to be done independently of this file. For example, I want to use getfile-> infeAvroSchema-> Create the table in Postgresl and then use PutdatabaseRecod. Once I have my schema generated I do not know what exactly to use to create the table.
... View more
03-01-2018
02:58 PM
Hello, Using InferAvroSchema I got an avro Schema of my file. I want to create at able in PostregSql using this avroSchema. Which processor I have to use and connect to InferAvroSchema to create my table. Thank you!
... View more
02-28-2018
08:46 AM
1 Kudo
@Timothy Spann
... View more
02-27-2018
05:59 PM
I have always the same error! I share my processor configuration
... View more