Cloudera Community
AmiraKhalifa
user rank
Contributor
My Accepted Solutions
Show All

Re: How to adapt the schema generated by inverAvro...

by Contributor in Support Questions
‎03-16-2018 01:56 PM
‎03-16-2018 01:56 PM
I used UpdateAttribut in which I get inferred.avro.schema with getAttribut and apply a 'replace' expression ... View more

Re: How to adapt the schema generated by inverAvro...

by Contributor in Support Questions
‎03-07-2018 11:52 AM
‎03-07-2018 11:52 AM
Hi @Pierre Villard Thank you for your response. I want to automate the work as much as possible. For a user of the application I want him to be able to specify only the file to import and then nifi does the necessary. I did a function that created a sql table automatically from inferAvroSchema and I want to adapt this schema. Avro Registry works manually. For each file I have to fill it manually. ... View more

How to adapt the schema generated by inverAvroSche...

by Contributor in Support Questions
‎03-06-2018 04:19 PM
‎03-06-2018 04:19 PM
Hello, I use the processor inferavroschema to define the schema of my table SQL then I use PutDatabaseRecord except for some columns I have null values not detectable by inferavroschema For example for a column in my file csv it can have null values but so the schema avro they have string values. During the insertion I have errors I would like to know if there is a way to change the schema Avro without going through a manual writing of the structure of the table with Schema Registry. Thank you ... View more
Labels:

Re: Read a flowfile with groovy error

by Contributor in Support Questions
‎03-05-2018 06:15 PM
‎03-05-2018 06:15 PM
My file extension is avro i can't use def schema = ff.read().withReader("UTF-8"){newJsonSlurper().parse(it)} I have to use each and then parse schema. Do you an example plz @Matt Burgess ... View more

Re: Read a flowfile with groovy error

by Contributor in Support Questions
‎03-05-2018 05:00 PM
‎03-05-2018 05:00 PM
Not for me 😞 def createTable = "create table ${schema.name} (" + schema.fields.collect{ "\n ${it.name.padRight(39)} ${typeMap[it.type]}" }.join(',') + "\n)" No such a no such property : name for class : java.lang.string And i Used this to escape the first error : def schema = '' session.read(ff, {inputStream -> schema= IOUtils.toString(inputStream, StandardCharsets.UTF_8) } as InputStreamCallback) ... View more

Re: Read a flowfile with groovy error

by Contributor in Support Questions
‎03-05-2018 04:26 PM
‎03-05-2018 04:26 PM
Thank you for your response. Actually it's my post. I used SQL.mydb.execute(createTable.toString()) but I got the same error which due to this instruction def schema = ff.read().withReader("UTF-8"){newJsonSlurper().parse(it). So I changed the code import groovy.json.JsonSlurper import org.apache.commons.io.IOUtils import java.nio.charset.StandardCharsets def ff = session.get() if(!ff)return def schema = '' session.read(ff, {inputStream -> schema= IOUtils.toString(inputStream, StandardCharsets.UTF_8) } as InputStreamCallback) //define type mapping def typeMap = [ "string" : "varchar(255)", "long" : "numeric(10)", [ "null", "string" ]: "varchar(255)", [ "null", "long" ] : "numeric(10)", ] //build create table statement def createTable = "create table ${schema.name} (" + schema.fields.collect{ "\n ${it.name.padRight(39)} ${typeMap[it.type]}" }.join(',') + "\n)" //execute statement through the custom defined property //SQL.mydb references http://docs.groovy-lang.org/2.4.10/html/api/groovy/sql/Sql.html object SQL.mydb.execute(createTable.toString()) //transfer flow file to success REL_SUCCESS << ff error : groovy.lang.missingpropertyexception no such property for class java.lang.string ... View more

Re: Read a flowfile with groovy error

by Contributor in Support Questions
‎03-05-2018 03:28 PM
‎03-05-2018 03:28 PM
@Matt Burgess Help plz ... View more

Read a flowfile with groovy error

by Contributor in Support Questions
‎03-05-2018 03:09 PM
‎03-05-2018 03:09 PM
I'm using groovy in Execute processor to read an avro schema to create a sql table. My code is : import groovy.json.JsonSlurper def ff = session.get() if(!ff)return def schema = ff.read().withReader("UTF-8"){ new JsonSlurper().parse(it) } def typeMap = [ "string" : "varchar(255)", "long" : "numeric(10)", [ "null", "string" ]: "varchar(255)", [ "null", "long" ] : "numeric(10)", ] def createTable = "create table ${schema.name} (" + schema.fields.collect{ "\n ${it.name.padRight(39)} ${typeMap[it.type]}" }.join(',') + "\n)" SQL.mydb.execute(createTable) REL_SUCCESS << ff And I got this error can someone help me plz ERROR [Timer-Driven Process Thread-3] o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=e65b733e-0161-1000-45f0-3264d6fb51dd] ExecuteSc$ Possible solutions: getId(), find(), grep(), each(groovy.lang.Closure), find(groovy.lang.Closure), grep(java.lang.Object); rolling back session: {} org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingMethodException: No signature of m$ Possible solutions: getId(), find(), grep(), each(groovy.lang.Closure), find(groovy.lang.Closure), grep(java.lang.Object) at org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:230) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingMethodException: No signature of method: org.apache.nifi.controller.repositor$ Possible solutions: getId(), find(), grep(), each(groovy.lang.Closure), find(groovy.lang.Closure), grep(java.lang.Object) ... View more
Labels:

Re: create SQL table from AVRO Schema

by Contributor in Support Questions
‎03-01-2018 03:59 PM
‎03-01-2018 03:59 PM
@Maik Krisanov ... View more

Re: create SQL table from AVRO Schema

by Contributor in Support Questions
‎03-01-2018 03:12 PM
‎03-01-2018 03:12 PM
for my test file I have 10 columns but I want the work to be done independently of this file. For example, I want to use getfile-> infeAvroSchema-> Create the table in Postgresl and then use PutdatabaseRecod. Once I have my schema generated I do not know what exactly to use to create the table. ... View more
Contact Me
Online
Offline
Last Visited
‎08-08-2016 05:05 AM
Community Statistics
Member Since ‎06-30-2016 06:01 AM
Last Visited ‎08-08-2016 05:05 AM
Posts 44
Kudos received 2