About AmiraKhalifa

sujay1848 · ‎08-21-2019

Hi @mburgess, I was looking for a processor that would let me pull a query out of a field of the incoming flow file, instead of turning the entire flow file into a query. PutDatabaseRecord allowed to do that (which is when I discovered that the Hive connection does not support an explicit call of conn.commit()). I want to keep as much of the flow file intact as possible, is there a way to do that? Thank you.

mburgess · ‎06-29-2018

As of NiFi 1.5.0 (via NIFI-4522), you can issue a SQL query in PutSQL while still retaining the incoming flow file contents. For your case, you could send the CSV file to PutSQL and execute a "CREATE TABLE IF NOT EXISTS" statement, which will create the table the first time but allow the CSV to proceed to the "real" destination processor, likely PutDatabaseRecord.

Shu_ashu · ‎06-29-2018

@Amira khalifa Use one of the way from the above shared link to take out only the header from the csv file then in replace text keep the then search for (&|$|$|\/_|\s) and in Replacement value keep as empty string, now we are searching for all the special characters in the header flowfile then replacing with empty string.Now add this header flowfile with the other non header flowfile. all the explanation and template.xml are shared in this link.

Shu_ashu · ‎05-20-2018

@Amira khalifa You are having timestamp column format as 2008:05:17 17:23:01(with colon delimiter). Use the ReplaceText processor with below configs Search Value (\d{4}:\d{2}:\d{2})(\s+)(\d{2}:\d{2}:\d{2}) Replacement Value '$1$2$3' Character Set UTF-8 Maximum Buffer Size 1 MB //change the value as per your flowfile size Replacement Strategy Regex Replace Evaluation Mode Entire text Input Flowfile: 2008:05:17 17:23:01 --other fields Outputflowfile: '2008:05:17 17:23:01' --other fields - If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.

mburgess · ‎03-27-2018

The relevant part of the log is "Address already in use", looks like you've configured ListenHttp to listen on port 8081 but some other process is using that port already.

AmiraKhalifa · ‎03-16-2018

I used UpdateAttribut in which I get inferred.avro.schema with getAttribut and apply a 'replace' expression

AmiraKhalifa · ‎03-05-2018

My file extension is avro i can't use def schema = ff.read().withReader("UTF-8"){newJsonSlurper().parse(it)} I have to use each and then parse schema. Do you an example plz @Matt Burgess

cstanca · ‎03-03-2018

@Amira khalifa There is no standard processor capable to address your requirement. You would have to build a custom processor to generate the DDL or use ExecuteSQL or ExecuteScript processor. Anyhow, keep in mind that avro data types are not an exact match with Postgresql data types to not mention that your avro may be hierarchical. For that situation I suggest you to follow the steps: ConvertAvroToJson -> FlattenJson -> ConvertJsonToSQL -> ExecuteSQL. This is just one approach, you can do it in several ways including using Record processors.

TimothySpann · ‎02-28-2018

1. Turn translate fields name to true 2. You must specify a schema registry 3. Change your schema access strategy. It's usually not from header Where is your schema name? Where is schema stored?

jsensharma · ‎02-26-2018

@Amira khalifa As this issue is resolved and it was because of the classname issue of the driver class, hence can you please mark this HCC thread as Answered by clicking on the "Accept" button that way the mentioned troubleshooting steps can be useful for other HCC users to quickly find/troubleshoot the issue when they see similar issue.

Online	Offline
Last Visited	‎08-08-2016 05:05 AM

Member Since	‎06-30-2016 06:01 AM
Last Visited	‎08-08-2016 05:05 AM
Posts	44
Kudos received	2

Cloudera Community

Re: How to adapt the schema generated by inverAvro...

Re: Error: PutDatabaseRecord + Hive connection poo...

Re: Need a processor to test if a table sql exists...

Re: ReplaceText Processor edit only the header of ...

Re: Error with replaceText timestamp add a quota

Re: send data from python to nifi http request

Re: How to adapt the schema generated by inverAvro...

Re: Read a flowfile with groovy error

Re: create SQL table from AVRO Schema

Re: none of the fields in the records map to the c...

Re: Error can't load postgresql jdbc driver