Member since
06-30-2016
44
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1558 | 03-16-2018 01:56 PM |
08-21-2019
08:11 PM
Hi @mburgess, I was looking for a processor that would let me pull a query out of a field of the incoming flow file, instead of turning the entire flow file into a query. PutDatabaseRecord allowed to do that (which is when I discovered that the Hive connection does not support an explicit call of conn.commit()). I want to keep as much of the flow file intact as possible, is there a way to do that? Thank you.
... View more
07-09-2018
03:24 PM
InferAvroSchema requires you to enter the record name yourself, if you need access to that record name later, you could set a variable on the process group and use that in Expression Language for the Record Name, then you'd have access to that same variable everywhere in the process group and wouldn't have to extract it. If you don't have access to the value injected into the schema by InferAvroSchema, and the schema is in an attribute (let's say "inferred.avro.schema"), then you can use the jsonPath() function in NiFi Expression Language to extract the record name into a separate attribute. You'd need an UpdateAttribute to set "record.name" to the following: ${inferred.avro.schema:jsonPath("$.name")} If your schema is in the content of the flow file, then since it is a JSON object you can use EvaluateJSONPath to get the record name into an attribute, using the following JSONPath expression: $.name ExtractAvroMetadata is for Avro files, so if your flow file contained Avro (with an embedded schema), you could use ExtractAvroMetadata (adding "avro.schema" to the list of metadata to extract) in order to get the schema. But this processor doesn't work for CSV files.
... View more
06-29-2018
03:06 PM
As of NiFi 1.5.0 (via NIFI-4522), you can issue a SQL query in PutSQL while still retaining the incoming flow file contents. For your case, you could send the CSV file to PutSQL and execute a "CREATE TABLE IF NOT EXISTS" statement, which will create the table the first time but allow the CSV to proceed to the "real" destination processor, likely PutDatabaseRecord.
... View more
06-29-2018
09:18 AM
@Amira khalifa Use one of the way from the above shared link to take out only the header from the csv file then in replace text keep the then search for (&|\(|\)|\/_|\s) and in Replacement value keep as empty string, now we are searching for all the special characters in the header flowfile then replacing with empty string.Now add this header flowfile with the other non header flowfile. all the explanation and template.xml are shared in this link.
... View more
05-20-2018
12:29 PM
@Amira khalifa You are having timestamp column format as 2008:05:17 17:23:01(with colon delimiter). Use the ReplaceText processor with below configs Search Value
(\d{4}:\d{2}:\d{2})(\s+)(\d{2}:\d{2}:\d{2})
Replacement Value
'$1$2$3'
Character Set
UTF-8
Maximum Buffer Size
1 MB //change the value as per your flowfile size
Replacement Strategy
Regex Replace
Evaluation Mode
Entire text Input Flowfile: 2008:05:17 17:23:01 --other fields Outputflowfile: '2008:05:17 17:23:01' --other fields - If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of issues.
... View more
05-14-2018
01:43 PM
@Amira khalifa Please mark the answer as accepted if it helped to resolve your problem
... View more
03-27-2018
12:16 PM
The relevant part of the log is "Address already in use", looks like you've configured ListenHttp to listen on port 8081 but some other process is using that port already.
... View more
03-16-2018
01:56 PM
I used UpdateAttribut in which I get inferred.avro.schema with getAttribut and apply a 'replace' expression
... View more
03-05-2018
06:15 PM
My file extension is avro i can't use def schema = ff.read().withReader("UTF-8"){newJsonSlurper().parse(it)} I have to use each and then parse schema. Do you an example plz @Matt Burgess
... View more