About johndcal

johndcal · ‎07-27-2020

Ignore, I figured it out. Hate to say it, but I am working with the UCS satellite database as input - it has 36 fields - Eccentricity is one of the fields. I forgot to add a default value for this field for handling nulls.

johndcal · ‎07-24-2020

I created a small input file that produces 9 splits. In each split, none of the fields contained a null value, so nulls are not the problem as someone in StackOverflow suggested. MergeRecord still threw the error when it tried to merge the 9 records.

johndcal · ‎07-24-2020

Does anyone know what the error in the subject line of this message means? I am using SplitRecord processor, checking for duplicate records (removed for testing), then trying to use MergeRecord to put all the splits back into one file, but MergeRecord fails with "Field Eccentricity Cannot Be Null" Error. I have not found information on the meaning of this error. Thanks in advance if you know.

johndcal · ‎06-19-2020

This post does not need any more views, I have solved the problem: The first line of my data is column headers. In latter case, in CSVReader, when Schema Access Strategy is set to Use 'Schema Name' Property, then Treat First Line as Header needs to be set to True. The property Ignore CSV Header Column Names also needs to be set to True otherwise if there are field name mismatches between the input file and the schema, the input file names will be used instead of the schema. Alternative: I was able to set Schema Access Strategy to Use String Fields From Header, and that worked, but it ignores your schema, if you have one set up. jc

johndcal · ‎06-18-2020

Update: template for flow titled CountryCodeLookup.xml is in GitHub: https://github.com/johndcal/NiFi-Tutorial-5-Files Hello, Can anyone help me with this? Below is the data that I am trying to parse and the schema being used to parse it. Should I be looking at something else? Will provide more info if needed. Thanks, jc Data: name,loc_id,launch,norad_number,classification NOAA 19,US,2009-02-06,33591,Weather GOES 13,US,2006-05-24,29155,TV SAUDISAT 1C,SA,2002-12-20,27607,Education METEOR M2,RU,2014-07-08,40069,Weather MEASAT 3,MY,2006-12-11,29648,TV Schema: { "type": "record", "namespace": "satellites", "name": "satdata_in", "fields": [ { "name": "name", "type": "string" }, { "name": "loc_id", "type": "string" }, { "name": "launch", "type": "string" }, { "name": "norad_number", "type": "int" }, { "name": "classification", "type": "string" } ] }

johndcal · ‎05-15-2020

Is the namespace field in an AVRO schema required? Does the namespace first have to be created if you provide it in the schema? Not sure how this works. Thanks, johndcal

johndcal · ‎05-12-2020

Steven, Thanks much for your quick reply. You were correct; I had my GetFile scheduled to zero, so I reset it to 30 seconds. I do have a CSVReader configured, and made one change so that the first line would not be read as column headers (there are two settings for that, I had the first one correct, but not the second.) My current text data file looks like this: sat-01 sat-02 ... sat-06 After a quick test, only the first value went into the database. Looking at the Provenance Event for my ExtractText processor, here are the attributes it captured: satellite_name sat-01 No value set satellite_name.0 sat-01 No value set satellite_name.1 sat-01 No value set satellite_name.2 sat-02 No value set satellite_name.3 sat-03 No value set satellite_name.4 sat-04 No value set satellite_name.5 sat-05 No value set satellite_name.6 sat-06 No value set I wish there was one attribute (satellite_name) and each name was picked up as a separate record, but not in an array. The array is ok, I just don't know how to deal with it using SQL in NiFi expression language. The ExtractText Output Claim content is perfect; again, only first name is going to the database. Thanks again for your help.

johndcal · ‎05-12-2020

Hello, Is there anyone out there that can help me with the following two problems? Thanks in advance. Problem #1: I have a very simple CSV input file (it has 6 mock satellite names). I am using the ExtractText processor to capture the satellite name into a user-defined property that becomes an attribute [Property: satellite_name, Value: (.{1,200})] ExtractText makes 3 copies of the same attribute; I only want one. I can see this when I look at the processor's data provenance - it makes: satellite_name, satellite_name.0, satellite_name.1, fills these with the first of the 6 mock satellite names, then picks up the other five and puts them in satellite_name.2, satellite_name.3, etc. Problem #2: I have tried a lot of different settings that I found in Google searches, but when I run the processor group, it does not stop inserting the flowfile data into the database, and it only inserts the first satellite name. The "matchted" flow of my NiFi processors is: GetFile -> ExtractText -> PutSQL -> LogAttribute. (The "unmatched" flow is GetFile -> ExtractText -> (different) LogAttribute.) I do not have any problems using NiFi to access the Postgres database. NiFi is running on Windows, Postgres is running on a Linux VM (on the same PC). In the PutSQL processor, this is my insert statement: INSERT INTO satellite.satname (satellite_name) VALUES('${satellite_name}'); That's the vital information about my problems; I can provide screenshots if needed if you think you can help and would like more information. I sincerely appreciate any time/effort anyone puts forward to helping me. I am new to NiFi and have come a long way getting it to talk to a database running on a VM. Thanks again,

Online	Offline
Last Visited	‎10-19-2020 04:24 PM

Member Since	‎05-12-2020 05:30 AM
Last Visited	‎10-19-2020 04:24 PM
Posts	10

Cloudera Community

Cloudera Community

Re: NiFi MergeRecord Processor throwing "Field Ecc...

Re: NiFi: LookupRecord Could Not Parse Incoming D...

Re: NiFi MergeRecord Processor throwing "Field Ecc...

Re: NiFi MergeRecord Processor throwing "Field Ecc...

NiFi MergeRecord Processor throwing "Field Eccentr...

Re: NiFi: LookupRecord Could Not Parse Incoming D...

NiFi: LookupRecord Could Not Parse Incoming Data

AVRO Schema Registry Namespace

Re: NiFi / Postgres Database Problems Inserting Re...

NiFi / Postgres Database Problems Inserting Record...