Member since
11-16-2015
902
Posts
664
Kudos Received
249
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
137 | 09-30-2025 05:23 AM | |
607 | 06-26-2025 01:21 PM | |
443 | 06-19-2025 02:48 PM | |
687 | 05-30-2025 01:53 PM | |
9685 | 02-22-2024 12:38 PM |
11-08-2016
08:11 PM
1 Kudo
Avro doesn't like the dots in the attribute names (which become field names), perhaps you could rename them (with UpdateAttribute or at the source processor(s)) using underscores (or other valid characters like alphanumeric characters, see the link for the rules).
... View more
11-08-2016
07:51 PM
1 Kudo
I don't believe it is currently possible to convert/use the decimal type in the Avro processors, I have written up NIFI-3000 to cover this improvement. If you are comfortable with a scripting language like Javascript or Groovy, you could use the ExecuteScript processor, point the Module Directory property at JAR(s) for Avro 1.8.0, and code the conversion yourself. However beware that you don't do additional Avro processing downstream, as that will likely break (because of the bug in NIFI-3000).
... View more
11-08-2016
04:20 PM
I have written up an improvement Jira to add better validation for the Avro Record Name property.
... View more
11-08-2016
04:13 PM
What do your search and replace regular expressions look like? You might be matching the whole text looking for the delimiter and replacing with the delimiter, when really it sounds like you want to match the delimiter and replace with something else (a comma, e.g.)
... View more
11-08-2016
02:58 PM
1 Kudo
Does the term "avro-schema" (from your error message) appear in the schema? If so, then the error is because it doesn't like the dash, you can only use upper- and lower-case letters and underscores for the first character, then that set plus digits for all remaining characters (see Naming section of Avro doc). Try replacing the dash with an underscore and it should work.
... View more
11-06-2016
03:15 PM
3 Kudos
The code for ScanContent looks like it watches the specified file for changes; otherwise it shouldn't refresh the dictionary file (unless something weird happens with the internal search mechanism). Alternatively, I answered a Stack Overflow question with a similar use case, using ExecuteScript to check the JSON (and in their case, replace the value from an external file). That example also reads the file every time, but you could use a similar approach with InvokeScriptedProcessor to read the file in the initialize() method, then it will not be re-read during onTrigger (which is called when the processor is scheduled).
... View more
11-03-2016
06:25 PM
You could try Presto, that should let you hook up to multiple DBs. Then you can set up a DBCPConnectionPool in NiFi to connect to Presto and issue the query from there. Also perhaps Apache Calcite has this functionality but I'm not sure.
... View more
11-03-2016
03:08 PM
1 Kudo
How do you know which delimiter is used for a particular file? If you can determine that from the content, you might be able to use RouteContent to send all \u0001-delimited files to one ConvertCSVToAvro (using the technique I describe above), all \u0002 files to another, and so on. Likewise if you can somehow extract the delimiter into an attribute you can use RouteOnAttribute rather than RouteContent. Why would you like to avoid ReplaceText? The content of the flow files will be altered when converting to Avro, so you won't have the original input at that point. If it is a performance issue, do you think my suggestion above would work for your use case?
... View more
11-02-2016
07:07 PM
1 Kudo
On a Mac, I found a useful procedure here for enabling the pasting of a Unicode character into the current text box. Using this, I opened the ConvertCSVToAvro processor dialog, then the CSV Delimiter property value dialog. Then using the procedure I selected character \u0001, which pastes it into the property (although it is not a visible character so you won't see it on the screen). Click OK then Apply and the delimiter should be set to \u0001. I tried this with a simple example and it worked. On Windows I think you can use the Character Map or something similar, but the idea is to either have some utility copy a unicode character to the clipboard for pasting into the property value dialog, or perhaps it will paste for you (like the Mac utility). Once NIFI-2369 is resolved, there might be a way to use Expression Language to make this more visible, like ${literal('\u0001')} or something. Alternatively, you could use a scripting processor like ExecuteScript and do the split with code (Javascript, Groovy, e.g.)
... View more
11-02-2016
05:34 PM
1 Kudo
In your snippet above, the property set to "KERBEROS" is "hive.server2.authentication", not "hadoop.security.authentication". If "hadoop.security.authentication" is set to "kerberos" in your core-site.xml, ensure the path to your core-site.xml is in the Hive Configuration Resources property. That property accepts a comma-separated list of files, so you can include your hive-site.xml (as you've done in your above screenshot) as well as the core-site.xml file (which has the aforementioned property set).
... View more