Member since
01-07-2019
220
Posts
23
Kudos Received
30
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
4856 | 08-19-2021 05:45 AM | |
1781 | 08-04-2021 05:59 AM | |
863 | 07-22-2021 08:09 AM | |
3595 | 07-22-2021 08:01 AM | |
3295 | 07-22-2021 07:32 AM |
02-12-2021
01:44 AM
1 Kudo
Looks like the files are available here: https://repo.hortonworks.com/content/repositories/releases/org/apache/nifi/nifi-hive-nar/
... View more
02-04-2021
01:57 AM
Hello Dennis, thank you for the reply. It really helped a lot! Q1: That worked very well with the updateAttrbute processor. Q2: This also worked. I had the the settings of the csvWriter service (UpdateRecord) messed up. But it works fine. Q3: That is a bummer, I hoped it will would be a piece of cake to implement that. But i will look into one of the mentioned tools and figure it out. Q4: True that. The file is going to explode with data.
... View more
02-03-2021
08:22 AM
Apologies to disturb this old thread, but it seems people are still landing on this via search: The discussion here is outdated, especially the exclusions around gateway nodes. Please contact your Cloudera Representative for the latest terms an conditions.
... View more
02-02-2021
08:26 AM
@peter_coppens While the HTTP spec may be case insensitive, NiFi Expression Language (EL) is case sensitive. Are we talking about an undefined number of unique header case sensitivity here? I would think you have a short defined list of possible header values here, correct? There are multiple EL functions you can use here: 1: ${http.headers.x-My-Header:isNull():ifElse('${http.headers.x-my-header}','${http.headers.x-My-Header}'):......} In above if subject "http.headers.x-My-Header" does not exist on the FlowFile it returns the value from "http.headers.x-my-header" attribute instead; otherwise it returns value from "http.headers.x-My-Header". Then you can continue the EL statement to do whatever additional manipulation needed. 2: ${allMatchingAttributes('(?i)http\.headers\.x-my-header'):join(''):....} In above it uses a case insensitive Java regular expression within the "AllMatchingAttribtes( )" EL function to return values from all FlowFile attributes that satisfy that regex and then use the "join( )" function to merge them all together. Since we expect that only one attribute will actually match, the result is just a single value. This may be best option. 3: ${http.headers.x-My-Header:replaceNull(${http.header.X-my-header:replaceNull(${http.headers.x-my-header})}):....} In above we chain multiple "replaceNull( )" functions to iterate through all case permutations till we find one that exist as an attribute on the FlowFile. Note: In any of the above you would replace that last ":...." with the rest of your EL statement fucntions needed to manipulate the subject returned as you need. Options 1 and 3 require you to know all case permutations that may exist. Option 2 may be best choice as you do not need to know all the permutations, but can be an issue if for some reason you have multiple permutations of the same attribute on the same FlowFile. Hope this helps, Matt
... View more
02-02-2021
07:02 AM
@kk_nifi This is more likely an issue in the specific Java 11 version your are using. https://bugs.openjdk.java.net/browse/JDK-8243541 Somethings you may want to try: 1. Try a different update version of Java 11 2. add the following line to the NiFi bootstrap.conf file: java.arg.<some unused number>=-Djava.locale.providers=COMPAT Note: A restart of NiFi is needed with any config file change except logback.xml. Hope this helps, Matt
... View more
02-02-2021
06:41 AM
1 Kudo
@Arash In your 4 node NiFi cluster, what value do you have set in the "nifi.remote.input.host" property in the nifi.properties file for each of the 4 nodes? It should be the FQDN for each node and not be the same value on all 4 nodes. Form the host where MiNiFi is running, can all 4 of those FQDNs be resolved and reachable over the network? If not, MiNiFI RPG is only going to be able to send successfully to one FQDN it can reach. When the RPG is started it reaches out to the URL configured in the RPG to obtain S2S details from the target host. That target host collects the host details for all currently connected nodes in the cluster and communicates that back to the client (MiNiFi). If all 4 nodes report the same configured FQDN in the "nifi.remote.input.host" property, then client only knows of one FQDN to which it can send FlowFiles over Site-To-Site (S2S). To improve redundancy in the RPG, you can provide a comma separated list of URLS in the RPG configuration so if any one node is down, the RPG can try fetch S2S details from the next host in the comma separated list. Hope this helps, Matt
... View more
02-01-2021
06:32 AM
This can likely not be resolved without much more detail, so it may be good to log a support ticket for this. The first thing that comes to mind: Are you able to use the rest API at all, does your request reach the node or is there possibly a problem in getting the request there (Firewall, hostname, typo).
... View more
02-01-2021
06:27 AM
9 out of 10 times this message is caused because you run the GetHDFS on multiple nodes. Both nodes see it, perhaps even try to pick it up, but clearly not both of these can delete it. In old versions of NiFi you can fix this by setting the GetHDFS to run only on the primary node. However, that will ofcourse burden the primary node more than it should. So in recent versions (and likely yours) you will find the ListHDFS and FetchHDFS processors (and similar sets for different data sources). The lightweight List processor can then run on the primary node, and loadbalance to all nodes which will then Fetch.
... View more
02-01-2021
06:24 AM
As you can see the Mergecontent processor has several parameters for ensuring messages with a commonality get merged. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.6.0/org.apache.nifi.processors.standard.MergeContent/index.html This could be used in the way that you let each of the two processors give the messages an attribute, and only merge things with the same attribute. You appear to be looking for the opposite. I am not aware of any such parameter, and am also not sure if this is in the spirit of what mergecontent was designed for. Perhaps it is time to step back and see if mergecontent (or even NiFi) is the best tool for this specific job.
... View more
02-01-2021
06:08 AM
The first thing I would recommend, is attempting to avoid a custom processor. If you need to add columns to a record, the UpdateRecord processor should enable this. (At first I expected the QueryRecord processor to do it as well, but am unable to find any references to this). https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.UpdateRecord/index.html https://community.cloudera.com/t5/Support-Questions/Adding-columns-to-sql-in-nifi/td-p/224598
... View more