Member since
09-29-2015
871
Posts
723
Kudos Received
255
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 4261 | 12-03-2018 02:26 PM | |
| 3203 | 10-16-2018 01:37 PM | |
| 4308 | 10-03-2018 06:34 PM | |
| 3165 | 09-05-2018 07:44 PM | |
| 2425 | 09-05-2018 07:31 PM |
11-08-2016
02:32 PM
What you described is the correct approach, you need to delete flow.xml.gz, users.xml, and authorizations.xml, you also need to make sure authorizers.xml is the same on all nodes. You can see from your log that it was authorizations that were the problem "Proposed Authorizations do not match current Authorizations". You can either copy authorizers.xml from the good node to the others to ensure it is the same, or you can make sure the other nodes have no initial admin and no legacy authorized users file so that they will inherit the authorizations/users from the cluster.
... View more
11-07-2016
06:52 PM
1 Kudo
If it failed to write the data to HDFS it would transfer the flow file to the failed relationship, which you route back to itself to keep re-trying, or route to whatever you want to do to handle the error. If the NiFi node crashed while it was in process of writing the data, but hadn't committed the session in the processor, then when the node restarts that flow file will still be in the queue before the processor and will be tried again. There should not be data loss.
... View more
11-02-2016
04:11 PM
2 Kudos
In addition to what Tim Spann said, you can also implement a custom ReportingTask to send metrics to any system you like. See the AmbariReportingTask for reference: https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-ambari-bundle/nifi-ambari-reporting-task/src/main/java/org/apache/nifi/reporting/ambari/AmbariReportingTask.java
... View more
11-01-2016
10:52 PM
When you choose "Merge Format" of "Avro" it is not doing binary concatenation. It is merging all of the Avro files into a new valid Avro file with single header/schema entry. What you described would be "Merge Format" of "Binary Concatenation". You can see the possible values for "Merge Format" here: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MergeContent/index.html There are also unit tests that show it merging Avro records and then parsing the resulting Avro: https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestMergeContent.java#L70
... View more
11-01-2016
01:12 PM
1 Kudo
The MergeContent processor has a merge type of Avro which will merge together Avro messages that have the same schema. If you are sending in Avro messages with different schemas you will want to use the Correlation Attribute property to only merge messages of the same schema.
... View more
11-01-2016
01:08 PM
2 Kudos
If you want to restrict which users can see parts of the flow then you need security enabled. If you don't care who can see what, then it is fine to have many people working on the same canvas. Teams usually create process groups to contain their part of the flow.
... View more
10-31-2016
01:03 PM
1 Kudo
In general you tell NiFi about HBase by copying /etc/hbase/conf/hbase-site.xml and /etc/hadoop/conf/core-site.xml from the server where NiFi is running, and pointing to these files in the HBaseClientService in NiFi (Configuration Resources Property). In your case you will also have to ensure all of the ports and hostnames that are referenced are accessible from your HDF sandbox to your HDP sandbox. It would probably be a lot easier to just run HDF on your HDP sandbox to not deal with networking related issues.
... View more
10-28-2016
02:09 AM
1 Kudo
The accepted answer here shows how to connect PublishKafka from HDF 2.0 to a Kerberized Kafka: https://community.hortonworks.com/questions/63180/error-in-nifi-flow.html
... View more
10-27-2016
06:43 PM
2 Kudos
You could use Apache NiFi with a ListHDFS + FetchHDFS processor followed by PublishKafka.
... View more
10-27-2016
01:31 PM
4 Kudos
The problem is a whitespace issue... If you look at your log you can see "CN=BrowserClient, OU=GUI, O="Test Inc "" with a space before the OU. If you look at the Initial Admin and users.xml there is no space "CN=Browser Client,OU=GUI,O="Test Inc " so its not lining up. You can fix this by stopping NiFi, updating authorizers.xml so the Initial Admin has the exact value from the logs, and then deleting users.xml and authorizations.xml, and restart.
... View more