Member since
09-29-2015
871
Posts
723
Kudos Received
255
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 4309 | 12-03-2018 02:26 PM | |
| 3249 | 10-16-2018 01:37 PM | |
| 4344 | 10-03-2018 06:34 PM | |
| 3210 | 09-05-2018 07:44 PM | |
| 2448 | 09-05-2018 07:31 PM |
09-19-2017
01:01 PM
1 Kudo
The reader on the SAM side is trying to read the encoded schema reference, but it is likely not there. The AvroRecordSetWriter being used by PublishKafkaRecord_0_10 must be configured with a "Schema Write Strategy" of "Hortonworks Content Encoded Schema Reference".
... View more
09-19-2017
12:56 PM
1 Kudo
Change Correlation Attribute Name to "filename" instead of "${filename}".
... View more
09-19-2017
12:44 PM
1 Kudo
You seem to have a good understanding of the situation... There isn't really a big pro or con with NiFI's authorizer vs Ranger... If you are already using Ranger for other things then it gives you a nice central place to manage all policies across your enterprise and a nice way to look at audit logs. The current downside would be that when using an external authorizer like Ranger, NiFi's UI currently doesn't show anything about the policies, so you have to jump back and forth a bit between two UIs sometimes. This is only temporary though and in a future release there are plans to offer a read-only view of policies on the NiFi side when using an external authorizer. Sometimes it is helpful to start out with NiFi's authorizer to get an understanding of the policies before attempting to set them up in Ranger. Regarding the RPG pattern, you are correct that this is one unfortunate scenario regarding multi-tenancy... I think the real way that this should be addressed is by creating a more user-friendly way to do load balancing of a connection across the cluster, most likely a special way to connect two components and indicate the connection should transfer data to each node across the cluster. The RPG approach is used simply because that is all that exists, but site-to-site was really meant for transfer between two separate instances. An alternative option is always to have each team with its own NiFi instance and then they can manage the root canvas themselves, and they can send data across teams via site-to-site, but I realize this requires managing more clusters, just throwing it out there.
... View more
09-15-2017
04:57 PM
2 Kudos
I think you might be able to do this with LookupAttribute and SimpleCsvFileLookupService, but you might need to do two separate look ups... With the SimpleCsvFileLookupService you provide it with a path to a CSV file and then a key column and a value column, and it then loads those two columns into a lookup map. So you would probably need two instances of this service, one where the lookup column was "key" and value column was "col1" and then the second instance where the look up column was "key" and the value column was "col2". With that setup you could then have two LookupAttribute processors in a row where each one used one of the loop up services above. LookupRecord would be used if you have record-oriented data in your flow file (csv, json, avro) and you want to enrich each record based on a lookup. I'm assuming you want flow file attributes based on your description, which would be LookupAttribute.
... View more
09-15-2017
04:47 PM
2 Kudos
Currently this isn't something that is configurable... processors that are restricted are marked with a @Restricted annotation in the source code, and during start up of the application, the framework identifies processors with the annotation and then provides the restrictions based on the presence of the annotation.
... View more
09-08-2017
08:12 PM
Something is not set up correctly because the log is showing alvin@NIFI.COM in some places and alvin@NIFI.COM in other places. What are you entering as the username when you login? What is entered for the Initial Admin in authorizers.xml?
... View more
09-08-2017
06:41 PM
1 Kudo
If you are setting up authentication for users accessing NiFi's UI, then you only need the spnego properties as shown in this post. If you need NiFi to authenticate to other services, for example to talk Ranger when Ranger is kerberized, then you need the service principal and keytab.
... View more
09-01-2017
01:47 PM
You may want to take a look at the record related processors. There is a processor called PartitionRecord that works kind of like RouteText, but a lot nicer, and you could send in a big set of JSON documents in, and have it partition the records based on a field in each record (i.e. eventType). From there you can use MergeContent with a Correlation Attribute of eventType in order to merge together the appropriate number of events for each type, and then PutHDFS with a Directory of something like /data/${eventType}, this way you need only one MergeContent and one PutHDFS to handle all of the event types. The unfortunate thing is that PartitionRecord may not work for your data since technically JSON documents separated by new lines is not a valid JSON document in itself, but it might be worth a try... If you have control of the upstream data you could always make it a valid array of JSON documents. If you can't do any of that then I think RouteText is the best option... After RouteText you could have each route go to an UpdateAttribute processor that adds an attribute called eventType with the corresponding type, then send them all to a MergeContent and PutHDFS using the approach I described above with Correlation Attribute and the dynamic HDFS directory.
... View more
08-31-2017
05:03 PM
I believe Chrome 39 is too old and not really a supported browser, the latest Chrome is up to version 60. https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#browser-support The version of Java should not matter.
... View more
08-31-2017
04:36 PM
What browser are you using and are there are any errors in browser when this happens (may need to open specific dev console to see them)?
... View more