Member since
03-20-2017
6
Posts
2
Kudos Received
0
Solutions
08-25-2017
10:05 AM
Hello, So I'm trying to create a flow wherein one flow will export data from a table to an S3 bucket in avro format and the other flow will import that data into a table. It would be best if we could do an UPSERT, however as there are many tables, was trying to keep the flow as dynamic and small as possible. The schema exists in the avro registry thus the PutDatabaseRecord works perfectly without writing any sql query. Tried creating a flow wherein we could separate each avro record into 1 record and if INSERT fails then update, but with over a million records, that is extremely slow. The last option we have is to TRUNCATE and re-insert all the data, but those get separated into 2 flows, truncate in a separate flow and then insert in a different flow. Has anyone else run into this issue? Would like to know if its possible to combine truncate and insert into one transaction so we are sure there is always data available. Secondly, what is the best way to do an UPSERT if any? The data source doesn't have a timestamp hence it's hard to check what really was created/updated hence the upsert reasoning and the source tables cannot be changed.
... View more
04-19-2017
10:23 AM
When creating controller services, it was noticed that there are 2 mentions of it in the UI. One is within the root canvas wherein you can click on the cog icon and can reach controller settings from there. The other way is to click on the list icon, on the right of the 'log out' text, and controller settings can be accessed from there. The question is, whats the difference between the two? Creating it in A does not show up in B and vice versa. Are there specific use cases for this differentiation? What is the best way to implement controller services if we want to make it available to all process groups only to read and connect to those sources but not modify them.
... View more
Labels:
03-28-2017
07:30 AM
2 Kudos
Hello, We have created a custom variable registry file with a list of items. The file contains credentials to data sources. Is there any possible way to encrypt this? I've checked the nifi-toolkit and when running the help command, it shows values within that file can be encrypted. Is this possible for a custom variable registry file?
... View more
03-23-2017
10:55 AM
In the nifi.variable.registry.properties we can configure a file to list all the variables we want to have available, however is it possible to have variables limited per group? Lets say we have GroupA and Group B. We have var1, var2, var3, var4, and var5. We want to provide GroupA with access to all 5 variables, however to GroupB we would only like them to access var4 and var5. Is there a way this can be implemented in any way possible?
... View more
Labels:
03-21-2017
03:13 AM
Hello @Pierre Villard Thank you for the answer. Definitely helps.
... View more
03-20-2017
07:34 AM
LDAP has been successfully integrated with Apache NiFi 1.1.2, however the main question is, how do we specify permissions based on groups rather than users? Setting Initial Admin Identity to as : cn=userA,ou=xyz,dc=xyz,dc=xyz Lets say there is groupA (posix group) and groupB (normal group) in the LDAP Directory and userA and userB. userA is the default admin so it already has access to NiFi.
How do we provide access to userB based on groups rather than adding the user manually in NiFi first? Created groupA and groupB in NiFi and added all policies necessary. 1) When NiFi checks in LDAP, does it validate against the posix group in LDAP or just the normal group? 2) Tried using both USE_DN and USE_USERNAME in the Identity Strategy but it still says no permissions for userB. 3) Added userB within NiFi and linked it to the above NiFi groups and now login to NiFi works with the password available within LDAP. How can we configure NiFi to allow different permissions to different LDAP Groups and without adding the users within LDAP into NiFi ?
... View more
Labels: