Member since
07-30-2019
920
Posts
196
Kudos Received
91
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1081 | 10-05-2021 01:53 PM | |
14655 | 09-23-2019 06:03 AM | |
5698 | 05-04-2019 08:42 PM | |
1184 | 06-11-2018 12:45 PM | |
10677 | 06-04-2018 01:11 PM |
04-10-2017
04:20 PM
1 Kudo
@Sherif Eldeeb Have
you looked at using the JoltTransformJson processor? You could use it
with a Jolt specification and add the additional properties at the same
time. Also, the Jolt specification can keep the format of the
variables.
... View more
04-08-2017
10:30 PM
@Sherif Eldeeb What are you trying to do with your flow?
... View more
04-08-2017
04:39 PM
2 Kudos
@Sherif Eldeeb I am not aware of any automatic way for NiFi to convert all of the Json content into one for one attributes. I will mention you should be cautious when doing this depending on the the size of your Json files and the volume of data in your flow. Having a large number of flow files with very large attributes will require a large size JVM for NiFi to run smoothly. NiFi keeps all of the attributes of the flow files in the graph in the JVM memory configured in the bootstrap.conf file.
... View more
04-06-2017
10:25 PM
@Harish S Have you tried using the JoltTransformJSON processor? You should be able create a Jolt specification to extract the information you want.
... View more
03-08-2017
02:16 PM
Good to hear.
... View more
03-07-2017
05:20 PM
1 Kudo
There should be more information in the nifi-app.log file.
Is kerberos enabled on the HDFS component?
... View more
02-16-2017
05:08 PM
2 Kudos
Try this with a ReplaceText processor: Here is the complete Search Value property: ((?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)).((?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)).((?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)).((?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)) @anshuman Ghosh
... View more
02-02-2017
04:34 PM
10 Kudos
Assumptions The following assumptions are made. The openldap clients and server are already installed. The basic setup of the ldap server has been completed and users "nifi admin", "nifi user1" and "nifi user2" are in the ldap database. LDAPS System Configuration The example below is being configured on system
nifi-sme-20. The CA
certificate being used, aka truststore, is called, all-trusted.jks and the server
certificate, aka keystore, is called nifi-sme-20.cert.pfx. There is also the encrypted private key for
the server, nifi-sme-20.key.enc.pem which is needed for the configuration of the ldaps service. List the current
certificates in the database, the default location is /etc/openldap/certs, using the following command: certutil -d /etc/openldap/certs/ -L Output will look like the following If your CA is in pem format, then it can be imported into the NSS database. If you have a CA that is in jks format, it first must be converted before it can be imported. Converting it can be done in two steps: keytool -importkeystore -srckeystore all-trusted.jks -destkeystore all-trusted.p12 -deststoretype PKCS12
openssl pkcs12 -in all-trusted.p12 -out all-trusted.pem Now the truststore can be imported into the database certutil -d /etc/openldap/certs/ -A -n "CAcert" -t CT,, -a -i /opt/configuration-resources/certs/all-trusted.pem This command adds a CA certificate stored in the PEM (ASCII) formatted
file named /opt/configuration-resources/certs/all-trusted.pem, the -t CT,, means that the certificate is
trusted to be a CA issuing certs for use in TLS clients and servers. To verify the CA has been imported use the certutil command from above to list Now import the server certificate in the database. certutil -d /etc/openldap/certs/ -A -n "nifi-sme-20" -t u,u,u -a -i /opt/configuration-resources/certs/nifi-sme-20-cert.cer This command adds the server certificate, the -t u,u,u, means the certificate can be used for authentication or signing. Now list the contents in the database, you see the following Next update the slapd service to use the CA and server certificate. This is done by updating the /etc/openldap/slapd.d/cn=config.ldif file. The file cannot be edited manually, you have to update the file by using the ldapmodify command. One way to do this is create your own file.ldif file with the updates needed and then use this file as a parameter on the ldapmodify command. For this article, I created a file called tls-enable.ldif, here is a copy of the file: TLSCertificateFile this directive specifies the file that contains the slapd server certificate TLSCACertificateFile this directive specifies the PEM-format file containing certificates for the CA's that slapd will trust TLSCertificateKeyFile this directive specifies the file that contains the private key that matches the certificate stored in the TLSCertificateFile file. Note: To use the private key, we need to decrypt it, this can done with the following command: openssl rsa -in nifi-sme-20.key.enc.pem -out nifi-sme-20.key.pem The command used to update the cn=config.ldif is: ldapmodify -Y EXTERNAL -H ldapi:/// -f tls-enable.ldif Now restart the slapd service systemctl status slapd Now verify that you are able to connect to the slapd service, run the following command: openssl s_client -connect nifi-sme-20:636 -debug -state -CAfile /opt/configuration-resources/certs/all-trusted.pem If the commands works, output similar to this is displayed This will put you into a shell , which you can use control-c to exit. In addition, if you check the status of the slapd service, you will also see the connection from the above command LDAPS NiFi Configuration Now that you have successfully configured the slapd service,
there are a few steps to setup NiFi to use LDAPS. First, configure NiFi to perform user
authentication over HTTPS, the following sections in the nifi.properties file section
need to be completed. Again for this example, the configuration is being done on system nifi-sme-20. Make sure to set the web section to use https host and port In addition, fill in the security section with the keystore and truststore. In this example, I use the same CA certificate in the nifi.properties as the ldaps service, but it isn't a requirement for it to work with NiFi. The CA used in the configuration of the login-identity-provider.xml has to be the same as the one used in the configuration of the ldaps service. Notice also that the nifi.security.user.login.identity.provider is set to ldap-provider. Now edit the login-identity-provider.xml file and add the keystore, truststore and all of the other TLS properties. Once you set the authentication strategy to LDAPS, all of the other properties are required to have some value. Inside the file is short explanation of each property and the possible values. If this is the first time to secure the NiFi instance, the last step is to set the initial admin identity in the authorizers.xml file. Now restart/start NiFi. This is what you will see when you go to the NiFi UI in the browser: And there you go, you have successfully configured NiFi to use LDAPS.
... View more
Labels:
11-03-2016
03:28 PM
2 Kudos
@Anwaar Siddiqui Anwaar, The myid file value has to match the list of zookeeper servers in the order they are listed in the state-management.xml file. For example in the state-management file, we list the zookeeper servers <property name="Connect String">daves-nifi-cluster-2:2181,daves-nifi-cluster-3:2181,daves-nifi-cluster-4:2181</property> so on server daves-nifi-cluster-2 the value in the myid file on that server would be 1 so on server daves-nifi-cluster-3 the value in the myid file on that server would be 2 so on server daves-nifi-cluster-4 the value in the myid file on that server would be 3 In HDF 2.0/NiFi 1.0 you need to create a state/zookeeper directory on the same level as the conf, not inside of conf, directory, make sure you put the myid file in that state/zookeeper directory
... View more
11-01-2016
02:10 PM
1 Kudo
The easiest way is to stop NiFi, delete all files in the content and flowfile repositories, then restart NiFi.
... View more