Member since
07-30-2019
155
Posts
106
Kudos Received
33
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2954 | 04-18-2019 08:27 PM | |
951 | 12-31-2018 07:36 PM | |
1839 | 12-03-2018 06:47 PM | |
652 | 06-02-2018 02:35 AM | |
1692 | 06-02-2018 01:30 AM |
08-14-2017
06:26 PM
2 Kudos
Alvin, The encrypted content repository feature is actively being worked on. As a rule, we cannot make claims about the delivery dates or versions of active development features. Hope this helps. If you have compliance requirements that sensitive data (such as PII, PCI/payment details, EPHI, etc.) is never stored on disk in plaintext, you can explore using the volatile content repository, but be aware there is the risk of data loss in the event of power failure, and this applies to all content objects, not just the sensitive records. With Apache NiFi 1.3.0, you can also use the RecordReader and RecordSetWriter approach -- while the EncryptedRecordReader and EncryptedRecordSetWriter controller services are not yet available, you could use a custom ScriptedRecordReader and ScriptedRecordSetWriter to decrypt and re-encrypt on the fly. The intermediate "record" object is never persisted to disk, so at no point would the plaintext data be written outside of volatile memory regardless of the content repository implementation.
... View more
08-11-2017
06:54 PM
Hi DH. As of Apache NiFi 1.2.0, only TLSv1.2 is supported for incoming connections [Release Notes - Security]. This is independent of the version of OpenSSL installed, and unrelated to the selection of TLS/SSL protocol in StandardSSLContextService (see discussion on PR 1986 about removing invalid protocol version options from the dropdown because they are not supported). However, because your client should be the same instance as your server (the SiteToSite reporting task is pointing back to itself, correct?) the same protocol versions and cipher suites should be available to both roles. Can you do a remote debugging session and trace the outgoing connection to determine what protocol versions and cipher suites are available to the SSLContext object at that time?
... View more
08-08-2017
12:19 AM
2 Kudos
What issue do you get when using
openssl s_client ? You should definitely be able to connect using s_client, although you may need to provide -CAfile option to trust the issued server certificate and -cert and -key options to provide a client certificate as below:
openssl s_client -connect nifi.nifi.apache.org:9443 -debug -state -CAfile conf/nifi-cert.pem -cert conf/client.pem -key conf/client.key
As for the handshake_failure issue, this may be because:
The certificate has expired or is invalid
The provided keystore or key password is incorrect and the controller service cannot access the keystore
The server and client cannot agree on a suitable cipher suite during negotiation
Diagnosing with s_client is definitely the correct tool. You can also enable TLS debugging via the bootstrap.conf file with the line:
java.arg.15=-Djavax.net.debug=ssl,handshake
This output will be in logs/nifi-bootstrap.log .
... View more
06-29-2017
06:32 PM
The --subjectAlternativeNames flag can be used to pass SAN values to the certificate generation process.
--subjectAlternativeNames <arg> Comma-separated list of domains to use as Subject Alternative Names in the certificate
... View more
05-17-2017
07:16 PM
What version of Apache NiFi are you using? That feature was added in version 1.2.0 (see NIFI-2908).
... View more
05-09-2017
06:49 PM
2 Kudos
Eric, The public certificate they provided you is not sufficient to secure your system. You (eventually) need a Java Keystore (*.jks) file that contains the PrivateKeyEntry -- the private key that corresponds to the public key you currently have contained in a certificate. There are many guides on the internet for building a keystore from the private key (you'll have to go through an intermediate step of importing it into a PKCS12 keystore first then converting that to JKS). Once you have done this, you configure nifi.properties with the keystore path, password, and type. You will need to either obtain the private key from your DevOps team, or generate your own and send them a Certificate Signing Request (CSR) which allows you to keep the private key on your system and not expose it to anyone else during the signing process. Once you have the private key, follow the instructions linked above to build the keystore.
... View more
04-19-2017
05:38 PM
As noted in the REST API, clusterNodeId is a required field in the GET /provenance-events/{id} request when running in clustered mode. Use the Developer Tools of your browser to watch the requests that the UI fires when you get the details of a provenance event and you should see where to include the clusterNodeId in the request (my instance is not running in a cluster so it is not present in my request).
... View more
04-03-2017
06:27 PM
As @Hellmar Becker noted, SplitContent allows you to split on arbitrary byte sequences, but if you are looking for a specific word, SplitText will also achieve what you want. You may also want to look at RouteText , which allows you to apply a literal or regular expression to every line in the flowfile content and route each individually based on their matching results.
Finally, if you only care about the occurrence count of a specific word or sequence in the flowfile, you could use a small script in ExecuteScript or even ExecuteStreamCommand and use a terminal command like $ tr ' ' '\n' < FILE | grep WORD | wc -l (from here).
... View more
04-03-2017
06:19 PM
Can you clarify what you mean by "the ListenSyslog processor is unable to process the flow pattern"? The Syslog pattern is defined in RFC 5424 and 3164, and the ListenSyslog processor conforms to that pattern. If the incoming messages are not in a valid syntax or there is a bug in the ListenSyslog processor, please report that, but you can also use the data massaging capabilities of NiFi to transform that data into parsable values, either with ReplaceText or ExecuteScript.
... View more
04-03-2017
05:00 PM
1 Kudo
The error is saying that one node in the cluster is attempting to make a heartbeat connection to the other node, but it is not providing a valid client certificate in order to authenticate itself during the TLS handshake negotiation. There are a few possible reasons for this error:
The node is not sending the client certificate. Ensure that nifi.security.needClientAuth=true and nifi.cluster.protocol.is.secure=true are present in your nifi.properties file.
The truststore on the receiving node does not contain the public key certificate of the connecting node. When you followed the instructions from that link, how did you generate the respective certificates? Using the Apache NiFi TLS Toolkit as described by Pierre should ensure that all node certificates are signed by the same CA and that the CA is imported into the common truststore. If you manually generated your certificates, ensure that they are trusted on each node (you can do this with OpenSSL's s_client tool).
... View more
04-03-2017
04:50 PM
You need to tell s_client that a self-signed certificate is ok by providing the -CAfile flag and the path to the CA's certificate, exported as PEM.
$ openssl s_client -connect HKLPATHAS03.example.com:9090 -CAfile /opt/certs/ca.pem
... View more
03-23-2017
06:53 PM
You can use curl to make a REST API call to your NiFi instance to start the workflow. This does not require opening the UI and can be scripted/automated. You can also use MiNiFi as a headless (no UI) solution.
... View more
03-23-2017
06:51 PM
2 Kudos
Currently, NiFi supports encrypting/decrypting data through the EncryptContent processor, but the pre/post state of the data would still be stored in plaintext in the content repository. In general, transparent disk encryption/OS-level data encryption is recommended in conjunction with strict OS-level/POSIX access controls. There is a current effort to provide encrypted implementations of the flowfile (attribute), content, and provenance repositories. As Dan mentioned, a combination of encrypted payload and plaintext metadata for routing can work very well if the payload does not need to be processed/transformed inside NiFi.
... View more
03-22-2017
10:31 PM
Any reason why Ambari still enables SSLv2 and SSLv3 by default? Those were considered insecure 7+ years ago. Since Feb 2014, all modern browsers have supported TLSv1.2.
... View more
03-22-2017
06:42 PM
1 Kudo
If you really want to modify the colors you can play around with the values in graph.css.
... View more
03-21-2017
02:48 AM
1 Kudo
This might be because your Linux sandbox does not have current/any CA public certificates with which to authenticate the Twitter server certificate. You can find instructions for updating the default CA certificates for CentOS (the base OS of the sandbox) online (basically just
$ yum install ca-certificates && update-ca-trust force-enable ), and more information on adding custom CA certs if necessary
here.
... View more
03-08-2017
11:17 PM
1 Kudo
A default login page for an admin user served over HTTP provides the illusion of security -- security theater -- but does absolutely nothing to improve the security of the system while adding obstacles to ease of use.
... View more
02-27-2017
06:22 PM
1 Kudo
You can achieve this by using the
HashContent processor or the ExecuteStreamCommand processor to calculate the MD5 hash of the content and compare it to the contents of the prepared *.md5 digest file, which you can read with GetFile and then ExtractText/RouteOnContent/RouteText to compare the MD5 value in a flowfile attribute with the contents of the file.
... View more
02-20-2017
09:51 PM
3 Kudos
Arun,
Edit:
Sorry, I got this confused with another question I was answering at the same time. To access dynamic properties in the script body, just reference them as a variable name.
From
Matt Burgess' blog:
Dynamic Properties: Any dynamic properties defined in ExecuteScript are passed to the script engine as variables set to the string value of the property values. This means you must be aware of the variable naming properties for the chosen script engine. For example, Groovy does not allow periods (.) in variable names, so don't use something like "my.property" as a dynamic property name.
Here is the example output from a LogAttribute processor following an ExecuteScript processor where I update the flowfile by adding an attribute with the content of a dynamic property on the ExecuteScript processor.
Processor config:
Groovy script (note casting
dynamic to String because by default it is of type org.apache.nifi.attribute.expression.language.StandardPropertyValue 😞
def flowfile = session.get()
if (!flowfile) return
log.info("Dynamic property value: ${dynamic as String}")
flowfile = session.putAttribute(flowfile, "dynamic", dynamic as String)
session.transfer(flowfile, REL_SUCCESS)
2017-02-20 17:22:04,072 INFO [Timer-Driven Process Thread-3] o.a.n.processors.standard.LogAttribute LogAttribute[id=5e4107e1-015a-1000-5efa-735bf688a3d0] logging for flow file StandardFlowFileRecord[uuid=4edb12f2-a020-4c3e-b42b-0796b8e91031,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1487639391782-1, container=default, section=1], offset=725, length=29],offset=0,name=420715910236852,size=29]
--------------------------------------------------
Standard FlowFile Attributes
Key: 'entryDate'
Value: 'Mon Feb 20 17:22:04 PST 2017'
Key: 'lineageStartDate'
Value: 'Mon Feb 20 17:22:04 PST 2017'
Key: 'fileSize'
Value: '29'
FlowFile Attribute Map Content
Key: 'dynamic'
Value: 'This is the value of a dynamic processor property set by Andy.'
Key: 'filename'
Value: '420715910236852'
Key: 'path'
Value: './'
Key: 'uuid'
Value: '4edb12f2-a020-4c3e-b42b-0796b8e91031'
--------------------------------------------------
This is the flowfile content.
Original answer (referencing attributes):
You extract them from the
flowfile object and write them to the flowfile via the session object. flowfile.getAttribute(attribute_name) returns the attribute_value , and session.putAttribute(flowfile, attribute_name, attribute_value) returns the new flowfile instance.
Example:
flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename').split('.')[0]+'_translated.json')
... View more
02-14-2017
06:14 AM
In general you need to populate the
truststore of the StandardSSLContextService to interact with remote HTTPS endpoints. Populating the keystore provides the certificates used to identify the NiFi service and the private keys used to sign (and decrypt) data by NiFi. Configuring the truststore (and pointing it to cacerts as you did) allows NiFi to verify the public key certificate chain presented by the remote endpoint. Can you try populating the truststore values and re-try your flow? I am not sure why this would work natively on your OS X machine, as there is an existing open Jira to provide this "out-of-the-box" functionality (and it is noted there that InvokeHTTP works without custom StandardSSLContextService creation while GetHTTP requires it).
... View more
01-27-2017
07:42 PM
When you set the Initial Admin Identity, NiFi does populate the roles for that user, including "view the UI", which allows you to then configure other users via the interface. Obviously there was an issue with generating your authorizations.xml, so that policy doesn't exist at this time. If you provide the authorizations.xml, we can fix it.
... View more
01-27-2017
07:29 PM
1 Kudo
<role> elements are from NiFi 0.x . As you are on 1.x , you need to populate the authorizations.xml file with the appropriate mappings of the user (identified by the UUID) to the specific policies which will grant them access to perform the desired behavior. If you paste the contents of your authorizations.xml file here, we can correct any issues. Specifically, to view the UI, your user needs READ on the "view the UI" policy.
... View more
11-28-2016
07:19 PM
Glad it helped, Roger. I just updated the answer with a link to the list of current MiNiFi Java processors.
... View more
11-28-2016
06:43 PM
1 Kudo
1. Currently, MiNiFi C++ can use TailFile/GetFile or ListenSyslog processors to ingest data. In the future, more IoT-focused processors will be made available to communicate directly using the protocol specified by the edge device. At this time, MiNiFi Java has access to more processors because of the common development platform. 2. @slachterman's answer is very good. 3. I describe some of the interaction and use cases for NiFi and MiNiFi here.
... View more
11-23-2016
04:06 AM
You may be able to run the TLS Toolkit command on Windows as follows: tls-toolkit.bat ""standalone" -n "localhost" -C "CN=nifistos, OU=mycompany" -o "./target""
... View more
11-23-2016
03:33 AM
Your truststore has two
trustedCertEntries -- one for the server cert and one for the client. I don't know what browser you are using to attempt to access the UI, so I don't know why it is not prompting you for a client certificate. I recommend you use the following OpenSSL command to attempt a client connection from the command line and either verify that it successfully negotiates a mutually-authenticated handshake or fails and you can provide more error output. The command is on this Gist I have published.
$ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>
If you do not have your various certificates and keys in the correct format to use with this command, sections 2, 3, and 6 will allow you to export them to the right format.
It also appears from your earlier comment that you generated the client certificate and then stored it in your keystore. This is unnecessary and incorrect. The various keystores and certificates should be as follows:
Keystore
Used by NiFi application
Contains private key and public certificate (single PrivateKeyEntry entry) used by the server to identify itself
The public certificate should contain multiple certificates in the chain (in the same entry) to reach a trusted signing certificate (either one you control and manually trust, or a publicly-trusted certificate)
Truststore
Used by the NiFi application
Contains arbitrary number of trustedCertEntry entries (all public certificate(s) only) to verify "peer" certificates (either incoming client connections or connections NiFi makes to external resources, i.e. to ingest data from an HTTPS site)
Client certificate
Usually stored in PKCS12 keystore ( *.p12 )
Must contain the private key and public certificate with CN=<user name>
Certificate must be explicitly trusted or signed by known trusted certificate in truststore
Certificate must have complete chain in order for truststore to build certificate validation path
... View more
11-20-2016
04:03 AM
1 Kudo
@Karthik is correct that the provenance, content, and flowfile repositories are stored on disk unencrypted. Current recommendations are to restrict access to said repositories using OS-level access control (e.g. POSIX) and to use encrypted storage volumes. There is an existing security feature roadmap entry for transparent data encryption of the various repositories so that the values are never written to the file system in an unencrypted form. Obviously there are performance implications to take into consideration when developing this feature and an admin choosing to enable it. Just because the repository format on disk is "human unreadable" binary does not preclude the security concerns here -- an arbitrary process with OS permission can read those files, and the serialization logic is open source.
... View more
11-18-2016
12:13 AM
You did not answer my question about what type of error you are actually encountering.
The setting nifi.security.needClientAuth applies only to cluster communications and has no bearing on UI/API access. The Admin Guide discusses this.
If you did not configure LDAP or Kerberos as a user authentication mechanism, then you will definitely need a client certificate in order to access the secured UI. If you manually generated your keystore and truststore, you will need to do the same with the client certificate and have it be signed by a key that is loaded into your truststore. You can also use the TLS Toolkit that is provided with NiFi to generate all of the necessary files. There are many step-by-step tutorials on this site and others. An easy one is here. Do not load the server certificate identifying the NiFi application into the browser to act as a client certificate. They are different, and exposing the private key of the NiFi certificate is not a good idea.
... View more
11-17-2016
09:37 PM
Did you load your client certificate into the browser? What do you mean you are not able to access the site? Are you getting a TLS error and nothing comes up, or are you getting a NiFi message that says you are not authorized? You need to ensure that the DN of your client cert is in conf/authorizers.xml as the Initial Admin Identity and that it matches the user defined in conf/users.xml . You can find more information by tailing $NIFI_HOME/logs/nifi-app.log and $NIFI_HOME/logs/nifi-user.log .
... View more
11-17-2016
08:17 PM
In Windows you need to prepend the file path with " file:/// ". Try " file:///F:/Tools/HDF-2.0.1.0/nifi/conf/nifidev.keystore.jks ".
... View more