Member since
05-01-2017
17
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3298 | 09-25-2017 01:50 PM |
08-23-2018
10:09 AM
1 Kudo
Hey @Justin Brock, As @Andy LoPresto mentions securing NiFi with an internal CA is 100% possible. In fact Andy's advice helped me significantly when I configured our secure NiFi instance so all the credit goes to him or I'd still be stuck to this date (and he's a top bloke to boot!!!). I detailed the step by step process that I took here securing our NiFi instance using an internal CA (the example uses NiFi 1.3 but it's just as applicable now) so this combined with Andy's detail below is hopefully enough to get you over the line. Best of luck.
... View more
01-07-2018
06:29 AM
Hi Kevin, Are you using a
version of NiFi greater than or equal to v1.2? If so you will have
access to the Record-Oriented processors available in these versions
of NiFi which provide a great way to handle CSV data and should
provide significant performance improvements. Mark Payne provides an excellent example here of some of these processors
and their advantages. There are also other articles on
Record-Oriented processors on this community site here and from other
NiFi rockstars like @Bryan Bende on their personal blogs here. I’m
not sure of your downstream usecase but in my experience storing CSV
data as attributes on flow files is most commonly used for flow file
routing or data restructuring and persistence (hdfs, a database
etc.). If this is one of your intended downstream purposes once the
data is in Record-Oriented form you get quite a lot of flexibility in
what you can do with it including executing SQL queries against your
records using a QueryRecord processor, and inserting your records
directly into a database using a PutDatabaseRecord processor. There
is a little additional set-up with this process as you need to define
an Avro schema that represents your CSV file (and select/define a
schema registry) but conveniently NiFi even has an out of the box
InferAvroSchema processor to help you fast track this process (See @Timothy Spann article here). If you check the release notes for each NiFi
version you’ll be able to see what processors are available to you
as the Record-Oriented paradigm is relatively new and growing quickly
with each NiFi release. If you’re not at NiFi v1.2 I’d suggest
upgrading to the latest NiFi version if possible as having gone
through the upgrade process a number of times it’s a relatively straight
forward provided you have configured your NiFi instance according
to the suggested best practice. The Record-Oriented processors are
an exciting addition to the NiFi toolkit so I’d suggest investing
the time to embrace them as they will make your flows cleaner, more
performant, and provide greater flexibility in how you handle your
data.
... View more
11-09-2017
08:29 PM
Hi @David Sheard, @Bryan Bende, @Steven O'Neill, Myself and a colleague of mine noticed a similar behaviour with BLOB's and a legacy Oracle database we were using (Community link here). To extract the data we built an external Java program (jdbc + ResultSets) to get the BLOB bytes and write files to disk for NiFi ingestion. Alternatively, using NiFi's ExecuteScript processor and Groovy's database helper libraries the entire process can be managed through NiFi (or whatever scripting language you are comfortable with). Whichever of the above techniques you choose, you will still have to maintain your own business logic to manage your delta records. Not the most elegant solution but we just could not get the bundled NiFi processers to play nicely with the Oracle BLOB's.
... View more
11-01-2017
11:49 PM
Hi guys, I've been having issues viewing data provenance through the NiFi UI (v1.3, single instance) lately and wondering if anyone could offer some troubleshooting advice? What I've configured/observed: - No workload running on the NiFi instance other than the flow I'm testing provenance activity against. - Secure NiFi instance and the user has "query provenance" access policy (internally managed NiFi access, not through ranger). The user has full policy access to all NiFi functionality. - The Lucene provenance indexes are created as expected with timestamps on the file system aligning with NiFi activity. - There is no filter applied to the UI provenance search to hide provenance events between a certain date range. - My nifi.properties file has the below settings (only change other than size, duration, index/query threads is the use of the WriteAheadProvenanceRepository for performance). # Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
nifi.provenance.repository.debug.frequency=1_000_000
nifi.provenance.repository.encryption.key.provider.implementation=
nifi.provenance.repository.encryption.key.provider.location=
nifi.provenance.repository.encryption.key.id=
nifi.provenance.repository.encryption.key=
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=/data/nifi_content/Sandpit/repo/provenance_repository
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=10 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=4
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.journal.count=16
- Used repository size is well below the configured 10GB and time of viewing provenance is < 24 hours after flow execution (512Mb of provenance vs 10GB configured, provenance checked at 5,10,15,20.... min intervals after flow executed). Attachment "provsize.png". - If I list the queue of the connection after the processor/s I'm viewing provenance activity against, all flow files are listed and visible as expected. - The UI displays "no value set" as the begin time of the provenance UI search window. Attachment "provsearch.png" I'm a little stumped at my missing provenance activity so any community advice would be greatly appreciated. Thanks DH.
... View more
Labels:
- Labels:
-
Apache NiFi
09-25-2017
01:50 PM
Hi
James,
I
recently configured our NiFi 1.3 installation using certificates from
our internal CA and found I had to do a little bit of certificate and
keystore manipulation to get things working. I too had things
working nicely using the nifi-toolkit and then found myself
scratching my head and doing a lot of SSL handshake debugging
wondering what had changed with my configuration. The NiFi
toolkit is so great it’s hard to believe how much it actually does
for you using a single line of code …client AND server certificate
creation, strong passwords on all of your stores, keystore and
truststore creation with the appropriate private, public, and trusted
cert entries ….pretty impressive stuff!
Anyway
here’s some to the configuration I had to do to get my internal
certificates playing nicely with NiFi. Firstly, I had our pki team
issue two certificates for us which were conveniently issued in two
formats .p12 and .cer
1. Client
cert (myClientCert.pem.p12 and myClientCert.pem.cer). This
certificate only requires a EKU of clientAuth. This certificate is
imported into your browser.
2. Server
cert (myServerCert.pem.p12 and myServerCert.pem.cer). This
certificate requires an EKU of both clentAuth and serverAuth. These
are used on the server and not in your browser.
Client
cert – This certificate needs to have a DN with credentials exactly
matching those you have configured in your authorizers.xml file
“Initial Admin Identity” property. Exactly matching means
spaces, punctuation etc. need to match in your cert and your
authroizers.xml file. This is the certificate you need to
upload to your browser so you can log into the UI and configure
access for other users (i.e. if in windows double click on the .p12
certificate and import it with the appropriate password). Your client
configuration should be good to go.
Server
cert – Matt’s configuration details above are spot on. As
he mentions your server certificates KU and EKU look good (…which
was more than I can say about ours!). One thing I did
find in our CA issued certificates is that they did not contain the
complete certificate chain for our environment so I needed to add
this information in to get things working smoothly. To do this
in windows complete the following:
Import
the certificate into the windows certificate manager by double
clicking on the .p12 file and follow the import wizard.
Tick
the "Mark this key as exportable. This will allow you to back
up or transport your keys at a later time" check box.
Follow
the remaining wizard prompts to import the certificate.
Open
a Windows command prompt and run "certmgr.msc" (Capi).
Browse
to "Personal" and select the certificate that was imported
in step 1. Right click the certificate and choose "All
Tasks" -> "Export".
Select
"Yes, export the private key" radio button when prompted.
Select
"Include all certificates in the certification path if
possible" and "Export all extended properties" check
boxes.
Follow
the remaining wizard prompts to export the certificate which now
includes the entire certificate chain.
Upload
the exported certificate with the complete certificate chain to the
server environment that is being configured.
Keystore
creation
The
keystore exported from Capi will not be in jks format so you need to
convert it into a format that NiFi can use keytool -importkeystore -srckeystore <certificateLocation> -destkeystore <destinationKeystore> -srcstoretype pkcs12
You
now have a jks keystore with a single private key entry (with a chain
length of three) containing the complete certificate chain.
Truststore
creation
You
need to add your root and intermediate certificate as trusted entries
in your truststore. To extract the root and intermediate
certificates once again use certmgr.msc and browse the directories for
your root and intermediate ca entries. Export both of these
certificates and upload them to your server.
We
can now create a truststore file using the JDK's Keytool utility. keytool -importcert -v -trustcacerts –alias <certificateAliasName> -file <locationToCertificate> -keystore <TrustStoreName> –storepass <TrustStorePassword> –noprompt
Repeat
this process for both the root and intermediate certificates.
Be
careful with the passwords that you use for importing and exporting
the certificates as it is easy to change them and lose track of what
the final password of your keystore and trustsores are. Be sure to do
a verbose list of both the keystore and truststore so you know they
are correct and match the entries defined in your nifi.properties
file. That’s
pretty much it …if you’ve done everything correctly you should be
prompted for you admin user cert by your browser when you hit the
NiFi url and be greeted by the NiFi user interface. Good
luck.
... View more
09-04-2017
11:20 PM
Excellent
article as usual @Matt
Clarke. As an additional side note and to keep this excellent article
evolving I did notice when recently configuring our local NiFi instance that
the HDF3 doco now suggests better performance using the WriteAheadProvenanceRepository instead of the default PersistentProvananceRepository particularly for environments which
consume a lot of small files (available for NiFi v1.2 and above).
"When
used in a NiFi instance that is responsible for processing large volumes of
small FlowFiles, the PersistentProvenanceRepository can quickly become a bottleneck.
The WriteAheadProvenanceRepository was then written to provide the same
capabilities as the PersistentProvenanceRepository while providing far better
performance. Changing to the WriteAheadProvenanceRepository is easy to accomplish, as the
two repositories support most of the same properties." (https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.0/bk_administration/content/provenance-repository.html). NiFi users
meeting this criteria and starting with an instance greater than v1.2 might
want to consider this configuration as well.
... View more
08-23-2017
02:19 AM
Thanks for the info Andy this is good to know. I'm very close to getting there and now have a successful handshake taking place between NiFi and itself thanks to your useful advice (I've learnt more about SSL debugging in the process than I'd care to admit!). The problem has been the strict format required for the NiFi certs which were issued to us by our CA team and not generated using the NiFi toolkit. This has been complicated by not having a recent version of openssl capable of debugging the TLS1.2 version the Jetty server requires. For interest and other users out there make sure your certs have an EKU of both clientAuth and serverAuth specified (if your EKU is critical) and your KU has DigitalSignature specified. I've now hit the same problem as you've assisted with here so I'll try to work through your suggestions (http://apache-nifi.1125220.n5.nabble.com/NiFi-1-1-1-Secure-Cluster-Setup-Issue-HTTPS-Wrong-Hostname-wrong-should-be-lt-my-ip-address-gt-td15361.html). Thanks again.
... View more
08-10-2017
11:56 PM
Hi Andy, unfortunately I'm still having TLS cipher connection compatibility problems. NiFi always appears to present a ClientHello TLSv1.2 message irrespective of my selection on the StandardSSLContextService (v1.3.0) SSL Protocol. I assume the ClientHello is launched by NiFi using a java SSL connection and has no dependency on my underlying openssl version (OpenSSL 0.9.8j-fips)?
... View more
08-08-2017
07:26 AM
Thanks for the logging tip Andy this has been very useful. I had previously tried the openssl s_client command you mention above with the same, cert, key, and CA and connected without issue to another service on the server (knox) so I was confident of valid credentials. The bootstrap log is now showing a SSL_NULL_WITH_NULL_NULL and NiFi Web Server-228, fatal error: 40: no cipher suites in common which I was unable to see before. I'll explore further tomorrow and get back with a confirmation. Thanks again I appreciate the assistance.
... View more