Minifi C++ 0.4.0 secure communication

[#1]

Has anyone got Minifi C++ v0 4 0 working over a secure connection to a Nifi cluster. My certs seem to be fine as I can log into UI of the secured cluster through the browser using a cert I created for a nifiadmin user. I've got minifi (collecting Squid logs) working unsecured but every time I try to get it connecting securely it won't authenticate to be able to retreive s2s settings. The logs are:

[2018-03-10 19:13:58.149] [org::apache::nifi::minifi::controllers::SSLContextService] [info] not good [2018-03-10 19:13:58.149] [org::apache::nifi::minifi::controllers::SSLContextService] [warning] still not good [2018-03-10 19:13:58.230] [org::apache::nifi::minifi::core::ProcessSession] [info] Transferring 2ef87a2c-2497-11e8-81ff-be2999981f84 from Get_access.log to relationship success [2018-03-10 19:13:58.230] [org::apache::nifi::minifi::processors::TailFile] [info] TailFile access.log for 2481982 bytes [2018-03-10 19:13:58.230] [org::apache::nifi::minifi::processors::UpdateAttribute] [info] Set attribute 'Store State' of flow file '2ef87a2c-2497-11e8-81ff-be2999981f84' with value 'Do not store state' [2018-03-10 19:13:58.230] [org::apache::nifi::minifi::processors::UpdateAttribute] [info] Set attribute 'host_name' of flow file '2ef87a2c-2497-11e8-81ff-be2999981f84' with value 'gs1' [2018-03-10 19:13:58.230] [org::apache::nifi::minifi::processors::UpdateAttribute] [info] Set attribute 'tenant' of flow file '2ef87a2c-2497-11e8-81ff-be2999981f84' with value ''TheBerties'' [2018-03-10 19:13:58.230] [org::apache::nifi::minifi::core::ProcessSession] [info] Transferring 2ef87a2c-2497-11e8-81ff-be2999981f84 from UpdateAttribute to relationship success [2018-03-10 19:13:58.426] [org::apache::nifi::minifi::utils::HTTPClient] [error] curl_easy_perform() failed Peer certificate cannot be authenticated with given CA certificates [2018-03-10 19:13:58.426] [org::apache::nifi::minifi::RemoteProcessorGroupPort] [error] ProcessGroup::refreshRemoteSite2SiteInfo -- curl_easy_perform() failed [2018-03-10 19:13:58.426] [org::apache::nifi::minifi::c2::C2Agent] [info] Class is RESTSender [2018-03-10 19:13:58.428] [org::apache::nifi::minifi::io::Socket] [error] Could not bind to socket [2018-03-10 19:13:58.429] [org::apache::nifi::minifi::FlowController] [info] Started Flow Controller [2018-03-10 19:13:58.429] [main] [info] MiNiFi started [2018-03-10 19:13:58.444] [org::apache::nifi::minifi::utils::HTTPClient] [error] curl_easy_perform() failed Peer certificate cannot be authenticated with given CA certificates [2018-03-10 19:13:58.444] [org::apache::nifi::minifi::RemoteProcessorGroupPort] [error] ProcessGroup::refreshRemoteSite2SiteInfo -- curl_easy_perform() failed [2018-03-10 19:13:58.444] [org::apache::nifi::minifi::RemoteProcessorGroupPort] [info] no protocol, yielding

I can't find much in the way of documentation for secure minifi cpp operations. I'm using the Centos 7 version on Hortonworks repository. Has anyone been successful and are you able to suggest what I'm doing wrong. I haven't reverted back to compiling from github yet but guess that is the next on the list.

Many thanks in advance.

Tom

[#2]

There is more detail from the logs and the security elements of minifi.properties at https://pastebin.com/gg2H7HEP in case that is of use. I haven't set up any SSLContextService so it should be drawing straight from the minifi.properties if I understand correctly. I'm connecting over HTTP rather than RAW. Any help would be appreciated; I'm sure I'm doing something very basic wrong, but if I can crack this then I see nifi as the jewel in the metron crown!

[#3]

See: https://community.hortonworks.com/articles/85549/how-to-secure-minifi-c-site-to-site.html

[#4]

Thanks @Timothy Spann, but those are the instructions started with and it is throwing the errors I describe. Given that it dates back to a year ago with an earlier Minifi c++ release and and older version of config.yml I'm wondering what has changed.

[#5]

Tom,

I responded to your E-mail on the user group. The information here seems a little bit different. I see the error "Peer certificate cannot be authenticated" This is an error through curl when target SSL chain cannot be authenticated. I see some messages which show empty certs. Can you provide your configuration yaml?

The agent should attempt to use the procedures linked by @Timothy Spann for backwards compatibility ; however, it seems that from the following lines, there is a context service or the paths found were not good ( it's actually supposed to print a path before 'not good') . If you can provide your configuration YAML we should be able to help:

1. [2018-03-10 19:13:58.149] [org::apache::nifi::minifi::controllers::SSLContextService] [info] not good
2. [2018-03-10 19:13:58.149] [org::apache::nifi::minifi::controllers::SSLContextService] [warning] still not good

Also please make the following change.

nifi.https.*

should be

nifi.remote.input.secure=true

nifi.security.need.ClientAuth=true

nifi.security.client.certificate=XXX

nifi.security.client.private.key=XXX

nifi.security.client.pass.phrase=XXX

nifi.security.client.ca.certificate=XXX

nifi.remote.input.socket.port=XXX

Are you attempting to use HTTP site to site?

You will then need to add the following

nifi.remote.input.http.enabled=true

Please note you will not need to define an SSLContextService. To maintain backwards compatibility we will generate one internally for you using the above settings. Thanks

[#6]

Thanks for your reply @mparisi, and also through the forums. My config.yml is below but to answer/query a few of your other queries:

- Yes, I'm using HTTP rather than RAW. Its worked fine with all SSL security switched off at both ends but fails when I try to put SSL in place.

- I have nifi.remote.input.http.enabled=true at the nifi end, and assume if I didn't then it wouldn't work with or without security?

- I have all of those settings you have specified in minifi.properties set up (you will see on the pastebin I linked) but as nifi.https... rather than nifi.security... is this a change because I was using the minifi.properties file that came from the repo? The only setting I don't have is nifi-remote.input.socket.port but is this required for HTTP or just for RAW?

MiNiFi Config Version: 3
Flow Controller:
  name: collect_squid-0 0 3
  comment: ''
Core Properties:
  flow controller graceful shutdown period: 10 sec
  flow service write delay interval: 500 ms
  administrative yield duration: 30 sec
  bored yield duration: 10 millis
  max concurrent threads: 1
  variable registry properties: ''
FlowFile Repository:
  partitions: 256
  checkpoint interval: 2 mins
  always sync: false
  Swap:
    threshold: 20000
    in period: 5 sec
    in threads: 1
    out period: 5 sec
    out threads: 4
Content Repository:
  content claim max appendable size: 10 MB
  content claim max flow files: 100
  always sync: false
Provenance Repository:
  provenance rollover time: 1 min
  implementation: org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository
Component Status Repository:
  buffer size: 1440
  snapshot frequency: 1 min
Security Properties:
  keystore: ''
  keystore type: ''
  keystore password: ''
  key password: ''
  truststore: ''
  truststore type: ''
  truststore password: ''
  ssl protocol: ''
  Sensitive Props:
    key:
    algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
    provider: BC
Processors:
- id: 7e56473a-44c9-3c1a-0000-000000000000
  name: Get_access.log
  class: org.apache.nifi.processors.standard.TailFile
  max concurrent tasks: 1
  scheduling strategy: TIMER_DRIVEN
  scheduling period: 10 sec
  penalization period: 30 sec
  yield period: 1 sec
  run duration nanos: 0
  auto-terminated relationships list: []
  Properties:
    File Location: Local
    File to Tail: /var/log/squid/access.log
    Initial Start Position: Beginning of File
    Rolling Filename Pattern:
    tail-base-directory:
    tail-mode: Single file
    tailfile-lookup-frequency: 10 minutes
    tailfile-maximum-age: 24 hours
    tailfile-recursive-lookup: 'false'
- id: de9e8a3f-3cc3-356b-0000-000000000000
  name: UpdateAttribute
  class: org.apache.nifi.processors.attributes.UpdateAttribute
  max concurrent tasks: 1
  scheduling strategy: TIMER_DRIVEN
  scheduling period: 0 sec
  penalization period: 30 sec
  yield period: 1 sec
  run duration nanos: 0
  auto-terminated relationships list: []
  Properties:
    Delete Attributes Expression:
    Stateful Variables Initial Value:
    Store State: Do not store state
    host_name: ${hostname()}
    tenant: '''TheBerties'''
  annotation data: |-
    <criteria>
        <flowFilePolicy>USE_ORIGINAL</flowFilePolicy>
    </criteria>
Controller Services: []
Process Groups: []
Input Ports: []
Output Ports: []
Funnels: []
Connections:
- id: 2f59bfb7-7564-31d9-0000-000000000000
  name: Get_access.log/success/UpdateAttribute
  source id: 7e56473a-44c9-3c1a-0000-000000000000
  source relationship names:
  - success
  destination id: de9e8a3f-3cc3-356b-0000-000000000000
  max work queue size: 10
  max work queue data size: 500 MB
  flowfile expiration: 0 sec
  queue prioritizer class: ''
- id: e82df80d-1992-3ce4-0000-000000000000
  name: UpdateAttribute/success/05664ac9-0162-1000-0000-00000c9a533d
  source id: de9e8a3f-3cc3-356b-0000-000000000000
  source relationship names:
  - success
  destination id: 05664ac9-0162-1000-0000-00000c9a533d
  max work queue size: 10
  max work queue data size: 500 MB
  flowfile expiration: 0 sec
  queue prioritizer class: ''
Remote Process Groups:
- id: 42ac92f5-eeba-33a1-0000-000000000000
  name: ''
  url: https://nifi1.dev.cyhesion.com:9091/nifi
  comment: ''
  timeout: 30 sec
  yield period: 10 sec
  transport protocol: HTTP
  proxy host: ''
  proxy port: ''
  proxy user: ''
  proxy password: ''
  local network interface: ''
  Input Ports:
  - id: 05664ac9-0162-1000-0000-00000c9a533d
    name: Received_from_Minifi
    comment: ''
    max concurrent tasks: 1
    use compression: false
  Output Ports: []
NiFi Properties Overrides: {}

Many thanks for your support,

Tom

[#7]

Further to my last, I've changed the nifi.https... to nifi.security... (sorry @Timothy Spann, I hadn't noticed the difference between the provided minifi.properties and the article). I have also put in an entry for nifi.remote.input.http.enabled=true and one for the nifi.remote.input.socket.port=XXX. My minfi.properties is now as follows:

# Core Properties #
nifi.version=0.1.0
nifi.flow.configuration.file=./conf/config.yml
nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=10 millis
# Provenance Repository #
nifi.provenance.repository.directory.default=${MINIFI_HOME}/provenance_repository
nifi.provenance.repository.max.storage.time=1 MIN
nifi.provenance.repository.max.storage.size=1 MB
nifi.flowfile.repository.directory.default=${MINIFI_HOME}/flowfile_repository
nifi.database.content.repository.directory.default=${MINIFI_HOME}/content_repository
nifi.remote.input.secure=true
nifi.security.need.ClientAuth=true
nifi.security.client.certificate=./conf/keystore.pem
nifi.security.client.private.key=./conf/keystore.key
nifi.security.client.pass.phrase=./conf/password
nifi.security.client.ca.certificate=./conf/nifi-cert.pem
nifi.remote.input.socket.port=8022
nifi.remote.input.http.enabled=true
#nifi.rest.api.user.name=admin
#nifi.rest.api.password=password
## enable the controller socket provider on port 9998
controller.socket.host=localhost
controller.socket.port=9998

This has removed the SSLContextService issues but it still isn't working. The outstanding errors are listed below.

Thank you for your support and patience. Tom

[2018-03-11 21:09:04.071] [org::apache::nifi::minifi::utils::HTTPClient] [error] curl_easy_perform() failed Peer certificate cannot be authenticated with given CA certificates

[2018-03-11 21:09:04.071] [org::apache::nifi::minifi::RemoteProcessorGroupPort] [error] ProcessGroup::refreshRemoteSite2SiteInfo -- curl_easy_perform() failed

[2018-03-11 21:09:04.071] [org::apache::nifi::minifi::c2::C2Agent] [info] Class is RESTSender
[2018-03-11 21:09:04.073] [org::apache::nifi::minifi::io::Socket] [error] Could not bind to socket
[2018-03-11 21:09:04.074] [org::apache::nifi::minifi::FlowController] [info] Started Flow Controller
[2018-03-11 21:09:04.074] [main] [info] MiNiFi started

[#8]

is there a firewall? something running on that port? all SSL installed? what user is running minifi? can you run with sudo or logged in as root?

check here: https://lists.apache.org/thread.html/%3CCAEzjzLm4foJ4BGJpiignL8VLiu5Gw-OXoKTyR6EpVnoohT6tZQ@mail.gma...

Check this one: https://community.hortonworks.com/questions/54993/remote-instance-of-nifi-is-not-configured-to-allow...

Seems resolved: https://github.com/apache/nifi-minifi-cpp/pull/263

Have you read this free book: http://discover.attunity.com/apache-nifi-for-dummies-en-report-go-c-lp8558.html?utm_source=google&ut...

[#9]

Thanks again @Timothy Spann for your support and extensive reply. I've got a couple of questions as a result (-) and replies to yours (+):

- Does minifi use anything other than the https REST API port (in my case 9091) when communicating over secure http sms or does the raw socket also get used?

- Do you know which version from GitHub was used to build the binaries on the Hortonworks centos 7 repo?

+ At the moment I'm operating as root with firewalls disabled at both ends while I try to get things working.

+ I did see the thread in your first link, but it was unclear whether the issue was resolved.

+ Your second link seems to relate to RAW s2s; it's this relevant for http s2s?

+ the book was a great help from the outset, and I used it extensively to get things working. But it hasn't been particularly useful in getting an https link in place

I think my next option had to be building from the latest source, but can't help but feel that there is something fairly simple I'm doing wrong of lots of other people are using the bins on the repo effectively.

Yours, Tom

[#10]

Tom,

https://github.com/apache/nifi-minifi-cpp/blob/master/conf/minifi.properties is the repo properties. Is this different than what you've been using? You noted that it was nifi.https.* for those properties, so I'd be happy to fix the repo of origin.

The error we see: [2018-03-11 21:09:04.071] [org::apache::nifi::minifi::utils::HTTPClient] [error] curl_easy_perform() failed Peer certificate cannot be authenticated with given CA certificates

indicates that we have a cert chain validation issue.

For testing purposes you can take the options from https/github.com/phrocker/nifi-minifi-cpp#sitetosite-security-configuration . You probably only need to disable peer verification. I wouldn't advise running this in production but can get you beyond the immediate issue for further testing.

nifi.security.client.disable.host.verification=true
nifi.security.client.disable.peer.verification=true

On CENTOS you may need to perform the following procedures to get it working in :

1. sudo yum install ca-certificates
2. sudo update-ca-trust enable
3. sudo cp /path/to/your_new_cert.crt /etc/pki/ca-trust/source/anchors/
4. sudo update-ca-trust extract

I've taken this from :

https://stackoverflow.com/questions/14682894/getting-error-in-curl-peer-certificate-cannot-be-authen...

Are you using the toolkit to generate these certificates? I'll see if I can arrive at the same issue and if we can create a more user friendly way to get around it.

[#11]

@mparisi, that is an awesome response, which I will try to action this evening when I'm able to get close enough to my environment to play around. The guilty minifi.properties file is the one contained in the Hortonworks bin for Centos 7 at http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.1.1.0/tars/nifi-minifi-cpp/nifi-minif...

Many thanks and I'll get back to you (hopefully with a positive resolution), Tom

[#12]

And to answer your other question @mparisi, yes I'm using Nifi CA and generating certs with tls-toolkit.

[#13]

Building the latest source is usually good for odd problems.

But this is pretty standard and should work. Engineering will post some more information.

Have you tested your certificates?

https://www.cyberciti.biz/faq/test-ssl-certificates-diagnosis-ssl-certificate/

Was your MiniFi C++ built with SSL enabled?

type curl -V

[#14]

Thanks for your response and suggestions @Timothy Spann. I've tested the certs and they seem to be OK. I'm getting a self-signed response of 19 and when I have copied the CA cert into /etc/pki/ca-trust/source/anchors/ it responds 0(OK). Still not joy in getting things connected though. I'm about to start a fresh build from source in the hope that will change things.

[#15]

Sounds good. Someone from engineering hopefully will help you run this down. I am trying to think what other logs might be useful to post to spot something odd.

[#16]

@Tom Burton

Sorry for the delay. I responded but duplicated my response and upon deleting it must have deleted both. Yesterday I submitted https://github.com/apache/nifi-minifi-cpp/pull/276

I tested this on CENTOS 7 . I first was able to get it working by adding my nifi-cert to the ca-cert bundle in /etc/ssl ; however, I recognize that not everyone wants to do this, so I created the above pull request for review. This approach may not be correct in all cases, so I may update the PR to give users an option to use the system defined CA bundle ( my preference in certain deployments); however, if you have the chance to test it you may be able to get around not having to disable peer verification and updating the system's CA bundle ( that is curl's default CAFile/CAPath).

[#17]

In addition to the above response, on my prior response I mentioned that I was getting a domain error after adding my cert to the CA bundle and using the above fix but re-issues my cert so that the domain matched the certificate.

[#18]

A quick update @Timothy Spann and @mparisi. I've built from the current master on GitHub and am getting largely the same problems (just slightly different phrasing), which has led me to think again about what might be different about my environment. My standard centos VM is built with a NIST 800-171 security profile applied, which is where my thinking is at the moment. I'll have to look at that profile now to see what clamps may be causing the problem and gradually unwind them. Any ideas you may have would be great. It's going to be important to identify the blocker (if this is the cause) because many of my clients will have various security policies applied to the servers.

[#19]

@Tom Burton Thanks for the info. I'll investigate with that profile and make any changes that are needed in the PR, above. Thanks!

[#20]

That may lock down more things. Maybe you need a stronger cert instead of defaults.

[#21]

@Tom Burton I pulled up a clean Centos 7 VM with security profiles enabled and ran into the same issues. I noticed that the system's curl is not built with OpenSSL and instead is using NSS, which we are not using.

Did you use the bootstrap? I will begin checking all distros to ensure that we perform the download and install of libcurl-openssl when the default impl does not contain OpenSSL. Thanks for helping identify this.