Created on 01-08-2024 03:50 AM - edited 01-08-2024 03:51 AM
[begin_log]2024-01-08 18:57:00,406+0800|ERROR|pool-47-thread-1|o.s.s.s.TaskUtils$LoggingErrorHandler|Unexpected error occurred in scheduled task
org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal Server Error: "javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown"
at org.springframework.web.client.HttpServerErrorException.create(HttpServerErrorException.java:100)
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:170)
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122)
at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:825)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:783)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:717)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:608)
at com.orchsym.trace.alerts.api.timer.Timer.getBulletinBoardDTO(Timer.java:162)
at com.orchsym.trace.alerts.api.timer.Timer.getBulletinBoard(Timer.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:95)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Created 01-08-2024 07:33 PM
Ignore the difference between runtime-0.runtime-statefulset.default.svc.cluster.local and runtime-0.runtime-statefulseheadless.default.svc.cluster.local, because I'm putting the dns of the current cluster node from the former to the latter.
Created 01-09-2024 07:03 AM
@JamesZhang
The logs shared indicate a TLS exchange issue.
Have you looked at the output of openssl to see what your running NiFi responds with:
openssl s_client -connect runtime-0.runtime-statefulset.default.svc.cluster.local:443 -showcerts
and
openssl s_client -connect runtime-1.runtime-statefulset.default.svc.cluster.local:443 -showcerts
Created 01-09-2024 09:15 AM
[root@runtime-1 /opt/orchsym/runtime-ee]# openssl s_client -connect runtime-0.runtime-statefulset.default.svc.cluster.local:443 -showcerts
CONNECTED(00000003)
depth=1 OU = orchsym.com, CN = ca
verify error:num=19:self signed certificate in certificate chain
---
Certificate chain
0 s:/OU=orchsym.com/CN=runtime-0.runtime-statefulset.default.svc.cluster.local
i:/OU=orchsym.com/CN=ca
-----BEGIN CERTIFICATE-----
MIIDZTCCAk2gAwIBAgIJAMjrw8P09eTSMA0GCSqGSIb3DQEBCwUAMCMxFDASBgNV
BAsMC29yY2hzeW0uY29tMQswCQYDVQQDDAJjYTAgFw0yNDAxMDkxNzAwMzVaGA8y
MDUxMDUyNjE3MDAzNVowWDEUMBIGA1UECwwLb3JjaHN5bS5jb20xQDA+BgNVBAMM
N3J1bnRpbWUtMC5ydW50aW1lLXN0YXRlZnVsc2V0LmRlZmF1bHQuc3ZjLmNsdXN0
ZXIubG9jYWwwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCYDBOx7r3e
33zFDM8VilMZU4J/oWYKUe0eesd9gWsqIMUm26/ImQVN0aQIrOylLOLftcEXkQp8
BAkuo+IgbBzoQBEqDmHsktwcLld+04tRQMijL7RbieqN0sMqoHs/XRdB7bhfel73
ffnBQ2nctZCynuTQ7aem5ubzKMm5oQRPXPB5jJ3A5FwKy/F4lpdJsEZRVVohl0xt
kTIxpxvEu8OpuElajh34Lhn59yVNS4qkubsOE7ll+RPzHve0YeuUZXEjK41N3zLI
zNe5HDVGYpI6sQdGinY/u+2lP5Vm7LDFm67PjT/LfrQ/g5CRzo1dHxyniN0zuSg2
VTIV94Z8takhAgMBAAGjZTBjMEIGA1UdEQQ7MDmCN3J1bnRpbWUtMC5ydW50aW1l
LXN0YXRlZnVsc2V0LmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwHQYDVR0lBBYw
FAYIKwYBBQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEBCwUAA4IBAQDFAmP0YLbN
Cy0d/QPiXYhbOLWEGC+/Y1xMMIxfo6uxlhThp97IU0AFk9Q9sC/DTZ092+Mccp3w
eImOswQWiHjT+CKYpwpgnD5lQLf1l/6WNT/ffnRoCVH/iq6kkRp38KUI0l205kAw
2ZKlbS8AC1GC6U4ZEETgUjN1kSbgo3iA6oq9RKd0vi9gC3OfZg9NeSUGbil1rFrt
9jtGQgqu0WGe/mVFJ6wqS4yXvSavAVCpm7AQh00CwgtCGTIZ/zZmO9YtW/LwdTfC
h80ypeUVyekzpFANNPSjMp2JgP4PuwUX+RITq86n2biIQAgPf5KgGvvOgd4cEY2w
E6m0oHH8zgY0
-----END CERTIFICATE-----
1 s:/OU=orchsym.com/CN=ca
i:/OU=orchsym.com/CN=ca
-----BEGIN CERTIFICATE-----
MIIDGTCCAgGgAwIBAgIJANdiG5dyjQzgMA0GCSqGSIb3DQEBCwUAMCMxFDASBgNV
BAsMC29yY2hzeW0uY29tMQswCQYDVQQDDAJjYTAeFw0yNDAxMDgxNTUwMzRaFw0z
NDAxMDUxNTUwMzRaMCMxFDASBgNVBAsMC29yY2hzeW0uY29tMQswCQYDVQQDDAJj
YTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANsBJ0dfZkk1efw2EgKt
3b8AynVDq3B/UKRmtkVJmvH+Ja4lfqlpcZMO8L4cCNOHJzEPHFSitlQoagTmiBL0
axnrd5upk3UlM/JctZOCBVwR9d2t0RE6Z7P7HAaFyxJXGj7oYC7xjYxuVVuN56B9
BTZWX6X9k2Dz659cTsLsQGc0Uf69chuUvN0kycm3DpKBRVSg2kc8e9Rbnn+w69J6
fE6goEixE5ysZAwzDTUHnx9GiRI0l8BEOqki8yoGahRZzEBw3OpWfvStqfXROMN/
+mPzN9EHAowyNGLbjbusmDAsJ7ojB39klxm8qvUDY71sVY7stGoCUxXLvTRgXAct
xo0CAwEAAaNQME4wHQYDVR0OBBYEFGU5/OVYAsw5Vg6b9KTuu6y5/OmzMB8GA1Ud
IwQYMBaAFGU5/OVYAsw5Vg6b9KTuu6y5/OmzMAwGA1UdEwQFMAMBAf8wDQYJKoZI
hvcNAQELBQADggEBAKnim+IdTeDy7KmWZxAyj2qGyz/cSK2dqkYU1iLcc492mXFU
RtD+ZTI7zGOFfZ1i7TIX7+Or2SjJ1EeCBUJLVt0nHnESWQR7TlTn03wFwLyf95Bd
3e+OqDUdj3DhWp1bfb0JIbWBA6nLBNLOjgCjpV/X8m9o0+3E6FV/zjbjUNlpZXra
Gwmi839Ko+9KX/44tTgLMQKB34H28k4HBnunnD/GUImXYchzeSnlmFpheKQ0/MVM
LVDSX3BZFPpImmmaqithUOT+MRRfQL/MRpVqLy1oja5RVpP+kKPZxo2p9wn3heyP
PjaC4NSkV6E4hdddOkVIz01jO9Bxse9aCpfPo34=
-----END CERTIFICATE-----
---
Server certificate
subject=/OU=orchsym.com/CN=runtime-0.runtime-statefulset.default.svc.cluster.local
issuer=/OU=orchsym.com/CN=ca
---
Acceptable client certificate CA names
/OU=orchsym.com/CN=ca
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2290 bytes and written 483 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-SHA384
Session-ID: 659D7F10CB1256340096AE6B793A0EF99256807F5742D7B70EC637F0C1C8B5B6
Session-ID-ctx:
Master-Key: 3954CAAFF578E3D28D47394B42DBD2CE432D0D86C1D2C1D560BB2AF1E6AF982E812B40AD0D6142A2990622726C4B5399
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1704820496
Timeout : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
---
Created 01-09-2024 09:15 AM
[root@runtime-1 /opt/orchsym/runtime-ee]# openssl s_client -connect runtime-1.runtime-statefulset.default.svc.cluster.local:443 -showcerts
CONNECTED(00000003)
depth=1 OU = orchsym.com, CN = ca
verify error:num=19:self signed certificate in certificate chain
---
Certificate chain
0 s:/OU=orchsym.com/CN=runtime-1.runtime-statefulset.default.svc.cluster.local
i:/OU=orchsym.com/CN=ca
-----BEGIN CERTIFICATE-----
MIIDZTCCAk2gAwIBAgIJAOda8vSMjty7MA0GCSqGSIb3DQEBCwUAMCMxFDASBgNV
BAsMC29yY2hzeW0uY29tMQswCQYDVQQDDAJjYTAgFw0yNDAxMDkxNjU5NTlaGA8y
MDUxMDUyNjE2NTk1OVowWDEUMBIGA1UECwwLb3JjaHN5bS5jb20xQDA+BgNVBAMM
N3J1bnRpbWUtMS5ydW50aW1lLXN0YXRlZnVsc2V0LmRlZmF1bHQuc3ZjLmNsdXN0
ZXIubG9jYWwwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCiqFVT+BcV
L/7RdRz26cXeUw8ifP3omnTm3f4MzRHOOvlJMqQaoUdsDTooReYl4uF07vPmewGG
iOKhU4R4veucf9WNIzCY52PaDlcnPDcQhJisytHK+L+Cca5kNZ+eUzk8ywe5zR1a
t760THdweuHNeh9UaKkXgjDu0XdWh80VQ2rWOrbsJzikyUlAZ7olV/boGXD05EtX
mUG0a5K9KOccPn7HLOv3nOas0fqWDj2bYhxhCU8dwT2LaiNbsIyph7INZGp8ZxzT
T70ZpDJKguzGOSZwRTEyvCC3CjqjS4CWPB5RPQEYKHrPc0t5bXuixToITySgIX1/
BPL8RxftkpDFAgMBAAGjZTBjMEIGA1UdEQQ7MDmCN3J1bnRpbWUtMS5ydW50aW1l
LXN0YXRlZnVsc2V0LmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwHQYDVR0lBBYw
FAYIKwYBBQUHAwEGCCsGAQUFBwMCMA0GCSqGSIb3DQEBCwUAA4IBAQCS1zXSEU72
GF7K9L6Cjdc9dTB8/+d31IDPzQBPwtBTjHODz3PSYCaXnf08CZzEcM4KrzXrBfeM
LFjRnfD7tpM06hfRAqnACfAF5I9M6P6tXopaTQ5YOHerDnJJgvStdYd0yAh19/zu
8+Qvmjd5bdZ1h9adA1wXbvWfL1hEbJUHs/Zjx0qDYP4R06pM+TR6SbjCNxqvsJDJ
8ELpNp8Ykda7ht0vFqILAhJgNK4OV6Akklfv/Tkk0KXTMmws/tLfhz+MuLu/uj2f
p0BHlwUniIo2IthM0DAOSBJblZhGdCbMeNh2SiLMQ1Xg2QX3L0g5CZK84TRnnuKH
MNWaCMfYo6Yv
-----END CERTIFICATE-----
1 s:/OU=orchsym.com/CN=ca
i:/OU=orchsym.com/CN=ca
-----BEGIN CERTIFICATE-----
MIIDGTCCAgGgAwIBAgIJANdiG5dyjQzgMA0GCSqGSIb3DQEBCwUAMCMxFDASBgNV
BAsMC29yY2hzeW0uY29tMQswCQYDVQQDDAJjYTAeFw0yNDAxMDgxNTUwMzRaFw0z
NDAxMDUxNTUwMzRaMCMxFDASBgNVBAsMC29yY2hzeW0uY29tMQswCQYDVQQDDAJj
YTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANsBJ0dfZkk1efw2EgKt
3b8AynVDq3B/UKRmtkVJmvH+Ja4lfqlpcZMO8L4cCNOHJzEPHFSitlQoagTmiBL0
axnrd5upk3UlM/JctZOCBVwR9d2t0RE6Z7P7HAaFyxJXGj7oYC7xjYxuVVuN56B9
BTZWX6X9k2Dz659cTsLsQGc0Uf69chuUvN0kycm3DpKBRVSg2kc8e9Rbnn+w69J6
fE6goEixE5ysZAwzDTUHnx9GiRI0l8BEOqki8yoGahRZzEBw3OpWfvStqfXROMN/
+mPzN9EHAowyNGLbjbusmDAsJ7ojB39klxm8qvUDY71sVY7stGoCUxXLvTRgXAct
xo0CAwEAAaNQME4wHQYDVR0OBBYEFGU5/OVYAsw5Vg6b9KTuu6y5/OmzMB8GA1Ud
IwQYMBaAFGU5/OVYAsw5Vg6b9KTuu6y5/OmzMAwGA1UdEwQFMAMBAf8wDQYJKoZI
hvcNAQELBQADggEBAKnim+IdTeDy7KmWZxAyj2qGyz/cSK2dqkYU1iLcc492mXFU
RtD+ZTI7zGOFfZ1i7TIX7+Or2SjJ1EeCBUJLVt0nHnESWQR7TlTn03wFwLyf95Bd
3e+OqDUdj3DhWp1bfb0JIbWBA6nLBNLOjgCjpV/X8m9o0+3E6FV/zjbjUNlpZXra
Gwmi839Ko+9KX/44tTgLMQKB34H28k4HBnunnD/GUImXYchzeSnlmFpheKQ0/MVM
LVDSX3BZFPpImmmaqithUOT+MRRfQL/MRpVqLy1oja5RVpP+kKPZxo2p9wn3heyP
PjaC4NSkV6E4hdddOkVIz01jO9Bxse9aCpfPo34=
-----END CERTIFICATE-----
---
Server certificate
subject=/OU=orchsym.com/CN=runtime-1.runtime-statefulset.default.svc.cluster.local
issuer=/OU=orchsym.com/CN=ca
---
Acceptable client certificate CA names
/OU=orchsym.com/CN=ca
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2290 bytes and written 483 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-SHA384
Session-ID: 659D7F31DA4A0985B0E70BC8EBF9000310D5D5959F18ADB88E42283E98010508
Session-ID-ctx:
Master-Key: 0E9CE0E6F358A489908FA748D77876B1A66B6D8FDF9BC906BEC55442700D0A59EBF62AED6A88D42FD4FF4A375BBE1438
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1704820529
Timeout : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
---
Created 01-09-2024 09:17 AM
I've looked at its output via openssl and it indicates a self-signed certificate for the grant.
The output I fed back above. I don't know if you have viewed any errors that I haven't noticed.
Created 01-11-2024 01:22 PM
@JamesZhang
Certainly a challenging issue you have here. The shared output all points to good certificates, but gets you no closer to why the mutualTLS exchange between your two Nifi nodes is no yielding a successful mutual TLS handshake.
I guess I would start by looking at the configuration of NiFi on both nodes to make sure configurations in the nifi.properties files on both nodes match. Verify that both nodes NiFi's are using same Java version. You may need to look at the network traffic between both nodes as well. Is there some device (load balancer, firewall, etc) between those nodes on the network that may be interfering with the certificate exchange.
Matt