Skip to main content
blueb
blueb
Silver ⭐️
Aug 12, 2021 11:37 AM UTC
Solved

在启用kerberos的集群flink程序如何连接集群外未启用认证的kafka

我在一个启用了kerberos的CDH6.3集群里，提交一个flink1.9版本程序到yarn，需消费集群外一个0.8版本的未启用安全认证的kafka，需要如何设置他不使用安全的方式去连接这个集群外kafka的zookeeper呢？谢谢。

 

org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient - Client will use GSSAPI as SASL mechanism.
org.apache.flink.shaded.zookeeper.org.apache.zookeeper.ClientCnxn - Opening socket connection to server ark1.analysys.xyz/10.2.33.189:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
org.apache.flink.shaded.zookeeper.org.apache.zookeeper.client.ZooKeeperSaslClient - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state.
DennisJaheruddi
DennisJaheruddiAnswer
Platinum ⭐️
Aug 19, 2021 12:45 PM UTC

Mindset

Kafka is a popular message bus, which has many applications producing into it or consuming from it.

 

Though Flink is not limited to Kafka, the primary source and sink is Kafka.

 

Therefore, when looking for Flink-Kafka information, the best place is probably with your Flink supplier.

 

Practical

Though I do not fully understand the described security limitations, Cloudera has two ways to interact with flink.

1. With the SQL Streambuilder: A Graphical Interface to write SQL which automatically gets converted into a Flink job. When using the SQL streambuilder you only need to connect a cluster once, and then all your jobs can use it.

2. Directly with Flink: There are many examples available on connecting with Kafka, for example: https://docs.cloudera.com/csa/1.2.0/datastream-connectors/topics/csa-kafka.html

 

Further information

 

If this does not help, first of all make sure to use the right version of the software (It is always recommended to use the Cloudera distribution, which makes integration easier).

If you are already using the Cloudera distribution of Flink and need further assistance, please contact the Cloudera Support organization, as they can help you with more detailed information.

 

- Dennis Jaheruddin If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'.
View in thread

6 replies

VidyaSargur
Community Manager
VidyaSargur
Community Manager
Aug 12, 2021 12:13 PM UTC

English Translation of the issue: 

 

TITLE: How does the flink program of a kerberos-enabled cluster connect to Kafka that is not authenticated outside the cluster?

 

I am in a CDH6.3 cluster with kerberos enabled, and I submit a flink1.9 version program to yarn. I need to consume a 0.8 version of Kafka without security authentication outside the cluster. How to set it not to use a secure way to connect to this What about the zookeeper of Kafka outside the cluster? thanks.

Translate
Regards, Vidya Sargur Madabhushi| Senior Community Manager | Was your question answered? Make sure to mark the answer as the Best Answer. If you find a reply useful, say thanks by clicking on the thumbs up button.
DennisJaheruddi
Platinum ⭐️
DennisJaheruddiAnswer
Platinum ⭐️
Aug 19, 2021 12:45 PM UTC

Mindset

Kafka is a popular message bus, which has many applications producing into it or consuming from it.

 

Though Flink is not limited to Kafka, the primary source and sink is Kafka.

 

Therefore, when looking for Flink-Kafka information, the best place is probably with your Flink supplier.

 

Practical

Though I do not fully understand the described security limitations, Cloudera has two ways to interact with flink.

1. With the SQL Streambuilder: A Graphical Interface to write SQL which automatically gets converted into a Flink job. When using the SQL streambuilder you only need to connect a cluster once, and then all your jobs can use it.

2. Directly with Flink: There are many examples available on connecting with Kafka, for example: https://docs.cloudera.com/csa/1.2.0/datastream-connectors/topics/csa-kafka.html

 

Further information

 

If this does not help, first of all make sure to use the right version of the software (It is always recommended to use the Cloudera distribution, which makes integration easier).

If you are already using the Cloudera distribution of Flink and need further assistance, please contact the Cloudera Support organization, as they can help you with more detailed information.

 

Translate
- Dennis Jaheruddin If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'.
VidyaSargur
Community Manager
VidyaSargur
Community Manager
Aug 23, 2021 5:50 AM UTC

@blueb, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. 

Translate
Regards, Vidya Sargur Madabhushi| Senior Community Manager | Was your question answered? Make sure to mark the answer as the Best Answer. If you find a reply useful, say thanks by clicking on the thumbs up button.
Terms & ConditionsAccessibility statement