Deadlock Problems in Spark Structured Streaming writing to Teradata using JDBC Driver

[#1]

Hi everyone,

We are facing some difficulties:

Application description: Spark subscribes to Kafka topic, flattens out Json content and writes it using JDBC driver to Teradata Table. The Teradata Table is queried by a Talend Job that transfers the data to a DWH 3N model.

Problem: There are multiple deadlock problems in the Spark App. Is there any best practice or technical solution to tackle this problem?

Software components:

• Apache Spark 2.2.0
• Teradata 16.20

Kafka Topic has 5 partitions.

Any comment will be appreciated.