Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Deadlock Problems in Spark Structured Streaming writing to Teradata using JDBC Driver

Highlighted

Deadlock Problems in Spark Structured Streaming writing to Teradata using JDBC Driver

New Contributor

Hi everyone,

We are facing some difficulties:

Application description: Spark subscribes to Kafka topic, flattens out Json content and writes it using JDBC driver to Teradata Table. The Teradata Table is queried by a Talend Job that transfers the data to a DWH 3N model.

Problem: There are multiple deadlock problems in the Spark App. Is there any best practice or technical solution to tackle this problem?

Software components:

  • Apache Spark 2.2.0
  • Teradata 16.20

Kafka Topic has 5 partitions.

Any comment will be appreciated.