Additional Information: As we can see, even though there are 3 stages active, only 1 task each is running in Production as well as Default pools.    My basic question is - how can we increase the parallelism within pools?  In other words, how can I make sure that the Stage ID "8" in the above screenshot also runs in parallel with the other 2    Thanks and Regards, Sudhindra ... View more

Hi @senthh,    Thanks a lot for your reply.  I have monitored the Spark Streaming logs and verified that the connection to broker is established correctly.    Given below are the logs confirming the same:   The interesting thing is the same Spark Streaming job is working outside Kubernetes setup, without any issues.    Please help!!   9/12/23 15:56:59.562 INFO AppInfoParser: Kafka version: 2.2.1 19/12/23 15:56:59.563 INFO AppInfoParser: Kafka commitId: 55783d3133a5a49a 19/12/23 15:56:59.566 DEBUG KafkaConsumer: [Consumer clientId=consumer-1, groupId=test] Kafka consumer initialized 19/12/23 15:56:59.569 INFO KafkaConsumer: [Consumer clientId=consumer-1, groupId=test] Subscribed to partition(s): table-update-0 19/12/23 15:56:59.593 DEBUG NetworkClient: [Consumer clientId=consumer-1, groupId=test] Initialize connection to node 10.20.0.44:29092 (id: -1 rack: null) for sending metadata request 19/12/23 15:56:59.596 DEBUG NetworkClient: [Consumer clientId=consumer-1, groupId=test] Initiating connection to node 10.20.0.44:29092 (id: -1 rack: null) using address /10.20.0.44 19/12/23 15:56:59.640 DEBUG Metrics: Added sensor with name node--1.bytes-sent 19/12/23 15:56:59.642 DEBUG Metrics: Added sensor with name node--1.bytes-received 19/12/23 15:56:59.642 DEBUG Metrics: Added sensor with name node--1.latency 19/12/23 15:56:59.643 DEBUG Selector: [Consumer clientId=consumer-1, groupId=test] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1 19/12/23 15:56:59.928 DEBUG NetworkClient: [Consumer clientId=consumer-1, groupId=test] Completed connection to node -1. Fetching API versions. 19/12/23 15:56:59.929 DEBUG NetworkClient: [Consumer clientId=consumer-1, groupId=test] Initiating API versions fetch from node -1. 19/12/23 15:56:59.962 DEBUG NetworkClient: [Consumer clientId=consumer-1, groupId=test] Recorded API versions for node -1: (Produce(0): 0 to 7 [usable: 7], Fetch(1): 0 to 10 [usable: 10], ListOffsets(2): 0 to 5 [usable: 5], Metadata(3): 0 to 7 [usable: 7], LeaderAndIsr(4): 0 to 2 [usable: 2], StopReplica(5): 0 to 1 [usable: 1], UpdateMetadata(6): 0 to 5 [usable: 5], ControlledShutdown(7): 0 to 2 [usable: 2], OffsetCommit(8): 0 to 6 [usable: 6], OffsetFetch(9): 0 to 5 [usable: 5], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): ... View more