Hello @intersoldi,  Thanks for reaching our community. Glad to have you here.  The error looks clear on this message:  2024-07-11T07:30:57,262 WARN com.snowflake.kafka.connector.internal.SnowflakeInternalStage: [kafka_snowflake_poc_2|task-0] [SF_KAFKA_CONNECTOR] uploadWithoutConnection encountered an exception:JDBC driver encountered IO error. Message: Encountered exception during upload: null. for filePath:kafka_snowflake_poc_2/existing_table1/0/1_1_1720683056775.json.gz in Storage:AZURE 2024-07-11T07:30:57,265 ERROR com.snowflake.kafka.connector.internal.InternalUtils: [kafka_snowflake_poc_2|task-0] [SF_KAFKA_CONNECTOR] Retry count:0 caught an exception for operation:UPLOAD_FILE_TO_INTERNAL_STAGE_NO_CONNECTION with message:[SF_KAFKA_CONNECTOR] Exception: Failed to execute cached put Error Code: 5018 Detail: Error in cached put command Message: JDBC driver encountered IO error. Message: Encountered exception during upload: null. Kafka on CDP have access to the Snowflake domain, but the put on Azure storage is failing.  Most likely there is a proxy or firewall that is not helping on this connection.  You can test with this command:  curl -v https://azure_name.blob.core.windows.net  If you have a proxy, try to add it to the KAFKA_OPTS with these options -Dhttps.proxyHost=<proxy_host> -Dhttps.proxyPort=<proxy_port>   Also, if you change the connector to Snowpipe Streaming method, that upload can be avoided and possible this issue gets solved.  To do that you can update the JSON for the connector with this setting:  "snowflake.ingestion.method": "SNOWPIPE_STREAMING"    Check those points and see if it helps.  ... View more

Spark lazily evaluates the workload. The actual execution happens when you write the output, usually. If it gets stuck there that means the workload is not tuned with respect to your infrastructure. Try checking the logs, most likely you need to tune your shuffle partition and/or repartition your data.🙂 ... View more