Default configurations are the tested and for that reason are set in that way, but those are configurable for a reason, sometimes, and depending on the environment, the use case and much more, they need to be tuned in an specific way.
... View more
If you're using Conda Create the environment conda create -n pyspark_env python=3.9 numpy Activate it conda activate pyspark_env Tell Spark to use it export PYSPARK_PYTHON=$(which python) export PYSPARK_DRIVER_PYTHON=$(which python)
... View more
@LSIMS PageWriter.java#L99 is where your operation is failing We need simple steps to reproduce the same error on a brand new environment to investigate further
... View more
@LSIMS As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks.
... View more
@LSIMS you mentioned it is occasional , does it mean that it is failing only on few nodes ? Can you check with Informatica team on how to pass the kerberos keytab cerds .. I found this Informatica Article on passing the keytab details for spark+kafka setup . https://docs.informatica.com/data-engineering/data-engineering-integration/10-2-2/big-data-management-administrator-guide/connections/configuring-hadoop-connection-properties/spark-advanced-properties.html
... View more
@LSIMS As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks.
... View more
