Hello, I am running this from the company network and I believe we have some sort of certificate for using cloudera-impala. When I copy the URL from the impala_prod it gives me at the end also a uid(which is my ID) and a password which is a standard password(not given by me at any point in time).    So when I run this script this is the error I recieve: java.sql.SQLException: java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](500170) Error occurred while setting up ALTUS Dynamic Discovery: Unable to load credentials from provider files.   Do you have any ideas how can I fix this? ... View more

@cr @PowerofAI You might have to make sue that Impala packages is installed and then import the UDF something like this may be:    import pandas as pdfrom pyspark.sql.functions import pandas_udf, PandasUDFType from pyspark.sql import Window df = spark.createDataFrame( [(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)], ("id", "v")) @pandas_udf("double", PandasUDFType.GROUPED_AGG) def pandas_mean(v): return v.sum() df.select(pandas_mean(df['v'])).show()df.groupby("id").agg(pandas_mean(df['v'])).show()df.select(pandas_mean(df['v']).over(Window.partitionBy('id'))).show() Also we have to make sure that  pip install ibis-framework pip install imapala is there that might causing the issue.  ... View more