Hi Chris,   You can use the SparkSession object to query your Hive tables. You don't need to create a HiveContext.   spark.catalog.listTables.show() spark.sql("describe database default").show spark.sql("describe formatted default.mytable").show   I don't think that this change would fix your issue.   This doesn't seem like a problem with your CDSW installation. Could you execute the same commands from a Session terminal or directly on the CDSW host machine using the spark2-shell cli tool to verify this?   Could you double check that a Hive gateway is deployed to your CDSW host? Can you show that the table exists from eg. beeline?   Regards, Peter   ... View more

Hello Chris,   What are you trying to achieve? Why are you trying to run in local mode? Can you use the expected yarn client mode instead?   From the documentation: "Unlike PySpark or Sparklyr, you can access a SparkContext assigned to the spark (SparkSession) and sc(SparkContext) objects on console startup, just as when using the Spark shell." https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_spark_and_scala.html   Can you access to your hive tables using the spark SparkSession object?   Regards, Peter ... View more

Hi NES,   I'm glad that your issue is resolved! Could you let us know what was the issue for the other topic followers?   Thanks, Peter ... View more

Hi NES,   I see that the web pods are stuck in ContainerCreating state.  I would try to submit a kubectl describe command on one of the web pods and check if we have any associated events.  $ kubectl describe pod web-3826671331-5b7wk   If there are no events I would check the kubelet logs, maybe it shows the reason why are we stuck: # get the kubelet logs in less $ journalctl -u kubelet # tail the kubelet logs $ journalctl -u kubelet -o cat -l -f   Regards, Peter ... View more

Hi NES,   Could you share the full 'cdsw status' output? I would like to see what system pods are not ready.   There is a db-migrate-XXXXX-xxxxx Kubernetes job which should make sure that the tables are created. When you submit the 'cdsw status' command, can you see this job in the output? Did you see a Completed status for this?   When you changed the cdsw configuration, did you submit a 'cdsw reset' before retrying the 'cdsw init'? If not, you will need to do a 'cdsw reset' and 'cdsw init' again.     'cdsw logs' generates a tar.gz file with different diagnostic information. If you want to validate your cluster, I recommend to use the 'cdsw validate' instead.   Thanks, Peter ... View more

Hi Krishna,   There is the true flag under the Stateful column, just it's not rendered nicely, please disregard the previous post. Could you try to restart your cdsw application and upload the `cdsw status` command output again? We shouldn't see multiple database pods there.   Thanks, Peter ... View more

Hi Krishna,   One issue we see is the master node is missing the stateful true flag. Something might went wrong with your installation. The code which would put the stateful tag on the node is using the `hostname` command output. Could you check if the `hostname` output matches the node name you see in the `cdsw status` output? I would start by doing `cdsw stop` and `cdsw start` on the master node and see if the stateful flag gets there. If this is not working you will probably need to do a `cdsw reset` and `cdsw init` for the master and workers also.   Regards, Peter ... View more