01-22-2018 01:05 PM
We are still getting familiar with CDSW. One thing I'm wondering is if someone knows any reasons why the Cloudera ODBC drivers are not immediately included in the base image?
We currently run our data science jobs on a linux edge node. Although Spark is useful, we still do a lot of data preparation in both R and Python with Impala (using ODBC - respectively the odbc and turbodbc packages).
I was hoping that the Impala ODBC driver would have been included in the base image. It does not look like that is the case. Unfortunately I also found out that you cannot install OS packages directly (no root access). Only option is to change/improve the base image and build a custom image.
Customized images is certainly useful, but it requires admin intervention. It feels a bit strange that is needed to deploy Cloudera software.
Similarly, the documentation states that it "currently" does not support customization of system packages that require root access. I am wondering if there is already a roadmap here and how allowing data scientists to install OS packages would work.
02-15-2018 08:01 PM - edited 02-15-2018 08:02 PM
I have the same issue here. Our enterprise data warehouse is in Teradata and the fastest way to communicate with that for pushing data out of simulations is to use ODBC drivers. However, I was surprised to find out that this isn't supported -- no root level access and an odbc driver manager is not pre-installed in the base image. If either iODBC or unixODBC come pre-installed this would be so helpful
02-15-2018 08:08 PM
The current best approach to install system packages is to use the custom engine functionality. This does require administration access however.
I have also filed a feature request internally to package the Impala ODBC drivers and ensure that the base ODBC drivers are part of the standard base image.
02-15-2018 08:18 PM
I'm not sure whether root access is necessary for the driver side, I am supposing it is. But I know that at least for the driver manager it is absolutely necessary.
I am not even sure who would have administration privileges, is there a way to file a ticket? And get support on this?
02-15-2018 08:29 PM
02-24-2018 05:07 AM
Tristan, thanks for adding it on your list.
JDBC is indeed something that users will be able to do without admin intervention. That should help us out in the meantime.
Within the R/tidyverse, the odbc package is becoming popular and although it still has few problems we are pushing users to use that approach.
Within the context of making Impala available to CDSW users, would it be possible to preconfigure the container impala-shell with the cluster info? It's a small thing and we can specify all the info when using it, but explaining all that to users takes focus and time away on getting things done.
In our case we will also use impala-shell in a small internal package to allow transfer of data from R back to an Impala table with a single function.