question Re: Local Data combined with HDFS in Support Questions

Local Data combined with HDFS

mpaulus — Fri, 16 Sep 2022 12:24:18 GMT

Hi All,

Just looking through the CDSW documentation and have found the following : If you want to create a new project around one or more data files on your computer, select the Localoption when creating the project.

We're looking at creating projects that combine Local data, data from HDFS, etc. Is this possible? Or can you only use local files in a project that's marked as 'Local'?

Thanks a lot.

Re: Local Data combined with HDFS

srowen — Mon, 16 Oct 2017 14:55:28 GMT

Really, this is just saying you can upload data at project creation time or later from your local computer to the local file system that the Python/R/Scala sessions see in their local file system.

Those jobs then see those local files as simple files, and can do what they like with them.

But you can also within the same program access whatever data you want, anywhere you want; you just need to write code that does so. Via Spark or whatever library you want you can also access whatever data sources you want, as well.

There is no either/or here.

Re: Local Data combined with HDFS

mpaulus — Tue, 17 Oct 2017 08:13:26 GMT

Hi, thanks for the information! This is exactly what I expected but just wanted to make sure.