question files transer(.csv) to Unix sever from CDSW in Support Questions

files transer(.csv) to Unix sever from CDSW

krishna2023 — Mon, 29 May 2023 10:58:27 GMT

Hello all,

How to transfer .csv or txt file from CDSW environment to Unix server using sftp?

Or do we have any inbuild service to transfer files from CDSW to Unix?

- Sharing public key to UNIX server

- run sftp scripts to trassfer using python or unix script

How to enable sftp protocol?

Re: files transer(.csv) to Unix sever from CDSW

DianaTorres — Mon, 29 May 2023 12:09:48 GMT

@krishna2023 Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our CDSW experts @Gopinath and @Mike who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: files transer(.csv) to Unix sever from CDSW

Mike — Tue, 30 May 2023 13:39:42 GMT

Hi, do you have access to the actual CDSW host environment on the CDSW master host? If so, you will be ablet o use sftp - the project files will be located in /var/lib/cdsw/current/projects/projects/[number]/[number] where the first [number] is usually 0 and the next number will be the id of your project. You can find what this ID is by running something like:

kubectl exec -it $(kubectl get pods -l role=db -o jsonpath='{.items[*].metadata.name}') -- psql -P pager=off --expanded -U sense -c "select id from projects where name='my_test_project'"

Another way you could transfer files from the CDSW project would be to simply open the project, start a session, open the _Terminal, and run:

python -m http.server 8080 --bind 127.0.0.1

This will start the SimpleHTTPServer. Let this run, and go back to the project. Where the little 9 dot menu is, there will be a new option to view the directory listing:

You can just click this, or manually go to the generated URL (https://public-9hcxj43aavpxk7gt.cdsw.company.com/ in your browser (where the public- is required, and the next chunk is the user session) or use wget or something to download files.

There are probably a lot of other ways to pull files directly from CDSW, but these two methods should get you started on whatever you are trying to do.

Re: files transer(.csv) to Unix sever from CDSW

krishna2023 — Thu, 01 Jun 2023 05:19:29 GMT

Thank you for reply,I will keep update with you.

Re: files transer(.csv) to Unix sever from CDSW

DianaTorres — Mon, 05 Jun 2023 13:28:52 GMT

@krishna2023 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

Re: files transer(.csv) to Unix sever from CDSW

krishna2023 — Thu, 15 Jun 2023 08:11:09 GMT

This is not fitting to us.

Re: files transer(.csv) to Unix sever from CDSW

krishna2023 — Thu, 15 Jun 2023 08:11:35 GMT

It's not fixed.

Re: files transer(.csv) to Unix sever from CDSW

BubbaBoBobBrain — Fri, 20 Oct 2023 18:53:30 GMT

Hi Krishna

We have done this by pushing commands out to the shell after setting up a trusted SSH connection between CDSW and the Unix server.

This is the python function we use:

user_name = "username" unix_server = "my.unix.host" unix_path = "/some/path" file_to_transfer = "my_csv_file.csv" def scp_file_to_sas(local_path, file_name, user_name, unix_server, unix_path): p = subprocess.Popen( [ "scp", "-v", local_path + file_name, user_name + "@" + unix_server + ":/" + unix_path + "/" + file_name, ] ) sts = os.waitpid(p.pid, 0)