files transer(.csv) to Unix sever from CDSW

[#1]

Hello all,

How to transfer .csv or txt file from CDSW environment to Unix server using sftp?

Or do we have any inbuild service to transfer files from CDSW to Unix? 

 - Sharing public key to UNIX server

- run sftp scripts to trassfer using python or unix script

How to enable sftp protocol?  

[#2]

@krishna2023 Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our CDSW experts @Gopinath and @Mike  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

Hi, do you have access to the actual CDSW host environment on the CDSW master host? If so, you will be ablet o use sftp - the project files will be located in /var/lib/cdsw/current/projects/projects/[number]/[number] where the first [number] is usually 0 and the next number will be the id of your project. You can find what this ID is by running something like:

kubectl exec -it $(kubectl get pods -l role=db -o jsonpath='{.items[*].metadata.name}') -- psql -P pager=off --expanded -U sense -c "select id from projects where name='my_test_project'"

Another way you could transfer files from the CDSW project would be to simply open the project, start a session, open the _Terminal, and run:

python -m http.server 8080 --bind 127.0.0.1

This will start the SimpleHTTPServer.  Let this run, and go back to the project. Where the little 9 dot menu is, there will be a new option to view the directory listing:

You can just click this, or manually go to the generated URL (https://public-9hcxj43aavpxk7gt.cdsw.company.com/ in your browser (where the public- is required, and the next chunk is the user session) or use wget or something to download files.

There are probably a lot of other ways to pull files directly from CDSW, but these two methods should get you started on whatever you are trying to do.

[#4]

Thank you for reply,I will keep update with you.

[#5]

This is not fitting to us. 

[#6]

@krishna2023 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

[#7]

It's not fixed.

[#8]

Hi Krishna

We have done this by pushing commands out to the shell after setting up a trusted SSH connection between CDSW and the Unix server.  

This is the python function we use:

user_name = "username"
unix_server = "my.unix.host"
unix_path = "/some/path"
file_to_transfer = "my_csv_file.csv"

def scp_file_to_sas(local_path, file_name, user_name, unix_server, unix_path):

    p = subprocess.Popen(
        [
            "scp",
            "-v",
            local_path + file_name,
            user_name + "@" + unix_server + ":/" + unix_path + "/" + file_name,
        ]
    )
    sts = os.waitpid(p.pid, 0)