files transer(.csv) to Unix sever from CDSW

@krishna2023 Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our CDSW experts @Gopinath and @Mike  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Hi, do you have access to the actual CDSW host environment on the CDSW master host? If so, you will be ablet o use sftp - the project files will be located in /var/lib/cdsw/current/projects/projects/[number]/[number] where the first [number] is usually 0 and the next number will be the id of your project. You can find what this ID is by running something like:

kubectl exec -it $(kubectl get pods -l role=db -o jsonpath='{.items[*].metadata.name}') -- psql -P pager=off --expanded -U sense -c "select id from projects where name='my_test_project'"

Another way you could transfer files from the CDSW project would be to simply open the project, start a session, open the _Terminal, and run:

python -m http.server 8080 --bind 127.0.0.1

This will start the SimpleHTTPServer.  Let this run, and go back to the project. Where the little 9 dot menu is, there will be a new option to view the directory listing:

You can just click this, or manually go to the generated URL (https://public-9hcxj43aavpxk7gt.cdsw.company.com/ in your browser (where the public- is required, and the next chunk is the user session) or use wget or something to download files.

There are probably a lot of other ways to pull files directly from CDSW, but these two methods should get you started on whatever you are trying to do.

@krishna2023 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

It's not fixed.

Hi Krishna

We have done this by pushing commands out to the shell after setting up a trusted SSH connection between CDSW and the Unix server.  

This is the python function we use:

user_name = "username"
unix_server = "my.unix.host"
unix_path = "/some/path"
file_to_transfer = "my_csv_file.csv"

def scp_file_to_sas(local_path, file_name, user_name, unix_server, unix_path):

    p = subprocess.Popen(
        [
            "scp",
            "-v",
            local_path + file_name,
            user_name + "@" + unix_server + ":/" + unix_path + "/" + file_name,
        ]
    )
    sts = os.waitpid(p.pid, 0)