How to use docker bind mount directory in executestreamcommand processor nifi

[#1]

Hello everyone, I am new to docker and I have been researching and working with nifi recently. I have one query for my use case where I need to use code repositories that are outside the docker container (I have already bind them while docker run command) and I can access the folder via exec but I need to access these scripts in the executestreamcommand processor's working directory and command path fields. Even after putting the destination directories into the fields I am getting no such file or directory exists. So, please help me out.

[#2]

@rupeshh Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi expert @steven-matison   who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

Thanks! @DianaTorres . I will be waiting for @steven-matison to response.

[#4]

@rupeshh In order for nifi to be able to see the mounted directory or files within it, it needs to be properly owned to the same user that is running nifi.  For example:

chown nifi:nifi /some/path   

 Then nifi will be able to see the directories and files.

[#5]

Hello @steven-matison thanks for replying but I am getting permission denied everytime I am using chmod or chown in the docker shell and sudo does not work in the shell so I don't know how exactly I should change the ownership. My directory is /tmp/nifi/src/respository_1 and /tmp/nifi/src/respository_2 and repository_1 and repository_2 are already owned by nifi user but the /tmp/nifi/src all these 3 owned by root and I think I will have to change the ownership of each directory because in the docker run command I have mounted to the /tmp/nifi/src/repository_1 and /tmp/nifi/src/repository_2. Can you please help me out?

[#6]

I have created a new container with the new bind mount where I did not mount the repositories in any other directories.

docker run --name nifi2 -p 8443:8443 -d -e NIFI_WEB_HTTPS_PORT=8443 -v /path/to/host-directory/repository_1:/repository_1 -v /path/to/host-directory/repository_2:/repository_2 --privileged apache/nifi:latest

 And owner is the user nifi.

same goes for another repository, but I am still getting the same error.

and this is the error from bulletin board

I hope this makes it clear. Please let me know if you need anything else.

[#7 · Accepted solution]

@rupeshh Docker container are never fun for this and other reasons.   I still think you have missing permissions on the file.  I cannot see the ownership of the file listing, and i cannot see the path in the error.  At any rate, the error definitely suggests the processor does not see the file.     

One suggestion would be to use the nifi user and cli on docker to ls the directory and files.  If that user cannot see the files,  that would indicate the same issue the error states (directory path or file does not exist, or not seen due to permissions).

[#8]

Thanks @steven-matison I found out it was permission issue actually. I am able to access the directory now. A big thanks!

[#9]

Hi @rupeshh I am also facing a similar situation and wanted to clarify from you a few things. I see in the Configuration for the ExecuteStreamCommand you have referred to the venv inside the repository for the command path, does that mean you have the venv within the docker container? or are you referring to the one outside the docker container (in the host machine)?