Skip to main content
KonstaKyriako
KonstaKyriako
Silver ⭐️
March 8, 2024
Question

CDSW Docker Daemon stuck in busy state

Following a rolling restart of the cluster, one CDSW Docker Daemon is stuck in "busy" state. I can't start or stop it. I checked in the server and docker or any other related process are not running. Due to the fact that this Docker Daemon runs on the server where the master and application services run, I cant start docker and k8, so the whole CDSW application is down. When I try to start all the services on that server: CDSW Docker Daemon, Docker master, Docker Application at the same time, I see in the logs that the Daemon starts but very soon received a signal to be terminated. This is something that I cant explain. Again, I checked for any background process which could be responsible for this but didn't find any. Most probably, the fact that the Daemon service is stuck in the "busy" state creates this problem.

    2 replies

    DianaTorres
    Community Manager
    DianaTorres
    Community Manager
    March 9, 2024

    @KonstaKyriako Welcome to the Cloudera Community!

    To help you get the best possible solution, I have tagged our CDP experts @venkatsambath @Gopinath @aakulov 
    Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

    Translate
    Regards, Diana Torres, Senior Community Moderator
      KonstaKyriako
      KonstaKyriakoAuthor
      Silver ⭐️
      March 9, 2024

      After checking the status of the underlying processes(the process for docker worker & the process which gracefully stopped docker), it was clear that all processes had been stopped. There were not any processes related to docker etc. that were running in the relevant host. Hence the "busy" state of the docker worker was false.

      I manually updated the state of the service in the underlying cloudera manager database and set it to "stopped" to reflect the actual status of the service similarly to the other services. After I applied that, cloudera manager reflected properly the docker daemon as stopped. Then, I was able to start the docker daemon and the whole CDSW service without any issues. The CDSW service is now fully operational and without any issues.

      Apparently, some hickup or glitch had prevented the update of the docker daemon state in the database that cloudera manager uses, after the daemon had been stopped. Please, let me know if that is a known bug and what may have caused the issue. Would it make sense to have a reset option for the state of the service in the UI?

      Translate
        Terms & ConditionsAccessibility statement