Error: schema registry fails to start if user is changed to anything other than default

[#1]

Hello,

I have been trying a fresh install of HDF (latest ambari and HDF), the schema registry service fails to start however.

I have had to change the service account to something other than "registry" as that user is already taken by someone in my ldap.

However the followoing error occurs:

(i have even tried setting the 2 directories to 777 permissions but still i get the error, even when running the script as root -- i am assuming that somewhere in the script there is a line calling the command as the original "registry" user but i can't find where)

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/REGISTRY/0.3.0/package/scripts/registry_server.py", line 193, in <module>
    RegistryServer().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/REGISTRY/0.3.0/package/scripts/registry_server.py", line 120, in start
    not_if=no_op_test
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
    returns=self.resource.returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'source /usr/hdf/current/registry/conf/registry-env.sh ; /usr/hdf/current/registry/bin/registry start' returned 1. Starting Registry ksh: /var/run/registry/registry.pid: cannot create [Permission denied]
ksh: /var/log/registry/registry.out: cannot create [Permission denied]
failed.

[#2]

@Lorenc Hysi

Just wondaring if you are using "ksh" shell (instead of Bash) ?

Ideally the following following directories should get created as soon as the registry process starts.

# ls -ld /var/run/registry
drwxr-xr-x 2 registry hadoop 60 Sep 18 05:27 /var/run/registry

# ls -ld /var/log/registry
drwxr-xr-x 2 registry hadoop 66 Sep 18 05:22 /var/log/registry

# ls -ls /var/run/registry/registry.pid 
4 -rw-r--r-- 1 registry hadoop 6 Sep 18 05:27 /var/run/registry/registry.pid

# ls -ls /var/log/registry/registry.log 
12 -rw-r--r-- 1 registry hadoop 9808 Sep 18 05:27 /var/log/registry/registry.log

- Also Just to isolate the issue can you please try changing the following directories from Ambari UI to something else and then try again:

Ambari UI --> Registry --> Configs --> Advanced --> Advanced registry-env

export PID_DIR={{registry_pid_dir}}
export LOG_DIR={{registry_log_dir}}

.

Try changing the Directories to something else outside "/var" and then see if it works for you. This may not be a solution but it will help in isolating the issue if it is specific to the Path "/var/run/registry/" and "/var/log/registry/" on your host OR if it is due to some other issue.

[#3]

So to clarify, the default shell is bash, the folders are created:

[root]# ls -ld /var/run/registry
drwxr-xr-x. 2 sc_registry hadoop 40 Sep 17 15:17 /var/run/registry
[root]# ls -ld /var/log/registry
drwxr-xr-x. 2 sc_registry hadoop 6 Sep 17 15:17 /var/log/registry
[root]#

I ended up changing the the PID and LOG DIR and put them in /tmp which is very permissive and still got a permission denied error.

[root@siem-hdo-ewa00m tmp]# ls -ld /tmp/registry
drwxr-xr-x. 2 sc_registry hadoop 6 Sep 18 07:40 /tmp/registry
[root@siem-hdo-ewa00m tmp]#

Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 982, in restart
    self.status(env)
  File "/var/lib/ambari-agent/cache/common-services/REGISTRY/0.3.0/package/scripts/registry_server.py", line 146, in status
    check_process_status(status_params.registry_pid_file)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status
    raise ComponentIsNotRunning()
ComponentIsNotRunning

The above exception was the cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/REGISTRY/0.3.0/package/scripts/registry_server.py", line 193, in <module>
    RegistryServer().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute
    method(env)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 993, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/REGISTRY/0.3.0/package/scripts/registry_server.py", line 120, in start
    not_if=no_op_test
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
    returns=self.resource.returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'source /usr/hdf/current/registry/conf/registry-env.sh ; /usr/hdf/current/registry/bin/registry start' returned 1. Starting Registry ksh: /tmp/registry/registry.pid: cannot create [Permission denied]
ksh: /tmp/registry/registry.out: cannot create [Permission denied]
failed.

I ran in to a similar issue with SAM when I changed the streamline user to something else. I changed it back to streamline and it doesn't have the same error of permission denied (granted I cant finish SAm because it needs schema registry which is where it fails).

This leads me to believe that no matter what we change the user to there are scripts being run at the "default" username which is causing the permissison denied, because another user with that same default username exists in the environment but that user is not part of Hadoop group or has any permissions on this server he just exists in LDAP which conflicts.

But that is just an assumption on my part.

[#4]

So to clarify, the default shell is bash, the folders are created:

[root]# ls -ld /var/run/registry
drwxr-xr-x. 2 sc_registry hadoop 40 Sep 17 15:17 /var/run/registry
[root]# ls -ld /var/log/registry
drwxr-xr-x. 2 sc_registry hadoop 6 Sep 17 15:17 /var/log/registry
[root]#

I ended up changing the the PID and LOG DIR and put them in /tmp which is very permissive and still got a permission denied error.

[root@siem-hdo-ewa00m tmp]# ls -ld /tmp/registry
drwxr-xr-x. 2 sc_registry hadoop 6 Sep 18 07:40 /tmp/registry
[root@siem-hdo-ewa00m tmp]#

Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 982, in restart
    self.status(env)
  File "/var/lib/ambari-agent/cache/common-services/REGISTRY/0.3.0/package/scripts/registry_server.py", line 146, in status
    check_process_status(status_params.registry_pid_file)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status
    raise ComponentIsNotRunning()
ComponentIsNotRunning

The above exception was the cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/REGISTRY/0.3.0/package/scripts/registry_server.py", line 193, in <module>
    RegistryServer().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute
    method(env)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 993, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/REGISTRY/0.3.0/package/scripts/registry_server.py", line 120, in start
    not_if=no_op_test
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
    returns=self.resource.returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'source /usr/hdf/current/registry/conf/registry-env.sh ; /usr/hdf/current/registry/bin/registry start' returned 1. Starting Registry ksh: /tmp/registry/registry.pid: cannot create [Permission denied]
ksh: /tmp/registry/registry.out: cannot create [Permission denied]
failed.

I ran in to a similar issue with SAM when I changed the streamline user to something else. I changed it back to streamline and it doesn't have the same error of permission denied (granted I cant finish SAm because it needs schema registry which is where it fails).

This leads me to believe that no matter what we change the user to there are scripts being run at the "default" username which is causing the permissison denied, because another user with that same default username exists in the environment but that user is not part of Hadoop group or has any permissions on this server he just exists in LDAP which conflicts.

But that is just an assumption on my part.

[#5]

@Jay Kumar SenSharma

Some more updates, which leads me more to believe there is something hardcoded somewhere in the schema registry scripts that is preventing it to start if the user is changed from the default. Can someone please point out some some of the files and locations that could be the culprit so i can start walking the code at least.

I reinstalled the whole HDF cluster and made sure to change the pid and log dirs from the very beginning:

yet when i try to start it you can see in the text that the pid dir it is trying to write the pid file to is still the default location /var/run/registry

stderr: /var/lib/ambari-agent/data/errors-614.txt

Traceback (most recent call last):
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 982, in restart
    self.status(env)
  File "/var/lib/ambari-agent/cache/common-services/REGISTRY/0.3.0/package/scripts/registry_server.py", line 146, in status
    check_process_status(status_params.registry_pid_file)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status
    raise ComponentIsNotRunning()
ComponentIsNotRunning

The above exception was the cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/REGISTRY/0.3.0/package/scripts/registry_server.py", line 193, in <module>
    RegistryServer().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute
    method(env)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 993, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/REGISTRY/0.3.0/package/scripts/registry_server.py", line 120, in start
    not_if=no_op_test
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
    returns=self.resource.returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'source /usr/hdf/current/registry/conf/registry-env.sh ; /usr/hdf/current/registry/bin/registry start' returned 1. Starting Registry ksh: /var/run/registry/registry.pid: cannot create [Permission denied]
ksh: /data/registry/registry.out: cannot create [Permission denied]
failed.

stdout: /var/lib/ambari-agent/data/output-614.txt

2018-09-25 14:44:33,313 - Stack Feature Version Info: Cluster Stack=3.2, Command Stack=None, Command Version=3.2.0.0-520 -> 3.2.0.0-520
2018-09-25 14:44:33,520 - Stack Feature Version Info: Cluster Stack=3.2, Command Stack=None, Command Version=3.2.0.0-520 -> 3.2.0.0-520
2018-09-25 14:44:33,525 - Group['ranger'] {}
2018-09-25 14:44:33,526 - Group['nifiregistry'] {}
2018-09-25 14:44:33,526 - Group['hadoop'] {}
2018-09-25 14:44:33,526 - Group['nifi'] {}
2018-09-25 14:44:33,527 - User['streamline'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-09-25 14:44:33,530 - User['logsearch'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-09-25 14:44:33,531 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-09-25 14:44:33,532 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-09-25 14:44:33,533 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-09-25 14:44:33,534 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-09-25 14:44:33,535 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-09-25 14:44:33,536 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['ranger', 'hadoop'], 'uid': None}
2018-09-25 14:44:33,537 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-09-25 14:44:33,538 - User['nifiregistry'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifiregistry'], 'uid': None}
2018-09-25 14:44:33,539 - User['nifi'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifi'], 'uid': None}
2018-09-25 14:44:33,541 - User['scregistry'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-09-25 14:44:33,541 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-09-25 14:44:33,543 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-09-25 14:44:33,550 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2018-09-25 14:44:33,567 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2018-09-25 14:44:33,578 - Skipping Execute[('setenforce', '0')] due to only_if
2018-09-25 14:44:33,578 - Skipping unlimited key JCE policy check and setup since the Java VM is not managed by Ambari
2018-09-25 14:44:33,964 - Stack Feature Version Info: Cluster Stack=3.2, Command Stack=None, Command Version=3.2.0.0-520 -> 3.2.0.0-520
2018-09-25 14:44:33,978 - Directory['/data/registry'] {'group': 'hadoop', 'cd_access': 'a', 'create_parents': True, 'recursive_ownership': True, 'owner': 'scregistry', 'mode': 0755}
2018-09-25 14:44:33,981 - Directory['/data/registry'] {'group': 'hadoop', 'cd_access': 'a', 'create_parents': True, 'recursive_ownership': True, 'owner': 'scregistry', 'mode': 0755}
2018-09-25 14:44:33,981 - Directory['/usr/hdf/current/registry/conf'] {'group': 'hadoop', 'cd_access': 'a', 'create_parents': True, 'recursive_ownership': True, 'owner': 'scregistry', 'mode': 0755}
2018-09-25 14:44:33,982 - Directory['/var/lib/ambari-agent/data/registry'] {'group': 'hadoop', 'cd_access': 'a', 'create_parents': True, 'mode': 0755, 'owner': 'scregistry', 'recursive_ownership': True}
2018-09-25 14:44:33,983 - Execute['source /usr/hdf/current/registry/conf/registry-env.sh; /usr/hdf/current/registry/bin/registry stop'] {'user': 'scregistry'}
2018-09-25 14:44:34,071 - File['/data/registry/registry.pid'] {'action': ['delete']}
2018-09-25 14:44:34,072 - Deleting File['/data/registry/registry.pid']
2018-09-25 14:44:34,072 - Pid file /data/registry/registry.pid is empty or does not exist
2018-09-25 14:44:34,074 - Directory['/data/registry'] {'group': 'hadoop', 'cd_access': 'a', 'create_parents': True, 'recursive_ownership': True, 'owner': 'scregistry', 'mode': 0755}
2018-09-25 14:44:34,075 - Directory['/data/registry'] {'group': 'hadoop', 'cd_access': 'a', 'create_parents': True, 'recursive_ownership': True, 'owner': 'scregistry', 'mode': 0755}
2018-09-25 14:44:34,075 - Directory['/usr/hdf/current/registry/conf'] {'group': 'hadoop', 'cd_access': 'a', 'create_parents': True, 'recursive_ownership': True, 'owner': 'scregistry', 'mode': 0755}
2018-09-25 14:44:34,076 - Directory['/var/lib/ambari-agent/data/registry'] {'group': 'hadoop', 'cd_access': 'a', 'create_parents': True, 'mode': 0755, 'owner': 'scregistry', 'recursive_ownership': True}
2018-09-25 14:44:34,081 - File['/usr/hdf/current/registry/conf/registry-env.sh'] {'content': InlineTemplate(...), 'owner': 'scregistry'}
2018-09-25 14:44:34,082 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'}
2018-09-25 14:44:34,082 - Directory['/hdf/registry'] {'owner': 'scregistry', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2018-09-25 14:44:34,085 - File['/etc/security/limits.d/registry.conf'] {'content': Template('registry.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2018-09-25 14:44:34,096 - File['/usr/hdf/current/registry/conf/registry.yaml'] {'owner': 'scregistry', 'content': Template('registry.yaml.j2'), 'group': 'hadoop', 'mode': 0644}
2018-09-25 14:44:34,097 - Writing File['/usr/hdf/current/registry/conf/registry.yaml'] because contents don't match
2018-09-25 14:44:34,099 - Directory['/usr/lib/ambari-logsearch-logfeeder/conf'] {'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2018-09-25 14:44:34,099 - Generate Log Feeder config file: /usr/lib/ambari-logsearch-logfeeder/conf/input.config-registry.json
2018-09-25 14:44:34,099 - File['/usr/lib/ambari-logsearch-logfeeder/conf/input.config-registry.json'] {'content': Template('input.config-registry.json.j2'), 'mode': 0644}
2018-09-25 14:44:34,100 - File['/usr/lib/ambari-agent/DBConnectionVerification.jar'] {'content': DownloadSource('http://siem-hdo-ewa00m.cs.boeing.com:8080/resources/DBConnectionVerification.jar')}
2018-09-25 14:44:34,100 - Not downloading the file from http://siem-hdo-ewa00m.cs.boeing.com:8080/resources/DBConnectionVerification.jar, because /var/lib/ambari-agent/tmp/DBConnectionVerification.jar already exists
2018-09-25 14:44:34,101 - Execute['export JAVA_HOME=/usr/java/latest ; source /usr/hdf/current/registry/conf/registry-env.sh ; /usr/hdf/current/registry/bootstrap/bootstrap-storage.sh migrate'] {'user': 'root'}
2018-09-25 14:44:35,211 - Execute['source /usr/hdf/current/registry/conf/registry-env.sh ; /usr/hdf/current/registry/bin/registry start'] {'not_if': 'ls /data/registry/registry.pid >/dev/null 2>&1 && ps -p `cat /data/registry/registry.pid` >/dev/null 2>&1', 'user': 'root'}
2018-09-25 14:44:38,397 - Execute['find /data/registry -maxdepth 1 -type f -name '*' -exec echo '==> {} <==' \; -exec tail -n 40 {} \;'] {'logoutput': True, 'ignore_failures': True, 'user': 'scregistry'}
==> /data/registry/registry.out <==

==> /data/registry/registry.err <==
 at  at ch.qos.logback.core.recovery.ResilientFileOutputStream.<init>(ResilientFileOutputStream.java:26)
 at  at ch.qos.logback.core.FileAppender.openFile(FileAppender.java:204)
 at  at ch.qos.logback.core.FileAppender.start(FileAppender.java:127)
 at  at ch.qos.logback.core.rolling.RollingFileAppender.start(RollingFileAppender.java:100)
 at  at io.dropwizard.logging.FileAppenderFactory.build(FileAppenderFactory.java:233)
 at  at io.dropwizard.logging.DefaultLoggingFactory.configure(DefaultLoggingFactory.java:143)
 at  at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:81)
 at  at io.dropwizard.cli.Cli.run(Cli.java:75)
 at  at io.dropwizard.Application.run(Application.java:93)
 at  at com.hortonworks.registries.webservice.RegistryApplication.main(RegistryApplication.java:296)
11:19:22,634 |-INFO in ch.qos.logback.classic.AsyncAppender[async-file-appender] - Attaching appender named [file-appender] to AsyncAppender.
11:19:22,634 |-INFO in ch.qos.logback.classic.AsyncAppender[async-file-appender] - Setting discardingThreshold to 51

java.lang.RuntimeException: java.lang.ClassNotFoundException: com.mysql.jdbc.jdbc2.optional.MysqlDataSource
 at com.zaxxer.hikari.util.PoolUtilities.createInstance(PoolUtilities.java:141)
 at com.zaxxer.hikari.util.PoolUtilities.initializeDataSource(PoolUtilities.java:153)
 at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:146)
 at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:113)
 at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:78)
 at com.hortonworks.registries.storage.impl.jdbc.connection.HikariCPConnectionBuilder.prepare(HikariCPConnectionBuilder.java:49)
 at com.hortonworks.registries.storage.impl.jdbc.connection.HikariCPConnectionBuilder.<init>(HikariCPConnectionBuilder.java:38)
 at com.hortonworks.registries.storage.impl.jdbc.provider.QueryExecutorFactory.getHikariCPConnnectionBuilder(QueryExecutorFactory.java:78)
 at com.hortonworks.registries.storage.impl.jdbc.provider.QueryExecutorFactory.get(QueryExecutorFactory.java:44)
 at com.hortonworks.registries.storage.impl.jdbc.JdbcStorageManager.init(JdbcStorageManager.java:240)
 at com.hortonworks.registries.webservice.RegistryApplication.getStorageManager(RegistryApplication.java:268)
 at com.hortonworks.registries.webservice.RegistryApplication.registerResources(RegistryApplication.java:184)
 at com.hortonworks.registries.webservice.RegistryApplication.run(RegistryApplication.java:80)
 at com.hortonworks.registries.webservice.RegistryApplication.run(RegistryApplication.java:66)
 at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43)
 at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:85)
 at io.dropwizard.cli.Cli.run(Cli.java:75)
 at io.dropwizard.Application.run(Application.java:93)
 at com.hortonworks.registries.webservice.RegistryApplication.main(RegistryApplication.java:296)
Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.jdbc2.optional.MysqlDataSource
 at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
 at com.zaxxer.hikari.util.PoolUtilities.createInstance(PoolUtilities.java:126)
 ... 18 more

Command failed after 1 tries

[#6]

Can u check for any hung process?

ps -ef | grep registry

ps -ef | grep nifi

You can also create the empty pid file with right permissions for the service account.

[#7]

@Pranay Vyas

so there are no hung processes. I also crated the /var/run/registry/registry.pid file myself put 777 permission on it. But the file and the whole var/run/registry folder get recreated everytime the service is started so the pid file no longer gets deleted and no longer exists and we are back to the same error.

Can someone please point me to the specific file and or piece of code that would create the /var/run/registry folder and pid file?

[#8]

Here is your culprit. as i have stated from he beginning this looked like a hard coded variable issue. usually the simplest answer is the right answer.

In the file /usr/hdf/current/registry/bin/registry you have this line of code:

# User Name for setup parameter

[ -z $REGISTRY_USER ] && REGISTRY_USER=registry

As you can see the registry user is hardcoded as registry -- so no matter if you change the user during the amabri setup it will always run as "registry" user and the permission denieds will follow.

Please send this to the appropriate team as a bug for whoever looks at these.

For anyone else in the meantime please change that value to be your actual user.

Another bug as well is with the /usr/hdf/current/registry/conf/registry-env.sh file.

in there the PID direcotry will not be set, no matter the value you put in -- therefore there are discrepencies and the PID directory is not available to all scripts.

Enjoy!