After a cluster restart it worked ... will continue to trace why that happened ... ... View more

Thanks Neeraj, that worked. If I am now creating a table in hive, it has the owner "admin" and group "hdfs" on hdfs. As what I understand, my Ambari-Server is initialized by user "root" and group "root" What are the mechanics behind this user transfer and how can I manipulate it e.g. if want to have another owner/group than "admin/hdfs" on hdfs for my freshly created table? best regards, Rainer ... View more

OK ... here is where I am ... I changed my primary group of user nifi to nifi ... i stopped nifi through "nifi.sh stop" I cleared /var/log/nifi and /var/run/nifi when I start nifi now from ambari there is nothing in stderr and I get the following feedback in ambari: stderr: /var/lib/ambari-agent/data/errors-1806.txt None stdout: /var/lib/ambari-agent/data/output-1806.txt 2016-01-30 16:31:57,093 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485 2016-01-30 16:31:57,094 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0 2016-01-30 16:31:57,094 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1} 2016-01-30 16:31:57,115 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '') 2016-01-30 16:31:57,115 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False} 2016-01-30 16:31:57,135 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0') 2016-01-30 16:31:57,136 - Ensuring that hadoop has the correct symlink structure 2016-01-30 16:31:57,136 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2016-01-30 16:31:57,244 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.4.0-3485 2016-01-30 16:31:57,244 - Checking if need to create versioned conf dir /etc/hadoop/2.3.4.0-3485/0 2016-01-30 16:31:57,245 - call['conf-select create-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1} 2016-01-30 16:31:57,265 - call returned (1, '/etc/hadoop/2.3.4.0-3485/0 exist already', '') 2016-01-30 16:31:57,266 - checked_call['conf-select set-conf-dir --package hadoop --stack-version 2.3.4.0-3485 --conf-version 0'] {'logoutput': False, 'sudo': True, 'quiet': False} 2016-01-30 16:31:57,286 - checked_call returned (0, '/usr/hdp/2.3.4.0-3485/hadoop/conf -> /etc/hadoop/2.3.4.0-3485/0') 2016-01-30 16:31:57,287 - Ensuring that hadoop has the correct symlink structure 2016-01-30 16:31:57,287 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2016-01-30 16:31:57,288 - Group['hadoop'] {} 2016-01-30 16:31:57,289 - Group['nifi'] {} 2016-01-30 16:31:57,289 - Group['users'] {} 2016-01-30 16:31:57,289 - User['zookeeper'] {'gid': 'hadoop', 'groups': [u'hadoop']} 2016-01-30 16:31:57,290 - User['ams'] {'gid': 'hadoop', 'groups': [u'hadoop']} 2016-01-30 16:31:57,290 - User['ambari-qa'] {'gid': 'hadoop', 'groups': [u'users']} 2016-01-30 16:31:57,291 - User['kafka'] {'gid': 'hadoop', 'groups': [u'hadoop']} 2016-01-30 16:31:57,291 - User['hdfs'] {'gid': 'hadoop', 'groups': [u'hadoop']} 2016-01-30 16:31:57,292 - User['yarn'] {'gid': 'hadoop', 'groups': [u'hadoop']} 2016-01-30 16:31:57,292 - User['nifi'] {'gid': 'hadoop', 'groups': [u'hadoop']} 2016-01-30 16:31:57,293 - User['mapred'] {'gid': 'hadoop', 'groups': [u'hadoop']} 2016-01-30 16:31:57,293 - User['hbase'] {'gid': 'hadoop', 'groups': [u'hadoop']} 2016-01-30 16:31:57,294 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2016-01-30 16:31:57,295 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2016-01-30 16:31:57,299 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if 2016-01-30 16:31:57,299 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'} 2016-01-30 16:31:57,300 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2016-01-30 16:31:57,300 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'} 2016-01-30 16:31:57,304 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if 2016-01-30 16:31:57,304 - Group['hdfs'] {'ignore_failures': False} 2016-01-30 16:31:57,305 - User['hdfs'] {'ignore_failures': False, 'groups': [u'hadoop', u'hdfs']} 2016-01-30 16:31:57,305 - Directory['/etc/hadoop'] {'mode': 0755} 2016-01-30 16:31:57,317 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'} 2016-01-30 16:31:57,317 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0777} 2016-01-30 16:31:57,328 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'} 2016-01-30 16:31:57,335 - Skipping Execute[('setenforce', '0')] due to only_if 2016-01-30 16:31:57,335 - Directory['/var/log/hadoop'] {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'} 2016-01-30 16:31:57,337 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'} 2016-01-30 16:31:57,337 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'} 2016-01-30 16:31:57,341 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'} 2016-01-30 16:31:57,342 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'} 2016-01-30 16:31:57,343 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644} 2016-01-30 16:31:57,350 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'} 2016-01-30 16:31:57,350 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755} 2016-01-30 16:31:57,354 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'} 2016-01-30 16:31:57,357 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755} 2016-01-30 16:31:57,536 - File['/opt/nifi-1.1.1.0-12/conf/nifi.properties'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi'} 2016-01-30 16:31:57,539 - File['/opt/nifi-1.1.1.0-12/conf/bootstrap.conf'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi'} 2016-01-30 16:31:57,543 - File['/opt/nifi-1.1.1.0-12/conf/logback.xml'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi'} 2016-01-30 16:31:57,543 - Execute['echo pid file /var/run/nifi/nifi.pid'] {} 2016-01-30 16:31:57,546 - Execute['echo JAVA_HOME=/usr/jdk64/jdk1.8.0_60'] {} 2016-01-30 16:31:57,549 - Execute['export JAVA_HOME=/usr/jdk64/jdk1.8.0_60;/opt/nifi-1.1.1.0-12/bin/nifi.sh start >> /var/log/nifi/nifi-setup.log'] {'user': 'nifi'} 2016-01-30 16:32:00,606 - Execute['cat /opt/nifi-1.1.1.0-12/bin/nifi.pid | grep pid | sed 's/pid=\(\.*\)/\1/' > /var/run/nifi/nifi.pid'] {} 2016-01-30 16:32:00,625 - Execute['chown nifi:nifi /var/run/nifi/nifi.pid'] {} Still though the service goes down after some green blinking ... when I am going to bash there is also no nifi service running ... in /var/log/nifi there are 4 log files nifi-app.log nifi-bootstrap.log nifi-setup.log nifi-user.log --------------------------------------------------- [nifi@nifi1n1 nifi]$ cat /var/log/nifi/* 2016-01-30 17:10:40,677 INFO [main] org.apache.nifi.bootstrap.RunNiFi No Bootstrap Notification Services configured. 2016-01-30 17:10:40,679 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is not running Java home: /usr/jdk64/jdk1.8.0_60 NiFi home: /opt/nifi-1.1.1.0-12 Bootstrap Config File: /opt/nifi-1.1.1.0-12/conf/bootstrap.conf [nifi@nifi1n1 nifi]$ ---------------------------------------------------------- /var/run/nifi Directory is empty Access Rights are set appropriately for /var/log/nifi and /var/run/nifi [nifi@nifi1n1 log]$ ls -lisa /var/log | grep nifi 8621255 0 drwxr-xr-x. 2 nifinifi 91 Jan 30 17:10 nifi [nifi@nifi1n1 run]$ cd /var/run [nifi@nifi1n1 run]$ ls -lisa | grep nifi 62714 0 drwxr-xr-x. 2 nifi hadoop 40 Jan 30 17:11 nifi if I execute export JAVA_HOME=/usr/jdk64/jdk1.8.0_60;/opt/nifi-1.1.1.0-12/bin/nifi.sh start >> /var/log/nifi/nifi-setup.log from bash the service goes up. ... and I am getting crazy 🙂 Any Advise??? ... View more

I think I am messing up things here 🙂 ... I initially put the user nifi in the groop hadoop ... and did the nifi install ... as what it looks like ambari requires user "nifi" to be in group "nifi" ... I will start to change a couple of access rights and provide further feedback when it's done ... ... View more

I did that already in order to trick Ambari after starting nifi through bash (and then I inserted a valid PID in /var/run/nifi) and restarted the service through Amari. Obviously Ambari found that PID and got green ... but still no luck starting the service through Ambari 😞 ... View more

Hi Neeray, thanks for your feedback ... I am now getting more detailed logs ... even though the problem didn't go aways, please see above. Thanks, Rainer ... View more

Based on Neerjas comment below I enabled user nifi to write into /var/log/ ... no at least I am getting an error in the Ambari Console when starting the service that I attached in errors-1795.txt. In addition I posted the logs of /var/log/nifi. Again, when I am starting "nifi.sh start" from bash, nifi starts successfully. Thanks, in advance, for any further advise. br, Rainer ... View more