zookeeper gets permission problem on /var/lib/zookeeper

[#1]

Hello

zookeeper could not start because of the java.io.FileNotFoundException: /var/lib/zookeeper/version-2/snapshot.0 (Permission denied) error

Normally zookeeper user could write in this directory

[root@testos1 ~]# ll /var/lib | grep zoo
drwxr-xr-x. 3 zookeeper zookeeper 4096 Aug 12 18:54 zookeeper
You have new mail in /var/spool/mail/root
[root@testos1 ~]# ll /var/lib/zookeeper/
total 4
drwxr-xr-x. 2 zookeeper zookeeper 4096 Aug 13 11:31 version-2
[root@testos1 ~]# ll /var/lib/zookeeper/version-2/
total 0
[root@testos1 ~]#

My environment has 1 server and tried to install from parcels

cloudera-manager-agent.x86_64 5.4.3-1.cm543.p0.258.el6 @/cloudera-manager-agent-5.4.3-1.cm543.p0.258.el6.x86_64
cloudera-manager-daemons.x86_64 5.4.3-1.cm543.p0.258.el6 @/cloudera-manager-daemons-5.4.3-1.cm543.p0.258.el6.x86_64
cloudera-manager-server.x86_64 5.4.3-1.cm543.p0.258.el6 @/cloudera-manager-server-5.4.3-1.cm543.p0.258.el6.x86_64

var/log/zookeeper/zookeeper-cmf-zookeeper-SERVER-testos1.localdomain.log file has following error lines

Aug 13, 1:08:54.630 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.name=cloudera-scm

Aug 13, 1:08:54.630 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.home=/home/cloudera-scm

Aug 13, 1:08:54.630 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.dir=/var/run/cloudera-scm-agent/process/115-zookeeper-server

Aug 13, 1:08:54.631 PM DEBUG org.apache.zookeeper.server.persistence.FileTxnSnapLog

Opening datadir:/var/lib/zookeeper snapDir:/var/lib/zookeeper

Aug 13, 1:08:54.631 PM INFO org.apache.zookeeper.server.ZooKeeperServer

tickTime set to 2000

Aug 13, 1:08:54.631 PM INFO org.apache.zookeeper.server.ZooKeeperServer

minSessionTimeout set to 4000

Aug 13, 1:08:54.632 PM INFO org.apache.zookeeper.server.ZooKeeperServer

maxSessionTimeout set to 60000

Aug 13, 1:08:54.654 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

binding to port 0.0.0.0/0.0.0.0:2181

Aug 13, 1:08:54.674 PM INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog

Snapshotting: 0x0 to /var/lib/zookeeper/version-2/snapshot.0

Aug 13, 1:08:54.675 PM ERROR org.apache.zookeeper.server.ZooKeeperServer

Severe unrecoverable error, exiting
java.io.FileNotFoundException: /var/lib/zookeeper/version-2/snapshot.0 (Permission denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at org.apache.zookeeper.server.persistence.FileSnap.serialize(FileSnap.java:225)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.save(FileTxnSnapLog.java:275)
at org.apache.zookeeper.server.ZooKeeperServer.takeSnapshot(ZooKeeperServer.java:270)
at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:265)
at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:377)
at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:122)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:118)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

Aug 13, 1:08:56.903 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

Reading configuration from: /var/run/cloudera-scm-agent/process/115-zookeeper-server/zoo.cfg

Aug 13, 1:08:56.918 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

Invalid configuration, only one server specified (ignoring)

Aug 13, 1:08:56.919 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

autopurge.snapRetainCount set to 5

Aug 13, 1:08:56.920 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

autopurge.purgeInterval set to 24

Aug 13, 1:08:56.920 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

Either no config or no quorum defined in config, running in standalone mode

[#2]

Hi,

Did you check your dataDir in the zoo.cfg? Your configuration file is correct?

BRs,

Spyros
 

[#3]

Hello

First of all thanks for your input

I changed directory permission with chmod -R 777 /var/lib/zookeeper  and then tried again

zookeeper has started

But snapshot.0's owner is cloudera-scm 

Is this normal ?

[root@testos1 lib]# cd zookeeper/version-2/
[root@testos1 version-2]# ll
total 4
-rw-r--r-- 1 cloudera-scm cloudera-scm 296 Aug 13 15:40 snapshot.0
[root@testos1 version-2]#

Regarding to your question there are lots of zoo.cfg file in the server, which one is current configuration file that I must share with you ?

[root@testos1 version-2]# find / -name zoo.cfg -exec ls -l {} \;
-rw-r--r--. 1 cloudera-scm cloudera-scm 518 Jun 25 06:11 /opt/cloudera/parcels/CDH-5.4.3-1.cdh5.4.3.p0.6/share/doc/solr-doc-4.10.3+cdh5.4.3+256/example/solr/zoo.cfg
-rw-r--r--. 1 cloudera-scm cloudera-scm 518 Jun 25 06:11 /opt/cloudera/parcels/CDH-5.4.3-1.cdh5.4.3.p0.6/share/doc/solr-doc-4.10.3+cdh5.4.3+256/example/multicore/zoo.cfg
-rw-r--r--. 1 cloudera-scm cloudera-scm 518 Jun 25 06:11 /opt/cloudera/parcels/CDH-5.4.3-1.cdh5.4.3.p0.6/etc/solr/conf.dist/zoo.cfg
-rw-r--r--. 1 cloudera-scm cloudera-scm 1269 Jun 25 06:09 /opt/cloudera/parcels/CDH-5.4.3-1.cdh5.4.3.p0.6/etc/zookeeper/conf.dist/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:40 /var/run/cloudera-scm-agent/process/123-zookeeper-server/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:07 /var/run/cloudera-scm-agent/process/118-zookeeper-init/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:10 /var/run/cloudera-scm-agent/process/120-zookeeper-init/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:10 /var/run/cloudera-scm-agent/process/121-zookeeper-server/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:04 /var/run/cloudera-scm-agent/process/117-zookeeper-server/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:40 /var/run/cloudera-scm-agent/process/122-zookeeper-init/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:50 /var/run/cloudera-scm-agent/process/124-zookeeper-server/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:10 /var/run/cloudera-scm-agent/process/119-zookeeper-init/zoo.cfg

Thanks

[#4]

Hi,

In the /var/run/cloudera-scm-agent/process/124-zookeeper-server/zoo.cfg you can check the latest configuration of your zookeeper.

I think the best way to change your configuration is from cloudera manager.

Furthermore, the owner of the snapshot I think that have to be as:

-rw-r--r-- 1 zookeeper zookeeper

How did you start your service? From cmd as a specific user or from cloudera manager?

BRs,

Spyros

[#5]

Hello again

zoo.cfg contents are in following lines

[root@testos1 124-zookeeper-server]# more zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
jute.maxbuffer=4194304
dataDir=/var/lib/zookeeper
dataLogDir=/var/lib/zookeeper
clientPort=2181
maxClientCnxns=60
minSessionTimeout=4000
maxSessionTimeout=40000
autopurge.purgeInterval=24
autopurge.snapRetainCount=5
server.1=testos1.localdomain:3181:4181
leaderServes=yes

Normally I try to use CM for all changes

I have VM versiopn of Cloudera. I just checked from VM and snapshot file's owner is zookeeper in the  VM version

I started the service through CM

Thanks

[#6 · Accepted solution]

Hi again,

Your configuration is correct and looks like mine.

I think the problem is the permission of the folder and specially the period at the end of the permission.
[root@testos1 ~]# ll /var/lib | grep zoo
drwxr-xr-x. 3 zookeeper zookeeper 4096 Aug 12 18:54 zookeeper

You have to check your ACL and the SeLinux. It is better to disable the SeLinux on your server.

BRs,

Spyros

[#7]

Hello

I already disabled SELinux and also IPTables

Folder's ACL like following lines

[root@testos1 lib]# getfacl zookeeper/
# file: zookeeper/
# owner: zookeeper
# group: zookeeper
user::rwx
group::rwx
other::rwx

By the way I also enabled HBase from CM and I got similar error for /var/log/hbase directory. I changed permission to 777 and then issue fixed (?) I could start HBase service through CM

[root@testos1 lib]# getfacl hbase
getfacl: hbase: No such file or directory
[root@testos1 lib]# cd /var/log
[root@testos1 log]# ll | grep hbase
drwxrwxrwx. 4 hbase        hbase          4096 Aug 13 16:05 hbase
[root@testos1 log]# ll hbase
total 280
drwx------ 2 cloudera-scm cloudera-scm   4096 Aug 13 16:04 audit
-rw-r--r-- 1 cloudera-scm cloudera-scm 137248 Aug 13 21:39 hbase-cmf-hbase-MASTER-testos1.localdomain.log.out
-rw-r--r-- 1 cloudera-scm cloudera-scm 138631 Aug 13 21:40 hbase-cmf-hbase-REGIONSERVER-testos1.localdomain.log.out
drwxr-xr-x 2 cloudera-scm cloudera-scm   4096 Aug 13 16:04 stacks
[root@testos1 log]# getfacl hbase
# file: hbase
# owner: hbase
# group: hbase
user::rwx
group::rwx
other::rwx

I think I must not get this issues,  why hbase and zookeeper daemons/services  do not use their users to write their directories instead of cloudera-scm

How can I fix that issue ?

Best Regards

Murat

[#8]

Hi,

Try to change the directory and sub-directories with chown -R zookeeper:zookeeper and restart again the service.

Furthermore about the permissions of the folder I think the correct one is 755 but you can check it after the owner's change.

BRs,

Spyros 

[#9]

Hello

I already tried it :S no change, same error
I just changed owner of files in /var/lib/zookeper/version2 directory with zookeeper:zookeeper while permission of /var/lib/zookeper is 755. But I got same error

Any other idea?

Thanks

[#10]

Hi,

Did you change the owner only on folder version-2 or on all the files,path and the parent folder?

Brs,

Spyros