Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

zookeeper gets permission problem on /var/lib/zookeeper

avatar
Contributor

Hello

zookeeper could not start because of the java.io.FileNotFoundException: /var/lib/zookeeper/version-2/snapshot.0 (Permission denied) error

Normally zookeeper user could write in this directory

 

[root@testos1 ~]# ll /var/lib | grep zoo
drwxr-xr-x. 3 zookeeper zookeeper 4096 Aug 12 18:54 zookeeper
You have new mail in /var/spool/mail/root
[root@testos1 ~]# ll /var/lib/zookeeper/
total 4
drwxr-xr-x. 2 zookeeper zookeeper 4096 Aug 13 11:31 version-2
[root@testos1 ~]# ll /var/lib/zookeeper/version-2/
total 0
[root@testos1 ~]#

 

My environment has 1 server and tried to install from parcels

 

cloudera-manager-agent.x86_64 5.4.3-1.cm543.p0.258.el6 @/cloudera-manager-agent-5.4.3-1.cm543.p0.258.el6.x86_64
cloudera-manager-daemons.x86_64 5.4.3-1.cm543.p0.258.el6 @/cloudera-manager-daemons-5.4.3-1.cm543.p0.258.el6.x86_64
cloudera-manager-server.x86_64 5.4.3-1.cm543.p0.258.el6 @/cloudera-manager-server-5.4.3-1.cm543.p0.258.el6.x86_64


var/log/zookeeper/zookeeper-cmf-zookeeper-SERVER-testos1.localdomain.log file has following error lines


Aug 13, 1:08:54.630 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.name=cloudera-scm

Aug 13, 1:08:54.630 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.home=/home/cloudera-scm

Aug 13, 1:08:54.630 PM INFO org.apache.zookeeper.server.ZooKeeperServer

Server environment:user.dir=/var/run/cloudera-scm-agent/process/115-zookeeper-server

Aug 13, 1:08:54.631 PM DEBUG org.apache.zookeeper.server.persistence.FileTxnSnapLog

Opening datadir:/var/lib/zookeeper snapDir:/var/lib/zookeeper

Aug 13, 1:08:54.631 PM INFO org.apache.zookeeper.server.ZooKeeperServer

tickTime set to 2000

Aug 13, 1:08:54.631 PM INFO org.apache.zookeeper.server.ZooKeeperServer

minSessionTimeout set to 4000

Aug 13, 1:08:54.632 PM INFO org.apache.zookeeper.server.ZooKeeperServer

maxSessionTimeout set to 60000

Aug 13, 1:08:54.654 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory

binding to port 0.0.0.0/0.0.0.0:2181

Aug 13, 1:08:54.674 PM INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog

Snapshotting: 0x0 to /var/lib/zookeeper/version-2/snapshot.0

Aug 13, 1:08:54.675 PM ERROR org.apache.zookeeper.server.ZooKeeperServer

Severe unrecoverable error, exiting
java.io.FileNotFoundException: /var/lib/zookeeper/version-2/snapshot.0 (Permission denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at org.apache.zookeeper.server.persistence.FileSnap.serialize(FileSnap.java:225)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.save(FileTxnSnapLog.java:275)
at org.apache.zookeeper.server.ZooKeeperServer.takeSnapshot(ZooKeeperServer.java:270)
at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:265)
at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:377)
at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:122)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:118)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

Aug 13, 1:08:56.903 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig

Reading configuration from: /var/run/cloudera-scm-agent/process/115-zookeeper-server/zoo.cfg

Aug 13, 1:08:56.918 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig

Invalid configuration, only one server specified (ignoring)

Aug 13, 1:08:56.919 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

autopurge.snapRetainCount set to 5

Aug 13, 1:08:56.920 PM INFO org.apache.zookeeper.server.DatadirCleanupManager

autopurge.purgeInterval set to 24

Aug 13, 1:08:56.920 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain

Either no config or no quorum defined in config, running in standalone mode

 

 

1 ACCEPTED SOLUTION

avatar
Explorer

Hi again,

 

Your configuration is correct and looks like mine.


I think the problem is the permission of the folder and specially the period at the end of the permission.
[root@testos1 ~]# ll /var/lib | grep zoo
drwxr-xr-x. 3 zookeeper zookeeper 4096 Aug 12 18:54 zookeeper

                       

You have to check your ACL and the SeLinux. It is better to disable the SeLinux on your server.

 

BRs,

Spyros

View solution in original post

10 REPLIES 10

avatar
Explorer

Hi,

 

Did you check your dataDir in the zoo.cfg? Your configuration file is correct?

 

BRs,

Spyros
 

avatar
Contributor

Hello

 

First of all thanks for your input

 

I changed directory permission with chmod -R 777 /var/lib/zookeeper  and then tried again

zookeeper has started

 

But snapshot.0's owner is cloudera-scm 

Is this normal ?

 

[root@testos1 lib]# cd zookeeper/version-2/
[root@testos1 version-2]# ll
total 4
-rw-r--r-- 1 cloudera-scm cloudera-scm 296 Aug 13 15:40 snapshot.0
[root@testos1 version-2]#

 

Regarding to your question there are lots of zoo.cfg file in the server, which one is current configuration file that I must share with you ?

 

[root@testos1 version-2]# find / -name zoo.cfg -exec ls -l {} \;
-rw-r--r--. 1 cloudera-scm cloudera-scm 518 Jun 25 06:11 /opt/cloudera/parcels/CDH-5.4.3-1.cdh5.4.3.p0.6/share/doc/solr-doc-4.10.3+cdh5.4.3+256/example/solr/zoo.cfg
-rw-r--r--. 1 cloudera-scm cloudera-scm 518 Jun 25 06:11 /opt/cloudera/parcels/CDH-5.4.3-1.cdh5.4.3.p0.6/share/doc/solr-doc-4.10.3+cdh5.4.3+256/example/multicore/zoo.cfg
-rw-r--r--. 1 cloudera-scm cloudera-scm 518 Jun 25 06:11 /opt/cloudera/parcels/CDH-5.4.3-1.cdh5.4.3.p0.6/etc/solr/conf.dist/zoo.cfg
-rw-r--r--. 1 cloudera-scm cloudera-scm 1269 Jun 25 06:09 /opt/cloudera/parcels/CDH-5.4.3-1.cdh5.4.3.p0.6/etc/zookeeper/conf.dist/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:40 /var/run/cloudera-scm-agent/process/123-zookeeper-server/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:07 /var/run/cloudera-scm-agent/process/118-zookeeper-init/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:10 /var/run/cloudera-scm-agent/process/120-zookeeper-init/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:10 /var/run/cloudera-scm-agent/process/121-zookeeper-server/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:04 /var/run/cloudera-scm-agent/process/117-zookeeper-server/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:40 /var/run/cloudera-scm-agent/process/122-zookeeper-init/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:50 /var/run/cloudera-scm-agent/process/124-zookeeper-server/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:10 /var/run/cloudera-scm-agent/process/119-zookeeper-init/zoo.cfg

 

 

Thanks

 

avatar
Explorer

Hi,

 

In the /var/run/cloudera-scm-agent/process/124-zookeeper-server/zoo.cfg you can check the latest configuration of your zookeeper.

I think the best way to change your configuration is from cloudera manager.

 

Furthermore, the owner of the snapshot I think that have to be as:

-rw-r--r-- 1 zookeeper zookeeper

 

How did you start your service? From cmd as a specific user or from cloudera manager?

BRs,

Spyros

 

 

 

avatar
Contributor

Hello again

 

zoo.cfg contents are in following lines

 

[root@testos1 124-zookeeper-server]# more zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
jute.maxbuffer=4194304
dataDir=/var/lib/zookeeper
dataLogDir=/var/lib/zookeeper
clientPort=2181
maxClientCnxns=60
minSessionTimeout=4000
maxSessionTimeout=40000
autopurge.purgeInterval=24
autopurge.snapRetainCount=5
server.1=testos1.localdomain:3181:4181
leaderServes=yes

 

Normally I try to use CM for all changes

 

I have VM versiopn of Cloudera. I just checked from VM and snapshot file's owner is zookeeper in the  VM version

 

I started the service through CM

 

Thanks

 

 

 

avatar
Explorer

Hi again,

 

Your configuration is correct and looks like mine.


I think the problem is the permission of the folder and specially the period at the end of the permission.
[root@testos1 ~]# ll /var/lib | grep zoo
drwxr-xr-x. 3 zookeeper zookeeper 4096 Aug 12 18:54 zookeeper

                       

You have to check your ACL and the SeLinux. It is better to disable the SeLinux on your server.

 

BRs,

Spyros

avatar
Contributor

Hello

 

I already disabled SELinux and also IPTables

 

Folder's ACL like following lines

 

[root@testos1 lib]# getfacl zookeeper/
# file: zookeeper/
# owner: zookeeper
# group: zookeeper
user::rwx
group::rwx
other::rwx

 

By the way I also enabled HBase from CM and I got similar error for /var/log/hbase directory. I changed permission to 777 and then issue fixed (?) I could start HBase service through CM

 

 

[root@testos1 lib]# getfacl hbase
getfacl: hbase: No such file or directory
[root@testos1 lib]# cd /var/log
[root@testos1 log]# ll | grep hbase
drwxrwxrwx. 4 hbase        hbase          4096 Aug 13 16:05 hbase
[root@testos1 log]# ll hbase
total 280
drwx------ 2 cloudera-scm cloudera-scm   4096 Aug 13 16:04 audit
-rw-r--r-- 1 cloudera-scm cloudera-scm 137248 Aug 13 21:39 hbase-cmf-hbase-MASTER-testos1.localdomain.log.out
-rw-r--r-- 1 cloudera-scm cloudera-scm 138631 Aug 13 21:40 hbase-cmf-hbase-REGIONSERVER-testos1.localdomain.log.out
drwxr-xr-x 2 cloudera-scm cloudera-scm   4096 Aug 13 16:04 stacks
[root@testos1 log]# getfacl hbase
# file: hbase
# owner: hbase
# group: hbase
user::rwx
group::rwx
other::rwx

 

I think I must not get this issues,  why hbase and zookeeper daemons/services  do not use their users to write their directories instead of cloudera-scm

 

How can I fix that issue ?

 

Best Regards

Murat

avatar
Explorer

Hi,

 

Try to change the directory and sub-directories with chown -R zookeeper:zookeeper and restart again the service.

Furthermore about the permissions of the folder I think the correct one is 755 but you can check it after the owner's change.

 

BRs,

 

Spyros 

avatar
Contributor
Hello

I already tried it :S no change, same error
I just changed owner of files in /var/lib/zookeper/version2 directory with zookeeper:zookeeper while permission of /var/lib/zookeper is 755. But I got same error

Any other idea?

Thanks

avatar
Explorer

Hi,

 

Did you change the owner only on folder version-2 or on all the files,path and the parent folder?

 

Brs,

Spyros