Created on 08-13-2015 03:36 AM - edited 09-16-2022 02:37 AM
Hello
zookeeper could not start because of the java.io.FileNotFoundException: /var/lib/zookeeper/version-2/snapshot.0 (Permission denied) error
Normally zookeeper user could write in this directory
[root@testos1 ~]# ll /var/lib | grep zoo
drwxr-xr-x. 3 zookeeper zookeeper 4096 Aug 12 18:54 zookeeper
You have new mail in /var/spool/mail/root
[root@testos1 ~]# ll /var/lib/zookeeper/
total 4
drwxr-xr-x. 2 zookeeper zookeeper 4096 Aug 13 11:31 version-2
[root@testos1 ~]# ll /var/lib/zookeeper/version-2/
total 0
[root@testos1 ~]#
My environment has 1 server and tried to install from parcels
cloudera-manager-agent.x86_64 5.4.3-1.cm543.p0.258.el6 @/cloudera-manager-agent-5.4.3-1.cm543.p0.258.el6.x86_64
cloudera-manager-daemons.x86_64 5.4.3-1.cm543.p0.258.el6 @/cloudera-manager-daemons-5.4.3-1.cm543.p0.258.el6.x86_64
cloudera-manager-server.x86_64 5.4.3-1.cm543.p0.258.el6 @/cloudera-manager-server-5.4.3-1.cm543.p0.258.el6.x86_64
var/log/zookeeper/zookeeper-cmf-zookeeper-SERVER-testos1.localdomain.log file has following error lines
Aug 13, 1:08:54.630 PM INFO org.apache.zookeeper.server.ZooKeeperServer
Server environment:user.name=cloudera-scm
Aug 13, 1:08:54.630 PM INFO org.apache.zookeeper.server.ZooKeeperServer
Server environment:user.home=/home/cloudera-scm
Aug 13, 1:08:54.630 PM INFO org.apache.zookeeper.server.ZooKeeperServer
Server environment:user.dir=/var/run/cloudera-scm-agent/process/115-zookeeper-server
Aug 13, 1:08:54.631 PM DEBUG org.apache.zookeeper.server.persistence.FileTxnSnapLog
Opening datadir:/var/lib/zookeeper snapDir:/var/lib/zookeeper
Aug 13, 1:08:54.631 PM INFO org.apache.zookeeper.server.ZooKeeperServer
tickTime set to 2000
Aug 13, 1:08:54.631 PM INFO org.apache.zookeeper.server.ZooKeeperServer
minSessionTimeout set to 4000
Aug 13, 1:08:54.632 PM INFO org.apache.zookeeper.server.ZooKeeperServer
maxSessionTimeout set to 60000
Aug 13, 1:08:54.654 PM INFO org.apache.zookeeper.server.NIOServerCnxnFactory
binding to port 0.0.0.0/0.0.0.0:2181
Aug 13, 1:08:54.674 PM INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog
Snapshotting: 0x0 to /var/lib/zookeeper/version-2/snapshot.0
Aug 13, 1:08:54.675 PM ERROR org.apache.zookeeper.server.ZooKeeperServer
Severe unrecoverable error, exiting
java.io.FileNotFoundException: /var/lib/zookeeper/version-2/snapshot.0 (Permission denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at org.apache.zookeeper.server.persistence.FileSnap.serialize(FileSnap.java:225)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.save(FileTxnSnapLog.java:275)
at org.apache.zookeeper.server.ZooKeeperServer.takeSnapshot(ZooKeeperServer.java:270)
at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:265)
at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:377)
at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:122)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:118)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:91)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:121)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)
Aug 13, 1:08:56.903 PM INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig
Reading configuration from: /var/run/cloudera-scm-agent/process/115-zookeeper-server/zoo.cfg
Aug 13, 1:08:56.918 PM ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig
Invalid configuration, only one server specified (ignoring)
Aug 13, 1:08:56.919 PM INFO org.apache.zookeeper.server.DatadirCleanupManager
autopurge.snapRetainCount set to 5
Aug 13, 1:08:56.920 PM INFO org.apache.zookeeper.server.DatadirCleanupManager
autopurge.purgeInterval set to 24
Aug 13, 1:08:56.920 PM WARN org.apache.zookeeper.server.quorum.QuorumPeerMain
Either no config or no quorum defined in config, running in standalone mode
Created 08-13-2015 06:35 AM
Hi again,
Your configuration is correct and looks like mine.
I think the problem is the permission of the folder and specially the period at the end of the permission.
[root@testos1 ~]# ll /var/lib | grep zoo
drwxr-xr-x. 3 zookeeper zookeeper 4096 Aug 12 18:54 zookeeper
You have to check your ACL and the SeLinux. It is better to disable the SeLinux on your server.
BRs,
Spyros
Created 08-13-2015 05:08 AM
Hi,
Did you check your dataDir in the zoo.cfg? Your configuration file is correct?
BRs,
Spyros
Created 08-13-2015 05:54 AM
Hello
First of all thanks for your input
I changed directory permission with chmod -R 777 /var/lib/zookeeper and then tried again
zookeeper has started
But snapshot.0's owner is cloudera-scm
Is this normal ?
[root@testos1 lib]# cd zookeeper/version-2/
[root@testos1 version-2]# ll
total 4
-rw-r--r-- 1 cloudera-scm cloudera-scm 296 Aug 13 15:40 snapshot.0
[root@testos1 version-2]#
Regarding to your question there are lots of zoo.cfg file in the server, which one is current configuration file that I must share with you ?
[root@testos1 version-2]# find / -name zoo.cfg -exec ls -l {} \;
-rw-r--r--. 1 cloudera-scm cloudera-scm 518 Jun 25 06:11 /opt/cloudera/parcels/CDH-5.4.3-1.cdh5.4.3.p0.6/share/doc/solr-doc-4.10.3+cdh5.4.3+256/example/solr/zoo.cfg
-rw-r--r--. 1 cloudera-scm cloudera-scm 518 Jun 25 06:11 /opt/cloudera/parcels/CDH-5.4.3-1.cdh5.4.3.p0.6/share/doc/solr-doc-4.10.3+cdh5.4.3+256/example/multicore/zoo.cfg
-rw-r--r--. 1 cloudera-scm cloudera-scm 518 Jun 25 06:11 /opt/cloudera/parcels/CDH-5.4.3-1.cdh5.4.3.p0.6/etc/solr/conf.dist/zoo.cfg
-rw-r--r--. 1 cloudera-scm cloudera-scm 1269 Jun 25 06:09 /opt/cloudera/parcels/CDH-5.4.3-1.cdh5.4.3.p0.6/etc/zookeeper/conf.dist/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:40 /var/run/cloudera-scm-agent/process/123-zookeeper-server/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:07 /var/run/cloudera-scm-agent/process/118-zookeeper-init/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:10 /var/run/cloudera-scm-agent/process/120-zookeeper-init/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:10 /var/run/cloudera-scm-agent/process/121-zookeeper-server/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:04 /var/run/cloudera-scm-agent/process/117-zookeeper-server/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:40 /var/run/cloudera-scm-agent/process/122-zookeeper-init/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:50 /var/run/cloudera-scm-agent/process/124-zookeeper-server/zoo.cfg
-rw-r----- 1 cloudera-scm cloudera-scm 311 Aug 13 15:10 /var/run/cloudera-scm-agent/process/119-zookeeper-init/zoo.cfg
Thanks
Created 08-13-2015 06:15 AM
Hi,
In the /var/run/cloudera-scm-agent/process/124-zookeeper-server/zoo.cfg you can check the latest configuration of your zookeeper.
I think the best way to change your configuration is from cloudera manager.
Furthermore, the owner of the snapshot I think that have to be as:
-rw-r--r-- 1 zookeeper zookeeper
How did you start your service? From cmd as a specific user or from cloudera manager?
BRs,
Spyros
Created on 08-13-2015 06:22 AM - edited 08-13-2015 06:23 AM
Hello again
zoo.cfg contents are in following lines
[root@testos1 124-zookeeper-server]# more zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
jute.maxbuffer=4194304
dataDir=/var/lib/zookeeper
dataLogDir=/var/lib/zookeeper
clientPort=2181
maxClientCnxns=60
minSessionTimeout=4000
maxSessionTimeout=40000
autopurge.purgeInterval=24
autopurge.snapRetainCount=5
server.1=testos1.localdomain:3181:4181
leaderServes=yes
Normally I try to use CM for all changes
I have VM versiopn of Cloudera. I just checked from VM and snapshot file's owner is zookeeper in the VM version
I started the service through CM
Thanks
Created 08-13-2015 06:35 AM
Hi again,
Your configuration is correct and looks like mine.
I think the problem is the permission of the folder and specially the period at the end of the permission.
[root@testos1 ~]# ll /var/lib | grep zoo
drwxr-xr-x. 3 zookeeper zookeeper 4096 Aug 12 18:54 zookeeper
You have to check your ACL and the SeLinux. It is better to disable the SeLinux on your server.
BRs,
Spyros
Created 08-13-2015 12:04 PM
Hello
I already disabled SELinux and also IPTables
Folder's ACL like following lines
[root@testos1 lib]# getfacl zookeeper/
# file: zookeeper/
# owner: zookeeper
# group: zookeeper
user::rwx
group::rwx
other::rwx
By the way I also enabled HBase from CM and I got similar error for /var/log/hbase directory. I changed permission to 777 and then issue fixed (?) I could start HBase service through CM
[root@testos1 lib]# getfacl hbase
getfacl: hbase: No such file or directory
[root@testos1 lib]# cd /var/log
[root@testos1 log]# ll | grep hbase
drwxrwxrwx. 4 hbase hbase 4096 Aug 13 16:05 hbase
[root@testos1 log]# ll hbase
total 280
drwx------ 2 cloudera-scm cloudera-scm 4096 Aug 13 16:04 audit
-rw-r--r-- 1 cloudera-scm cloudera-scm 137248 Aug 13 21:39 hbase-cmf-hbase-MASTER-testos1.localdomain.log.out
-rw-r--r-- 1 cloudera-scm cloudera-scm 138631 Aug 13 21:40 hbase-cmf-hbase-REGIONSERVER-testos1.localdomain.log.out
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Aug 13 16:04 stacks
[root@testos1 log]# getfacl hbase
# file: hbase
# owner: hbase
# group: hbase
user::rwx
group::rwx
other::rwx
I think I must not get this issues, why hbase and zookeeper daemons/services do not use their users to write their directories instead of cloudera-scm
How can I fix that issue ?
Best Regards
Murat
Created 08-14-2015 04:53 AM
Hi,
Try to change the directory and sub-directories with chown -R zookeeper:zookeeper and restart again the service.
Furthermore about the permissions of the folder I think the correct one is 755 but you can check it after the owner's change.
BRs,
Spyros
Created 08-14-2015 05:48 AM
Created 08-14-2015 05:56 AM
Hi,
Did you change the owner only on folder version-2 or on all the files,path and the parent folder?
Brs,
Spyros