After a successful fresh Installation of HDP 3.1.0 on a 2 node Ubuntu 18.0.4 instances, we were able to get all the services up and running. But after a night off, Datanode doesn't start. Following is the error: ERROR datanode.DataNode (DataNode.java:secureMain(2883)) - Exception in secureMain java.io.IOException: The path component: '/' in '/var/lib/hadoop-hdfs/dn_socket' has permissions 0777 uid 0 and gid 0. It is not protected because it is world-writable. This might help: 'chmod o-w /'. For more information: https://wiki.apache.org/hadoop/SocketPathSecurity     at org.apache.hadoop.net.unix.DomainSocket.validateSocketPathSecurity0(Native Method)     at org.apache.hadoop.net.unix.DomainSocket.bindAndListen(DomainSocket.java:193)     at org.apache.hadoop.hdfs.net.DomainPeerServer.<init>(DomainPeerServer.java:40)     at org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:1194)     at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:1161)     at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1416)     at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:500)     at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2782)     at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2690)     at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2732)     at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2876)     at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2900) 2019-04-08 17:36:52,452 INFO  util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: java.io.IOException: The path component: '/' in '/var/lib/hadoop-hdfs/dn_socket' has permissions 0777 uid 0 and gid 0. It is not protected because it is world-writable. This might help: 'chmod o-w /'. For more information: https://wiki.apache.org/hadoop/SocketPathSecurity 2019-04-08 17:36:52,456 INFO  datanode.DataNode (LogAdapter.java:info(51)) - SHUTDOWN_MSG I did check the socket file and it's parent directories, I don't see the 777 permission at any level. But still the error appears while starting the datanode. Couldn't find any solution, so posted here for help. I have also uploaded the complete logs for the datanode. hadoop-hdfs-datanode.txtRegards, Vinay MP ... View more

How can I change the Apache kylin port ? The default listen port is 7070, I have salt-bootstrap running on the Azure VMs in that port. I went through kylin.properties and didn't find a relevant property for listen port. ... View more

Ambari version: 2.2.2.18 HDP stack: 2.4.3 OS: centos 7.3 Issue description: Ambari-server can't communicate with Ambari agent. I can see below error in the ambari-agent logs: ERROR 2017-09-18 06:35:34,684 NetUtil.py:84 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) ERROR 2017-09-18 06:35:34,684 NetUtil.py:85 - SSLError: Failed to connect. Please check openssl library versions. I am facing this issue recently and it appears this can be replicated consistently after the instances are restarted. (I am using EC2 instances). I am able to register agent nodes successfully, install HDP cluster, run yarn jobs etc... no problem at all. Once i restart my instances, I see this problem. There are some solutions already posted for this problem like: Downgrade the Python from 2.7 to lower. This is a known problem of Ambari with Python 2.7 Control the certificate verification by disabling it. Set "verify = disable"; under /etc/python/cert-verification.cfg I don't want to play with Python as it can disrupt lot many things like Cassandra, yum package manager etc... Second work around is very much easy and it works well! Now comes my question :- Is it safe to disable the certificate verification in Python ? i.e. by setting property verify = disable Regards, Vinay MP ... View more

Hey All, I faced this issue in HDP 2.4 cluster running on centos. When I run 'hive' command, it always used to halt like below: Tried to add the proxyuser properties for hosts and groups and finally found that's not actually causing this. I stop Spark thrift server, immediately it gives me the hive prompt and I can work with hive cli. Anybody faced similar problems ??? Regards, Vinay MP ... View more

Hello, I am running below command from the map reduce examples for PI, it is failing and I can see socket timeout exception in the logs. I am not able to find a solution anywhere till now, would be glad if someone can help. Command: yarn jar hadoop-mapreduce-examples.jar pi 5 10 (From the directory: /usr/hdp/2.3.0.0-2557/hadoop-mapreduce) Below is the log trace: 2016-04-20 06:12:48,333 WARN [RMCommunicator Allocator] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.17.0.2:53751 remote=node1/172.17.0.2:8030] 2016-04-20 06:12:51,884 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM. java.io.IOException: Failed on local exception: java.io.IOException: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.17.0.2:53751 remote=node1/172.17.0.2:8030]; Host Details : local host is: "node1/172.17.0.2"; destination host is: "node1":8030; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773 I can see the property in advances yarn site -- yarn.resourcemanager.scheduler.address node1:8030 Hosts file entry: [root@node1 ~]# cat /etc/hosts 172.17.0.2 node1 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters [root@node1 ~]# Not sure what is the problem. I can ping localhost / node1/127.0.0.1 from node1 terminal. Regards, Vinay MP ... View more