CDH 5.15 Single Node custer installed using CM on CentOS 7.x on AWS Ec2 instance. 8 CPU, 64 GB RAM.
Verfied WebHDFS is running and I am connecting from a remote machine (non-hadoop client), after being connected to the environment using SSH key.
using PyWebHDFSClient library to list, read and write files off HDFS.
The following code works -
hdfs = PyWebHdfsClient(host='IP_ADDR', port='50070', user_name='hdfs', timeout=1) # your Namenode IP & username here my_dir = 'ds-datalake/misc' pprint(hdfs.list_dir(my_dir))
{u'FileStatuses': {u'FileStatus': [{u'accessTime': 1534856157369L, u'blockSize': 134217728, u'childrenNum': 0, u'fileId': 25173, u'group': u'supergroup', u'length': 28, u'modificationTime': 1534856157544L, u'owner': u'centos', u'pathSuffix': u'sample.txt', u'permission': u'644', u'replication': 3, u'storagePolicy': 0, u'type': u'FILE'}]}}
But, when I try to read/write at same location, using something like this:
my_file = 'ds-datalake/misc/sample.txt' print(hdfs.read_file(my_file))
I get the following error:
requests.exceptions.ConnectionError: HTTPConnectionPool(host='HOST_NAME', port=50075): Max retries exceeded with url: /webhdfs/v1/ds-datalake/misc/sample.txt?op=OPEN&user.name=hdfs&namenoderpcaddress=HOST_NAME:8020&offset=0 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000000068F4828>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed',))
This is what the HDFS folder looks like:
hadoop fs -ls /ds-datalake/misc Found 1 items -rwxrwxrwx 3 centos supergroup 28 2018-08-21 12:55 /ds-datalake/misc/sample.txt
Can you please help me? I have two single node test clusters and this happens on both. HDFS Namenode UI comes up fine from the CM site and all services look healthy.
Thanks.
... View more
