Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

CDH4.3 httpfs setup

CDH4.3 httpfs setup

New Contributor

I  like to setup httpfs to enable REST API access to my Kerberos secured HDFS using CHD4.3. After looking thru Cloudera documentation, there are a couple of things I like to clarify. Can anyone who are more expiernced help to anser it?

1) what is the major different between webhdfs and httpfs? 2) do I need to configure webhdfs in order to run httpfs? 3) do I need to install httpfs on both namenode and all datandoes? or it can be installed on only namenode?

Thanks a lot for your reply in advanced!

5 REPLIES 5

Re: CDH4.3 httpfs setup

HttpFS is a server that provides a REST HTTP gateway supporting all HDFS
File System operations (read and write). And it is inteoperable with the
webhdfs REST HTTP API.

Please read:
http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-hdfs-httpfs/index.html
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/c...

Regards,
Gautam Gopalakrishnan
Highlighted

Re: CDH4.3 httpfs setup

New Contributor

Thanks GautamG for your reply!   I did go thru the documentation, but it didn't mention if the httpfs should be on the namenode only, or it should be installed on all datanodes.  Can you help to clarify this for me? Thanks,  

Re: CDH4.3 httpfs setup

Please read these documents.
http://archive.cloudera.com/cdh4/cdh/4/hadoop/hadoop-hdfs-httpfs/index.html
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/c...

​If you use Cloudera Manager, HttpFS is another role within the HDFS
service. Either way, the HttpFS service is a standalone daemon that can be
installed anywhere on the cluster. It is not installed on each datanode. It
may or may not be alongside the namenode​
​, this depends on the hardware and the traffic you expect to handle.


Regards,
Gautam Gopalakrishnan

Re: CDH4.3 httpfs setup

New Contributor

Thanks again Gatuam!

 

I had httpfs configured with kerberos security enabled.  When I tried to use curl to fetch pages, I am getting "404 error" as below

 

" curl --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt http://localhost:14000/webhdfs/v1/tmp/hosts; echo
<html><head><title>Apache Tomcat/6.0.35 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 404 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The requested resource () is not available.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/6.0.35</h3></body></html>"

 

I can't find any log info from /var/log/hadoop-httpfs.  Could you please let me know how to enable extended logging and how to troubleshooting this issue?

 

Thanks a lot,

Wei

 

 

Re: CDH4.3 httpfs setup

Please open a new thread and the current thread can be closed. Let's stick
to one question per thread. Thanks


Regards,
Gautam Gopalakrishnan