Support Questions
Find answers, ask questions, and share your expertise

HTTPFS Installation & Startup Issues

Highlighted

HTTPFS Installation & Startup Issues

Hi there, I am using HTTPFS with my 6 node cluster with NameNode HA enabled.

I installed the HTTPFS, but when I browse to <server>:14000/webhdfs/v1/ I either get "404 Not Authorized" (with ANY user, ambari users or "hdfs" user - any advice/information regarding httpfs / webhdfs users would be much appreciated), or I get the following message:

{"RemoteException":{"message":"Missing Operation parameter [op]","exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException"}}

It seems I get the latter when Hadoop is running. For my configuration of HTTPFS, I added this to my core-sites.xml:

<property>
    <name>hadoop.proxyuser.hdfs.hosts</name>
    <value>httpfs-host.foo.com</value>
  </property>
  <property>
    <name>hadoop.proxyuser.hdfs.groups</name>
    <value>*</value>
  </property>

And replaced httpfs-host.foo.com with my Httpfs host's FQDN. My httpfs.sh looks like this:

#!/bin/bash
# Autodetect JAVA_HOME if not defined 
if [ -e /usr/libexec/bigtop-detect-javahome ]; then   
  . /usr/libexec/bigtop-detect-javahome 
elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then   
  . /usr/lib/bigtop-utils/bigtop-detect-javahome
fi 
export JAVA_HOME=/usr/jdk64/jdk1.8.0_60/ 
export HADOOP_HOME=${HADOOP_HOME:-/usr/hdp/current/hadoop-client}
export HADOOP_LIBEXEC_DIR=${HADOOP_HOME}/libexec 
export CATALINA_BASE=/usr/hdp/current/hadoop-httpfs
export HTTPFS_CATALINA_HOME=/etc/hadoop-httpfs/tomcat-deployment
export HTTPFS_CONFIG=/etc/hadoop-httpfs/conf export HTTPFS_LOG=/var/log/hadoop/httpfs 
export HTTPFS_TEMP=/tmp/httpfs
exec /usr/hdp/current/hadoop-httpfs/sbin/httpfs.sh.distro "$@"

Any help you could offer would me much appreciated, as I am quite lost.

7 REPLIES 7
Highlighted

Re: HTTPFS Installation & Startup Issues

Hi @Savanna Endicott

Looks like url is not complete.

Try http://httpfs-host:14000/webhdfs/v1/user?op=list and share the result.

Highlighted

Re: HTTPFS Installation & Startup Issues

Highlighted

Re: HTTPFS Installation & Startup Issues

Hi @Rahul Pathak. Thanks for the link. I had seen this and gone through the setup there.

With your link suggestion, I got this page (which was new for me):

httpfs-err2.png

It might help to share my results of the "curl -i "http://<HTTPFSHOSTNAME>:14000?user.name=hdfs&op=homedir" test:

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Accept-Ranges: bytes
ETag: W/"674-1450236394000"
Last-Modified: Wed, 16 Dec 2015 03:26:34 GMT
Content-Type: text/html
Content-Length: 674
Date: Mon, 21 Nov 2016 08:11:12 GMT

<?xml version="1.0" encoding="UTF-8"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
  http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<html>
<body>
<b>HttpFs service</b>, service base URL at /webhdfs/v1.
</body>
</html>
Highlighted

Re: HTTPFS Installation & Startup Issues

@Savanna Endicott

I referred - http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_hadoop-ha/content/ha-nn-deploy-hue.html

and was able to implement is successfully.

Can you check output of "$netstat -taupen |grep 1400"

Highlighted

Re: HTTPFS Installation & Startup Issues

Also can you paste output of - "$ps -aef |grep httpfs"

Highlighted

Re: HTTPFS Installation & Startup Issues

@Sagar Shimpi from "$netstat -taupen | grep 14000" I got:

tcp        0      0 0.0.0.0:14000               0.0.0.0:*                   LIST                 
EN      0          13928      2608/java
tcp        0      0 127.0.0.1:14001             0.0.0.0:*                   LIST                 
EN      0          14170      2608/java

From "$ps -aef | grep httpfs" I got:

root      2608     1  0 08:10 pts/0    00:00:12 /usr/jdk64/jdk1.8.0_60//bin/java -Djava.util.logging.config.file=/usr/hdp/current/hadoop-httpfs/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dhttpfs.home.dir=/usr/hdp/current/hadoop-httpfs -Dhttpfs.config.dir=/etc/hadoop-httpfs/conf -Dhttpfs.log.dir=/var/log/hadoop/httpfs -Dhttpfs.temp.dir=/tmp/httpfs -Dhttpfs.admin.port=14001 -Dhttpfs.http.port=14000 -Dhttpfs.http.hostname=mo-ffcc9cc72.mo.sap.corp -Dhttpfs.ssl.enabled=false -Dhttpfs.ssl.keystore.file=/root/.keystore -Dhttpfs.ssl.keystore.pass=password -Djava.endorsed.dirs=/etc/hadoop-httpfs/tomcat-deployment/endorsed -classpath /etc/hadoop-httpfs/tomcat-deployment/bin/bootstrap.jar -Dcatalina.base=/usr/hdp/current/hadoop-httpfs -Dcatalina.home=/etc/hadoop-httpfs/tomcat-deployment -Djava.io.tmpdir=/usr/hdp/current/hadoop-httpfs/temp org.apache.catalina.startup.Bootstrap start
root      9084  2313  0 08:43 pts/0    00:00:00 grep httpfs
Highlighted

Re: HTTPFS Installation & Startup Issues

Explorer

was this ever resolved @Savanna Endicott for you ? I am running into same issue with 404 error on httpfs.

,

Was this ever resolved for you @Savanna Endicott the 404 error ? If so what was the solution, I am facing similar issue.

Don't have an account?