Sorry it took me a while to response here but I was putting together a working sample. The first important point is that I think people tend to overestimate the complexity of dealing with the REST APIs, especially WebHDFS. The point of having REST APIs after all is supposed to be very thin clients. I played with a few different Java HTTP client libraries and to my surprise the venerable Java HttpsUrlConnection resulted in the cleanest examples. The Apache HttpClient is certainly an option and might be warranted in more complex situations.
IMPORTANT: Before you continue however please note that these examples are setup to circumvent both SSL hostname and certificate validation. This is not acceptable in production but often helps in samples to make sure they don't become a barrier to success.
I'll show the heart of the solution below but the full answer can be found here: https://github.com/kminder/knox-webhdfs-client-examples. Specifically here: https://github.com/kminder/knox-webhdfs-client-examples/blob/master/src/test/java/net/minder/KnoxWebHdfsJavaClientExamplesTest.java
Now for the code. The first is an example of the simplest of operations: GETHOMEDIRECTORY.
@Test
public void getHomeDirExample() throws Exception {
HttpsURLConnection connection;
InputStream input;
JsonNode json;
connection = createHttpUrlConnection( WEBHDFS_URL + "?op=GETHOMEDIRECTORY" );
input = connection.getInputStream();
json = MAPPER.readTree( input );
input.close();
connection.disconnect();
assertThat( json.get( "Path" ).asText(), is( "/user/"+TEST_USERNAME ) );
}
Next a more complicated sample that writes and reads a file to HDFS via the CREATE and OPEN operations.
@Test
public void putGetFileExample() throws Exception {
HttpsURLConnection connection;
String redirect;
InputStream input;
OutputStream output;
String data = UUID.randomUUID().toString();
connection = createHttpUrlConnection( WEBHDFS_URL + "/tmp/" + data + "/?op=CREATE" );
connection.setRequestMethod( "PUT" );
assertThat( connection.getResponseCode(), is(307) );
redirect = connection.getHeaderField( "Location" );
connection.disconnect();
connection = createHttpUrlConnection( redirect );
connection.setRequestMethod( "PUT" );
connection.setDoOutput( true );
output = connection.getOutputStream();
IOUtils.write( data.getBytes(), output );
output.close();
connection.disconnect();
assertThat( connection.getResponseCode(), is(201) );
connection = createHttpUrlConnection( WEBHDFS_URL + "/tmp/" + data + "/?op=OPEN" );
assertThat( connection.getResponseCode(), is(307) );
redirect = connection.getHeaderField( "Location" );
connection.disconnect();
connection = createHttpUrlConnection( redirect );
input = connection.getInputStream();
assertThat( IOUtils.toString( input ), is( data ) );
input.close();
connection.disconnect();
}
Now of course you have probably noticed that all of the "magic" is hidden in that createHttpUrlConnection method. Not really magic at all but this is where the "un-securing" of SSL happens. This also takes care of setting up HTTP BasicAuth for authentication and disables redirects which should be done when using the WebHDFS REST APIs. private HttpsURLConnection createHttpUrlConnection( URL url ) throws Exception {
HttpsURLConnection conn = (HttpsURLConnection)url.openConnection();
conn.setHostnameVerifier( new TrustAllHosts() );
conn.setSSLSocketFactory( TrustAllCerts.createInsecureSslContext().getSocketFactory() );
conn.setInstanceFollowRedirects( false );
String credentials = TEST_USERNAME + ":" + TEST_PASSWORD;
conn.setRequestProperty( "Authorization", "Basic " + DatatypeConverter.printBase64Binary(credentials.getBytes() ) );
return conn;
}
private HttpsURLConnection createHttpUrlConnection( String url ) throws Exception {
return createHttpUrlConnection( new URL( url ) );
}
... View more
