Member since
09-24-2015
49
Posts
67
Kudos Received
16
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3581 | 03-29-2016 03:02 PM | |
1591 | 03-21-2016 01:34 PM | |
2091 | 03-07-2016 09:12 PM | |
1409 | 01-12-2016 10:01 PM | |
525 | 01-11-2016 10:04 PM |
04-11-2016
03:23 PM
2 Kudos
It is technically possible to have more than one authentication provider in a given topology but the result is unlikely to be what is expected. The first reason might be that all but one of them is enabled=false so that there is in effect only one. The other possibility is that a given custom service in a topology requires a specific authentication provider implementation. In this case the first enabled authentication provider in the topology would be the default and the custom service would identify a specific authentication provide by role and name in its service.xml file.
... View more
04-05-2016
02:27 PM
and /var/log/knox/gateway.out if it isn't empty while you are collecting things.
... View more
03-29-2016
03:02 PM
2 Kudos
The Apache Knox User's Guide contains a section (http://knox.apache.org/books/knox-0-6-0/user-guide.html#High+Availability) on using the Apache HTTP server as a load balancer for Knox. Someone familiar with another load balancer should be able to extrapolate the information provided in the Apache Knox User's Guide to setup that product.
... View more
03-22-2016
04:51 PM
@Sushil Kumar < If the answers below don't address your questions can you be more specific?
For the "This command bin/gateway.sh must not be run as root." issue you should either not be logged in as root or you should execute the command as a different user via sudo (e.g. su -l knox -c "bin/gateway.sh start")
For the "integration with open Apache Hadoop" you just need to modify the topology files (e.g. conf/topologies/sandbox.xml) to match the network locations for your installation.
... View more
03-21-2016
01:34 PM
1 Kudo
The LDAP configuration goes in the topology files. These are located in <GATEWAY_HOME>/conf/topologies. Out of the box there is a topology file named sandbox.xml with some sample configuration. That would need to be modified to integrate with a different LDAP or ActiveDirectory. The ldap.jar that you show running is a demo LDAP server that is provided with Knox that is not intended for production environments. If you haven't seen the Knox User's Guide you should check that out: http://knox.apache.org/books/knox-0-8-0/user-guide.html for more information about configuring things.
... View more
03-07-2016
09:12 PM
Can you confirm you have a topology file named knox_sample in your {GATEWAY_HOME}/conf/topologies directory? You should also check to make sure that it is valid XML. If it is malformed the topology will not deploy and you will get 404s.
... View more
02-29-2016
07:26 PM
2 Kudos
Sorry it took me a while to response here but I was putting together a working sample. The first important point is that I think people tend to overestimate the complexity of dealing with the REST APIs, especially WebHDFS. The point of having REST APIs after all is supposed to be very thin clients. I played with a few different Java HTTP client libraries and to my surprise the venerable Java HttpsUrlConnection resulted in the cleanest examples. The Apache HttpClient is certainly an option and might be warranted in more complex situations.
IMPORTANT: Before you continue however please note that these examples are setup to circumvent both SSL hostname and certificate validation. This is not acceptable in production but often helps in samples to make sure they don't become a barrier to success.
I'll show the heart of the solution below but the full answer can be found here: https://github.com/kminder/knox-webhdfs-client-examples. Specifically here: https://github.com/kminder/knox-webhdfs-client-examples/blob/master/src/test/java/net/minder/KnoxWebHdfsJavaClientExamplesTest.java
Now for the code. The first is an example of the simplest of operations: GETHOMEDIRECTORY.
@Test
public void getHomeDirExample() throws Exception {
HttpsURLConnection connection;
InputStream input;
JsonNode json;
connection = createHttpUrlConnection( WEBHDFS_URL + "?op=GETHOMEDIRECTORY" );
input = connection.getInputStream();
json = MAPPER.readTree( input );
input.close();
connection.disconnect();
assertThat( json.get( "Path" ).asText(), is( "/user/"+TEST_USERNAME ) );
}
Next a more complicated sample that writes and reads a file to HDFS via the CREATE and OPEN operations.
@Test
public void putGetFileExample() throws Exception {
HttpsURLConnection connection;
String redirect;
InputStream input;
OutputStream output;
String data = UUID.randomUUID().toString();
connection = createHttpUrlConnection( WEBHDFS_URL + "/tmp/" + data + "/?op=CREATE" );
connection.setRequestMethod( "PUT" );
assertThat( connection.getResponseCode(), is(307) );
redirect = connection.getHeaderField( "Location" );
connection.disconnect();
connection = createHttpUrlConnection( redirect );
connection.setRequestMethod( "PUT" );
connection.setDoOutput( true );
output = connection.getOutputStream();
IOUtils.write( data.getBytes(), output );
output.close();
connection.disconnect();
assertThat( connection.getResponseCode(), is(201) );
connection = createHttpUrlConnection( WEBHDFS_URL + "/tmp/" + data + "/?op=OPEN" );
assertThat( connection.getResponseCode(), is(307) );
redirect = connection.getHeaderField( "Location" );
connection.disconnect();
connection = createHttpUrlConnection( redirect );
input = connection.getInputStream();
assertThat( IOUtils.toString( input ), is( data ) );
input.close();
connection.disconnect();
}
Now of course you have probably noticed that all of the "magic" is hidden in that createHttpUrlConnection method. Not really magic at all but this is where the "un-securing" of SSL happens. This also takes care of setting up HTTP BasicAuth for authentication and disables redirects which should be done when using the WebHDFS REST APIs. private HttpsURLConnection createHttpUrlConnection( URL url ) throws Exception {
HttpsURLConnection conn = (HttpsURLConnection)url.openConnection();
conn.setHostnameVerifier( new TrustAllHosts() );
conn.setSSLSocketFactory( TrustAllCerts.createInsecureSslContext().getSocketFactory() );
conn.setInstanceFollowRedirects( false );
String credentials = TEST_USERNAME + ":" + TEST_PASSWORD;
conn.setRequestProperty( "Authorization", "Basic " + DatatypeConverter.printBase64Binary(credentials.getBytes() ) );
return conn;
}
private HttpsURLConnection createHttpUrlConnection( String url ) throws Exception {
return createHttpUrlConnection( new URL( url ) );
}
... View more
02-16-2016
02:49 PM
1 Kudo
I didn't understand that beeline was working via Knox already. A few questions then: What application is making the HS2 call via Knox? Is the application using JDBC or ODBC drivers and what version? What does your JDBC connect string look like (without real hostname or passwords of course)?
... View more
02-12-2016
02:23 PM
2 Kudos
Looks like you are missing hadoop.proxyuser.knox.groups=users
hadoop.proxyuser.knox.hosts=* Also note that you should probably not have hadoop.proxyuser.guest.groups=users
hadoop.proxyuser.guest.hosts=* as this is essentially saying that the 'guest' user is allowed to impersonate anyone in the 'users' group. Beyond that you need to ensure that your user 'adpqa' is in group 'users'.
... View more
01-26-2016
08:41 PM
2 Kudos
The potentially confusing process of adding a partition to Apache Directory Studio is the reason we decided to include the pre-populated Demo LDAP server with Knox instead of just instructions for using ADS. To do this in ADS you need to switch to the "Servers" tab in the lower right and click on Local. Then in the Partitions view on the left press "Add..." and provide the Suffix: value for example dc=custom,dc=sample,dc=com. Set ID: to something unique. Then you should be able to add subentries to that partition and you would no longer use the Knox Demo LDAP server. Keep in mind that the port is typically 10389 instead of the 33389 used by the Knox Demo LDAP. See the "General" view tab when the Local server is selected for details. You can import a LDIF using the File>Import menu item. Select LDAP Browser>DIF into LDAP. Browse for your LDIF file and Import into Local. Make sure you check "Overwrite existing log..." if you have to repeat the process. One confusing part here is that there needs to be an entry in your LDIF file for the Suffix: entered above. For example if you are trying to import the users.ldif that comes with Knox the Suffix: you would use is dc=hadoop,dc=apache,dc=org because this is the root object in that users.ldif.
... View more
01-18-2016
09:38 PM
1 Kudo
Have you read the "Configuring SSL Verification" section (pg 19) of the Hortonworks Hive
ODBC Driver with SQL
Connector
User Guide. At a minimum you will need to "Enable SSL" and "Allow Self-signed Server
Certificate". You may possibly require "Allow Common Name Host Name Mismatch" if you have DNS setup issues at your site. I'm not totally sure if you need a PEM file if you are using Self-signed certs. Try the above first. http://hortonworks.com/wp-content/uploads/2015/10/Hortonworks-Hive-ODBC-Driver-User-Guide.pdf
... View more
01-14-2016
02:39 PM
3 Kudos
I was able to set this up using
Authentication Method: Simple Authentication Bind DN or user: uid=admin,ou=people,dc=hadoop,dc=apache,dc=org This of course assumes you haven't changed the users.ldif file. I'm guessing you are trying to use your real domain but haven't updated the user.ldif file to reflect that. For reference here is the entry for the admin user in the default demo users.ldif file. dn: uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
objectclass:top
objectclass:person
objectclass:organizationalPerson
objectclass:inetOrgPerson
cn: Admin
sn: Admin
uid: admin
userPassword:xxxxxxxxxxxxxx
... View more
01-12-2016
10:03 PM
Any chance you can capture/provide the gateway.log file content from the period of time during which this occurred?
... View more
01-12-2016
10:01 PM
5 Kudos
Currently Knox does not currently support proxying the Ranger UI. If/when Knox does support proxying the Ranger UI you are correct that it may be impossible to access the Ranger UI via Knox if the Range/Knox agent is installed and if the required users have not already been granted access. Presumably setting up the required policies would be done before hand or from "within" the cluster and not via Knox.
... View more
01-11-2016
10:04 PM
2 Kudos
The answer is going to depend on exactly what you are looking for. Knox has a general purpose WebAppSercurityProvider that currently supports layering Cross Site Request Forgery (CSRF) protection onto any of the REST APIs Knox currently supports. The WebAppSecurityProvider is also extensible to so that support for other common WebApp vulnerabilities could be developed and plugged in. Knox does not currently have any support for layering injection vulnerability protection to the supported REST APIs. This is possible for some services given the architecture but would require a much tighter coupling between Knox and those services than would be ideal. Can you please clarify what you mean by "broken authentication" before I tackle that one?
... View more
01-04-2016
06:16 PM
1 Kudo
Thinking about running it "within Hadoop" may be the wrong way to think about it. From a Knox perspective "within Hadoop" is usually discussed from a security perspective. This means that everything "within Hadoop" is protected by the same firewall setup, same Kerberos configuration, etc. It is really just a collection of hosts dedicated to the various components that make up the solution. So in your case the Tomcat hosted API could simply run on one of the hosts that is part of the infrastructure dedicated to Hadoop. This API would be accessed via Knox which would be running on one of the hosts considered to be at the "permitter" from a network security perspective. All of the above being said, it is actually possible to run your Jetty or Tomcat hosted API "on Hadoop" via Slider. In this case the life-cycle of your API server would be managed by Hadoop. This would present some challenges from a Knox perspective as the Hadoop resource manager YARN may run your API server on any compute node in the Hadoop cluster making the Knox configuration challenging.
... View more
12-23-2015
03:03 PM
1 Kudo
A primary benefit of using Knox is that it insulates the clients from needing to be aware of Kerberos. However, if the HDP cluster is configured with Kerberos then Knox will need to be configured to interact with the cluster securely via Kerberos with the cluster. The clients however will be unaffected.
... View more
12-23-2015
02:59 PM
1 Kudo
Unfortunately HDP 2.2 was not certified with jdk1.8 and Knox 0.5.0.2.2 in particular has an issue with a keystore API change in jdk 1.8 that prevents it from starting. The only solution is to either upgrade HDP or downgrade the jdk.
... View more
12-22-2015
02:54 PM
2 Kudos
By default Knox has special behavior for Hadoop services that use the Hadoop Auth module.
https://hadoop.apache.org/docs/stable/hadoop-auth/... So yes it adds the user.name query parameter by default. I'm curios as to why {$username} isn't working for you though. What version of Knox are you using?
... View more
12-21-2015
04:02 PM
3 Kudos
In your rewrite.xml you can use a rewrite function to retrieve the current effective username. You can see an example of this in WebHDFS.
{code}
<rule dir="IN" name="WEBHDFS/webhdfs/inbound/namenode/home/file" pattern="*://*:*/**/webhdfs/{version}/~/{path=**}?{**}">
<rewrite template="{$serviceUrl[WEBHDFS]}/{version}/user/{$username}/{path=**}?{**}"/>
</rule>
{code} However password is a different matter. There are several issues with this. Depending upon the authorization provider there may be no password. The general Knox model is to protect the password not to make it easy to access. So from this perspective perhaps we need to understand your use case a bit better to determine if there is a different way to accomplish your goals. Without more information I'm guessing you actually need a trusted proxy model where your target service needs to trust that Knox has pre-authenticated the user and therefore only the username is required.
... View more
12-18-2015
03:40 PM
3 Kudos
You may find Sample 5 in my recent blog here helpful. http://kminder.github.io/knox/2015/11/18/knox-with... The only quick tip I can give you here without more information is that your authorization provider configuration should probably look like this.
<provider>
<role>authorization</role>
<name>AclsAuthz</name>
<enabled>true</enabled>
<param name="WEBHBASE.acl" value="admin;*;*"/>
</provider> For your custom services all you need to do is match the value before the ".acl" with the role of your custom service. This example may help clarify.
<provider>
<role>authorization</role>
<name>AclsAuthz</name>
<enabled>true</enabled>
<param name="WEBHBASE.acl" value="admin;*;*"/>
<param name="CUSTOM.acl" value="guest;*;*"/>
</provider> Of course you can also use the Ranger authorization plugin and instead of this "AclsAuthz" plugin and define the policy in the Ranger policy UI.
... View more
12-17-2015
04:11 AM
@Christopher Lewis < This looks like progress. I'm guessing the Demo LDAP server isn't running on your VM.
... View more
12-17-2015
02:52 AM
3 Kudos
If you've installed Knox via Ambari the topology name used is default instead of sandbox so try this command. curl -iku guest:guest-password -X GET 'https://localhost:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS'
... View more
12-16-2015
05:55 PM
2 Kudos
Try: "jdbc:hive2://knoxserver.net:443/;ssl=false;transportMode=http;httpPath=knox/sandbox/hive", "username", "pwd" This assumes a few things that you should double check:
You actually have SSL disabled in knox. Check ssl.enabled in gateway-site.xml. Otherwise use transportMode=https;sslTrustStore=<trustStoreFileName>;trustStorePassword=<trustStorePassword>. The <trustStoreFileName> and <trustStorePassword> need to be replaced with the real values from your system. This trust store needs to contain the public certificate used by the Knox server.
You have gatway.path in gateway.site.xml set to knox. The default is gateway. Otherwise use httpPath=gateway/sandbox/hive Your topology file is named sandbox.xml. If for example you are using Ambari your topology file is probably named default.xml so you would use httpPath=knox/default/hive or gateway/default/hive as per #2.
... View more
12-15-2015
03:14 PM
Can you share what you have either via github or pasted here?
... View more
12-14-2015
02:52 PM
3 Kudos
This is definitely possible and encouraged. Have you taken a look at the Apache Knox Developer's Guide? I've also created a blog recently called Adding a service to Apache Knox that while not directly applicable to your use case, might answer some questions you didn't know you had.
... View more
11-18-2015
03:08 PM
I also recently created a blog related to this.
http://kminder.github.io/knox/2015/11/16/adding-a-service-to-knox.html
... View more
11-12-2015
04:35 PM
3 Kudos
These extensions are committed to the Apache Knox repo itself. They all use the config driven extension model so you need to look in the gateway-service-definitions module. In particular look in this directory. Now that you mention the openweathermap example, I need to update that to the new configuration based model at least as a comparison to the code based extension. The developers guide does briefly cover the config based extension.
... View more
11-12-2015
03:04 PM
1 Kudo
We would certainly recommend the use of Knox's extensibility models to cover any components without coverage before we get there ourselves. There have been several developed in the community already such as Falcon that we don't yet officially support. The same goes for UI coverage where the community has added coverage for things like the HDFS and YARN UIs among others. The Knox Developer's Guide is a great resource that the community has used to help them jump start these efforts. Of course looking at the implementation of the existing integrations is a great place to start as well.
... View more
11-02-2015
03:15 PM
1 Kudo
Just to add context to this correct answer, the password required here to access the gateway.jks keystore is the password provided as the Knox master secret in Ambari when Knox was installed. The Ambari install scripts for Knox use the described knoxcli.sh create-master command "under the covers".
... View more