Support Questions
Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Advanced Search

KNOX - GATEWAY - FILE READ OPERATION

Highlighted

KNOX - GATEWAY - FILE READ OPERATION

ssanupindi
Explorer

Created ‎04-24-2017 04:54 PM

Friends, can anyone please help me with the following KNOX Read a File issue:

This is currently working for "listing a file (using "ListStatus") with our Knox Gateway LOAD BALANCE R URL:

$ curl -i -k -L -u <user id> 'https://knoxgateway.<CORP DOMAIN>:443/gateway/default/webhdfs/v1/user/<user id>/servers?op=ListStatus' Enter host password for user '<user id>': HTTP/1.1 200 OK Set-Cookie: JSESSIONID=9m4tcprbrs1eapxa0ljk5sfj;Path=/gateway/default;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: no-cache Expires: Mon, 24 Apr 2017 13:25:28 GMT Date: Mon, 24 Apr 2017 13:25:28 GMT Pragma: no-cache Expires: Mon, 24 Apr 2017 13:25:28 GMT Date: Mon, 24 Apr 2017 13:25:28 GMT Pragma: no-cache Server: Jetty(6.1.26.hwx) Content-Type: application/json Content-Length: 281 {"FileStatuses":{"FileStatus":[{"accessTime":1492803070763,"blockSize":134217728,"childrenNum":0,"fileId":219423467,"group":"hdfs","length":249,"modificationTime":1492803071085,"owner":"<user id>","pathSuffix":"","permission":"777","replication":3,"storagePolicy":0,"type":"FILE"}

Now, i want to read this file using the same KNOX GATEWAY LB URL:

$curl -i -k -L -u <user id> 'https://knoxgateway.<CORP DOMAIN>:443/gateway/test/webhdfs/v1/user/<user id>/servers?op=OPEN' < Enter host password for user '<user id>': HTTP/1.1 307 Temporary Redirect Set-Cookie: JSESSIONID=1sopqfdu53c61xutx0ufk0hij;Path=/gateway/test;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: no-cache Expires: Mon, 24 Apr 2017 13:34:31 GMT Date: Mon, 24 Apr 2017 13:34:31 GMT Pragma: no-cache Expires: Mon, 24 Apr 2017 13:34:31 GMT Date: Mon, 24 Apr 2017 13:34:31 GMT Pragma: no-cache Location: https://knoxgateway.<CORP DOMAIN>/knox/test/webhdfs/data/v1/webhdfs/v1/user/<user id>/servers?_=AAAACAAAABAAAAEAlcGYLi4LTj7bhrrDPr1o2u6UIMEkO_aYiGAxiS4hu39uo-Homt5CbB2pwJ9p0Lkl2-7-l0vxINRjR70Ub7SA3D_ZKcoN46q0Bj97ceByV8hZgwEiIvyZmwSYEdKTVRCKV3VOhbuw1peDAJMhlS8SwYoPsRUOmPsdbmX5NLysp7mM7qktkmbHJyf_qXiAwNYuXmIhPBW_PZMmwjmQXckj7mDGAk61P-qWy1rSPoyPZ5oZ6y-7Uwijew0C3FNZzISDJICX6ePU2ptLEJOu1G8FaQonOUi37pvblYUuKSo-0wiLnBKRIvzrjfPzvh0tKrXi7FbCQnbn9sG0IyFjWssqlIoOlUVbf-Jo9eVF653ZyIqGjIYn9aX-7g Server: Jetty(6.1.26.hwx) Content-Type: application/octet-stream Content-Length: 0 HTTP/1.1 404 Not Found Cache-Control: must-revalidate,no-cache,no-store Content-Type: text/html;charset=ISO-8859-1 Content-Length: 1324 Server: Jetty(8.1.14.v20131031) <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 Not Found</title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /knox/default/webhdfs/data/v1/webhdfs/v1/user/<user id>/servers. Reason: <pre> Not Found</pre></p> <hr /><i><small>Powered by Jetty://</small></i>

I am really confused with this Error: "

Problem accessing /knox/default/webhdfs/data/v1/webhdfs/v1/user/<user id>/servers: --> For me this path is wrong and i guess it is redirecting to a wrong path using 'rewrite.xml'? but i don't think we ever modified this file during the KNOX setup.

Can anyone please help me / guide me for fixing this issue? I greatly appreciate your help.

thank you,

Reply
1,349 Views
0 Kudos
4 REPLIES 4
Highlighted

Re: KNOX - GATEWAY - FILE READ OPERATION

dvillarreal
Guru

Created ‎05-12-2017 06:23 AM

Here it is working on my server. Maybe this may shed some light.

[root@groot1 topologies]# curl -ivk -u dvillarreal 'https://localhost:8443/gateway/default/webhdfs/v1/zone_encr/?op=LISTSTATUS'

Enter host password for user 'dvillarreal':

* About to connect() to localhost port 8443 (#0)

* Trying 127.0.0.1... connected

* Connected to localhost (127.0.0.1) port 8443 (#0)

* Initializing NSS with certpath: sql:/etc/pki/nssdb

* warning: ignoring value of ssl.verifyhost

* skipping SSL peer certificate verification

* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

* Server certificate:

* subject: CN=groot1.openstacklocal,OU=Test,O=Hadoop,L=Test,ST=Test,C=US

* start date: Apr 27 20:08:39 2017 GMT

* expire date: Apr 27 20:08:39 2018 GMT

* common name: groot1.openstacklocal

* issuer: CN=groot1.openstacklocal,OU=Test,O=Hadoop,L=Test,ST=Test,C=US

* Server auth using Basic with user 'dvillarreal'

> GET /gateway/default/webhdfs/v1/zone_encr/?op=LISTSTATUS HTTP/1.1

> Authorization: Basic ZHZpbGxhcnJlYWw6aGFkb29wMTIzNDUh

> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2

> Host: localhost:8443

> Accept: */*

>

< HTTP/1.1 200 OK

HTTP/1.1 200 OK

< Date: Thu, 11 May 2017 23:33:08 GMT

Date: Thu, 11 May 2017 23:33:08 GMT

< Set-Cookie: JSESSIONID=ayrhe6eilreq1egldq6hc6uu4;Path=/gateway/default;Secure;HttpOnly

Set-Cookie: JSESSIONID=ayrhe6eilreq1egldq6hc6uu4;Path=/gateway/default;Secure;HttpOnly

< Expires: Thu, 01 Jan 1970 00:00:00 GMT

Expires: Thu, 01 Jan 1970 00:00:00 GMT

< Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Wed, 10-May-2017 23:33:09 GMT

Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Wed, 10-May-2017 23:33:09 GMT

< Cache-Control: no-cache

Cache-Control: no-cache

< Expires: Thu, 11 May 2017 23:33:09 GMT

Expires: Thu, 11 May 2017 23:33:09 GMT

< Date: Thu, 11 May 2017 23:33:09 GMT

Date: Thu, 11 May 2017 23:33:09 GMT

< Pragma: no-cache

Pragma: no-cache

< Expires: Thu, 11 May 2017 23:33:09 GMT

Expires: Thu, 11 May 2017 23:33:09 GMT

< Date: Thu, 11 May 2017 23:33:09 GMT

Date: Thu, 11 May 2017 23:33:09 GMT

< Pragma: no-cache

Pragma: no-cache

< X-FRAME-OPTIONS: SAMEORIGIN

X-FRAME-OPTIONS: SAMEORIGIN

< Content-Type: application/json; charset=UTF-8

Content-Type: application/json; charset=UTF-8

< Server: Jetty(6.1.26.hwx)

Server: Jetty(6.1.26.hwx)

< Content-Length: 1419

Content-Length: 1419

<

{"FileStatuses":{"FileStatus":[{"accessTime":0,"blockSize":0,"childrenNum":0,"encBit":true,"fileId":37375,"group":"hdfs","length":0,"modificationTime":1494457290326,"owner":"hdfs","pathSuffix":".Trash","permission":"1777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":1494520647770,"blockSize":134217728,"childrenNum":0,"encBit":true,"fileId":38869,"group":"hdfs","length":0,"modificationTime":1494520647770,"owner":"dvillarreal","pathSuffix":"Screen Shot 2017-04-28 at 3.40.25 PM.png","permission":"644","replication":3,"storagePolicy":0,"type":"FILE"},{"accessTime":1494521001429,"blockSize":134217728,"childrenNum":0,"encBit":true,"fileId":38879,"group":"hdfs","length":52624,"modificationTime":1494521002174,"owner":"dvillarreal","pathSuffix":"Screen Shot 2017-04-28 at 3.43.55 PM.png","permission":"644","replication":3,"storagePolicy":0,"type":"FILE"},{"accessTime":1494519148636,"blockSize":134217728,"childrenNum":0,"encBit":true,"fileId":38834,"group":"hdfs","length":0,"modificationTime":1494* Connection #0 to host localhost left intact

* Closing connection #0

519148636,"owner":"dvillarreal","pathSuffix":"mag7.jpg","permission":"644","replication":3,"storagePolicy":0,"type":"FILE"},{"accessTime":1494457615918,"blockSize":134217728,"childrenNum":0,"encBit":true,"fileId":37384,"group":"hdfs","length":28,"modificationTime":1494457616450,"owner":"dvillarreal","pathSuffix":"test.txt","permission":"644","replication":3,"storagePolicy":0,"type":"FILE

[root@groot1 topologies]# curl -ivLk -u dvillarreal 'https://localhost:8443/gateway/default/webhdfs/v1/zone_encr/test.txt?op=OPEN'

Enter host password for user 'dvillarreal':

* About to connect() to localhost port 8443 (#0)

* Trying 127.0.0.1... connected

* Connected to localhost (127.0.0.1) port 8443 (#0)

* Initializing NSS with certpath: sql:/etc/pki/nssdb

* warning: ignoring value of ssl.verifyhost

* skipping SSL peer certificate verification

* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

* Server certificate:

* subject: CN=groot1.openstacklocal,OU=Test,O=Hadoop,L=Test,ST=Test,C=US

* start date: Apr 27 20:08:39 2017 GMT

* expire date: Apr 27 20:08:39 2018 GMT

* common name: groot1.openstacklocal

* issuer: CN=groot1.openstacklocal,OU=Test,O=Hadoop,L=Test,ST=Test,C=US

* Server auth using Basic with user 'dvillarreal'

> GET /gateway/default/webhdfs/v1/zone_encr/test.txt?op=OPEN HTTP/1.1

> Authorization: Basic ZHZpbGxhcnJlYWw6aGFkb29wMTIzNDUh

> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2

> Host: localhost:8443

> Accept: */*

>

< HTTP/1.1 307 Temporary Redirect

HTTP/1.1 307 Temporary Redirect

< Date: Thu, 11 May 2017 23:33:53 GMT

Date: Thu, 11 May 2017 23:33:53 GMT

< Set-Cookie: JSESSIONID=cmi3xz9vz22aztv01vy60fje;Path=/gateway/default;Secure;HttpOnly

Set-Cookie: JSESSIONID=cmi3xz9vz22aztv01vy60fje;Path=/gateway/default;Secure;HttpOnly

< Expires: Thu, 01 Jan 1970 00:00:00 GMT

Expires: Thu, 01 Jan 1970 00:00:00 GMT

< Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Wed, 10-May-2017 23:33:54 GMT

Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Wed, 10-May-2017 23:33:54 GMT

< Cache-Control: no-cache

Cache-Control: no-cache

< Expires: Thu, 11 May 2017 23:33:54 GMT

Expires: Thu, 11 May 2017 23:33:54 GMT

< Date: Thu, 11 May 2017 23:33:54 GMT

Date: Thu, 11 May 2017 23:33:54 GMT

< Pragma: no-cache

Pragma: no-cache

< Expires: Thu, 11 May 2017 23:33:54 GMT

Expires: Thu, 11 May 2017 23:33:54 GMT

< Date: Thu, 11 May 2017 23:33:54 GMT

Date: Thu, 11 May 2017 23:33:54 GMT

< Pragma: no-cache

Pragma: no-cache

< X-FRAME-OPTIONS: SAMEORIGIN

X-FRAME-OPTIONS: SAMEORIGIN

< Content-Type: application/octet-stream

Content-Type: application/octet-stream

< Location: https://localhost:8443/gateway/default/webhdfs/data/v1/webhdfs/v1/zone_encr/test.txt?_=AAAACAAAABAAA...

Location: https://localhost:8443/gateway/default/webhdfs/data/v1/webhdfs/v1/zone_encr/test.txt?_=AAAACAAAABAAA...

< Server: Jetty(6.1.26.hwx)

Server: Jetty(6.1.26.hwx)

< Content-Length: 0

Content-Length: 0

<

* Connection #0 to host localhost left intact

* Issue another request to this URL: 'https://localhost:8443/gateway/default/webhdfs/data/v1/webhdfs/v1/zone_encr/test.txt?_=AAAACAAAABAAAAEQGeIZcVX_mUa9HOTHUCBIZ7b_iNiz924O7UBVlI3ZPZeYbhzO8LW0SVhKlX3zUvhuykF7TisStFefLuYdHNSYIOmsoeB3MPAoVIGUvnTHmlEBko2aDm6r7OvYm0Ytkk4WhS5Xtn-TSWPt5OGYsa-trOUi2OyTY5lkGw0Iy-iKrlSV_svcO_0hX53C73NnCCMBJYVV8NiCHUX0qpv7IzcYZGCS2wyiuwwNnhPexTUpJcCZhT40MjMCCDauex_uaUdgYHPZKFH1BzFtIJKWYUbGKe_KiB4goWEyVqF2NHj0R58-jLcYewuPClbmquX3A8VHt9O2YSw-_WWtb_nIsTx1HMYFC5iPajfqsk9FKxtSTBtpP0dkhrjBnWfa15chNgfrZIaZ5cr5Er4'

* Re-using existing connection! (#0) with host localhost

* Connected to localhost (127.0.0.1) port 8443 (#0)

* Server auth using Basic with user 'dvillarreal'

> GET /gateway/default/webhdfs/data/v1/webhdfs/v1/zone_encr/test.txt?_=AAAACAAAABAAAAEQGeIZcVX_mUa9HOTHUCBIZ7b_iNiz924O7UBVlI3ZPZeYbhzO8LW0SVhKlX3zUvhuykF7TisStFefLuYdHNSYIOmsoeB3MPAoVIGUvnTHmlEBko2aDm6r7OvYm0Ytkk4WhS5Xtn-TSWPt5OGYsa-trOUi2OyTY5lkGw0Iy-iKrlSV_svcO_0hX53C73NnCCMBJYVV8NiCHUX0qpv7IzcYZGCS2wyiuwwNnhPexTUpJcCZhT40MjMCCDauex_uaUdgYHPZKFH1BzFtIJKWYUbGKe_KiB4goWEyVqF2NHj0R58-jLcYewuPClbmquX3A8VHt9O2YSw-_WWtb_nIsTx1HMYFC5iPajfqsk9FKxtSTBtpP0dkhrjBnWfa15chNgfrZIaZ5cr5Er4 HTTP/1.1

> Authorization: Basic ZHZpbGxhcnJlYWw6aGFkb29wMTIzNDUh

> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2

> Host: localhost:8443

> Accept: */*

>

< HTTP/1.1 200 OK

HTTP/1.1 200 OK

< Date: Thu, 11 May 2017 23:33:54 GMT

Date: Thu, 11 May 2017 23:33:54 GMT

< Set-Cookie: JSESSIONID=vhf31ukoxmintdfe2h5ekg2y;Path=/gateway/default;Secure;HttpOnly

Set-Cookie: JSESSIONID=vhf31ukoxmintdfe2h5ekg2y;Path=/gateway/default;Secure;HttpOnly

< Expires: Thu, 01 Jan 1970 00:00:00 GMT

Expires: Thu, 01 Jan 1970 00:00:00 GMT

< Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Wed, 10-May-2017 23:33:54 GMT

Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Wed, 10-May-2017 23:33:54 GMT

< Access-Control-Allow-Methods: GET

Access-Control-Allow-Methods: GET

< Access-Control-Allow-Origin: *

Access-Control-Allow-Origin: *

< Content-Type: application/octet-stream

Content-Type: application/octet-stream

< Connection: close

Connection: close

< Server: Jetty(9.2.15.v20160210)

Server: Jetty(9.2.15.v20160210)

<

This is a test for enc zone

* Closing connection #0

Reply
729 Views
0 Kudos

Re: KNOX - GATEWAY - FILE READ OPERATION

dvillarreal
Guru

Created ‎05-12-2017 03:49 PM

Most likely something wrong with your load balancer configuration. Here is an example : http://knox.apache.org/books/knox-0-12-0/user-guide.html#High+Availability+with+Apache+HTTP+Server++...

Reply
729 Views
0 Kudos
Highlighted

Re: KNOX - GATEWAY - FILE READ OPERATION

ssanupindi
Explorer

Created ‎05-12-2017 09:01 PM

dvillarreal, Thank you for the response, we have actually found the root cause, apologies for not posting solution we implemented on this forum earlier, after enabling debug we found our knox hosts were not able to connect individual datanodes on port 1022, once this FW is resolved, our external tool user was able to read a file from HDFS, but we are still having an issue and it is: when i tried using curl command to read a file from an edgenode going through knox, i am still not able to connect and the output log shows, it was trying to hit :KNOX LB URL on port 8443 (KNOX LB listens on port 443), but in knox config, we got a front end URL: <KNOX LB URL>/gateway, but when we pursued with HW support, they told me that we need to change 'front end URL" to: <KNOX LB URL>:443/gateway, by default it was trying to hit KNOX LB on port 8443, so to avoid this, they asked me to include port: 443 also, but my concern is, our external tool user is able to use KNOX without any issues, so i am not sure to make this change or not? Can you please advise me?

Reply
729 Views
0 Kudos
Highlighted

Re: KNOX - GATEWAY - FILE READ OPERATION

sshelgao
Explorer

Created ‎05-23-2018 10:51 AM

Hi ,

I am also facing 403 Forbidden issue while reading a file from WEBHDFS (OPEN operation), where i have enabled KNOX and kerberos. But on the sam efile LISTSTATUS opertaion is working fine.

Here is the error: https://host:8443/gateway/default/webhdfs/v1/user/admin/hive/querystatus/stdout?op=OPEN

<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/> <title>Error 403 Forbidden</title> </head> <body><h2>HTTP ERROR 403</h2> <p>Problem accessing /gateway/default/webhdfs/data/v1/webhdfs/v1/user/knox/test/customers.csv. Reason: <pre> Forbidden</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/> </body> </html>

Please help me with this.

Thanks.

Reply
729 Views
0 Kudos
Don't have an account?
Register
Announcements
Product Announcements
[ANNOUNCE] New Cloudera JDBC 2.6.20 Driver for Apache Impala Released
Product Announcements
Transition to private repositories for CDH, HDP and HDF
Product Announcements
[ANNOUNCE] New Applied ML Research from Cloudera Fast Forward: Few-Shot Text Classification
Product Announcements
[ANNOUNCE] New JDBC 2.6.13 Driver for Apache Hive Released
Product Announcements
[ANNOUNCE] Refreshed Research from Cloudera Fast Forward: Semantic Image Search and Federated Learning
View More Announcements