Support Questions

VidyaSargur · ‎09-17-2019

All,

while installing cdsw tcp-ingress controller and ds_reconsiler pods are failing due to connection failure with Grps server.. from below web service logs i see grps server status failing with port binding (description":"Failed to bind to port","file":"../deps/grpc/src/core/lib/iomgr/tcp_uv.cc","file_line":73,
"grpc_status":14,"os_error":"invalid argument"}]}]}), any suggestions how to fix this...

[..]

reated":"@1568398264.632135163","description":"No address added out of total 1 resolved","file":"../deps/grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc",
"file_line":349,"referenced_errors":[{"created":"@1568398264.632132593","description":"Failed to add port to server","file":"../deps/grpc/src/core/lib/iomgr/tcp_server_custom.cc",
"file_line":402,"referenced_errors":[{"created":"@1568398264.632131142","description":"Failed to bind to port","file":"../deps/grpc/src/core/lib/iomgr/tcp_uv.cc","file_line":73,"grpc_status":14,"os_error":"invalid argument"}]}]}
E0913 18:11:04.634478860 1 server_secure_chttp2.cc:81] {"created":"@1568398264.634442264","description":"No address added out of total 1 resolved","file":"../deps/grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc",
"file_line":349,"referenced_errors":[{"created":"@1568398264.634440097","description":"Failed to add port to server",
"file":"../deps/grpc/src/core/lib/iomgr/tcp_server_custom.cc","file_line":402,"referenced_errors":[{"created":"@1568398264.634439104","description":"Failed to bind to port","file":"../deps/grpc/src/core/lib/iomgr/tcp_uv.cc","file_line":73,
"grpc_status":14,"os_error":"invalid argument"}]}]}
[{"code":"EXTRA_REFERENCE_PROPERTIES","message":"Extra JSON Reference properties will be ignored: type, description","path":["definitions","DescribeRunResponse"]},
{"code":"EXTRA_REFERENCE_PROPERTIES","message":"Extra JSON Reference properties will be ignored: type, description","path":["definitions","RepeatRunResponse"]},
{"code":"EXTRA_REFERENCE_PROPERTIES","message":"Extra JSON Reference properties will be ignored: type, description","path":["definitions","Run","properties","submitter"]},
{"code":"UNUSED_DEFINITION","message":"Definition is not used: #/definitions/ValidationError","path":["definitions","ValidationError"]}] 2
2019-09-13 18:12:08.090 1 INFO AppServer.HTTP Req-ff9b1780-d651-11e9-b240-a59f514efc13 GET chd6-cdsw.udl.net /editor/file/NativeFileError.js 304 34.273 ms - -
2019-09-13 18:39:14.744 1 INFO AppServer.HTTP Req-c92f3830-d655-11e9-b240-a59f514efc13 GET chd6-cdsw.udl.net /editor/file/NativeFileError.js 304 4.623 ms - -

AutoIN · ‎09-18-2019

@aahbs we recently observed this with CDSW 1.6 on hosts which have ipv6 disabled. If you're hitting this behaviour please check dmesg, it would likely show segfaults on node process. We are working internally to understand the GRPC behaviour and its connection with ipv6 but in the meantime, you might want to enable ipv6 per the RedHat article https://access.redhat.com/solutions/8709#rhel7enable

1. Edit /etc/default/grub and delete the entry ipv6.disable=1 from the GRUB_CMDLINE_LINUX, like the following sample:

   GRUB_CMDLINE_LINUX="rd.lvm.lv=rhel/swap crashkernel=auto rd.lvm.lv=rhel/root"

2. Run the grub2-mkconfig command to regenerate the grub.cfg file:

 # grub2-mkconfig -o /boot/grub2/grub.cfg

Alternatively, on UEFI systems, run the following:

   # grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

3. Delete the file /etc/sysctl.d/ipv6.conf which contains the entry:

# To disable for all interfaces 
net.ipv6.conf.all.disable_ipv6 = 1 
# the protocol can be disabled for specific interfaces as well. 
net.ipv6.conf.<interface>.disable_ipv6 = 1

4. Check the content of the file /etc/ssh/sshd_config and make sure the AddressFamily line is commented:

 #AddressFamily inet

5. Make sure the following line exists in /etc/hosts, and is not commented out:

 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

6. Enable ipv6 support on the ethernet interface. Double check /etc/sysconfig/network and /etc/sysconfig/network-scripts/ifcfg-* and ensure we've IPV6INIT=yes .This setting is required for IPv6 static and DHCP assignment of IPv6 addresses.

7. Stop CDSW service.
8. Reboot the CDSW hosts to enable IPv6 support.

9. Start CDSW service

View solution in original post

AutoIN · ‎09-20-2019

@aahbs good point. Certain organizations which makes use of firewall or proxies, can block websockets. If your browser shows problems with websockets using Chrome Developer Tools, it's likely the case. You might want to speak with your network admin and get this sorted. Regarding the extension, see if you can download the chrome extension on a machine which has internet connectivity and then scp install it manually on your laptop.

aahbs · ‎09-23-2019

@AutoIN Yes you are right, this might be causing the issue let me check and update with the outcome!

Thank you!

aahbs · ‎10-04-2019

Hi ,

1. We are still facing the issue,We have check with networks team they didn't see firewalls blocking this.

when we check with cloudera even in their lab they see below error in session pod logs, but after clearing the cache its start working , though its not a perminant solution. In our case we are not able to access session/terminal even after clearing the cache but we didnt see below error in session pods.

[..]

fication error","user":"cdsw"}
2019-10-04 23:35:39.713 1 ERROR EngineInit.BrowserSvcs jv89ty4xw6v4tgkm Finish isAuthorized, failed to get auth cookie subject data = {"err":"crypto/rsa: verification error","user":"cdsw"}
2019-10-04 23:35:39.713 1 ERROR EngineInit.BrowserSvcs jv89ty4xw6v4tgkm Error from memo.Get data = {"err":"crypto/rsa: verification error","user":"cdsw"}
2019-10-04 23:35:39.713 1 WARNING EngineInit.BrowserSvcs jv89ty4xw6v4tgkm Rejecting unauthorized request data = {"scope":"session-like","status":401,"user":"cdsw"}
2019-10-04 23:35:41.966 1 ERROR EngineInit.BrowserSvcs jv89ty4xw6v4tgkm Error from memo.Get data = {"err":"crypto/rsa: verification error","user":"cdsw"}
2019-10-04 23:35:41.966 1 WARNING EngineInit.BrowserSvcs jv89ty4xw6v4tgkm Re

[..]

2. after restarting cdsw we are seeing above error eventually after couple ours when we try to launch session we see different error as below in session pods..,wierd..

[..]

2019-10-01 17:37:20.248 1 ERROR EngineInit.BrowserSvcs uc1wc74ulb3xn7o5 Finish apiGet, failed to parse resp body data = {"err":"invalid character '\u003c' looking for beginning of value","user":"cdsw"}

[..]

Did you see any such behavour and any suggestions....please..

Thank You!

aahbs · ‎10-15-2019

After opening websocket traffic at firewall level we are able to launch the sessions and run basic commands.

But still not able to launch the session terminal, getting below error not sure where this is coming from.

[..]

0/16/2019 12:52 AM 1 ERROR EngineInit.BrowserSvcs 4ubgvkai9lozx6q6 Finish apiGet, failed to parse resp body data = {"err":"invalid character '\u003c' looking for beginning of value","user":"cdsw"}

10/16/2019 12:52 AM 1 ERROR EngineInit.BrowserSvcs 4ubgvkai9lozx6q6 Finish isAuthorized, failed to get user permissions from API data = {"err":"invalid character '\u003c' looking for beginning of value","user":"cdsw"}

10/16/2019 12:52 AM 1 ERROR EngineInit.BrowserSvcs 4ubgvkai9lozx6q6 Error from memo.Get data = {"err":"invalid character '\u003c' looking for beginning of value","user":"cdsw"}

10/16/2019 12:52 AM 1 ERROR EngineInit.BrowserSvcs 4ubgvk

AutoIN · ‎10-24-2019

@aahbs Thanks for the call today. Let's see if we can narrow those 401's to the browser level (Chrome).

aahbs · ‎10-24-2019

@AutoIN it was nice talking to you! it will be good if we can narrow it down there will keep updated with firefox, IE outcome.Thanks for all your support

aahbs · ‎10-24-2019

@AutoIN just now observed issue with firefox as well after restart. attached screenshots in case.

aahbs · ‎12-20-2019

Hi @AutoIN

Hope doing great!

We are planning to install cdsw 1.6.1 in new environment, just wanted to check if we still have to enable ipv6 as we did in 1.6.0.

Thanks,

Srinivas

AutoIN · ‎12-20-2019

Thanks Srinivas. All is well. Thx

Yes, ipv6 is still a requirement in 1.6.1. Efforts are still undergoing to investigate options to NOT require IPv6 to be enabled in future CDSW versions.

Cloudera Community

Support Questions

cdsw -tcp-ingress controller failing..grpc_status":14,"os_error":"invalid argument"}]}]}