Cloudera Community
MSharma
user rank
Expert Contributor
My Accepted Solutions
Show All

Re: kudu soft memory error

by Expert Contributor in Support Questions
‎06-23-2017 09:17 AM
‎06-23-2017 09:17 AM
  Ok , i have increased memory to 10GB .kudu version 1.3.0-1.cdh5.11.0.p0.12 one dir per tablet server  ,total 4 tablet server      748f9ff1944b5490f23631bcd351da (wewcw0hd3dn04.example.com:7050): Couldn't send request to peer 6e748f9ff1944b5490f23631bcd351da for tablet 0bcada0b3ff54885a5db5d234f51bc10. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 53 times. W0623 12:13:45.419793 22784 consensus_peers.cc:357] T 4a9cc150758744a0a2fd477c5dcb7ff3 P 63b4ef88fb84431ea93a79304c3b9bb8 -> Peer 6e748f9ff1944b5490f23631bcd351da (wewcw0hd3dn04.example.com:7050): Couldn't send request to peer 6e748f9ff1944b5490f23631bcd351da for tablet 4a9cc150758744a0a2fd477c5dcb7ff3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 43 times. W0623 12:13:45.453810 22784 consensus_peers.cc:357] T 4fdce491787a4be3b4e0cef44107b191 P 63b4ef88fb84431ea93a79304c3b9bb8 -> Peer b2ead65ab0164f5b8db24d700a2c474a (wewcw0hd3dn04.example.com:7050): Couldn't send request to peer b2ead65ab0164f5b8db24d700a2c474a for tablet 4fdce491787a4be3b4e0cef44107b191. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 29 times. W0623 12:13:45.498914 22784 consensus_peers.cc:357] T 4c8c4a5bcdc24f9a87ec5800818ca937 P 63b4ef88fb84431ea93a79304c3b9bb8 -> Peer 6e748f9ff1944b5490f23631bcd351da (wewcw0hd3dn04.example.com:7050): Couldn't send request to peer 6e748f9ff1944b5490f23631bcd351da for tablet 4c8c4a5bcdc24f9a87ec5800818ca937. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: NOT_STARTED. Retrying in the next heartbeat period. Already tried 263 times.   ... View more

kudu soft memory error

by Expert Contributor in Support Questions
‎06-23-2017 08:42 AM
‎06-23-2017 08:42 AM
Hi , I am doing upsert in kudu table and getting error  Kudu error(s) reported, first error: Timed out: Failed to write batch of 227 ops to tablet 8b19e4a0362e4b82941e54d33ac9c5a2 after 1 attempt(s): Failed to write to server: b2ead65ab0164f5b8db24d700a2c474a (wewcw0hd3dn02.example.com:7050): Write RPC to 10.11.100.85:7050 timed out after 179.977s (SENT)   when i checked log of tablet server i found    W0623 11:30:58.060768 13885 consensus_peers.cc:357] T 4c8c4a5bcdc24f9a87ec5800818ca937 P b2ead65ab0164f5b8db24d700a2c474a -> Peer 6e748f9ff1944b5490f23631bcd351da (wewcw0hd3dn04.example.com:7050): Couldn't send request to peer 6e748f9ff1944b5490f23631bcd351da for tablet 4c8c4a5bcdc24f9a87ec5800818ca937. Status: Remote error: Service unavailable: Soft memory limit exceeded (at 98.77% of capacity). Retrying in the next heartbeat period. Already tried 32 times. W0623 11:30:58.318686 13885 consensus_peers.cc:357] T 6bb3bdb188b44048bcddd70db21158dc P b2ead65ab0164f5b8db24d700a2c474a -> Peer 6e748f9ff1944b5490f23631bcd351da (wewcw0hd3dn04.example.com:7050): Couldn't send request to peer 6e748f9ff1944b5490f23631bcd351da for tablet 6bb3bdb188b44048bcddd70db21158dc. Status: Remote error: Service unavailable: Soft memory limit exceeded (at 96.85% of capacity). Retrying in the next heartbeat period. Already tried 40 times.   intitally i thought it is because of batchsize , i used the batch size 10000 and also  doubled memory hard limit in kudu to 2GB but still same error is coming . ... View more
Labels:

Re: kudu service are getting down frequently

by Expert Contributor in Support Questions
‎06-19-2017 07:45 AM
3 Kudos
‎06-19-2017 07:45 AM
3 Kudos
I was able to resolve it .   if you see error with ntptime mostly kudu service will go down , so you have to restart ntpd and then this error will go . [root@wcw0hd3dn02 ~]# ntptime ntp_gettime() returns code 5 (ERROR)   time dce7466c.fc37b000  Sun, Jun 11 2017  0:32:44.985, (.985225),   maximum error 16000000 us, estimated error 16 us, TAI offset 0 ntp_adjtime() returns code 5 (ERROR)   modes 0x0 (),   offset 0.000 us, frequency 0.000 ppm, interval 1 s,   maximum error 16000000 us, estimated error 16 us,   status 0x4041 (PLL,UNSYNC,MODE),   time constant 7, precision 1.000 us, tolerance 500 ppm,     this error comes if you run ntp with -x option    [root@wuwcw0hd3mn01 ~]# ps -ef|grep ntp root 3183 2731 0 10:38 pts/0 00:00:00 grep ntp ntp 20736 1 0 Jun13 ? 00:00:19 ntpd -x -u ntp:ntp -p /var/run/ntpd.pid -g   remove -X from belwo file and restart ntp  [root@wuwcw0hd3mn01 ~]# more /etc/sysconfig/ntpd # Drop root to id 'ntp:ntp' by default. OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid -g"   wait for ntp to synchronize ,after that i didnt see any issue in kudu service so far.   ntp_gettime() returns code 0 (OK) time dcf260c3.66c6abfc Mon, Jun 19 2017 10:40:03.401, (.401469911), maximum error 394157 us, estimated error 345 us, TAI offset 0 ntp_adjtime() returns code 0 (OK) modes 0x0 (), offset -707.277 us, frequency 20.094 ppm, interval 1 s, maximum error 394157 us, estimated error 345 us, status 0x6001 (PLL,NANO,MODE), time constant 10, precision 0.001 us, tolerance 500 ppm,     check this https://access.redhat.com/solutions/38542     ... View more

Re: kudu service are getting down frequently

by Expert Contributor in Support Questions
‎06-16-2017 09:51 AM
‎06-16-2017 09:51 AM
you need to be login as an administrator to add these values. ... View more

Re: kudu service are getting down frequently

by Expert Contributor in Support Questions
‎06-14-2017 09:11 PM
‎06-14-2017 09:11 PM
kudu --> configuration -- "Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile ... View more

Re: kudu service are getting down frequently

by Expert Contributor in Support Questions
‎06-14-2017 08:32 PM
‎06-14-2017 08:32 PM
not yet but restarting ntp service cause more trouble so i have put --use-hybrid-clock=false . but mostly it is a network delay between ntp server and kudu server which is causing this . i am still troubleshooting this problem ,will update here if we can do anything to reslove it ... View more

Re: Sqoop2 metastore configuration error

by Expert Contributor in Support Questions
‎06-13-2017 08:26 PM
‎06-13-2017 08:26 PM
CDH 5.7.5 ... View more

Sqoop2 metastore configuration error

by Expert Contributor in Support Questions
‎06-13-2017 02:10 PM
‎06-13-2017 02:10 PM
Hi , I am trying to configure postgres metastore from sqoop2 but getting error whiel starting the sqoop2 Sqoop server failed to start java.lang.RuntimeException: Failure in server initialization at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:63) at org.apache.sqoop.server.ServerInitializer.contextInitialized(ServerInitializer.java:41) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4709) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:802) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944) at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1068) at org.apache.catalina.core.StandardHost.start(StandardHost.java:822) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1060) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) at org.apache.catalina.core.StandardService.start(StandardService.java:525) at org.apache.catalina.core.StandardServer.start(StandardServer.java:759) at org.apache.catalina.startup.Catalina.start(Catalina.java:595) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: org.apache.sqoop.common.SqoopException: REPO_0002:Repository structures are not in suitable state, might require upgrade at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:130) at org.apache.sqoop.repository.RepositoryManager.initialize(RepositoryManager.java:78) at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:56) ... 25 more I have created the sqoop db in postgres  followed doc  https://www.cloudera.com/documentation/enterprise/5-5-x/topics/install_sqoop_ext_db.html     List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges -----------+----------+----------+-------------+-------------+----------------------- postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | sqoop | sqoop | UTF8 | en_US.UTF8 | en_US.UTF8 | template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres template1 | postgres | UTF8 | en_US.UTF8 | en_US.UTF8 | (4 rows)   ... View more
Labels:

Re: kudu service are getting down frequently

by Expert Contributor in Support Questions
‎06-10-2017 09:21 PM
‎06-10-2017 09:21 PM
@adar - yes DN not require NTP but if ntp is out of sync on these DNs CM will report clock offset . NTP is running on DNs . [root@wuwcw0hd3dn01 hadoop-hdfs]# ntptime ntp_gettime() returns code 5 (ERROR) time dce74398.988fc000 Sun, Jun 11 2017 0:20:40.595, (.595943), maximum error 16000000 us, estimated error 16 us, TAI offset 0 ntp_adjtime() returns code 5 (ERROR) modes 0x0 (), offset 0.000 us, frequency 0.000 ppm, interval 1 s, maximum error 16000000 us, estimated error 16 us, status 0x4041 (PLL,UNSYNC,MODE), time constant 7, precision 1.000 us, tolerance 500 ppm, [root@wuwcw0hd3dn01 hadoop-hdfs]# these all are physical servers ... View more

kudu service are getting down frequently

by Expert Contributor in Support Questions
‎06-09-2017 01:20 PM
‎06-09-2017 01:20 PM
Hi, Kudu crashing frequently with error Couldn't get the current time: Clock unsynchronized. Status: Service unavailable: Error reading clock. Clock considered unsynchronized however i am not seeing any clock offset error in cluster . kudu documents says this could be because of network delay between NTP server and kudu host. but kudu is sharing host with database while datanode is not reporting clock offset error . what could be the reason . ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎08-31-2018 11:15 AM
Community Statistics
Member Since ‎02-15-2016 01:41 PM
Last Visited ‎08-31-2018 11:15 AM
Posts 113
Kudos received 7