Member since
07-02-2018
26
Posts
0
Kudos Received
0
Solutions
01-29-2019
04:31 AM
Hbase master stopped after running for few seconds.region servers are running . hbase master logs: WARN [slnxhadoop01:16000.activeMasterManager] master.SplitLogManager: error while splitting logs in [hdfs://slnxhadoop01.noid.in.sopra:8020/apps/hbase/data/WALs/slnxhadoop04.dhcp.noid.in.sopra,16020,1536839826364-splitting] installed = 1 but only 0 done
2019-01-10 16:48:57,344 FATAL [slnxhadoop01:16000.activeMasterManager] master.HMaster: Failed to become active master
java.io.IOException: error or interrupted while splitting logs in [hdfs://slnxhadoop01.noid.in.sopra:8020/apps/hbase/data/WALs/slnxhadoop04.dhcp.noid.in.sopra,16020,1536839826364-splitting] Task = installed = 1 done = 0 error = 1
at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:290)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:429)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:339)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:330)
at org.apache.hadoop.hbase.master.HMaster.splitMetaLogBeforeAssignment(HMaster.java:1203)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:806)
at org.apache.hadoop.hbase.master.HMaster.access$900(HMaster.java:225)
at org.apache.hadoop.hbase.master.HMaster$3.run(HMaster.java:2038)
at java.lang.Thread.run(Thread.java:745)
2019-01-10 16:48:57,345 FATAL [slnxhadoop01:16000.activeMasterManager] master.HMaster: Master server abort: loaded coprocessors are: []
2019-01-10 16:48:57,345 FATAL [slnxhadoop01:16000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.
java.io.IOException: error or interrupted while splitting logs in [hdfs://slnxhadoop01.noid.in.sopra:8020/apps/hbase/data/WALs/slnxhadoop04.dhcp.noid.in.sopra,16020,1536839826364-splitting] Task = installed = 1 done = 0 error = 1
at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:290)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:429)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:339)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:330)
at org.apache.hadoop.hbase.master.HMaster.splitMetaLogBeforeAssignment(HMaster.java:1203)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:806)
at org.apache.hadoop.hbase.master.HMaster.access$900(HMaster.java:225)
at org.apache.hadoop.hbase.master.HMaster$3.run(HMaster.java:2038)
at java.lang.Thread.run(Thread.java:745)
2019-01-10 16:48:57,345 INFO [slnxhadoop01:16000.activeMasterManager] regionserver.HRegionServer: STOPPED: Unhandled exception. Starting shutdown.
2019-01-10 16:48:57,346 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] regionserver.HRegionServer: Stopping infoServer
2019-01-10 16:48:57,400 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:16010
2019-01-10 16:48:57,403 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] procedure2.ProcedureExecutor: Stopping the procedure executor
2019-01-10 16:48:57,403 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] wal.WALProcedureStore: Stopping the WAL Procedure Store
2019-01-10 16:48:57,417 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] regionserver.HRegionServer: stopping server slnxhadoop01.noid.in.sopra,16000,1547117834090
2019-01-10 16:48:57,417 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x36691ecadd9004c
2019-01-10 16:48:57,422 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] zookeeper.ZooKeeper: Session: 0x36691ecadd9004c closed
2019-01-10 16:48:57,422 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2019-01-10 16:48:57,423 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] regionserver.HRegionServer: stopping server slnxhadoop01.noid.in.sopra,16000,1547117834090; all regions closed.
2019-01-10 16:48:57,423 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] hbase.ChoreService: Chore service for: slnxhadoop01.noid.in.sopra,16000,1547117834090 had [[ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS]] on shutdown
2019-01-10 16:48:57,427 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x36691ecadd9004d
2019-01-10 16:48:57,428 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] zookeeper.ZooKeeper: Session: 0x36691ecadd9004d closed
2019-01-10 16:48:57,428 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] hbase.ChoreService: Chore service for: slnxhadoop01.noid.in.sopra,16000,1547117834090_splitLogManager_ had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown
2019-01-10 16:48:57,428 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] flush.MasterFlushTableProcedureManager: stop: server shutting down.
2019-01-10 16:48:57,428 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] ipc.RpcServer: Stopping server on 16000
2019-01-10 16:48:57,428 INFO [slnxhadoop01:16000.activeMasterManager-EventThread] zookeeper.ClientCnxn: EventThread shut down
2019-01-10 16:48:57,429 INFO [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: stopping
2019-01-10 16:48:57,430 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2019-01-10 16:48:57,430 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2019-01-10 16:48:57,436 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] zookeeper.RecoverableZooKeeper: Node /hbase-unsecure/rs/slnxhadoop01.noid.in.sopra,16000,1547117834090 already deleted, retry=false
2019-01-10 16:48:57,437 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] zookeeper.ZooKeeper: Session: 0x36691ecadd9004b closed
2019-01-10 16:48:57,437 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] regionserver.HRegionServer: stopping server slnxhadoop01.noid.in.sopra,16000,1547117834090; zookeeper connection closed.
2019-01-10 16:48:57,437 INFO [master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000] regionserver.HRegionServer: master/slnxhadoop01.noid.in.sopra/172.26.50.102:16000 exiting
2019-01-10 16:48:57,438 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
... View more
- Tags:
- Data Processing
- HBase
Labels:
01-10-2019
12:18 PM
Hi@subhash parise Thanks for the reply. I have checked ambari-server.log and ambari-agent.log file . Both the file does not show out of memory error. /var/log/messages logs systemd: Starting Session 7799 of user root.
Jan 10 17:00:01 slnxhadoop01 systemd: Started Session 7799 of user root.
Jan 10 17:01:01 slnxhadoop01 systemd: Starting Session 7800 of user root.
Jan 10 17:01:01 slnxhadoop01 systemd: Started Session 7800 of user root.
Jan 10 17:29:39 slnxhadoop01 NetworkManager[673]: <error> [1547121579.803564] [devices/nm-device.c:2303] activation_source_schedule(): (eno16777736): activation stage already scheduled
Jan 10 17:32:42 slnxhadoop01 NetworkManager[673]: <error> [1547121762.303526] [devices/nm-device.c:2303] activation_source_schedule(): (eno16777736): activation stage already scheduled
... View more
01-10-2019
11:00 AM
Hbase master is installed in namenode (namenode.in.sop). when I start Hbase master, It get started but after few minutes it gets stopped automatically. Alert message shown is : connection refused to namenode.in.sop:16000 Hbase logs : Java HotSpot(TM) 64-Bit Server VM (25.121-b13) for linux-amd64 JRE (1.8.0_121-b13), built on Dec 12 2016 16:36:53 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
Memory: 4k page, physical 8011900k(1931220k free), swap 8257532k(8236804k free)
CommandLine flags: -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -XX:InitialHeapSize=128190400 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=87244800 -XX:MaxTenuringThreshold=6 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError=kill -9 %p -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2019-01-10T16:27:12.008+0530: 1.191: [GC (Allocation Failure) 2019-01-10T16:27:12.008+0530: 1.191: [ParNew: 33856K->4223K(38080K), 0.0087958 secs] 33856K->5387K(122752K), 0.0089220 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2019-01-10T16:27:12.835+0530: 2.018: [GC (Allocation Failure) 2019-01-10T16:27:12.835+0530: 2.018: [ParNew: 38079K->3389K(38080K), 0.0174462 secs] 39243K->6595K(122752K), 0.0175369 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2019-01-10T16:27:13.207+0530: 2.389: [GC (Allocation Failure) 2019-01-10T16:27:13.207+0530: 2.389: [ParNew: 37245K->4223K(38080K), 0.0094610 secs] 40451K->9480K(122752K), 0.0095239 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2019-01-10T16:27:13.631+0530: 2.813: [GC (Allocation Failure) 2019-01-10T16:27:13.631+0530: 2.813: [ParNew: 38079K->3698K(38080K), 0.0074865 secs] 43336K->9365K(122752K), 0.0075744 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2019-01-10T16:27:13.835+0530: 3.017: [GC (Allocation Failure) 2019-01-10T16:27:13.835+0530: 3.017: [ParNew: 37554K->1240K(38080K), 0.0055719 secs] 43221K->7778K(122752K), 0.0056540 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2019-01-10T16:27:14.407+0530: 3.589: [GC (Allocation Failure) 2019-01-10T16:27:14.407+0530: 3.589: [ParNew: 35096K->2431K(38080K), 0.0061526 secs] 41634K->8969K(122752K), 0.0062370 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2019-01-10T16:27:14.426+0530: 3.608: [GC (CMS Initial Mark) [1 CMS-initial-mark: 6538K(84672K)] 10971K(122752K), 0.0040071 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2019-01-10T16:27:14.434+0530: 3.616: [CMS-concurrent-mark-start]
2019-01-10T16:27:14.468+0530: 3.650: [CMS-concurrent-mark: 0.034/0.034 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
2019-01-10T16:27:14.474+0530: 3.656: [CMS-concurrent-preclean-start]
2019-01-10T16:27:14.475+0530: 3.657: [CMS-concurrent-preclean: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2019-01-10T16:27:14.475+0530: 3.657: [CMS-concurrent-abortable-preclean-start]
2019-01-10T16:27:14.787+0530: 3.970: [GC (Allocation Failure) 2019-01-10T16:27:14.787+0530: 3.970: [ParNew: 36287K->2316K(38080K), 0.0095541 secs] 42825K->10118K(122752K), 0.0096321 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2019-01-10T16:27:15.104+0530: 4.286: [CMS-concurrent-abortable-preclean: 0.229/0.629 secs] [Times: user=0.59 sys=0.01, real=0.63 secs]
2019-01-10T16:27:15.115+0530: 4.298: [GC (CMS Final Remark) [YG occupancy: 28884 K (38080 K)]2019-01-10T16:27:15.115+0530: 4.298: [Rescan (parallel) , 0.0077814 secs]2019-01-10T16:27:15.123+0530: 4.305: [weak refs processing, 0.0000331 secs]2019-01-10T16:27:15.123+0530: 4.305: [class unloading, 0.0030827 secs]2019-01-10T16:27:15.126+0530: 4.309: [scrub symbol table, 0.0027604 secs]2019-01-10T16:27:15.129+0530: 4.311: [scrub string table, 0.0004649 secs][1 CMS-remark: 7801K(84672K)] 36685K(122752K), 0.0145743 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2019-01-10T16:27:15.141+0530: 4.323: [CMS-concurrent-sweep-start]
2019-01-10T16:27:15.144+0530: 4.326: [CMS-concurrent-sweep: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2019-01-10T16:27:15.144+0530: 4.326: [CMS-concurrent-reset-start]
2019-01-10T16:27:15.147+0530: 4.330: [CMS-concurrent-reset: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2019-01-10T16:27:15.155+0530: 4.337: [GC (Allocation Failure) 2019-01-10T16:27:15.155+0530: 4.337: [ParNew: 36172K->2520K(38080K), 0.0038867 secs] 41896K->8243K(122752K), 0.0039568 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2019-01-10T16:27:15.278+0530: 4.460: [GC (Allocation Failure) 2019-01-10T16:27:15.278+0530: 4.460: [ParNew: 36376K->1955K(38080K), 0.0046129 secs] 42099K->8052K(122752K), 0.0046859 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2019-01-10T16:27:15.503+0530: 4.686: [GC (Allocation Failure) 2019-01-10T16:27:15.503+0530: 4.686: [ParNew: 35811K->2243K(38080K), 0.0041595 secs] 41908K->8340K(122752K), 0.0042305 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2019-01-10T16:27:15.849+0530: 5.032: [GC (Allocation Failure) 2019-01-10T16:27:15.849+0530: 5.032: [ParNew: 36099K->2268K(38080K), 0.0057452 secs] 42196K->8366K(122752K), 0.0058221 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2019-01-10T16:27:16.155+0530: 5.337: [GC (Allocation Failure) 2019-01-10T16:27:16.155+0530: 5.337: [ParNew: 36124K->2377K(38080K), 0.0069522 secs] 42222K->8832K(122752K), 0.0070305 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2019-01-10T16:27:17.130+0530: 6.312: [GC (Allocation Failure) 2019-01-10T16:27:17.130+0530: 6.312: [ParNew: 36233K->4224K(38080K), 0.0082606 secs] 42688K->10740K(122752K), 0.0083634 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2019-01-10T16:27:17.265+0530: 6.447: [GC (Allocation Failure) 2019-01-10T16:27:17.265+0530: 6.447: [ParNew: 38080K->2058K(38080K), 0.0092968 secs] 44596K->10896K(122752K), 0.0093922 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2019-01-10T16:27:17.769+0530: 6.951: [GC (Allocation Failure) 2019-01-10T16:27:17.769+0530: 6.951: [ParNew: 35914K->2100K(38080K), 0.0057748 secs] 44752K->10938K(122752K), 0.0058689 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2019-01-10T16:27:18.092+0530: 7.274: [GC (Allocation Failure) 2019-01-10T16:27:18.092+0530: 7.275: [ParNew: 35956K->3100K(38080K), 0.0056527 secs] 44794K->11938K(122752K), 0.0057448 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2019-01-10T16:27:18.933+0530: 8.115: [GC (Allocation Failure) 2019-01-10T16:27:18.933+0530: 8.115: [ParNew: 36956K->2513K(38080K), 0.0082556 secs] 45794K->11973K(122752K), 0.0083492 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2019-01-10T16:27:19.466+0530: 8.648: [GC (Allocation Failure) 2019-01-10T16:27:19.466+0530: 8.649: [ParNew: 36369K->2498K(38080K), 0.0092841 secs] 45829K->12789K(122752K), 0.0093821 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] I am also getting GC allocation Failure in namenode logs Namenode logs: 2018-09-13T17:25:13.341+0530: 163.098: [GC (Allocation Failure) 2018-09-13T17:25:13.341+0530: 163.098: [ParNew: 176060K->5173K(184320K), 0.0493439 secs] 180804K->13604K(1028096K), 0.0494549 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
2018-09-13T17:25:15.043+0530: 164.800: [GC (Allocation Failure) 2018-09-13T17:25:15.043+0530: 164.800: [ParNew: 169013K->8331K(184320K), 0.1074649 secs] 177444K->16761K(1028096K), 0.1075756 secs] [Times: user=0.07 sys=0.00, real=0.10 secs]
2018-09-13T17:25:15.850+0530: 165.607: [GC (Allocation Failure) 2018-09-13T17:25:15.850+0530: 165.607: [ParNew: 172171K->5445K(184320K), 0.1568449 secs] 180601K->13876K(1028096K), 0.1569780 secs] [Times: user=0.06 sys=0.00, real=0.16 secs]
2018-09-13T17:25:16.709+0530: 166.467: [GC (Allocation Failure) 2018-09-13T17:25:16.709+0530: 166.467: [ParNew: 169285K->5954K(184320K), 0.0760959 secs] 177716K->14385K(1028096K), 0.0762063 secs] [Times: user=0.07 sys=0.00, real=0.08 secs]
2018-09-13T17:25:26.130+0530: 175.888: [GC (Allocation Failure) 2018-09-13T17:25:26.130+0530: 175.888: [ParNew: 169794K->8792K(184320K), 0.0456534 secs] 178225K->17223K(1028096K), 0.0457743 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
2018-09-13T17:30:54.415+0530: 504.173: [GC (Allocation Failure) 2018-09-13T17:30:54.415+0530: 504.173: [ParNew: 172632K->5359K(184320K), 0.0423570 secs] 181063K->13789K(1028096K), 0.0425261 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
2018-09-13T17:58:33.507+0530: 2163.265: [GC (Allocation Failure) 2018-09-13T17:58:33.507+0530: 2163.265: [ParNew: 169199K->3440K(184320K), 0.0451056 secs] 177629K->13056K(1028096K), 0.0452152 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
2018-09-13T18:21:41.935+0530: 3551.693: [GC (Allocation Failure) 2018-09-13T18:21:41.935+0530: 3551.693: [ParNew: 167280K->4107K(184320K), 0.0469423 secs] 176896K->14342K(1028096K), 0.0470430 secs] [Times: user=0.04 sys=0.00, real=0.05 secs]
2018-09-13T19:01:27.263+0530: 5937.020: [GC (Allocation Failure) 2018-09-13T19:01:27.263+0530: 5937.021: [ParNew: 167947K->1769K(184320K), 0.0432569 secs] 178182K->12011K(1028096K), 0.0433850 secs] [Times: user=0.05 sys=0.00, real=0.04 secs]
2018-09-13T19:43:59.792+0530: 8489.549: [GC (Allocation Failure) 2018-09-13T19:43:59.792+0530: 8489.549: [ParNew: 165609K->1652K(184320K), 0.0448316 secs] 175851K->11903K(1028096K), 0.0449217 secs] [Times: user=0.05 sys=0.00, real=0.04 secs]
... View more
Labels:
09-12-2018
11:10 AM
Hi @Akhil S Naik , I removed the repository and now i am getting error. I followed https://community.hortonworks.com/articles/102213/hdp-2603-package-installation-failing-with-error-i.html and did all 3 steps but still getting same error. 2018-09-12 16:26:17,632 - Package Manager failed to install packages. Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install mahout' returned 1. Error: Package: hadoop_2_6_5_0_292-hdfs-2.7.3.2.6.5.0-292.x86_64 (HDP-2.6-repo-106)
Requires: libtirpc-devel
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 423, in install_packages
retry_count=agent_stack_retry_count
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/package/__init__.py", line 57, in action_upgrade
self.upgrade_package(package_name, self.resource.use_repos, self.resource.skip_repos)
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/package/yumrpm.py", line 269, in upgrade_package
return self.install_package(name, use_repos, skip_repos, is_upgrade)
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/package/yumrpm.py", line 264, in install_package
self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput())
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/package/__init__.py", line 266, in checked_call_with_retries
return self._call_with_retries(cmd, is_checked=True, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/package/__init__.py", line 283, in _call_with_retries
code, out = func(cmd, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install mahout' returned 1. Error: Package: hadoop_2_6_5_0_292-hdfs-2.7.3.2.6.5.0-292.x86_64 (HDP-2.6-repo-106)
Requires: libtirpc-devel
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 487, in <module>
InstallPackages().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute
method(env)
File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 153, in actionexecute
raise Fail("Failed to distribute repositories/install packages")
resource_management.core.exceptions.Fail: Failed to distribute repositories/install packages
... View more
09-11-2018
04:38 PM
I am trying to upgrade HDP 2.5.3 to 2.6.5 and I am not sure if I am doing it right. I have taken following step: 1. extracted hdp 2.6.5 , hdp-utils, hdp-gpl tar file in /var/www/html/hdp-2.6. 2. registered hdp 2.6.5 through ambari by providing base url for hdp 2.6, hdp-utils, hdp-gpl. I have not created repo file and when I am trying to install HDP 2.6.5 , ambari-hdp-101.repo file is created in /etc/yum.repos.d. 3. installation of HDP 2.6.5 is getting failed with error /var/lib/ambari-agent/data/errors-3368.txt
2018-09-11 12:23:28,785 - Package Manager failed to install packages. Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install mahout' returned 1.
One of the configured repositories failed (MySQL Connectors Community),and yum doesn't have enough cached data to continue. At this point the onlysafe thing yum can do is fail. There are a few ways to work "fix" this:
1. Contact the upstream for the repository and get them to fix the problem.
2. Reconfigure the baseurl/etc. for the repository, to point to a working upstream. This is most often useful if you are using a newer distribution release than is supported by the repository (and the packages for the previous distribution release still work).
3. Disable the repository, so yum won't use it by default. Yum will then just ignore the repository until you permanently enable it again or use --enablerepo for temporary usage:
yum-config-manager --disable mysql-connectors-community
4. Configure the failing repository to be skipped, if it is unavailable. Note that yum will try to contact the repo. when it runs most commands, so will have to try and fail each time (and thus. yum will be be much slower). If it is a very temporary problem though, this is often a nice compromise:
yum-config-manager --save --setopt=mysql-connectors-community.skip_if_unavailable=true
failure: repodata/repomd.xml from mysql-connectors-community: [Errno 256] No more mirrors to try.
http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: [Errno 12] Timeout on http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: (28, 'Connection timed out after 30177 milliseconds')
http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: [Errno 12] Timeout on http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: (28, 'Connection timed out after 30045 milliseconds')
http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: [Errno 12] Timeout on http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: (28, 'Connection timed out after 30192 milliseconds')
http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: [Errno 12] Timeout on http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: (28, 'Connection timed out after 30052 milliseconds')
http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: [Errno 12] Timeout on http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: (28, 'Connection timed out after 30203 milliseconds')
http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: [Errno 12] Timeout on http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: (28, 'Connection timed out after 30037 milliseconds')
http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: [Errno 12] Timeout on http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: (28, 'Connection timed out after 30179 milliseconds')
http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: [Errno 12] Timeout on http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: (28, 'Connection timed out after 30041 milliseconds')
http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: [Errno 12] Timeout on http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: (28, 'Connection timed out after 30180 milliseconds')
http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: [Errno 12] Timeout on http://repo.mysql.com/yum/mysql-connectors-community/el/7/x86_64/repodata/repomd.xml: (28, 'Connection timed out after 30050 milliseconds')
... View more
Labels:
09-07-2018
07:14 AM
@Jay Kumar SenSharma Thanks for the reply. slnxhadoop01.noid.in is ambari server host and i am able to make successful curl calls and download xml file. For setting local repository for HDP 2.3.5.0 , I deleted existing HDP.repo file for HDP 2.5.3.0 and created new HDP.repo file for HDP version 2.6.5.0. I am able to register HDP 2.6.5.0 but while installing getting error: stderr: /var/lib/ambari-agent/data/errors-3360.txt 2018-09-07 14:46:16,114 - Package Manager failed to install packages. Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install mahout' returned 1. One of the configured repositories failed (HDP-2.5.3.0-37),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:
1. Contact the upstream for the repository and get them to fix the problem.
2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).
3. Disable the repository, so yum won't use it by default. Yum will then
just ignore the repository until you permanently enable it again or use
--enablerepo for temporary usage:
yum-config-manager --disable HDP-2.5.3.0-37
4. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:
yum-config-manager --save --setopt=HDP-2.5.3.0-37.skip_if_unavailable=true
failure: repodata/repomd.xml from HDP-2.5.3.0-37: [Errno 256] No more mirrors to try.
http://slnxhadoop01.noid.in.sopra/hdp/HDP/centos7/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 423, in install_packages
retry_count=agent_stack_retry_count
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/package/__init__.py", line 57, in action_upgrade
self.upgrade_package(package_name, self.resource.use_repos, self.resource.skip_repos)
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/package/yumrpm.py", line 269, in upgrade_package
return self.install_package(name, use_repos, skip_repos, is_upgrade)
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/package/yumrpm.py", line 264, in install_package
self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput())
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/package/__init__.py", line 266, in checked_call_with_retries
return self._call_with_retries(cmd, is_checked=True, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/package/__init__.py", line 283, in _call_with_retries
code, out = func(cmd, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
... View more
09-07-2018
06:38 AM
I am trying to upgrade HDP2.5.3 to HDP 2.6.5. To setup local repository with no internet access i have taken following steps: download and extracted tar file for HDP 2.6.5 , HDP_UTLIS, HDP-GPL in the node hosting ambari server. downloaded hdp.repo and hdp.gpl.repo file in all the host and changed base url. 2. now while registering version in ambari base url for HDP 2.6, HDP-UTILS, HDP-2.6-GPL failed validation. baseurl for HDP 2.6 = http://slnxhadoop01.noid.in/hdp/HDP/centos7/2.6.5.0-292 HDP-2.6-GPL = http://slnxhadoop01.noid.in/hdp/HDP-UTILS/centos7/1.1.0.22/ HDP-UTILS-1.1.0.22 = http://slnxhadoop01.noid.in/hdp/HDP-GPL/centos7/2.6.5.0-292/ ambari-baseurl-validation.pnghdp-265-repo.pnghdp-gpl-repo.pnghdp-utils-repo.png
... View more
09-06-2018
10:19 AM
I am trying to upgrade HDP 2.5.3 to 2.6.5 and while registering version HDP 2.6.5 ,save button is getting disabled. I have enabled Skip Repository Base URL validation but still save is disabled. I am using ambari 2.6.2.2 and have uploaded version definition file. hdp-265-register-error.png
... View more
08-31-2018
10:12 AM
@Jay Kumar SenSharma Thanks for the reply. I followed steps given in https://community.hortonworks.com/articles/79327/re-initializing-apache-accumulo-under-hdp.html and it worked.
... View more
08-31-2018
09:28 AM
@Jay Kumar SenSharma Thanks for the reply. I tried to drop table testtable but getting error : Thread "shell" stuck on IO to slnxhadoop03.dhcp.noid.in:9999 (0) for at least 120348 ms I am using HDP 2.5.3 and ambari version 2.4.2.0.
... View more
08-31-2018
07:17 AM
Accumulo is not showing any alert but accumulo.pngaccumulo-error.png Accumulo service check is getting failed with error: ERROR: org.apache.accumulo.core.client.TableExistsException: Table testtable exists
2018-08-30 17:10:46,508 [shell.Shell] ERROR: java.lang.IllegalStateException: Not in a table context. Please use 'table <tableName>' to switch to a table, or use '-t' to specify a table if option is available.
2018-08-30 17:10:46,509 [shell.Shell] ERROR: java.lang.IllegalStateException: Not in a table context. Please use 'table <tableName>' to switch to a table, or use '-t' to specify a table if option is available.
2018-08-30 17:10:46,509 [shell.Shell] ERROR: java.lang.IllegalStateException: Not in a table context. Please use 'table <tableName>' to switch to a table, or use '-t' to specify a table if option is available.
2018-08-30 17:10:46,509 [shell.Shell] ERROR: java.lang.IllegalStateException: Not in a table context. Please use 'table <tableName>' to switch to a table, or use '-t' to specify a table if option is available.
... View more
Labels:
08-30-2018
06:05 AM
@billie Thanks for the reply. I have set open files limit to 32768 and now accumulo Tserver is running but service check of accumulo is getting failed with error: ERROR: org.apache.accumulo.core.client.TableExistsException: Table testtable exists
2018-08-30 11:21:28,554 [shell.Shell] ERROR: java.lang.IllegalStateException: Not in a table context. Please use 'table <tableName>' to switch to a table, or use '-t' to specify a table if option is available.
2018-08-30 11:21:28,555 [shell.Shell] ERROR: java.lang.IllegalStateException: Not in a table context. Please use 'table <tableName>' to switch to a table, or use '-t' to specify a table if option is available.
2018-08-30 11:21:28,555 [shell.Shell] ERROR: java.lang.IllegalStateException: Not in a table context. Please use 'table <tableName>' to switch to a table, or use '-t' to specify a table if option is available.
2018-08-30 11:21:28,556 [shell.Shell] ERROR: java.lang.IllegalStateException: Not in a table context. Please use 'table <tableName>' to switch to a table, or use '-t' to specify a table if option is available.
... View more
08-29-2018
11:17 AM
accumulo TServer process giving error:connection refused to slnxhadoop04.in:9997. Accumulo-tserver.err log org.apache.thrift.transport.TTransportException: java.io.IOException: Too many open files
at org.apache.accumulo.server.rpc.TNonblockingServerSocket.acceptImpl(TNonblockingServerSocket.java:118)
at org.apache.accumulo.server.rpc.TNonblockingServerSocket.acceptImpl(TNonblockingServerSocket.java:44)
at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at org.apache.accumulo.server.rpc.CustomNonBlockingServer$SelectAcceptThread.handleAccept(CustomNonBlockingServer.java:250)
at org.apache.accumulo.server.rpc.CustomNonBlockingServer$SelectAcceptThread.select(CustomNonBlockingServer.java:225)
at org.apache.accumulo.server.rpc.CustomNonBlockingServer$SelectAcceptThread.run(CustomNonBlockingServer.java:184)
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.apache.accumulo.server.rpc.TNonblockingServerSocket.acceptImpl(TNonblockingServerSocket.java:109)
... 5 more
org.apache.thrift.transport.TTransportException: java.io.IOException: Too many open files
at org.apache.accumulo.server.rpc.TNonblockingServerSocket.acceptImpl(TNonblockingServerSocket.java:118)
at org.apache.accumulo.server.rpc.TNonblockingServerSocket.acceptImpl(TNonblockingServerSocket.java:44)
at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at org.apache.accumulo.server.rpc.CustomNonBlockingServer$SelectAcceptThread.handleAccept(CustomNonBlockingServer.java:250)
at org.apache.accumulo.server.rpc.CustomNonBlockingServer$SelectAcceptThread.select(CustomNonBlockingServer.java:225)
at org.apache.accumulo.server.rpc.CustomNonBlockingServer$SelectAcceptThread.run(CustomNonBlockingServer.java:184)
Caused by: java.io.IOException: Too many open files
... View more
Labels:
08-27-2018
07:15 AM
mysql-connector-error.pngI am runninf kafka connect cluster in distributed mode and i am usinf kafka(0.10.0.2.5) connect of ambari deployment. I gave debezium mysql connector path using export CLASSPATH=/path to connector/. Loaded connector configuration into Kafka Connect using the REST API : curl -i -X POST -H "Accept:application/json" \
-H "Content-Type:application/json" http://localhost:8083/connectors/ \
-d '{
"name": "MYSQL_CONNECTOR",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"database.hostname": "10.224.21.36",
"database.port": "3306",
"database.user": "root",
"database.password": "shobhna",
"database.server.id": "1",
"database.server.name": "demo",
"database.history.kafka.bootstrap.servers": "slnxhadoop04.dhcp.noid.in:6669",
"database.history.kafka.topic": "dbhistory.demo" ,
"include.schema.changes": "true"
}
}' Now after checking connector status I am getting error: {"name":"MYSQL_CONNECTOR","connector":{"state":"RUNNING","worker_id":"172.26.177.115:8083"},
"tasks":[{"state":"FAILED","trace":"org.apache.kafka.connect.errors.ConnectException: java.lang.NoClassDefFoundError: org/apache/kafka/clients/admin/AdminClient\n\tat io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:218)\n\tat io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:45)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:137)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)\n\tat java.util.concurrent.Executors$RunnableAdapter.cal(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: java.lang.NoClassDefFoundError: org/apache/kafka/clients/admin/AdminClient\n\tat io.debezium.relational.history.KafkaDatabaseHistory.initializeStorage(KafkaDatabaseHistory.java:336)\n\tat io.debezium.connector.mysql.MySqlSchema.intializeHistoryStorage(MySqlSchema.java:260)\n\tat io.debezium.connector.mysql.MySqlTaskContext.initializeHistoryStorage(MySqlTaskContext.java:194)\n\tat io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:126)\n\t... 9 more\nCaused by: java.lang.ClassNotFoundException: org.apache.kafka.clients.admin.AdminClient \n\tat java.net.URLClassLoader.findClass(URLClassLoader.java:381)\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:424)\n\tat sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:357)\n\t
... View more
Labels:
08-24-2018
10:30 AM
I am running HDP 2.5.3 kafka cluster. I don't find any feature in ambari to monitor cluster like confluent control center provide. Is there any way I can set up confluent control center on my hdp?
... View more
08-24-2018
10:12 AM
@Jordan Moore Thanks for the help.
... View more
08-22-2018
11:54 AM
I am trying to pull data from mysql and I am using kafka provided by ambari.I am new to kafka and have few doubts. 1. where I can find logs for running kafka connect cluster and debezium connectors? 2. I am not using confluent, do i need to configure schema registry and why it is used?
... View more
Labels:
08-21-2018
11:35 AM
@Jordan Moore Thanks for the quick reply. However, I am currently using Kafka version 0.10.0.2.5 , how can I give Debezium mysql connector path. I am getting error: ERROR Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder:183)
org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector andwhich name matches io.debezium.connector.mysql.MySqlConnector available connectors are: org.apache.kafka.connect.sink.SinkConnector,
org.apache.kafka.connect.tools.VerifiableSourceConnector,
org.apache.kafka.connect.file.FileStreamSinkConnector,
org.apache.kafka.connect.file.FileStreamSourceConnector,
org.apache.kafka.connect.source.SourceConnector,
org.apache.kafka.connect.tools.VerifiableSinkConnector,
org.apache.kafka.connect.tools.MockSourceConnector,
org.apache.kafka.connect.tools.MockConnector,
org.apache.kafka.connect.tools.MockSinkConnector
... View more
08-20-2018
05:02 PM
I am new to kafka and i am trying to use get data from mysql to kafka broker using debezium mysql connector?I am not able to understand how to run kafka connect in distributed mode to use debezium mysql connector.
... View more
Labels:
08-20-2018
09:57 AM
I am trying to run sqoop job to import incremental data from mysql through oozie and workflow is importing data from mysql but instead of importing incremental data from the table, It imports all the existing data from the table. Sqoop job works fine when I tried running it from CLI. workflow ends with the following error: 2018-08-20 15:10:36,469 WARN SqoopActionExecutor:523 - SERVER[slnxhadoop03.dhcp.noid.in.sopra] USER[shobhna] GROUP[-] TOKEN[] APP[ETL Workflow] JOB[0000016-180813165952618-oozie-oozi-W] ACTION[0000016-180813165952618-oozie-oozi-W@sqoop_extract] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], main() threw exception, org/json/JSONObject
2018-08-20 15:10:36,473 WARN SqoopActionExecutor:523 - SERVER[slnxhadoop03.dhcp.noid.in.sopra] USER[shobhna] GROUP[-] TOKEN[] APP[ETL Workflow] JOB[0000016-180813165952618-oozie-oozi-W] ACTION[0000016-180813165952618-oozie-oozi-W@sqoop_extract] Launcher exception: org/json/JSONObject
java.lang.NoClassDefFoundError: org/json/JSONObject
at org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:43)
at org.apache.sqoop.SqoopOptions.writeProperties(SqoopOptions.java:759)
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.createInternal(HsqldbJobStorage.java:399)
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.update(HsqldbJobStorage.java:445)
at org.apache.sqoop.tool.ImportTool.saveIncrementalState(ImportTool.java:164)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:528)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:243)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:298)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:225)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.main(Sqoop.java:243)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:202)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:182)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:48)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: org.json.JSONObject
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 32 more
... View more
Labels:
08-07-2018
12:14 PM
Hi @Akhil S Naik Instead of removing it from database can I add new host with new hostname and remove host existing earlier.
... View more
08-07-2018
09:47 AM
07 Aug 2018 15:04:49,587 ERROR [main] HostUpdateHelper:562 - Unexpected error, host names update failed
javax.persistence.RollbackException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "uq_hosts_host_name"
Detail: Key (host_name)=(slnxhadoop04.dhcp.noid.in.sopra) already exists.
Error Code: 0
Call: UPDATE hosts SET host_name = ? WHERE (host_id = ?)
bind => [2 parameters bound]
at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:159)
at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:153)
at org.apache.ambari.server.update.HostUpdateHelper.updateHostsInDB(HostUpdateHelper.java:405)
at org.apache.ambari.server.update.HostUpdateHelper.main(HostUpdateHelper.java:546)
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "uq_hosts_host_name"
Detail: Key (host_name)=(slnxhadoop04.dhcp.noid.in.sopra) already exists.
Error Code: 0
Call: UPDATE hosts SET host_name = ? WHERE (host_id = ?)
bind => [2 parameters bound]
at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1620)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:900)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeNoSelect(DatabaseAccessor.java:964)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:633)
at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatch(ParameterizedSQLBatchWritingMechanism.java:149)
at org.eclipse.persistence.internal.databaseaccess.ParameterizedSQLBatchWritingMechanism.executeBatchedStatements(ParameterizedSQLBatchWritingMechanism.java:134)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.writesCompleted(DatabaseAccessor.java:1845)
at org.eclipse.persistence.internal.sessions.AbstractSession.writesCompleted(AbstractSession.java:4300)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.writesCompleted(UnitOfWorkImpl.java:5592)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.acquireWriteLocks(UnitOfWorkImpl.java:1646)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitTransactionAfterWriteChanges(UnitOfWorkImpl.java:1614)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:285)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
... 3 more
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "uq_hosts_host_name"
Detail: Key (host_name)=(slnxhadoop04.dhcp.noid.in.sopra) already exists.
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:892)
... 15 more
07 Aug 2018 15:04:49,815 INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.4 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
07 Aug 2018 15:04:50,093 INFO [Stack Version Loading Thread] LatestRepoCallable:219 - Stack HDP-2.4 cannot resolve OS debian6 to the supported ones: suse11,redhat7,debian7,redhat6,ubuntu14,ubuntu12. Family: null
07 Aug 2018 15:04:50,450 INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.5 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
07 Aug 2018 15:04:51,107 INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.3.ECS from http://s3.amazonaws.com/dev.hortonworks.com/HDP/hdp_urlinfo.json
07 Aug 2018 15:04:52,507 INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.1.GlusterFS from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
07 Aug 2018 15:04:52,583 INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.3.GlusterFS from http://s3.amazonaws.com/dev.hortonworks.com/HDP/hdp_urlinfo.json
07 Aug 2018 15:04:53,291 INFO [Stack Version Loading Thread] LatestRepoCallable:80 - Loading latest URL info for stack HDP-2.0 from http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
... View more
08-07-2018
04:52 AM
Hi @amarnath reddy pappu thanks for replying . error message from server logs: ERROR [ambari-client-thread-377621] ContainerResponse:537 - Mapped exception to response: 500 (Internal Server Error)
org.apache.ambari.view.hive2.utils.ServiceFormattedException
at org.apache.ambari.view.hive2.resources.jobs.ProgressRetriever.jobCheck(ProgressRetriever.java:55)
at org.apache.ambari.view.hive2.resources.jobs.ProgressRetriever.getProgress(ProgressRetriever.java:42)
at org.apache.ambari.view.hive2.resources.jobs.JobService.getProgress(JobService.java:418)
at sun.reflect.GeneratedMethodAccessor2305.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:257)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.apache.ambari.server.security.authorization.jwt.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:96)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
at org.apache.ambari.server.security.authentication.AmbariAuthenticationFilter.doFilter(AmbariAuthenticationFilter.java:88)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
... View more
08-06-2018
05:22 PM
I have 3 datanodes and ip address and hostnames of these datanode are changed. I followed steps given in https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.18/bk_ambari-reference/content/ch_changing_host_names.html but getting update-host-name failed.
... View more
Labels:
07-06-2018
06:07 AM
Hi Thanks @ Geoffrey Shelton Okot I have given all privileges to root user but still I am getting same error. I have 4 nodes in cluster and each of them have all privileges in MYSQL. My oozie workfile document : <workflow-app xmlns = "uri:oozie:workflow:0.4" name = "simple-Workflow"> <start to = "run_sqoop_job" /> <!-- step 1 --> <action name = "run_sqoop_job"> <sqoop xmlns="uri:oozie:sqoop-action:0.2"> <job-tracker>${resourceManager}</job-tracker> <name-node>${nameNode}</name-node><command>job --meta-connect "jdbc:hsqldb:hsql://ooozie MetastoreUrl:16001/sqoop" --exec samplejob</command> </sqoop> <ok to = "end"/> <error to = "fail"/> </action> <kill name = "fail"> <message>workflow fail</message> </kill> <end name = "end"/> </workflow-app> property file: resourceManager = resource manager url:port nameNode = hdfs://hdfsurl.com:port oozie.wf.application.path= hdfs://hdfs url:port/shared/emproot/workflow/sqoop_job_run.xml oozie.use.system.libpath = true
... View more
07-02-2018
10:14 AM
I have set sqoop.metastore.client.record.password true in sqoop-site.xml.
Also shows that java.sql.SQLException: Access denied for user 'root'@'slnxhadoopUrl' (using password: NO)
java.sql.SQLException: Access denied for user 'root'@'slnxhadoopUrl' (using password: NO
)
access is given to the root user in MYSQL.
... View more
Labels:
