Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

thrift into hbase

thrift into hbase

Contributor

We are using storm to ingest data into hbase via thrift and we get thrift connection timeout we have a single node storm cluster and for some mysterious reason data is no longer being saved into hbase via thrift This started at a certain time and every restart of the cluster allows for a few records in and then dies. 2015-05-12T20:28:46.463+0000 b.s.t.ShellBolt [INFO] ShellLog pid:11027, name:transactions FAIL: thrift connection timeout kill: No such process eventually this cascades to hbase thrift crashing with too many open file descriptors.and the hbase thrift server also comes crashing Any help appreciated I am using storm version 0.9.3 cdh 5.3.3 hbase 0.98.6 storm.yaml storm.local.dir: "/mnt/storm" storm.zookeeper.servers: - "172.30.1.113" storm.zookeeper.port: 2181 nimbus.host: "172.30.1.113" nimbus.thrift.port: 6627 nimbus.thrift.threads: 64 nimbus.thrift.max_buffer_size: 20480000 nimbus.childopts: "-Xmx2048m" nimbus.task.timeout.secs: 60 nimbus.supervisor.timeout.secs: 90 nimbus.monitor.freq.secs: 10 nimbus.cleanup.inbox.freq.secs: 600 ui.port: 8772 supervisor.slots.ports: - 6700 - 6701 - 6702 - 6703 supervisor.childopts: "-Xmx512m -XX:MaxPermSize=256m -XX:+PrintGCDetails -XX:+P rintGCTimeStamps -verbose:gc -Xloggc:/mnt/storm/logs/gc-storm-supervisor.log" worker.childopts: "-Xmx1024m -XX:MaxPermSize=256m -XX:+PrintGCDetails -XX:+Prin tGCTimeStamps -verbose:gc -Xloggc:/mnt/storm/logs/gc-storm-worker-%ID%.log" Below is the supervisor log 2015-05-12T20:16:35.664+0000 b.s.d.supervisor [INFO] Launching worker with assignment #backtype.storm.daemon.supervisor.Loc alAssignment{:storm-id "ripple-ledger-importer-1-1431461267", :executors ([34 34] [67 67] [4 4] [37 37] [70 70] [7 7] [40 4 0] [10 10] [43 43] [13 13] [46 46] [16 16] [49 49] [19 19] [52 52] [22 22] [55 55] [25 25] [58 58] [28 28] [61 61] [31 31] [64 64] [1 1])} for this supervisor b020bea7-d69a-4b12-bd00-968ebfa7cd9d on port 6703 with id 0e189101-84b6-4483-ad93-612d4 94f3680 2015-05-12T20:16:35.667+0000 b.s.d.supervisor [INFO] Launching worker with command: 'java' '-server' '-Xmx1024m' '-XX:MaxPe rmSize=256m' '-XX:+PrintGCDetails' '-XX:+PrintGCTimeStamps' '-verbose:gc' '-Xloggc:/mnt/storm/logs/gc-storm-worker-6703.log ' '-Djava.library.path=/mnt/storm/supervisor/stormdist/ripple-ledger-importer-1-1431461267/resources/Linux-amd64:/mnt/storm /supervisor/stormdist/ripple-ledger-importer-1-1431461267/resources:/usr/lib/jvm/java-7-openjdk-amd64' '-Dlogfile.name=work er-6703.log' '-Dstorm.home=/var/opt/apache-storm-0.9.3' '-Dstorm.conf.file=' '-Dstorm.options=' '-Dstorm.log.dir=/var/opt/a pache-storm-0.9.3/logs' '-Dlogback.configurationFile=/var/opt/apache-storm-0.9.3/logback/cluster.xml' '-Dstorm.id=ripple-le dger-importer-1-1431461267' '-Dworker.id=0e189101-84b6-4483-ad93-612d494f3680' '-Dworker.port=6703' '-cp' '/var/opt/apache- storm-0.9.3/lib/kryo-2.21.jar:/var/opt/apache-storm-0.9.3/lib/json-simple-1.1.jar:/var/opt/apache-storm-0.9.3/lib/tools.log ging-0.2.3.jar:/var/opt/apache-storm-0.9.3/lib/ring-devel-0.3.11.jar:/var/opt/apache-storm-0.9.3/lib/math.numeric-tower-0.0 .1.jar:/var/opt/apache-storm-0.9.3/lib/clojure-1.5.1.jar:/var/opt/apache-storm-0.9.3/lib/commons-io-2.4.jar:/var/opt/apache -storm-0.9.3/lib/clout-1.0.1.jar:/var/opt/apache-storm-0.9.3/lib/ring-jetty-adapter-0.3.11.jar:/var/opt/apache-storm-0.9.3/ lib/servlet-api-2.5.jar:/var/opt/apache-storm-0.9.3/lib/jetty-util-6.1.26.jar:/var/opt/apache-storm-0.9.3/lib/commons-lang- 2.5.jar:/var/opt/apache-storm-0.9.3/lib/commons-exec-1.1.jar:/var/opt/apache-storm-0.9.3/lib/logback-core-1.0.13.jar:/var/o pt/apache-storm-0.9.3/lib/jline-2.11.jar:/var/opt/apache-storm-0.9.3/lib/commons-codec-1.6.jar:/var/opt/apache-storm-0.9.3/ lib/jetty-6.1.26.jar:/var/opt/apache-storm-0.9.3/lib/ring-servlet-0.3.11.jar:/var/opt/apache-storm-0.9.3/lib/objenesis-1.2. jar:/var/opt/apache-storm-0.9.3/lib/reflectasm-1.07-shaded.jar:/var/opt/apache-storm-0.9.3/lib/chill-java-0.3.5.jar:/var/op t/apache-storm-0.9.3/lib/clj-time-0.4.1.jar:/var/opt/apache-storm-0.9.3/lib/ring-core-1.1.5.jar:/var/opt/apache-storm-0.9.3 /lib/slf4j-api-1.7.5.jar:/var/opt/apache-storm-0.9.3/lib/carbonite-1.4.0.jar:/var/opt/apache-storm-0.9.3/lib/core.incubator -0.1.0.jar:/var/opt/apache-storm-0.9.3/lib/tools.cli-0.2.4.jar:/var/opt/apache-storm-0.9.3/lib/storm-core-0.9.3.jar:/var/op t/apache-storm-0.9.3/lib/tools.macro-0.1.0.jar:/var/opt/apache-storm-0.9.3/lib/clj-stacktrace-0.2.2.jar:/var/opt/apache-sto rm-0.9.3/lib/asm-4.0.jar:/var/opt/apache-storm-0.9.3/lib/log4j-over-slf4j-1.6.6.jar:/var/opt/apache-storm-0.9.3/lib/jgrapht -core-0.9.0.jar:/var/opt/apache-storm-0.9.3/lib/commons-fileupload-1.2.1.jar:/var/opt/apache-storm-0.9.3/lib/compojure-1.1. 3.jar:/var/opt/apache-storm-0.9.3/lib/disruptor-2.10.1.jar:/var/opt/apache-storm-0.9.3/lib/minlog-1.2.jar:/var/opt/apache-s torm-0.9.3/lib/commons-logging-1.1.3.jar:/var/opt/apache-storm-0.9.3/lib/logback-classic-1.0.13.jar:/var/opt/apache-storm-0 .9.3/lib/hiccup-0.3.6.jar:/var/opt/apache-storm-0.9.3/lib/snakeyaml-1.11.jar:/var/opt/apache-storm-0.9.3/lib/joda-time-2.0. jar:/var/opt/apache-storm-0.9.3/conf:/mnt/storm/supervisor/stormdist/ripple-ledger-importer-1-1431461267/stormjar.jar' 'bac ktype.storm.daemon.worker' 'ripple-ledger-importer-1-1431461267' 'b020bea7-d69a-4b12-bd00-968ebfa7cd9d' '6703' '0e189101-84 b6-4483-ad93-612d494f3680' 2015-05-12T20:16:35.694+0000 b.s.d.supervisor [INFO] 0e189101-84b6-4483-ad93-612d494f3680 still hasn't started 2015-05-12T20:16:36.197+0000 b.s.d.supervisor [INFO] 0e189101-84b6-4483-ad93-612d494f3680 still hasn't started 2015-05-12T20:16:36.698+0000 b.s.d.supervisor [INFO] 0e189101-84b6-4483-ad93-612d494f3680 still hasn't started 2015-05-12T20:16:37.198+0000 b.s.d.supervisor [INFO] 0e189101-84b6-4483-ad93-612d494f3680 still hasn't started 2015-05-12T20:16:37.699+0000 b.s.d.supervisor [INFO] 0e189101-84b6-4483-ad93-612d494f3680 still hasn't started 2015-05-12T20:16:38.199+0000 b.s.d.supervisor [INFO] 0e189101-84b6-4483-ad93-612d494f3680 still hasn't started 2015-05-12T20:16:38.700+0000 b.s.d.supervisor [INFO] 0e189101-84b6-4483-ad93-612d494f3680 still hasn't started 2015-05-12T20:16:39.201+0000 b.s.d.supervisor [INFO] 0e189101-84b6-4483-ad93-612d494f3680 still hasn't started 2015-05-12T20:16:39.701+0000 b.s.d.supervisor [INFO] 0e189101-84b6-4483-ad93-612d494f3680 still hasn't started 2015-05-12T20:29:08.416+0000 b.s.d.supervisor [INFO] Removing code for storm id ripple-ledger-importer-1-1431461267 2015-05-12T20:29:08.787+0000 b.s.d.supervisor [INFO] Shutting down and clearing state for id ad1bf237-0c2e-4de2-b9b2-3cd8aa 00c51d. Current supervisor time: 1431462548. State: :disallowed, Heartbeat: #backtype.storm.daemon.common.WorkerHeartbeat{: time-secs 1431462548, :storm-id "ripple-ledger-importer-1-1431461267", :executors #{[2 2] [35 35] [68 68] [5 5] [38 38] [8 8] [41 41] [11 11] [44 44] [14 14] [47 47] [17 17] [50 50] [20 20] [53 53] [23 23] [56 56] [26 26] [59 59] [29 29] [62 62] [-1 -1] [32 32] [65 65]}, :port 6702} 2015-05-12T20:29:08.787+0000 b.s.d.supervisor [INFO] Shutting down b020bea7-d69a-4b12-bd00-968ebfa7cd9d:ad1bf237-0c2e-4de2- b9b2-3cd8aa00c51d 2015-05-12T20:29:09.002+0000 b.s.util [INFO] Error when trying to kill 11019. Process is probably already dead. 2015-05-12T20:29:09.024+0000 b.s.util [INFO] Error when trying to kill 10887. Process is probably already dead. 2015-05-12T20:29:09.046+0000 b.s.util [INFO] Error when trying to kill 10943. Process is probably already dead. 2015-05-12T20:29:09.061+0000 b.s.util [INFO] Error when trying to kill 10565. Process is probably already dead. 2015-05-12T20:29:09.076+0000 b.s.util [INFO] Error when trying to kill 10928. Process is probably already dead. 2015-05-12T20:29:09.089+0000 b.s.util [INFO] Error when trying to kill 10747. Process is probably already dead. 2015-05-12T20:29:09.101+0000 b.s.util [INFO] Error when trying to kill 10791. Process is probably already dead. 2015-05-12T20:29:09.113+0000 b.s.util [INFO] Error when trying to kill 10829. Process is probably already dead. 2015-05-12T20:29:09.126+0000 b.s.util [INFO] Error when trying to kill 10814. Process is probably already dead. 2015-05-12T20:29:09.139+0000 b.s.util [INFO] Error when trying to kill 10999. Process is probably already dead. 2015-05-12T20:29:09.151+0000 b.s.util [INFO] Error when trying to kill 10754. Process is probably already dead. 2015-05-12T20:29:09.164+0000 b.s.util [INFO] Error when trying to kill 10774. Process is probably already dead. 2015-05-12T20:29:09.177+0000 b.s.util [INFO] Error when trying to kill 10802. Process is probably already dead. 2015-05-12T20:29:09.190+0000 b.s.util [INFO] Error when trying to kill 10897. Process is probably already dead. 2015-05-12T20:29:09.204+0000 b.s.util [INFO] Error when trying to kill 10953. Process is probably already dead. 2015-05-12T20:29:09.216+0000 b.s.util [INFO] Error when trying to kill 10857. Process is probably already dead. 2015-05-12T20:29:09.229+0000 b.s.util [INFO] Error when trying to kill 10843. Process is probably already dead. 2015-05-12T20:29:09.242+0000 b.s.util [INFO] Error when trying to kill 11027. Process is probably already dead. 2015-05-12T20:29:09.254+0000 b.s.util [INFO] Error when trying to kill 10918. Process is probably already dead. 2015-05-12T20:29:10.274+0000 b.s.util [INFO] Error when trying to kill 10909. Process is probably already dead. 2015-05-12T20:29:10.287+0000 b.s.util [INFO] Error when trying to kill 10973. Process is probably already dead. 2015-05-12T20:29:10.301+0000 b.s.util [INFO] Error when trying to kill 10962. Process is probably already dead. 2015-05-12T20:29:10.315+0000 b.s.util [INFO] Error when trying to kill 10982. Process is probably already dead. 2015-05-12T20:29:10.329+0000 b.s.util [INFO] Error when trying to kill 11019. Process is probably already dead. 2015-05-12T20:29:10.342+0000 b.s.util [INFO] Error when trying to kill 10887. Process is probably already dead. 2015-05-12T20:29:10.356+0000 b.s.util [INFO] Error when trying to kill 10943. Process is probably already dead. 2015-05-12T20:29:10.385+0000 b.s.util [INFO] Error when trying to kill 10565. Process is probably already dead. 2015-05-12T20:29:10.402+0000 b.s.util [INFO] Error when trying to kill 10928. Process is probably already dead. 2015-05-12T20:29:10.414+0000 b.s.util [INFO] Error when trying to kill 10747. Process is probably already dead. 2015-05-12T20:29:10.428+0000 b.s.util [INFO] Error when trying to kill 10791. Process is probably already dead. 2015-05-12T20:29:10.444+0000 b.s.util [INFO] Error when trying to kill 10829. Process is probably already dead. 2015-05-12T20:29:10.457+0000 b.s.util [INFO] Error when trying to kill 10814. Process is probably already dead. 2015-05-12T20:29:10.473+0000 b.s.util [INFO] Error when trying to kill 10999. Process is probably already dead. 2015-05-12T20:29:10.490+0000 b.s.util [INFO] Error when trying to kill 10754. Process is probably already dead. 2015-05-12T20:29:10.503+0000 b.s.util [INFO] Error when trying to kill 10774. Process is probably already dead. 2015-05-12T20:29:10.515+0000 b.s.util [INFO] Error when trying to kill 10802. Process is probably already dead. 2015-05-12T20:29:10.528+0000 b.s.util [INFO] Error when trying to kill 10897. Process is probably already dead. 2015-05-12T20:29:10.540+0000 b.s.util [INFO] Error when trying to kill 10953. Process is probably already dead. 2015-05-12T20:29:10.553+0000 b.s.util [INFO] Error when trying to kill 10857. Process is probably already dead. 2015-05-12T20:29:10.567+0000 b.s.util [INFO] Error when trying to kill 10843. Process is probably already dead. 2015-05-12T20:29:10.580+0000 b.s.util [INFO] Error when trying to kill 11027. Process is probably already dead. 2015-05-12T20:29:10.593+0000 b.s.util [INFO] Error when trying to kill 10918. Process is probably already dead. --

Don't have an account?
Coming from Hortonworks? Activate your account here