Member since
03-23-2017
41
Posts
5
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
614 | 01-19-2018 08:05 AM | |
3572 | 12-01-2017 06:46 PM | |
3915 | 04-19-2017 06:32 AM |
05-09-2018
09:53 AM
I just checked and it is opening, can you try again? @Krishna Srinivas
... View more
05-08-2018
06:34 AM
This question has been answered in this community before, please search for the same. Also have a look at https://superuser.blog/hbase-dead-regionserver/
... View more
04-26-2018
06:44 AM
I have used something similar only. You can use this template or what I used (https://superuser.blog/upgrading-apache-phoenix-hdp/) The only thing is see what you're doing before actually doing it. And try to make sense out of it.
... View more
04-25-2018
12:12 PM
if you set TTL to 20(seconds) then that may be the reason.
... View more
04-25-2018
12:10 PM
we also encountered many bugs with 4.7 and we had to manually upgrade phoenix. However they have fixed bugs with latest Hortonworks distribution but the version number still remains at 4.7
... View more
03-12-2018
05:51 AM
hey @Juan Castrilli let the community know how did you proceed 🙂
... View more
01-22-2018
10:40 AM
you can check queue size from regionserver UI (host-address:16030/rs-status) under Queues tab. Queue size increases when there are too many requests and your RS cannot handle it. This can happen in case of too many requests (CPU crunch) ongoing compation ongoing GC You should investigate and identify the cause. and accordingly you can have more cpu, schedule compactions during off-peak hours, optimize GC. Hope it helps!
... View more
01-22-2018
10:36 AM
check hbase gc logs. if there is a GC pause that correlates with these log lines then you got your problem. While master/RS is in GC, it fails to send heatbeat to zookeeper and the zookeeper node expires. In that case you can increase zookeeper timeout and increase ticktime in zookeeper config. this is not directly related but do checkout: https://superuser.blog/hbase-dead-regionserver/
... View more
01-19-2018
08:05 AM
1 Kudo
no it will not. during compaction, temporary files are created which will be compacted. later when it's done the smaller files are removed and huge compacted files will come it it's place. So at no time client requests are blocked. response time might increase because of cpu overhead of compaction process.
... View more
12-07-2017
06:43 PM
good that the data is there. here's the next things you can do: 1. check table size on src cluster and snapshot size on dest cluster [hadoop dfs -du-h /apps/hbase/data/<table> or something like that] 2. compare files in snapshots that are getting listed and one which is not. What happens during copying snapshot is first data gets copied into some temp dir on dest cluster and then it gets moved to actual folder. If you see some files missing [metadata file. with .snapshot extension probably], try to search for them in other places .tmp folder as far as I remember, if you happen to find them, move it to table snapshot folder. I don't exactly remember all the dir names and locations so you will have to explore around a bit
... View more
12-07-2017
01:41 PM
can you confirm the owner of copied directory in destination cluster? if it is different, you may want to change it. Also check the copied data on destination cluster. the data size, the metadata file (snapshot files) and if data is present or not. sometime it is there but in different folder, probably some tmp or hidden folder, move it if that is the case
... View more
12-07-2017
05:30 AM
in that case you would want to further increase cache timeout or break your query into multiple queries and use temporary tables.
... View more
12-05-2017
07:03 PM
would you care to provide more details? ie how much memory is it configured to use? (under Config tab of HBASE service in Ambari) how much is it going up to? ideally it should not go beyond set value.
... View more
12-05-2017
05:46 AM
good to know you solved the original problem. not sure what the warning means though. Maybe someone from community can help
... View more
12-01-2017
06:46 PM
1 Kudo
you may want to increase value of phoenix.coprocessor.maxServerCacheTimeToLiveMs Maximum living time (in milliseconds) of server caches. A cache entry expires after this amount of time has passed since last access. Consider adjusting this parameter when a server-side IOException(“Could not find hash cache for joinId”) happens. Getting warnings like “Earlier hash cache(s) might have expired on servers” might also be a sign that this number should be increased. http://phoenix.apache.org/tuning.html
... View more
11-29-2017
05:33 AM
If you're using zeppelin for query execution then you can have LDAP configured. Not sure if you're looking for this kind of solution though.
... View more
11-23-2017
02:11 AM
Thanks all for help! I carried out steps as I mentioned in question. Please include `chown -R` operation too before starting services as mentioned by @rmaruthiyodan We did it with approx 5min of downtime though, If anyone else carries this operation out without downtime/in rolling fashion, please let community know.
... View more
11-21-2017
07:34 AM
@Karthik Palanisamy I am trying to figure a rolling based approach and in my comment above, I suggest repeating step 2,3,4 for each zookeeper one after other. Is that correct way to go about this?
... View more
11-21-2017
06:54 AM
Thanks @rmaruthiyodan , So let me confirm the steps:
change dataDir conf in ambari . (dataLogDir is not separately configured.) shutdown zk node. copy contents to new dir, change permission of folder (myid and version-2/ ) start zk repeat 2-4 for other two zk. yes we have 3 zookeeper nodes. I wanted to ask if above steps can be executed while HBase is running. (they should be)
... View more
11-20-2017
07:42 AM
1 Kudo
Hello, Currently our zookeeper dataDir is at `/dfs/1/hadoop/zookeeper/` but unfortunately, `/dfs/1/` is the HDFS disk mount. Hence in current scenario, for us it is not possible to swap disks for HDFS as zookeeper is also using it. We wanted to move zookeeper dataDir to some other place like `/usr/lib/zookeeper` but I am not quite sure of what steps needs to be taken. Here's what I think should work. Create new dir. Stop zookeeper and hbase. copy data from old zk datadir to new zk datadir change zk conf to point dataDir to new dir start zk and hbase. Here what I'm unsure of is if copying the data is the correct way to do this. We do not have staging cluster hence seeking help from community 🙂 Much thanks! Sanket.
... View more
Labels:
11-18-2017
06:05 AM
If you are just testing out on sandbox, this should really help: https://superuser.blog/upgrading-apache-phoenix-hdp/ We did it on prod.
... View more
11-18-2017
06:02 AM
Adding to @Ankit Singhal's answer, we did try to replace the jars and it worked. I have a write-up here: https://superuser.blog/upgrading-apache-phoenix-hdp/
... View more
11-18-2017
05:59 AM
I managed to upgrade it to 4.10 as 4.7 had some serious bugs which would not allow it to be usable. https://superuser.blog/upgrading-apache-phoenix-hdp/ UPDATE: we were on HDP 2.5 and bug was related to count() method. It may have been resolved in newer patched version of phoenix available with latest HDP 2.6
... View more
06-08-2017
09:22 AM
1 Kudo
I have set major compaction frequency to 3 days. hbase.offpeak.start.hour = 19
hbase.offpeak.end.hour = 1 However I see in RS metrics, for a particular table number of total major compaction is 22 ("Namespace_default_table_user_states_metric_majorCompactionTime_num_ops": 22) and the cluster was last restarted 14 days ago. So this is >1 major compaction a day. Also in logs I see many major compaction running on single file. Despite it's set for 3 to 10 files. ( see the first line below) 2017-06-08 02:32:07,377 INFO [StoreOpener-1f254a063af7b15509846ff27ef9b5a8-1] compactions.CompactionConfiguration: size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 259200000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6
2017-06-08 00:28:36,581 INFO [regionserver/<host>/10.10.210.195:16020-shortCompactions-1495621699452] regionserver.HStore: Completed major compaction of 1 (all) file(s) in f of events,voodoo.com:ringo:call_log:1433665626397:r116qjoprsjhppq1,1435904225768.c901139ea575133398f86ceb4dc253fa. into f275a4737fe1485e9058532a2226d401(size=13.7 M), total size for store is 13.7 M. This selection was in queue for 0sec, and took 2sec to execute.2017-06-08
06:02:22,176 INFO [regionserver/<host>/10.10.210.195:16020-shortCompactions-1495621699452] regionserver.HStore: Completed major compaction of 1 (all) file(s) in f of events,voodoo.com:ringo:call_log:1458694092743:7azkyvazxry7rxkr,1461000798058.5703747043306593388ed83c0d245772. into 168f131831244c9f8aafdcc9105e37dc(size=304.3 M), total size for store is 304.3 M. This selection was in queue for 0sec, and took 27sec to execute.
2017-06-08 06:03:55,214 INFO [regionserver/<host>/10.10.210.195:16020-shortCompactions-1495621699452] regionserver.HStore: Completed major compaction of 1 (all) file(s) in f of events,voodoo.com:ringo:connectivity_log:1464497220532:y66a0afayoava6gn,1466007331891.d89b88bd976aab7e229116fcf4ac2fb0. into e7f6451d5f2a40a59795285c2d5c8feb(size=1.0 G), total size for store is 1.0 G. This selection was in queue for 28sec, and took 1mins, 32sec to execute. This log line I find several times in a day and it's not limited to off-peak hours. I do not think frequency of major compaction should be this high. Or maybe the log line is suggesting something else and not major compaction. hbase.hregion.max.filesize is 20G hbase.hstore.compaction.max is 10 Any help with figuring out what is happening is appreciated. Please let me know if any more information is required. Thank you!
... View more
Labels:
- Labels:
-
Apache HBase
05-30-2017
08:04 AM
copied from zookeeper docs:
tickTime: the length of a single tick, which is the basic time unit used by ZooKeeper, as measured in milliseconds. It is used to regulate heartbeats, and timeouts. For example, the minimum session timeout will be two ticks.
so minimum is twice the tickTime and maximum is 20 times of the same. And yes I agree for most systems default 40s shall suffice, but for exceptional case, one needs to increase. (as also recommanded in HBase book: http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired)
... View more
05-29-2017
07:05 AM
Adding to @Josh Elser 's answer, if you choose to increase timeout for zk session by increasing ticktime, adding those values in HBase conf won't work. It'll take zk ticktime in account for calculation (ticktime * 20) I was facing the same problem and later I wrote about it here: https://superuser.blog/hbase-dead-regionserver/
... View more
05-29-2017
07:02 AM
Setting timeouts from HBase conf did not work for me. tickTime in ZK was getting picked for session. Here's more info: https://superuser.blog/hbase-dead-regionserver/
... View more
04-19-2017
06:43 AM
thanks @ssingla , I found the issue. And thanks for pointing out something related, might help in future.
... View more