Member since
07-12-2016
17
Posts
4
Kudos Received
0
Solutions
06-12-2018
04:11 PM
As par phoenix documentation on UPDATE_CACHE_FREQUENCY : "A millisecond value indicates how long the client will hold on to its cached version of the metadata before checking back with the server for updates"
http://phoenix.apache.org/language/index.html#options So,. i set the value by following command(and i restarted client application as well): Alter table <TABLE_NAME> set UPDATE_CACHE_FREQUENCY=300000; But it does not seems to be working(client is not updating cache). Because i sent the request to client app (for this table's data) after 15 minutes and got same 'Cache Expired' message, rather cache should have been updated automatically every 5 minutes and it shouldn't have been updated in client query thread. Please update if my understanding is incorrect or something is missing
... View more
06-12-2018
01:47 PM
Is there any way to update client cache automatically Or what can be side effects if we keep 'UPDATE_CACHE_FREQUENCY' to 'NEVER' or very high value?
... View more
06-11-2018
07:34 AM
Thank Ankit! But Client cache is also not being refreshed automatically. As its mentioned in my original question when client queries some data , then only it goes for update and that particular request(query) takes long to execute( as it's refreshing cache as well) and after that for around 15 mins all queries works fine. Is there any way to automatically refreshes cache , so that it will not refresh in same client query thread.
... View more
06-08-2018
03:27 PM
Hi, I checked region server logs and couldn't found any errors. But one difference found when update stats are triggered manually. Logs when i trigger manually by command 'UPDATE STATISTICS...': 2018-06-08 16:30:13,389 INFO [RpcServer.FifoWFPBQ.default.handler=28,queue=1,port=16020-SendThread()] zookeeper.ClientCnxn: Session establishment complete on server , sessionid = 12345, negotiated timeout = 60000
2018-06-08 16:30:23,075 INFO [phoenix-update-statistics-3] coprocessor.UngroupedAggregateRegionObserver: UPDATE STATISTICS finished successfully for scanner: org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl@27ab2aa8....
2018-06-08 16:30:23,076 INFO [phoenix-update-statistics-3] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=12345
2018-06-08 16:30:23,078 INFO [phoenix-update-statistics-3] zookeeper.ZooKeeper: Session: 12345 closed (There are two more similar threads in the logs in same chain) Logs for other instances of update stats( as par thread id its trigerred due to compactions): 2018-06-08 16:39:52,308 INFO [regionserver-shortCompactions-22-SendThread()] zookeeper.ClientCnxn: Session establishment complete on server , sessionid = 12346
2018-06-08 16:39:52,538 INFO [phoenix-update-statistics-2] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=12346
2018-06-08 16:39:52,540 INFO [phoenix-update-statistics-2] zookeeper.ZooKeeper: Session: 12346 closed. (There is one more similar thread in same chain). For other instances: Number of threads are different, ThreadId is different and also message is different. So, it seems that Auto update of STATS is not triggering?
... View more
06-07-2018
01:36 PM
Hello, We've set 'phoenix.stats.updateFrequency' to 10 mins(and verified that this property exists in hbase-site.xml and restarted Hbase), still STATS are not being updated automatically. select * from SYSTEM.stats where "PHYSICAL_NAME"='TABLE_NAME' and "GUIDE_POSTS_WIDTH" is null; LAST_STATS_UPDATE_TIME : 2018-06-06 20:30:30 Current time :2018-07-07 13:20:26 But if i execute UPDATE STATISTICS command manually, thenSTATS are being updated and LAST_STATS_UPDATE_TIME is also being updated to latest time. What problem we're getting due to this : Whenever someone queries Phoenix table, then for the first time query takes so long(sometimes more than 15 secs) . and further queries becomes fast for next few(15) mins. We checked server logs for long response time and found out that TableStatsCache get expired and it tries to reload STATS cache in same thread and becomes slow. And we think that if stats get updated regularly, then queries will be faster as it will not go for Updating STATS cache. Please check and let us know if something more is required for Auto stats update.
... View more
Labels:
- Labels:
-
Apache Phoenix
04-05-2018
12:18 PM
i also faced the same issue , but it worked for me by changing timeout config(though it took some time for these chnages to come into effect). I changed the Phoenix Query Timeout config(via Ambari) and tried the query and i got the same timeoutException of default 60 secs. I verified the updated value in hbase-site.xml config file and tried the same query after some time(around 1hr or may be earlier as well it would have worked) and query executed successfully which took around 90secs. So, it took the timeout changes successfully.
... View more
04-04-2018
03:00 PM
Thanks Shubham! Now we can't see duplicates , we haven't deleted STATS, may be some metadata sync-up removed the issue(may be we'll see it again in some time).I've few queries: 1) Is it safe to delete these tables on production, i mean is phoenix automatically recreates these tables. 2) I tried to look for some web references where i can see that how Phoenix updates/refer these system tables and if there are some config impacting these scenarios.Please share if there is some reference pointing to this specific section of phoenix.
... View more
03-27-2018
10:52 PM
Hello, We're seeing lots of duplicates in phoenix table, while not in respective Hbase table. Result : The total count in phoenix table is 3 times to respective Hbase table(say total count, in phoenix is 30 millions and in respective Hbase table is 10 million only).We checked for specific row-keys as well, there are duplicates in Phoenix , but not in Hbase. More details :For this table we're using SALT_BUCKET property of phoenix and global index on one of the column and Phoenix version is 4.7. We're consuming data from kafka and storing in Pheonix via Storm-JDBC connector. Also this is reproducing only when there is so much concurrent requests.Till now we're unable to replicate on normal (dev) environment. Please guide us if we're missing some config or some other pointers. @Dhiraj
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache Phoenix
03-02-2018
10:40 AM
1 Kudo
Zeppelin is unable to get renewed kerberos ticket automatically. Kerboers config has ticket renewal after 24h. Zeppelin stops executing jdbc queries after 24h , even though kerberos ticket is valid(i've checked the same via klist on hbase node). Queries does not work even after waiting for 1h and even after logout-login again. Error is like : 'Couldn't find kerberos ticket..consider 'kinit'. but it works fine if i restart jdbc interpreter. So, It seems there any internal cache or something like that which has metadata of kerberos ticket and is there any config(way) to change the same so that we don't need to restart the interprator. More details :I am using phoenix queries and using ADM account and Zeppelin version is 0.7.0
... View more
Labels:
- Labels:
-
Apache Phoenix
-
Apache Zeppelin
07-28-2017
12:38 PM
PFA screenshot from Spout metrics from Storm UI . : storm-spout-executors.png And we have set distributed to 'true' while creating spout. So distribution(multithreading) is enabled. Also, though we have fixed some issues in bolts and overall performance is good, but we might have more load so if if spout level as well parrallism can work , that would be great. It might be that Solace(queue provider) does not supports multithreading?
... View more
07-06-2017
08:25 AM
Hello, We're consuming data from Solace queue(via JMS APIs) and we start the topology with 3 executors(and same tasks) for this spout.And problem we're getting is that all data is being consumed from one executor only and other executors are idle which makes the overall consumption very slow and after consuming few hundred thousand records this executor breaks with out of memory error. This was working fine with earlier version of HDP& Storm(1.0.1-2.5.3). The version we're using now ;Storm version : 1.1.0, HDP version : 2.6.1 So, we 're looking for solution of starting all executors at same time to distribute the load?
... View more
Labels:
- Labels:
-
Apache Storm
12-02-2016
10:41 AM
1 Kudo
@Larisa : We managed to fix this issue and messages are being consumed by kafka. We did following changes: 1) As Storm-kafka code(DynamicPartitionConnections class) is not supporting to communicate as PLAINTEXTSASL protocol. So for time being we modified that class in our codeto support secured protocol. @Artem : Can you please check if this change can be added in storm-kafka library. 2) Added permission to storm service user(which you can check in storm jaas file) for kafka topic via ranger or kafka ACL command. 3) Somehow value for host in brokers metadata on zookeeper was null. So we updated that value with broker hostname. Hopefully this will help you.
... View more
11-21-2016
04:30 PM
@Artem Ervits : I am following above guidelines.But i found that issue is that storm-kafka code is setting security protocol to unsecured one even if config is set to secured one.I checked by debugging that In PartitionConnection class instance, config is secured , but it is creating consumer with default(unsecured) protocol[Added more details in my previous comment]. And it is failing while getting offset as SimpleConsumer is trying to connect by unsecured protocol. So storm-kafka library should call another constructor of consumer which takes protocol as parameter..or if am still doing something wrong? PS: I tried by setting protocol to secured one while debugging the code, then i am getting Jaas related error, but at least it tries to connect in secure way to kafka.
... View more
11-21-2016
03:59 PM
We're getting same issue and we provided 'PLAINTEXTSASL' as security protocol in spout config. We checked that when storm kafka code creates kafka consumer, it calls first constructor of SimpleConsumer which sets the protocol to unsecured one(PLAINTEXT) , rather than calling second constructor which takes security protocol as parameter. For details , please refer : https://github.com/hortonworks/storm-release/blob/HDP-2.5.0.0-tag/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicPartitionConnections.java We're analyzing for solving the same..Any help is highly appriciated.
... View more
10-17-2016
09:18 AM
2 Kudos
We're using storm-jms for consuming data from queue,its working fine with Storm-0.10.0/HDP2.4. It starts breaking when we upgraded storm to 1.0.1 and HDP 2.5. We checked on HDP maven repo and HDP github as well, but could not find artifact for storm-jms 1.0.1. (maven group-id for storm-jms: org.apache.storm). Github repo for Storm-external on apache repo: https://github.com/apache/storm/tree/master/external Github repo for Storm-external on HDP repo: https://github.com/hortonworks/storm-release/tree/HDP-2.5.0.0-tag storm-jms is missing for apache 1.0.x tags and HDP2.5.0.0 tag.
... View more
Labels: