As par phoenix documentation on UPDATE_CACHE_FREQUENCY : "A millisecond value indicates how long the client will hold on to its cached version of the metadata before checking back with the server for updates" http://phoenix.apache.org/language/index.html#options So,. i set the value by following command(and i restarted client application as well): Alter table <TABLE_NAME> set UPDATE_CACHE_FREQUENCY=300000; But it does not seems to be working(client is not updating cache). Because i sent the request to client app (for this table's data) after 15 minutes and got same 'Cache Expired' message, rather cache should have been updated automatically every 5 minutes and it shouldn't have been updated in client query thread. Please update if my understanding is incorrect or something is missing ... View more

Is there any way to update client cache automatically Or what can be side effects if we keep 'UPDATE_CACHE_FREQUENCY' to 'NEVER' or very high value? ... View more

Thank Ankit! But Client cache is also not being refreshed automatically. As its mentioned in my original question when client queries some data , then only it goes for update and that particular request(query) takes long to execute( as it's refreshing cache as well) and after that for around 15 mins all queries works fine. Is there any way to automatically refreshes cache , so that it will not refresh in same client query thread. ... View more

Hi, I checked region server logs and couldn't found any errors. But one difference found when update stats are triggered manually. Logs when i trigger manually by command 'UPDATE STATISTICS...': 2018-06-08 16:30:13,389 INFO [RpcServer.FifoWFPBQ.default.handler=28,queue=1,port=16020-SendThread()] zookeeper.ClientCnxn: Session establishment complete on server , sessionid = 12345, negotiated timeout = 60000 2018-06-08 16:30:23,075 INFO [phoenix-update-statistics-3] coprocessor.UngroupedAggregateRegionObserver: UPDATE STATISTICS finished successfully for scanner: org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl@27ab2aa8.... 2018-06-08 16:30:23,076 INFO [phoenix-update-statistics-3] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=12345 2018-06-08 16:30:23,078 INFO [phoenix-update-statistics-3] zookeeper.ZooKeeper: Session: 12345 closed (There are two more similar threads in the logs in same chain) Logs for other instances of update stats( as par thread id its trigerred due to compactions): 2018-06-08 16:39:52,308 INFO [regionserver-shortCompactions-22-SendThread()] zookeeper.ClientCnxn: Session establishment complete on server , sessionid = 12346 2018-06-08 16:39:52,538 INFO [phoenix-update-statistics-2] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=12346 2018-06-08 16:39:52,540 INFO [phoenix-update-statistics-2] zookeeper.ZooKeeper: Session: 12346 closed. (There is one more similar thread in same chain). For other instances: Number of threads are different, ThreadId is different and also message is different. So, it seems that Auto update of STATS is not triggering? ... View more

i also faced the same issue , but it worked for me by changing timeout config(though it took some time for these chnages to come into effect). I changed the Phoenix Query Timeout config(via Ambari) and tried the query and i got the same timeoutException of default 60 secs. I verified the updated value in hbase-site.xml config file and tried the same query after some time(around 1hr or may be earlier as well it would have worked) and query executed successfully which took around 90secs. So, it took the timeout changes successfully. ... View more

Thanks Shubham! Now we can't see duplicates , we haven't deleted STATS, may be some metadata sync-up removed the issue(may be we'll see it again in some time).I've few queries: 1) Is it safe to delete these tables on production, i mean is phoenix automatically recreates these tables. 2) I tried to look for some web references where i can see that how Phoenix updates/refer these system tables and if there are some config impacting these scenarios.Please share if there is some reference pointing to this specific section of phoenix. ... View more

PFA screenshot from Spout metrics from Storm UI . : storm-spout-executors.png And we have set distributed to 'true' while creating spout. So distribution(multithreading) is enabled. Also, though we have fixed some issues in bolts and overall performance is good, but we might have more load so if if spout level as well parrallism can work , that would be great. It might be that Solace(queue provider) does not supports multithreading? ... View more

@Larisa : We managed to fix this issue and messages are being consumed by kafka. We did following changes: 1) As Storm-kafka code(DynamicPartitionConnections class) is not supporting to communicate as PLAINTEXTSASL protocol. So for time being we modified that class in our codeto support secured protocol. @Artem : Can you please check if this change can be added in storm-kafka library. 2) Added permission to storm service user(which you can check in storm jaas file) for kafka topic via ranger or kafka ACL command. 3) Somehow value for host in brokers metadata on zookeeper was null. So we updated that value with broker hostname. Hopefully this will help you. ... View more

@Artem Ervits : I am following above guidelines.But i found that issue is that storm-kafka code is setting security protocol to unsecured one even if config is set to secured one.I checked by debugging that In PartitionConnection class instance, config is secured , but it is creating consumer with default(unsecured) protocol[Added more details in my previous comment]. And it is failing while getting offset as SimpleConsumer is trying to connect by unsecured protocol. So storm-kafka library should call another constructor of consumer which takes protocol as parameter..or if am still doing something wrong? PS: I tried by setting protocol to secured one while debugging the code, then i am getting Jaas related error, but at least it tries to connect in secure way to kafka. ... View more

We're getting same issue and we provided 'PLAINTEXTSASL' as security protocol in spout config. We checked that when storm kafka code creates kafka consumer, it calls first constructor of SimpleConsumer which sets the protocol to unsecured one(PLAINTEXT) , rather than calling second constructor which takes security protocol as parameter. For details , please refer : https://github.com/hortonworks/storm-release/blob/HDP-2.5.0.0-tag/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicPartitionConnections.java We're analyzing for solving the same..Any help is highly appriciated. ... View more