@Vipul Choudhary 1) Please define actual size and performance numbers that you encountered. 2) Clarify what test beds you are referring and how did you use them? 3) Clarify what is the type of test case you execute? It is important to clarify because some tests can be disk I/O intensive, others can be memory intensive. After clarifying all the above, we can state that driving a bike is sometimes faster than driving a Ferrari. That may be because the bike is better suited for niche cases where there is a little space for a car to go through (narrow roads, etc). I would not generalize that easy. I am not sure about anything stated as "is always better". There is always an exception. Anyhow, you can set the desired engine the session level, if you wish to use MR or Tez. Thus, for cases where MR performs better, use it. It is not like you have to code it when you execute a Hive query. ... View more

@viswanath kammula The traceback shows maybe a symptom. Best place to start are the hiverserver2 logs and grep for errors. Most likely, your LLAP settings may not be appropriate, but until you mine the logs, that is just an assumption. Please provide excerpt from logs that indicate an error at the same timestamp. ... View more

@Artur Dushelyubov Attempted means that the initiator attempted to schedule a compaction but it failed. As such, there will be no compaction associated with those requests. Other than visually unpleasant, there is no reason to worry about. They are part of the metastore log and it will show upto the threshold set. See threshold for attempted below. They will not be displayed after that number. Compaction History hive.compactor.history.retention.succeeded Default: 3 Metastore Number of successful compaction entries to retain in history (per partition). hive.compactor.history.retention.failed Default: 3 Metastore Number of failed compaction entries to retain in history (per partition). hive.compactor.history.retention.attempted Default: 2 Metastore Number of attempted compaction entries to retain in history (per partition). hive.compactor.initiator.failed.compacts.threshold Default: 2 Metastore Number of of consecutive failed compactions for a given partition after which the Initiator will stop attempting to schedule compactions automatically. It is still possible to use ALTER TABLE to initiate compaction. Once a manually initiated compaction succeeds auto initiated compactions will resume. Note that this must be less than hive.compactor.history.retention.failed. hive.compactor.history.reaper.interval Default: 2m Metastore Controls how often the process to purge historical record of compactions runs. If this was helpful, please vote/accept best answer. ++++++ A little bit of theory below for others who may have a similar question. SHOW COMPACTIONS returns a list of all tables and partitions currently being compacted or scheduled for compaction when Hive transactions are being used, including this information: database name table name partition name (if the table is partitioned) whether it is a major or minor compaction the state the compaction is in, which can be: "initiated" – waiting in the queue to be compacted "working" – being compacted "ready for cleaning" – the compaction has been done and the old files are scheduled to be cleaned "failed" – the job failed. The metastore log will have more detail. "succeeded" – A-ok "attempted" – initiator attempted to schedule a compaction but failed. The metastore log will have more information. thread ID of the worker thread doing the compaction (only if in working state) the time at which the compaction started (only if in working or ready for cleaning state) Compactions are initiated automatically, but can also be initiated manually with an ALTER TABLE COMPACT statement. ... View more

@Kelvin Mitchell Your QueryTable processor queries continuously your table and generates a FlowFile with each query result. Right-click on QueryTable processor, then Configure and Settings tab and you will see that by default Run Schedule is set to 0 seconds. That means query after query. You can change the interval to more seconds as such a new query will be fired later. Keep in mind that your query can be also smart to pick-up only new records. Is query doing that? +++ If this helped, please vote/accept best answer. ... View more

@ripunjay godhani If your application has direct access to the Hadoop cluster, then that application server is your "edge" node. However, the fact that you don't need it in your special case, it does not mean is not a good practice because they are in the same network. That is not the explanation. @SBandaru explanation is valid and a best practice for those cases he mentioned. ... View more

@Subramaniyam KMV Go to: https://mosquitto.org/download/ and follow Centos 6 instructions. If this helps, please vote/accept best answer. Time spent to help should be appreciated. ... View more