@Hefei Li Great! Can you accept the answer then so we can close this question and others having similar issue get benefited? ... View more

For the sake of discussion, lets say the system is running at peak load 24 hours.. Out of 20K, there are 10K reads and 10K inserts per second So, after the first 30 mins of running, system will add additional 10K deletes per second so a total of 30K hits. Its definitely not that straight forward and HBase is going to batch the actual deletes somehow internally. 30K tps is not a lot for HBase but the question is how big of a cluster are we talking about? Also, other thing will be the memory available to the RegionServer... it makes sense to keep as much data in memory as possible so the I/O is minimal, as the data is to be deleted after 30 mins anyways. So, the next set of questions is - whats the memory available on the box and to region server? How big is each message? ... View more

Just to add to that.. if you want to make change, it would be easier for you to follow the code by looking at the source here - https://github.com/apache/ambari/ The ui code is here - https://github.com/apache/ambari/tree/trunk/ambari-web/app You can just fork the project and build it after making changes. ... View more

@Grace Xu Here is another approach for you (a script will definitely work as well but will not make use of oozie) I believe we can make use of decision node and do the following to get this done via oozie - Assumption: I am assuming the log table has some kind of id (log_entry_id) associated with the table names along with some other attributes that can be used by sqoop job like (HDFS/Hive destination, columns to import - IF not all etc) This flow can be adjusted to match whatever constraints and existing design you are working with. For e.g. you can use the table name from the log if you do not have an id. Also, you may have some tables that were relevant (active) in past but not any more so you may have active flag in log table that can be utilized in the 2) Shell action below to figure out if that table has to be imported or not.. etc etc.. you get the idea. This is a general structure that can be customized. the WorkFlow will have the following key steps - 1) Java action - Write a small java program that uses JDBC to connect to SQL Server, reads the data from log table and creates a comma delimited file on HDFS like /tmp/inegst-tables.txt 2) Shell action (Input parameter : log_entry_id) - read the file from HDFS and get the line starting with (log_entry_id+1). The script will output the value in Java Properties format (like param1=value1,param2=vaule2) etc.. The workflow node will capture the output value for use in the subsequent steps. If the shell script did not find the next row, it will return -1. 3) Decision Node - if the value is less than 0, then go to END else go to sqoop node 4) Sqoop Node - Execute sqoop task using the table name, destination etc received from the Shell action capture output. On success go to previous Shell Action else go to End Workflow Execution : When executing the workflow a default starting value of 0 will be provided for log_entry_id. ... View more

@Hefei Li Can you set the directory permissions to 755? (all the directories ,including parents, in the path that contains the jar files) ... View more

@Kumar Datla The error message does say - "Unable to validate the location with path: /stmp" which means either the path does not exist or due to permissions issue the program / process was unable to access the directory. In any case, looks like you have been able to move past the issue so I will close this thread. ... View more