About RajeshMadurai

RajeshMadurai · ‎04-02-2020

How to get entire table list in hive(cluster). Not for particular table and database.

RajeshMadurai · ‎01-23-2020

Output of my table with two dates. ========================== **** hdfs://*** 29GB 30233MB 1/23/2020 **** hdfs://*** 8GB 9040MB 1/23/2020 **** hdfs://*** 911GB 933122MB 1/23/2020 **** hdfs://*** 29GB 29795MB 1/23/2020 -(MINS) **** hdfs://*** 129GB 130233MB 1/24/2020 **** hdfs://*** 18GB 19040MB 1/24/2020 **** hdfs://*** 1911GB 1933122MB 1/24/2020 **** hdfs://*** 129GB 129795MB 1/24/2020 Output I need like this. **** hdfs://*** 29GB 30233MB **** hdfs://*** 8GB 9040MB **** hdfs://*** 911GB 933122MB **** hdfs://*** 29GB 29795MB I have tired two view creation with join select not working. create view alerts_archive.dbsizevw1 AS select * from alerts_archive.dbsize where date='Jan-23-2020'; create view alerts_archive.dbsizevw2 AS select * from alerts_archive.dbsize where date='Jan-24-2020'; select db_name,location, (a.size_in_mb-b.size_in_mb) as variance_MB, (a.size_in_gb-b.size_in_gb) as variance_GB from alerts_archive.dbsizevw2 a join alerts_archive.dbsizevw1 b; select (a.size_in_mb-b.size_in_mb) as variance_MB, (a.size_in_gb-b.size_in_gb) as variance_GB from alerts_archive.dbsizevw2 a join alerts_archive.dbsizevw1 b;

RajeshMadurai · ‎01-23-2020

Below is my table description. hive> describe dbsize; OK db_name string location string size_in_mb string size_in_gb string date string # Partition Information # col_name data_type comment date string Time taken: 0.335 seconds, Fetched: 10 row(s) Have to mins the column size_in_mb with different date in where condition. Having sample sql query for rdbms environment. select DB_NAME, LOCATION, c2-c1 "difference size in MB", c4-c3 "difference size in GB" from dbsize (select size_in_mb c1 from dbsize where date='01-23-2020'), (select size_in_mb c2 from dbsize where date='01-25-2020'), (select size_in_gb c3 from dbsize where date='01-23-2020'), (select size_in_gb c4 from dbsize where date='01-25-2020'); Need help on hive hql query to mins the columns for two different dates. In same single table. Please advise me.

RajeshMadurai · ‎02-21-2019

In non-partition table having multiple files in table location. When select statement triggered it worked. How it fetch the data where else without running msck repair command?

RajeshMadurai · ‎02-13-2019

We have taken backup one of the production database data and moved it to development local filesystem. In development movied data from local mountpoint to hive database hdfs location. Question1: Hive msck repair in managed partition table failed with below error message. hive> msck repair table testsb.xxx_bk1; FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask What does exception means. After dropping the table and re-create the table in external type. it worked successfully. hive> use testsb; OK Time taken: 0.032 seconds hive> msck repair table XXX_bk1; xxx_bk1:payloc=YYYY/client_key=MISSDC/trxdate=20140109 . . Repair: Added partition to metastore xxx_bk1:payloc=0002/client_key=MISSDC/trxdate=20110105 . . Time taken: 16347.793 seconds, Fetched: 94156 row(s) Can you please confirm why it not worked in managed table? Question:2. Where else select * from table; query able to fetch in non-partition table. Why? We have done testsb database creation and Table creation with ddl script. And moved the data from local to hdfs hive table location.

RajeshMadurai · ‎12-21-2018

How the directories are created ? Is it by some script / code ? Yes through the shell script executed two mins(1st time given hive prompt wrongly). But any second time it should exists file already available. #cat -e /tmp/hdfslist -you could see the end of each file name, also you can see any other chars if present other than $ sign After taking the output from above command. Have to remove the duplicate files hdfs dfs -rm rajesh/int_rajeshlake/Retirement/RS_Raw$ Please confirm

RajeshMadurai · ‎11-28-2018

#1. Below hdfs location having duplicate directories: ====================================================== hdfs dfs -ls /rajesh/int_datalake/Retirement/ drwxrwxr-x+ - root supergroup 0 2018-10-29 03:24 /rajesh/int_datalake/Retirement/RS_Access drwxrwxr-x+ - root supergroup 0 2018-10-29 03:17 /rajesh/int_datalake/Retirement/RS_Access drwxrwxr-x+ - root supergroup 0 2018-10-29 03:17 /rajesh/int_datalake/Retirement/RS_Raw drwxrwxr-x+ - root supergroup 0 2018-11-27 01:35 /rajesh/int_datalake/Retirement/RS_Raw1 drwxrwxrwx+ - root supergroup 0 2018-11-27 01:39 /rajesh/int_datalake/Retirement/RS_Raw_bk drwxrwxr-x+ - root supergroup 0 2018-10-29 03:24 /rajesh/int_datalake/Retirement/RS_Repos drwxrwxr-x+ - root supergroup 0 2018-10-29 03:17 /rajesh/int_datalake/Retirement/RS_Repos drwxrwxr-x+ - root supergroup 0 2018-10-29 03:24 /rajesh/int_datalake/Retirement/RS_Stage drwxrwxr-x+ - root supergroup 0 2018-10-29 03:17 /rajesh/int_datalake/Retirement/RS_Stage drwxrwxr-x+ - root supergroup 0 2018-10-29 03:24 /rajesh/int_datalake/Retirement/RS_Work drwxrwxr-x+ - root supergroup 0 2018-10-29 03:17 /rajesh/int_datalake/Retirement/RS_Work First we moved the file to bk name and tired to remove but not worked [root@ ]# [root@ ]# hdfs dfs -mv /rajesh/int_rajeshlake/Retirement/RS_Raw /rajesh/int_rajeshlake/Retirement/RS_Raw_bk [root@ ]# hdfs dfs -rm -r /rajesh/int_datalake/Retirement/RS_Raw rm: `/rajesh/int_datalake/Retirement/RS_Raw': No such file or directory hdfs dfs -mv /rajesh/int_datalake/Retirement/RS_Raw /rajesh/int_datalake/Retirement/RS_Raw123 mv: `/rajesh/int_datalake/Retirement/RS_Raw': No such file or directory #2. Finally removed directory rmdir below command and recreated the directory: ================================================================================ hdfs dfs -rmdir /rajesh/int_rajeshlake/Retirement/RS_Wor* hdfs dfs -ls /rajesh/int_rajeshlake/Retirement/ Found 9 items drwxrwxr-x+ - root supergroup 0 2018-10-29 03:24 /rajesh/int_rajeshlake/Retirement/RS_Access drwxrwxr-x+ - root supergroup 0 2018-10-29 03:17 /rajesh/int_rajeshlake/Retirement/RS_Access drwxrwxrwx+ - root supergroup 0 2018-11-27 01:39 /rajesh/int_rajeshlake/Retirement/RS_Raw drwxrwxr-x+ - root supergroup 0 2018-10-29 03:17 /rajesh/int_rajeshlake/Retirement/RS_Raw drwxrwxr-x+ - root supergroup 0 2018-11-27 01:35 /rajesh/int_rajeshlake/Retirement/RS_Raw1 drwxrwxr-x+ - root supergroup 0 2018-10-29 03:24 /rajesh/int_rajeshlake/Retirement/RS_Repos drwxrwxr-x+ - root supergroup 0 2018-10-29 03:17 /rajesh/int_rajeshlake/Retirement/RS_Repos drwxrwxr-x+ - root supergroup 0 2018-10-29 03:24 /rajesh/int_rajeshlake/Retirement/RS_Stage drwxrwxr-x+ - root supergroup 0 2018-10-29 03:17 /rajesh/int_rajeshlake/Retirement/RS_Stage #3. Tried mkdir duplicat manually, Command executed not error directory already exists. But not no duplication created: ======================================================================================================================= hdfs dfs -ls /rajesh/data/Corporate drwxrwxr-x - root supergroup 0 2018-11-27 04:23 /rajesh/data/Corporate/Corp_Access drwxrwxr-x - root supergroup 0 2018-11-27 04:24 /rajesh/data/Corporate/Corp_Raw drwxrwxr-x - root supergroup 0 2018-11-27 04:25 /rajesh/data/Corporate/Corp_Repos drwxrwxr-x - root supergroup 0 2018-11-27 04:25 /rajesh/data/Corporate/Corp_Stage drwxrwxr-x - root supergroup 0 2018-11-27 04:26 /rajesh/data/Corporate/Corp_Work hdfs dfs -mkdir -p /rajesh/data/Corporate/Corp_Work hdfs dfs -ls /rajesh/data/Corporate drwxrwxr-x - root supergroup 0 2018-11-27 04:23 /rajesh/data/Corporate/Corp_Access drwxrwxr-x - root supergroup 0 2018-11-27 04:24 /rajesh/data/Corporate/Corp_Raw drwxrwxr-x - root supergroup 0 2018-11-27 04:25 /rajesh/data/Corporate/Corp_Repos drwxrwxr-x - root supergroup 0 2018-11-27 04:25 /rajesh/data/Corporate/Corp_Stage drwxrwxr-x - root supergroup 0 2018-11-27 04:26 /rajesh/data/Corporate/Corp_Work Please confirm how the duplicate directories created? Directores are in empty file so we remove and recreated the file. If any table or data available what steps have to follow?

RajeshMadurai · ‎10-07-2018

While doing the manual failover in resource manager my schduled and running application id's will it move to stanby resource manager buy how? In hdfs name node journal nodes are monitoring the edit logs. In resource manager which daemon is monitoring?

RajeshMadurai · ‎10-07-2018

In rdbms database block size is 8kb and in hadoop block size is 64MB. In sqoop import example my rdbms tables size is 300mb. So it will split into 5 mapper ? Please confirm

RajeshMadurai · ‎09-26-2018

In sqoop import how mapreduce works in key & value pair in rdbms tables with structure data? Please explain.

Online	Offline
Last Visited	‎04-03-2020 10:27 AM

Date Registered	‎08-04-2017 04:05 AM
Last Visited	‎04-03-2020 10:27 AM
Total Messages Posted	20

Re: How to use HCatalog or Hive Metastore to list ...

Re: hive minus query

hive minus query

Re: Hive msck repair not working managed partition...

Hive msck repair not working managed partition tab...

Re: duplicate directories in hdfs location

duplicate directories in hdfs location

Re: Yarn HA does not work (both Resource manager s...

Re: In sqoop import how mapreduce works

In sqoop import how mapreduce works