About drgenious

drgenious · ‎09-30-2021

I have some tables in hive and I want to find the size of each table through the metastore (MySQL). I am trying the below but it returns paratition_params, notification_log, sds etc. My tables are stored in TABLES.TBLS but when I am running the below query with from information_schema.TABLES.TBLS it returns this: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '.TBLS ORDER BY (DATA_LENGTH + INDEX_LENGTH) DESC LIMIT 0, 200' at line 5 SELECT TABLE_NAME AS `Table`, ROUND((DATA_LENGTH + INDEX_LENGTH) / 1024 / 1024) AS `Size (MB)` FROM information_schema.TABLES ORDER BY (DATA_LENGTH + INDEX_LENGTH) DESC;

drgenious · ‎09-29-2021

Hi, I am trying to create a dashboard in CM with impala charts in order to see the maximum memory of impala and the one that my queries are using. I running tsquery for total_impala_admission_controller_local_backend_mem_reserved_across_impala_daemon_pools and total_impala_admission_controller_local_backend_mem_usage_across_impala_daemon_pools But it seems that the charts are representing exactly the same thing. Why is this happens?

drgenious · ‎09-24-2021

I am trying to learn the basics in cloudera and Impala and by reading this documentation : https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_cluster_util_custom.html#concept_jp4_4bh_hx I saw that with tsquery I can monitor Impala. I want to ask : 1) where can I run these kind of queries? 2) where can I find the attributes like category and clusterName in cloudera? Thanks in advance.

drgenious · ‎09-20-2021

How can I see the memory % that is used right now in Impala? Also I want to check the whole memory size

drgenious · ‎08-08-2021

Hi @ChethanYM , 1)I have created in impala. 2)result CREATE TABLE db.table ( ... ) PARTITIONED BY ( ... ) SORT BY ( ... ) STORED AS PARQUET LOCATION 'hdfs://path/' TBLPROPERTIES ('DO_NOT_UPDATE_STATS'='true', 'STATS_GENERATED'='TASK', 'impala.lastComputeStatsTime'='1627655409', 'numRows'='0', 'parquet.compression'='snappy', 'totalSize'='0') 3) I have listed the files and they seem ok 4)I can access it but returns again 0 results 5)It is not failing the query. It can not read the partitions as it did before.

drgenious · ‎08-07-2021

Hi, I have a table in Impala and it had some files in its location in hdfs. All of a sudden it can not read the files. I tried to invalidate and recover the partitions but nothing happened. I also moved the files from this folder into a new's folder path but it couldn't read them too. What can I do ? UPDATE Tried to repair the table in hive set hive.msck.repair.batch.size=1; set hive.msck.path.validation=ignore; msck REPAIR TABLE db.table but received this error : Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

drgenious · ‎08-04-2021

I have a table and when I am trying to query it through imapala it throws me this error: File 'hdfs:/path/table/1_data.0.parq' has an incompatible Parquet schema for column 'db.table.parameter_11'. Column type: STRING, Parquet schema: optional int64 amount [i:10 d:1 r:0] But when I query it through hive it returns results correctly. How can I fix it?

drgenious · ‎07-15-2021

I used to have some tables in HIVE but after an incident HIVE went down and some new tables erased and old tables went back to a previous schema state. What I want, is to ask if it is there any option to backup tables' schemas into MySQL which stores HIVE metastore in order to retrieve them if there occurred any incident again.

drgenious · ‎07-14-2021

I had a table that it vanished after an issue. I created again the table and then recover the partitions in order to read again the files that were in hdfs hive direcotry. Although all of a sudden now, it throws me the below error: File 'path/data.0.parq' has an incompatible Parquet schema for column 'db.table.parameter_11'. Column type: STRING, Parquet schema: optional int64 amount [i:10 d:1 r:0]

drgenious · ‎07-08-2021

I have a pyspark job with these configs: self.spark = SparkSession.builder.appName("example") \ .config("hive.exec.dynamic.partition", "true") \ .config("hive.exec.dynamic.partition.mode", "nonstrict") \ .config("hive.exec.max.dynamic.partitions", "5000000") \ .config("hive.exec.max.dynamic.partitions.pernode", "1000000") \ .enableHiveSupport() \ .getOrCreate() I can not find anywhere how to set a configuration to increase the max row size to 150mb. I found the command only in impala. Can you please help me ? Thanks in advance.

Online	Offline
Last Visited	‎06-26-2023 10:46 AM

Member Since	‎01-07-2020 06:44 AM
Last Visited	‎06-26-2023 10:46 AM
Posts	64
Kudos received	1

Cloudera Community

Find table's size in Hive metastore (MySQL)

What is the difference between mem_reserved and me...

Where can I run tsquery ?

Find the % of the used memory in impala

Re: Impala can not read the files stored in the hi...

Impala can not read the files stored in the hive d...

How to fix Parquet schema: optional int64 amount [...

Backup MySQL tables schemas used in hive

Parquet schema error

Increase max row size in HIVE