About jayes

jayes · ‎06-07-2024

Hi @ggangadharan , Can we export the table data into multiple files via some command like numfile or anything ?

jayes · ‎06-03-2024

Hello @ggangadharan , Can you please help me with the size per file ? or some setting where the size is mentioned ?

jayes · ‎06-03-2024

Command to dump the command : export table test to '/jayesh/testdump1';

jayes · ‎05-29-2024

i am trying to check the files that are created under data folder when we export table to hdfs location. I have single column table with 5000 rows but after export it created only one data file named as 000000_0 I have another table with partition, 2 partitions with 95 rows each approx. and for this table export is see 2 folders each for partition and then under it only one file. Can you please help me to know how are the data files for the exported table are created and how is it distributed? I mean is it only one data file for each table without partitioned? or it will create multiple files based on specific row count or size count or on what criteria it is split?

jayes · ‎05-07-2024

Hello @smruti , Thanks for the help.

jayes · ‎05-02-2024

Hello Team, I am unable to run the import command for Hive, since it is failing with error invalid path. I copied the table content to another folder and it has permission for hive user, but while running import command it says invalid path. i ran import command on the path where the source table location still it fails with same reason. Beeline version 3.1.3000.7.1.8.28-1 by Apache Hive [root@machine1 Log_Files]# sudo -u hive hadoop fs -ls /warehouse/tablespace/managed/hive/demo.db/test1/* Found 1 items -rw-r--r-- 3 hive hive 621 2024-05-02 16:05 /warehouse/tablespace/managed/hive/demo.db/test1/delta_0000004_0000004_0000/bucket_00000_0 [root@machine1 Log_Files]# sudo -u hive hadoop fs -ls /jay123/restore_jobid123243/jayesh/* Found 1 items drwxr-xr-x - hive hive 0 2024-05-02 16:05 /jay123/restore_jobid123243/jayesh/test1/delta_0000004_0000004_0000 [root@machine1 Log_Files]# sudo -E -u hive beeline -u "jdbc:hive2://machine1.testride.com:2181,machine22.testride.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;;principal=hive/_HOST@hdpcluster.test;" -n hive --showHeader=false --outputformat=tsv2 -e "use demo; show create table test1;" CREATE TABLE `test1`( `nm` int) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://machine1.testride.com:8020/warehouse/tablespace/managed/hive/demo.db/test1' TBLPROPERTIES ( 'bucketing_version'='2', 'last_modified_by'='hive', 'last_modified_time'='1713262492', 'transactional'='true', 'transactional_properties'='default', 'transient_lastDdlTime'='1714646145') 17 rows selected (0.329 seconds) 0: jdbc:hive2://machine1.testride.com:2181,machine2.t> use jayesh;import table test1 from '/jay123/restore_jobid123243/jayesh/test1'; INFO : Compiling command(queryId=hive_20240502173023_4e2433fa-dc38-4357-a5be-90892c65a217): use jayesh INFO : Semantic Analysis Completed (retrial = false) INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20240502173023_4e2433fa-dc38-4357-a5be-90892c65a217); Time taken: 0.007 seconds INFO : Executing command(queryId=hive_20240502173023_4e2433fa-dc38-4357-a5be-90892c65a217): use jayesh INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20240502173023_4e2433fa-dc38-4357-a5be-90892c65a217); Time taken: 0.014 seconds INFO : OK No rows affected (0.13 seconds) Error: Error while compiling statement: FAILED: SemanticException [Error 10027]: Invalid path (state=42000,code=10027) 0: jdbc:hive2://machine1.testride.com:2181,machine2.t> use jayesh;import table test1 from '/warehouse/tablespace/managed/hive/demo.db/test1'; INFO : Compiling command(queryId=hive_20240502173047_d1f036dc-8722-4fc2-be00-1bafd032971e): use jayesh INFO : Semantic Analysis Completed (retrial = false) INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20240502173047_d1f036dc-8722-4fc2-be00-1bafd032971e); Time taken: 0.006 seconds INFO : Executing command(queryId=hive_20240502173047_d1f036dc-8722-4fc2-be00-1bafd032971e): use jayesh INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20240502173047_d1f036dc-8722-4fc2-be00-1bafd032971e); Time taken: 0.006 seconds INFO : OK No rows affected (0.027 seconds) Error: Error while compiling statement: FAILED: SemanticException [Error 10027]: Invalid path (state=42000,code=10027) Can you please help me here to understand what's the issue and how can i resolve this.

jayes · ‎12-26-2023

Hello @smruti , Please let me know if any way where we can have call or meet to review the script ? I will in mean time will try to remove the -n options and check if that works.

jayes · ‎12-25-2023

Hi @smruti , Yes the exit code is 4. I tried to redirect the stdout and stderr to the file but there was no file created. Following is output of ps -eaf : loki 3936766 3935975 0 02:06 ? 00:00:00 bash /opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/bin/../lib/hive/bin/beeline -u 'jdbc:hive2://machine.domain.com:2181/default;password=**;principal=hive/_HOST@DEVEMC.COMMVAULT.COM;serviceDiscoveryMode=zooKeeper;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;trustStorePassword=***;user=loki;zooKeeperNamespace=hiveserver2' -n loki -n /tmp/cmdcvhive.3935975.140226804802880 --showHeader=false --outputformat=tsv2 --verbose=true -e 'use underscoretest; show tables;' >> /tmp/file 2>&1 loki 3936766 3935975 2 02:06 ? 00:00:00 bash /opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/bin/../lib/hive/bin/beeline -u 'jdbc:hive2://machine.domain.com:2181/default;password=**;principal=hive/_HOST@DEVEMC.COMMVAULT.COM;serviceDiscoveryMode=zooKeeper;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;trustStorePassword=**;user=loki;zooKeeperNamespace=hiveserver2' -n loki -n /tmp/cmdcvhive.3935975.140226804802880 --showHeader=false --outputformat=tsv2 --verbose=true -e 'use underscoretest; show tables;' >> /tmp/file 2>&1

jayes · ‎12-22-2023

Hi @smruti , The setup has kerberos authentication. There is no error message the moment the process is launched with loki user, the process exits with error 4. Sorry but the HiveUser and loki user is same its just replacing the username went wrong. The same command when i do su loki and then run the same command it works. or if the same command is ran using sudo it works. Issue is with launching it with other process.

jayes · ‎12-22-2023

Hi @cloude , Which logs should i share cause the moment the beeline command is launched in another process it fails. so i dont have a way to check why is it failing. it just says process exited with error 4. Can you please help here to know which logs file might have those error ?

Online	Offline
Last Visited	‎07-25-2024 01:08 AM

Member Since	‎09-26-2023 06:53 AM
Last Visited	‎07-25-2024 01:08 AM
Posts	20
Kudos received	3

Cloudera Community

Re: Export command creating files under data folde...

Re: Export command creating files under data folde...

Re: Export command creating files under data folde...

Export command creating files under data folder

Re: Not Able to run import command. it fails with ...

Not Able to run import command. it fails with inva...

Re: Beeline command failing with error 4

Re: Beeline command failing with error 4

Re: Beeline command failing with error 4

Re: Beeline command failing with error 4