Member since
09-26-2023
20
Posts
3
Kudos Received
0
Solutions
06-07-2024
02:53 AM
1 Kudo
Hi @ggangadharan , Can we export the table data into multiple files via some command like numfile or anything ?
... View more
06-03-2024
01:45 AM
Hello @ggangadharan , Can you please help me with the size per file ? or some setting where the size is mentioned ?
... View more
06-03-2024
01:44 AM
Command to dump the command : export table test to '/jayesh/testdump1';
... View more
05-29-2024
07:15 AM
i am trying to check the files that are created under data folder when we export table to hdfs location. I have single column table with 5000 rows but after export it created only one data file named as 000000_0 I have another table with partition, 2 partitions with 95 rows each approx. and for this table export is see 2 folders each for partition and then under it only one file. Can you please help me to know how are the data files for the exported table are created and how is it distributed? I mean is it only one data file for each table without partitioned? or it will create multiple files based on specific row count or size count or on what criteria it is split?
... View more
Labels:
- Labels:
-
Apache Hive
05-07-2024
03:03 AM
1 Kudo
Hello @smruti , Thanks for the help.
... View more
05-02-2024
05:03 AM
1 Kudo
Hello Team, I am unable to run the import command for Hive, since it is failing with error invalid path. I copied the table content to another folder and it has permission for hive user, but while running import command it says invalid path. i ran import command on the path where the source table location still it fails with same reason. Beeline version 3.1.3000.7.1.8.28-1 by Apache Hive [root@machine1 Log_Files]# sudo -u hive hadoop fs -ls /warehouse/tablespace/managed/hive/demo.db/test1/* Found 1 items -rw-r--r-- 3 hive hive 621 2024-05-02 16:05 /warehouse/tablespace/managed/hive/demo.db/test1/delta_0000004_0000004_0000/bucket_00000_0 [root@machine1 Log_Files]# sudo -u hive hadoop fs -ls /jay123/restore_jobid123243/jayesh/* Found 1 items drwxr-xr-x - hive hive 0 2024-05-02 16:05 /jay123/restore_jobid123243/jayesh/test1/delta_0000004_0000004_0000 [root@machine1 Log_Files]# sudo -E -u hive beeline -u "jdbc:hive2://machine1.testride.com:2181,machine22.testride.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;;principal=hive/_HOST@hdpcluster.test;" -n hive --showHeader=false --outputformat=tsv2 -e "use demo; show create table test1;" CREATE TABLE `test1`( `nm` int) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://machine1.testride.com:8020/warehouse/tablespace/managed/hive/demo.db/test1' TBLPROPERTIES ( 'bucketing_version'='2', 'last_modified_by'='hive', 'last_modified_time'='1713262492', 'transactional'='true', 'transactional_properties'='default', 'transient_lastDdlTime'='1714646145') 17 rows selected (0.329 seconds) 0: jdbc:hive2://machine1.testride.com:2181,machine2.t> use jayesh;import table test1 from '/jay123/restore_jobid123243/jayesh/test1'; INFO : Compiling command(queryId=hive_20240502173023_4e2433fa-dc38-4357-a5be-90892c65a217): use jayesh INFO : Semantic Analysis Completed (retrial = false) INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20240502173023_4e2433fa-dc38-4357-a5be-90892c65a217); Time taken: 0.007 seconds INFO : Executing command(queryId=hive_20240502173023_4e2433fa-dc38-4357-a5be-90892c65a217): use jayesh INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20240502173023_4e2433fa-dc38-4357-a5be-90892c65a217); Time taken: 0.014 seconds INFO : OK No rows affected (0.13 seconds) Error: Error while compiling statement: FAILED: SemanticException [Error 10027]: Invalid path (state=42000,code=10027) 0: jdbc:hive2://machine1.testride.com:2181,machine2.t> use jayesh;import table test1 from '/warehouse/tablespace/managed/hive/demo.db/test1'; INFO : Compiling command(queryId=hive_20240502173047_d1f036dc-8722-4fc2-be00-1bafd032971e): use jayesh INFO : Semantic Analysis Completed (retrial = false) INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20240502173047_d1f036dc-8722-4fc2-be00-1bafd032971e); Time taken: 0.006 seconds INFO : Executing command(queryId=hive_20240502173047_d1f036dc-8722-4fc2-be00-1bafd032971e): use jayesh INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20240502173047_d1f036dc-8722-4fc2-be00-1bafd032971e); Time taken: 0.006 seconds INFO : OK No rows affected (0.027 seconds) Error: Error while compiling statement: FAILED: SemanticException [Error 10027]: Invalid path (state=42000,code=10027) Can you please help me here to understand what's the issue and how can i resolve this.
... View more
Labels:
- Labels:
-
Apache Hive
12-26-2023
10:49 PM
Hello @smruti , Please let me know if any way where we can have call or meet to review the script ? I will in mean time will try to remove the -n options and check if that works.
... View more
12-25-2023
11:13 PM
Hi @smruti , Yes the exit code is 4. I tried to redirect the stdout and stderr to the file but there was no file created. Following is output of ps -eaf : loki 3936766 3935975 0 02:06 ? 00:00:00 bash /opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/bin/../lib/hive/bin/beeline -u 'jdbc:hive2://machine.domain.com:2181/default;password=**;principal=hive/_HOST@DEVEMC.COMMVAULT.COM;serviceDiscoveryMode=zooKeeper;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;trustStorePassword=***;user=loki;zooKeeperNamespace=hiveserver2' -n loki -n /tmp/cmdcvhive.3935975.140226804802880 --showHeader=false --outputformat=tsv2 --verbose=true -e 'use underscoretest; show tables;' >> /tmp/file 2>&1 loki 3936766 3935975 2 02:06 ? 00:00:00 bash /opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/bin/../lib/hive/bin/beeline -u 'jdbc:hive2://machine.domain.com:2181/default;password=**;principal=hive/_HOST@DEVEMC.COMMVAULT.COM;serviceDiscoveryMode=zooKeeper;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks;trustStorePassword=**;user=loki;zooKeeperNamespace=hiveserver2' -n loki -n /tmp/cmdcvhive.3935975.140226804802880 --showHeader=false --outputformat=tsv2 --verbose=true -e 'use underscoretest; show tables;' >> /tmp/file 2>&1
... View more
12-22-2023
03:19 AM
Hi @smruti , The setup has kerberos authentication. There is no error message the moment the process is launched with loki user, the process exits with error 4. Sorry but the HiveUser and loki user is same its just replacing the username went wrong. The same command when i do su loki and then run the same command it works. or if the same command is ran using sudo it works. Issue is with launching it with other process.
... View more
12-22-2023
03:16 AM
Hi @cloude , Which logs should i share cause the moment the beeline command is launched in another process it fails. so i dont have a way to check why is it failing. it just says process exited with error 4. Can you please help here to know which logs file might have those error ?
... View more