Created 09-04-2018 03:18 PM
Using Hive 1.2.1000.2.6.4.0-91.
When running a 'hive -e' from command line, I get a message about -chmod and hadoop fs. See below.
Any idea why I get this, and how to fix it? The -S doesn't suppress it either, as shown.
[ ~] hive -S -e 'select max(entry_timestamp) from xx.yy' log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender. -chmod: chmod : mode '0' does not match the expected pattern. Usage: hadoop fs [generic options] [-appendToFile <localsrc> ... <dst>] [-cat [-ignoreCrc] <src> ...] [-checksum <src> ...] [-chgrp [-R] GROUP PATH...] [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...] [-chown [-R] [OWNER][:[GROUP]] PATH...] [-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>] [-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] [-count [-q] [-h] [-v] [-t [<storage type>]] [-u] <path> ...] [-cp [-f] [-p | -p[topax]] <src> ... <dst>] [-createSnapshot <snapshotDir> [<snapshotName>]] [-deleteSnapshot <snapshotDir> <snapshotName>] [-df [-h] [<path> ...]] [-du [-s] [-h] <path> ...] [-expunge] [-find <path> ... <expression> ...] [-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] [-getfacl [-R] <path>] [-getfattr [-R] {-n name | -d} [-e en] <path>] [-getmerge [-nl] <src> <localdst>] [-help [cmd ...]] [-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [<path> ...]] [-mkdir [-p] <path> ...] [-moveFromLocal <localsrc> ... <dst>] [-moveToLocal <src> <localdst>] [-mv <src> ... <dst>] [-put [-f] [-p] [-l] <localsrc> ... <dst>] [-renameSnapshot <snapshotDir> <oldName> <newName>] [-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ...] [-rmdir [--ignore-fail-on-non-empty] <dir> ...] [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]] [-setfattr {-n name [-v value] | -x name} <path>] [-setrep [-R] [-w] <rep> <path> ...] [-stat [format] <path> ...] [-tail [-f] <file>] [-test -[defsz] <path>] [-text [-ignoreCrc] <src> ...] [-touchz <path> ...] [-truncate [-w] <length> <path> ...] [-usage [cmd ...]] Generic options supported are -conf <configuration file> specify an application configuration file -D <property=value> use value for given property -fs <local|namenode:port> specify a namenode -jt <local|resourcemanager:port> specify a ResourceManager -files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath. -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines. The general command line syntax is bin/hadoop command [genericOptions] [commandOptions] Usage: hadoop fs [generic options] -chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH... [ ~] 2018-08-31 12:42:14.41040
Created 03-14-2019 04:28 PM
I have the same trouble under Hive with an "analyze table <db_name>.<table_name> compute statistics ;" :
analyze table <db_name>.<table_name> compute statistics ;
-chmod: chmod : mode '0' does not match the expected pattern.
Usage: hadoop fs [generic options]
[-appendToFile <localsrc> ... <dst>]
For this moment, I have no solution. But I know that you can run a "hdfs dfs...." command from within the Hive CLI.
For this, drop the hdfs word from the command and add the semicolon at the end :
hive> dfs -ls / ;
Found 16 items
...
drwxrwxrwx - hdfs hdfs 0 2019-03-08 09:40 /tmp
drwxrwxrwx - hdfs hdfs 0 2019-03-05 11:13 /user
So I think there is a bug or a trouble with Hive CLI, and your Hive command is probably parsed and interpreted as a "hdfs dfs" command.
Created 03-14-2019 04:28 PM
I found the trouble.
At first, run Hive CLI in debug mode, like this :
hive --hiveconf hive.execution.engine=tez --hiveconf hive.root.logger=DEBUG,console
Run your Hive command and you will get a ton of logs !
For my part, because I want to analyze a table, there are implicit commands which are running.
The first implicit command creates a staging directory :
19/03/14 13:44:50 [main]: DEBUG ql.Context: Created staging dir = hdfs://<hostname>:8020/<myDirectories>/.hive-staging_hive_2019-03-14_13-44-49_446_4228566055524252294-1 for path = hdfs://<hostname>:8020/<myDirectories>
The second implicit command changes the group :
19/03/14 13:44:50 [main]: DEBUG shims.HadoopShimsSecure: {-chgrp,-R,hdfs,hdfs://<hostname>:8020/<myDirectories>/.hive-staging_hive_2019-03-14_13-44-49_446_4228566055524252294-1}
The third implicit command wants to change the HDFS permissions :
19/03/14 13:44:50 [main]: DEBUG shims.HadoopShimsSecure: {-chmod,-R,0,hdfs://<hostname>:8020/<myDirectories>/.hive-staging_hive_2019-03-14_13-44-49_446_4228566055524252294-1}
-chmod: chmod : mode '0' does not match the expected pattern.
Usage: hadoop fs [generic options]
As you can understand, the mode is false. This is not 0, but 000.
hdfs dfs -chmod 0 <path> is wrong
hdfs dfs -chmod 000 <path> is correct
We use the HDP 2.6.1 distribution, including Hive 1.2.1, and this is a bug. But I don't know how to solve it !
Created 09-27-2019 02:17 PM
@gilles_rouard How do you confirm this as a bug? Has this been reported to Cloudera or Hortonworks (since the version is 2.6.1). If so, could you include the link to the bug?
Also, chmod 0 or chmod 000 makes only root (or in Hadoop the hdfs user) access the file and no one else can. I don't think this is practical. I fail to understand why anyone would do this.