Hive -e throws chmod error

[#1]

Using Hive 1.2.1000.2.6.4.0-91.

When running a 'hive -e' from command line, I get a message about -chmod and hadoop fs. See below.

Any idea why I get this, and how to fix it? The -S doesn't suppress it either, as shown.

[ ~] hive -S -e 'select max(entry_timestamp) from xx.yy'
log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.
-chmod: chmod : mode '0' does not match the expected pattern.
Usage: hadoop fs [generic options]
        [-appendToFile <localsrc> ... <dst>]
        [-cat [-ignoreCrc] <src> ...]
        [-checksum <src> ...]
        [-chgrp [-R] GROUP PATH...]
        [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
        [-chown [-R] [OWNER][:[GROUP]] PATH...]
        [-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>]
        [-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
        [-count [-q] [-h] [-v] [-t [<storage type>]] [-u] <path> ...]
        [-cp [-f] [-p | -p[topax]] <src> ... <dst>]
        [-createSnapshot <snapshotDir> [<snapshotName>]]
        [-deleteSnapshot <snapshotDir> <snapshotName>]
        [-df [-h] [<path> ...]]
        [-du [-s] [-h] <path> ...]
        [-expunge]
        [-find <path> ... <expression> ...]
        [-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
        [-getfacl [-R] <path>]
        [-getfattr [-R] {-n name | -d} [-e en] <path>]
        [-getmerge [-nl] <src> <localdst>]
        [-help [cmd ...]]
        [-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [<path> ...]]
        [-mkdir [-p] <path> ...]
        [-moveFromLocal <localsrc> ... <dst>]
        [-moveToLocal <src> <localdst>]
        [-mv <src> ... <dst>]
        [-put [-f] [-p] [-l] <localsrc> ... <dst>]
        [-renameSnapshot <snapshotDir> <oldName> <newName>]
        [-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ...]
        [-rmdir [--ignore-fail-on-non-empty] <dir> ...]
        [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
        [-setfattr {-n name [-v value] | -x name} <path>]
        [-setrep [-R] [-w] <rep> <path> ...]
        [-stat [format] <path> ...]
        [-tail [-f] <file>]
        [-test -[defsz] <path>]
        [-text [-ignoreCrc] <src> ...]
        [-touchz <path> ...]
        [-truncate [-w] <length> <path> ...]
        [-usage [cmd ...]]


Generic options supported are
-conf <configuration file>     specify an application configuration file
-D <property=value>            use value for given property
-fs <local|namenode:port>      specify a namenode
-jt <local|resourcemanager:port>    specify a ResourceManager
-files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.


The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]


Usage: hadoop fs [generic options] -chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...
[ ~] 2018-08-31 12:42:14.41040

[#2]

I have the same trouble under Hive with an "analyze table <db_name>.<table_name> compute statistics ;" :

analyze table <db_name>.<table_name> compute statistics ;
-chmod: chmod : mode '0' does not match the expected pattern.
Usage: hadoop fs [generic options]
[-appendToFile <localsrc> ... <dst>]

For this moment, I have no solution. But I know that you can run a "hdfs dfs...." command from within the Hive CLI.

For this, drop the hdfs word from the command and add the semicolon at the end :

hive> dfs -ls / ;
Found 16 items
...

drwxrwxrwx - hdfs hdfs 0 2019-03-08 09:40 /tmp
drwxrwxrwx - hdfs hdfs 0 2019-03-05 11:13 /user

So I think there is a bug or a trouble with Hive CLI, and your Hive command is probably parsed and interpreted as a "hdfs dfs" command.

[#3]

I found the trouble.

At first, run Hive CLI in debug mode, like this :

hive --hiveconf hive.execution.engine=tez --hiveconf hive.root.logger=DEBUG,console

Run your Hive command and you will get a ton of logs !

For my part, because I want to analyze a table, there are implicit commands which are running.

The first implicit command creates a staging directory :

19/03/14 13:44:50 [main]: DEBUG ql.Context: Created staging dir = hdfs://<hostname>:8020/<myDirectories>/.hive-staging_hive_2019-03-14_13-44-49_446_4228566055524252294-1 for path = hdfs://<hostname>:8020/<myDirectories>

The second implicit command changes the group :

19/03/14 13:44:50 [main]: DEBUG shims.HadoopShimsSecure: {-chgrp,-R,hdfs,hdfs://<hostname>:8020/<myDirectories>/.hive-staging_hive_2019-03-14_13-44-49_446_4228566055524252294-1}

The third implicit command wants to change the HDFS permissions :

19/03/14 13:44:50 [main]: DEBUG shims.HadoopShimsSecure: {-chmod,-R,0,hdfs://<hostname>:8020/<myDirectories>/.hive-staging_hive_2019-03-14_13-44-49_446_4228566055524252294-1}
-chmod: chmod : mode '0' does not match the expected pattern.
Usage: hadoop fs [generic options]

As you can understand, the mode is false. This is not 0, but 000.

hdfs dfs -chmod 0 <path> is wrong

hdfs dfs -chmod 000 <path> is correct

We use the HDP 2.6.1 distribution, including Hive 1.2.1, and this is a bug. But I don't know how to solve it !

[#4]

@gilles_rouard How do you confirm this as a bug? Has this been reported to Cloudera or Hortonworks (since the version is 2.6.1). If so, could you include the link to the bug?

Also, chmod 0 or chmod 000 makes only root (or in Hadoop the hdfs user) access the file and no one else can. I don't think this is practical. I fail to understand why anyone would do this.