Hello All! I'm working as the snapshot directory owner, but hdfs dfs -deleteSnapshot /path/.snapshot s201605-17-115857.294 returns with deleteSnapshot: Modification on a read-only snapshot is disallowed Trying to change the permissions on the .snapshot directory also returns Modification...disallowed. What can I do? ... View more

Hi all, Odd question - I'm just starting out in Hadoop and am in the process of moving all my test work into production, however I get a strange message on the prod system when working in Hive: "number of reduce tasks is set to 0 since there's no reduce operator". The queries are not failing (yet...?), and there are no strange records in any logs I have looked at. I don't know how to troubleshoot this if indeed it is a problem at all. Any advice? Example: hive> select * from myTable where daily_date='2015-12-29' limit 10; Query ID = root_20160519113838_73d2b4dc-efb8-4ea6-b0a4-cdc4dc64c33a Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_1418226366907_2316, Tracking URL = http://hadoop-head01:8088/proxy/application_1418226366907_2316/ Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1418226366907_2316 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0 2016-05-19 11:39:07,038 Stage-1 map = 0%, reduce = 0% 2016-05-19 11:39:12,653 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.65 sec MapReduce Total cumulative CPU time: 2 seconds 650 msec Ended Job = job_1418226366907_2316 MapReduce Jobs Launched: Job 0: Map: 1 Cumulative CPU: 2.65 sec HDFS Read: 64722 HDFS Write: 831 SUCCESS Total MapReduce CPU Time Spent: 2 seconds 650 msec OK [..... records ....] Time taken: 15.876 seconds, Fetched: 10 row(s) Hadoop version- 2.4.0.2.1.3.0-563 Hive version...(?) 0.13.0.2.1.3.0-563 ... View more

Hello Hadoopers! I'm testing some stuff in the Sandbox, and I've come up against this problem. The regexp below works fine when run in the Hive view GUI in Ambari, but when run from a Hive shell in the command line, it fails with error: "....cannot recognize input near ',' ')' 'as' in expression specification" in the regexp line. INSERT OVERWRITE TABLE my_table SELECT trim(daily_date), trim(name), cast(trim(count) AS INT), regexp_replace(substr(from_unixtime(unix_timestamp(trim(daily_date), 'dd/MM/yyyy')),0,7),"-","") as month FROM staging_table ; Have tried combinations of backticks and single quote marks within the expression - nothing works. But the problem is why Ambari/Hive view would process without error (and with correct output!) but command line Hive throws up this error. Suspect it's to do with the odd characters in the expression, and how the shell treats them combined with the Hive environment... But I thought that all commands run within the Hive shell would be treated as Hive-QL and therefore wouldn't have the escaping issues that the command line does... Puzzled, and completely stuck. Any help troubleshooting this would be very gratefully received! ... View more

Hadoop experts! Say I have a table with: daily_date STRING fact1 STRING fact2 STRING fact_n STRING where daily_date is of format dd/MM/yyyy (oh horror!). I want to partition the data by date (either yyyyMMdd, or just yyyyMM), but the current date format is no good. How do I create a table (or multiple staging tables) to deal with this? At present my ingest script will load local data into an hdfs location, and into an un-partitioned hive staging table containing the daily_date column and the fact columns. From there I do: INSERT OVERWRITE TABLE {final table} PARTITION (daily_date=${data_date}) SELECT fact1, fact_n FROM {staging table} WHERE daily_date={$data_date} ; and where ${data_date} is a variable defined as the daily_date in the incoming data (which only comes in one day at a time). But I can't get the partitioning bit to work because of the dodgy daily_date format. I wrote this to convert dd/MM/yyyy to yyyyMM, but don't know where to use it. regexp_replace(substring(from_unixtime(unix_timestamp(daily_date, 'dd/MM/yyyy')),0,7),"-","") I'm a bit stuck with this, any help would be very gratefully received. (next thing I'll try is to load data into raw un-partitioned table, select into a new table with a new column nice_date, then select into a third table which is partitioned on nice_date). ... View more