Member since
06-24-2019
12
Posts
0
Kudos Received
0
Solutions
08-01-2019
12:26 PM
Hello, 1) Can we choose Hive Import/Export for ACID table within the same cluster in Hive 1.2. version? 2) Does distcp is supported for ACID table within the same cluster ? I know this is not supported for cross cluster. Thanks
... View more
Labels:
08-01-2019
04:57 AM
Hello, I'm trying to export hive table and the export is skipping the buckets in 2019-01-03 partition for 2 tables - 'users_info_af_a' and 'users_sysinfo_af_a' Command we used to do this: export table users_sysinfo_af_a to 'hdfs://clustername/user/test/exp/audit.users_sysinfo_af_a'; Hive log captures below error: 2019-07-31 12:44:51,462 INFO [partition-dump-thread-99]: Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1194)) - io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb 2019-07-31 12:44:51,462 INFO [partition-dump-thread-99]: Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1194)) - io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor 2019-07-31 12:44:51,527 ERROR [partition-dump-thread-99]: tools.DistCp (DistCp.java:run(133)) - Duplicate files in input path: org.apache.hadoop.tools.CopyListing$DuplicateFileException: File hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00000 and hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00000 would cause duplicates. Aborting at org.apache.hadoop.tools.CopyListing.validateFinalListing(CopyListing.java:165) at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:93) at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90) at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86) The hdfs file exists and i can access as listed below: hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00000 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00000 hive@hostname:~> hdfs dfs -ls hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00000 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00000 hive@hostname:~> hive@hostname:~> hdfs dfs -ls hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00000 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00000 hive@hostname:~> hive@hostname:~> hdfs dfs -ls hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/ Found 25 items -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00000 -rw-r--r-- 3 nifi hadoop 20675229 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00001 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00002 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00003 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00004 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00005 -rw-r--r-- 3 nifi hadoop 9160 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00006 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00007 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00008 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00009 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00010 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00011 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00012 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00013 -rw-r--r-- 3 nifi hadoop 13076 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00014 -rw-r--r-- 3 nifi hadoop 15306 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00015 -rw-r--r-- 3 nifi hadoop 18487 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00016 -rw-r--r-- 3 nifi hadoop 18891 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00017 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00018 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00019 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00020 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00021 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00022 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00023 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00024 hive@hostname:~> hive@hostname:~> hdfs dfs -ls hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/ Found 25 items -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00000 -rw-r--r-- 3 nifi hadoop 20615657 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00001 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00002 -rw-r--r-- 3 nifi hadoop 23901 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00003 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00004 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00005 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00006 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00007 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00008 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00009 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00010 -rw-r--r-- 3 nifi hadoop 12900 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00011 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00012 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00013 -rw-r--r-- 3 nifi hadoop 64126 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00014 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00015 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00016 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00017 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00018 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00019 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00020 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00021 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00022 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00023 -rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00024 hive@hostname:~> hive@hostname:~> hdfs dfs -ls hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/ Found 5 items -rw-r--r-- 3 nifi hadoop 4 2019-01-05 00:42 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/_orc_acid_version drwxrwxrwx - nifi hadoop 0 2019-01-05 00:44 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0327737_0327836 drwxrwxrwx - nifi hadoop 0 2019-01-05 00:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0327837_0327936 drwxrwxrwx - nifi hadoop 0 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636 drwxrwxrwx - nifi hadoop 0 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736 hive@hostname:~>
... View more
- Tags:
- Hadoop Core
- HDFS
- Hive
Labels:
07-05-2019
05:32 AM
Thanks @Geoffrey Shelton Okot ! This helped
... View more
07-01-2019
01:13 PM
Hi @Jay Kumar SenSharma Thanks for your reply and helping me ! Can you please have a look of below upgrade sequence and let me know your feedback: I'm planning to implement in the following sequence: 1) Upgrade HDF Mpack using the command: ambari-server upgrade-mpack --mpack=/root/hdf-ambari-mpack-3.1.0.0-564.tar.gz --verbose Restart ambari-server 2) Upgrade HDF to 3.1.0 from 3.0 Note: I will not be upgrading Ambari as of now since currently installed ambari version is 2.6.1 and it is compatible with HDF 3.1.0 as per matrix 3) Finally, i will upgrade Ambari in future if required to 2.7.3 from 2.6.1 Also, how can i verify the installed HDF Mpack version ? Please help ! Regards, Chethan
... View more
06-29-2019
02:16 PM
Hello All, Where can i download the HDF 3.1.0 version definition file (VDF) for SLES12 operating system ? VDF file for HDF 3.1. for SLES12 is not available in HDF 3.1 release notes https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.0/bk_release-notes/content/ch_hdf_relnotes.html My Goal is to upgrade the HDF version to 3.1.0 from 3..0.2. Ambari version we use is 2.6.1 Basically i have two separate clusters - HDP and HDF. Each clusters is managed by Ambari (one Ambari to manage HDF and another Ambari to manage HDP) Steps to reproduce the issue: In the Versions tab in Ambari of HDF cluster, i clicked on "Register version". Here i dont see 3.1.X.X to input target version In the drop down > Select "Add Version", I will get a popup to upload VDF file Appreciate your help to get me the VDF file for HDF 3.1. for SLES12 Thanks Chethan
... View more
Labels:
06-29-2019
02:16 PM
Hello, Where can i get or download the Version Definition File (VDF) of HDF 3.1.0 for SLES12 O.S ? Background: My Goal is to upgrade the HDF version to 3.1.0 from 3..0.2. Basically i have two separate clusters - HDP and HDF. Each clusters is managed by Ambari (one Ambari to manage HDF and another Ambari to manage HDP) Steps to reproduce the issue: In the Versions tab in Ambari of HDF cluster, i clicked on "Register version" In the drop down > Select "Add Version" I will get a popup to upload VDF file as show below: VDF file for HDF 3.1. for SLES12 is not available in HDF 3.1 release notes https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.0/bk_release-notes/content/ch_hdf_relnotes.html Appreciate help to get me the VDF file for HDF 3.1. for SLES12 Thanks Chethan
... View more
06-29-2019
02:16 PM
Hi @Michael Dennis "MD" Uanang I have similar issue where i can't find VDF file for HDF 3.1.0 version for SLES12 Operating system Can you please help me to find VDF file ? My Goal is to upgrade the HDF version to 3.1.0 from 3..0.2. Basically i have two separate clusters - HDP and HDF. Each clusters is managed by Ambari (one Ambari to manage HDF and another Ambari to manage HDP) Steps to reproduce the issue: In the Versions tab in Ambari of HDF cluster, i clicked on "Register version" In the drop down > Select "Add Version" I will get a popup to upload VDF file as show below: VDF file for HDF 3.1. for SLES12 is not available in HDF 3.1 release notes https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.0/bk_release-notes/content/ch_hdf_relnotes.html Appreciate help to get me the VDF file for HDF 3.1. for SLES12 Thanks Chethan
... View more
06-27-2019
05:11 AM
Thanks for your reply To arrive HDF 3.4.0, we also have to upgrade our Ambari as well. Basically we have two separate HDF and HDP clusters. Currently we are planning to upgrade only HDF cluster. We will not touch HDP cluster Current HDF version = 3.0.2 Current Ambari version = 2.6.1 So, my question is, what should be my upgrade sequence since I need to upgrade Ambari as well along with HDF ? Note, HDF version 3.1.0 is NOT compatible with Ambari version 2.7.3. Can you recommend which of the below sequence will be good to follow ? Sequence#1: Step 1) Upgrade first HDF to 3.1.0 from 3.0.2 This is because HDF 3.1.0 is compatible with Ambari 2.6.1 At the end of this upgrade you will have HDF version 3.1.0 and Ambari version 2.6.1 The first step in upgrading the HDF 3.1 is to upgrade to Ambari 2.6.1. Step 2) Upgrade Ambari alone to 2.7.3 from 2.6.1 At the end of this upgrade you will have HDF version 3.1.0 and Ambari version 2.7.3 Ref - https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.0/ambari-managed-hdf-upgrade/content/hdf-upgrade-paths.html Note: HDF version 3.1.0 is NOT compatible with Ambari version 2.7.3 Step 3) Upgrade “HDF” component to 3.3.0 from 3.1.0 At the end of this upgrade you will have HDF version 3.3.0 and Ambari version 2.7.3 HDF version 3.3.0 is compatible with Ambari version 2.7.3 Step 4) Upgrade “HDF” component to 3.4.0 from 3.3.0 At the end of this upgrade you will have HDF version 3.4.0 and Ambari version 2.7.3 HDF version 3.4.0 is compatible with Ambari version 2.7.3 OR Sequence#2: Step 1) Upgrade Ambari alone to 2.7.3 from 2.6.1 At the end of this upgrade you will have HDF version 3.0.2 and Ambari version 2.7.3 Note: HDF version 3.0.2 is NOT compatible with Ambari version 2.7.3 Step 2) Upgrade HDF to 3.1.0 from 3.0.2 At the end of this upgrade you will have HDF version 3.1.0 and Ambari version 2.7.3 Note: HDF version 3.1.0 is NOT compatible with Ambari version 2.7.3 Step 3) Upgrade “HDF” component to 3.3.0 from 3.1.0 At the end of this upgrade you will have HDF version 3.3.0 and Ambari version 2.7.3 HDF version 3.3.0 is compatible with Ambari version 2.7.3 Step 4) Upgrade “HDF” component to 3.4.0 from 3.3.0 At the end of this upgrade you will have HDF version 3.4.0 and Ambari version 2.7.3 HDF version 3.4.0 is compatible with Ambari version 2.7.3 Thanks Chethan
... View more
06-27-2019
05:11 AM
Thanks for your reply To arrive HDF 3.4.0, we also have to upgrade our Ambari as well. Basically we have two separate HDF and HDP clusters. Currently we are planning to upgrade only HDF cluster. We will not touch HDP cluster Current HDF version = 3.0.2 Current Ambari version = 2.6.1 So, my question is, what should be my upgrade sequence since I need to upgrade Ambari as well along with HDF ? Note, HDF version 3.1.0 is NOT compatible with Ambari version 2.7.3. Can you recommend which of the below sequence will be good to follow ? Sequence#1: Step 1) Upgrade first HDF to 3.1.0 from 3.0.2 This is because HDF 3.1.0 is compatible with Ambari 2.6.1 At the end of this upgrade you will have HDF version 3.1.0 and Ambari version 2.6.1 The first step in upgrading the HDF 3.1 is to upgrade to Ambari 2.6.1. Step 2) Upgrade Ambari alone to 2.7.3 from 2.6.1 At the end of this upgrade you will have HDF version 3.1.0 and Ambari version 2.7.3 Ref - https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.0/ambari-managed-hdf-upgrade/content/hdf-upgrade-paths.html Note: HDF version 3.1.0 is NOT compatible with Ambari version 2.7.3 Step 3) Upgrade “HDF” component to 3.3.0 from 3.1.0 At the end of this upgrade you will have HDF version 3.3.0 and Ambari version 2.7.3 HDF version 3.3.0 is compatible with Ambari version 2.7.3 Step 4) Upgrade “HDF” component to 3.4.0 from 3.3.0 At the end of this upgrade you will have HDF version 3.4.0 and Ambari version 2.7.3 HDF version 3.4.0 is compatible with Ambari version 2.7.3 OR Sequence#2: Step 1) Upgrade Ambari alone to 2.7.3 from 2.6.1 At the end of this upgrade you will have HDF version 3.0.2 and Ambari version 2.7.3 Note: HDF version 3.0.2 is NOT compatible with Ambari version 2.7.3 Step 2) Upgrade HDF to 3.1.0 from 3.0.2 At the end of this upgrade you will have HDF version 3.1.0 and Ambari version 2.7.3 Note: HDF version 3.1.0 is NOT compatible with Ambari version 2.7.3 Step 3) Upgrade “HDF” component to 3.3.0 from 3.1.0 At the end of this upgrade you will have HDF version 3.3.0 and Ambari version 2.7.3 HDF version 3.3.0 is compatible with Ambari version 2.7.3 Step 4) Upgrade “HDF” component to 3.4.0 from 3.3.0 At the end of this upgrade you will have HDF version 3.4.0 and Ambari version 2.7.3 HDF version 3.4.0 is compatible with Ambari version 2.7.3 Thanks Chethan
... View more
06-27-2019
05:11 AM
Thanks for your reply To arrive HDF 3.4.0, we also have to upgrade our Ambari as well. Basically we have two separate HDF and HDP clusters. Currently we are planning to upgrade only HDF cluster. We will not touch HDP cluster Current HDF version = 3.0.2 Current Ambari version = 2.6.1 So, my question is, what should be my upgrade sequence since I need to upgrade Ambari as well along with HDF ? Note, HDF version 3.1.0 is NOT compatible with Ambari version 2.7.3. Can you recommend which of the below sequence will be good to follow ? Sequence#1: Step 1) Upgrade first HDF to 3.1.0 from 3.0.2 This is because HDF 3.1.0 is compatible with Ambari 2.6.1 At the end of this upgrade you will have HDF version 3.1.0 and Ambari version 2.6.1 The first step in upgrading the HDF 3.1 is to upgrade to Ambari 2.6.1. Step 2) Upgrade Ambari alone to 2.7.3 from 2.6.1 At the end of this upgrade you will have HDF version 3.1.0 and Ambari version 2.7.3 Ref - https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.0/ambari-managed-hdf-upgrade/content/hdf-upgrade-paths.html Note: HDF version 3.1.0 is NOT compatible with Ambari version 2.7.3 Step 3) Upgrade “HDF” component to 3.3.0 from 3.1.0 At the end of this upgrade you will have HDF version 3.3.0 and Ambari version 2.7.3 HDF version 3.3.0 is compatible with Ambari version 2.7.3 Step 4) Upgrade “HDF” component to 3.4.0 from 3.3.0 At the end of this upgrade you will have HDF version 3.4.0 and Ambari version 2.7.3 HDF version 3.4.0 is compatible with Ambari version 2.7.3 OR Sequence#2: Step 1) Upgrade Ambari alone to 2.7.3 from 2.6.1 At the end of this upgrade you will have HDF version 3.0.2 and Ambari version 2.7.3 Note: HDF version 3.0.2 is NOT compatible with Ambari version 2.7.3 Step 2) Upgrade HDF to 3.1.0 from 3.0.2 At the end of this upgrade you will have HDF version 3.1.0 and Ambari version 2.7.3 Note: HDF version 3.1.0 is NOT compatible with Ambari version 2.7.3 Step 3) Upgrade “HDF” component to 3.3.0 from 3.1.0 At the end of this upgrade you will have HDF version 3.3.0 and Ambari version 2.7.3 HDF version 3.3.0 is compatible with Ambari version 2.7.3 Step 4) Upgrade “HDF” component to 3.4.0 from 3.3.0 At the end of this upgrade you will have HDF version 3.4.0 and Ambari version 2.7.3 HDF version 3.4.0 is compatible with Ambari version 2.7.3 Thanks Chethan
... View more
06-27-2019
05:10 AM
Thanks for your reply To arrive HDF 3.4.0, we also have to upgrade our Ambari as well. Basically we have two separate HDF and HDP clusters. Currently we are planning to upgrade only HDF cluster. We will not touch HDP cluster Current HDF version = 3.0.2 Current Ambari version = 2.6.1 So, my question is, what should be my upgrade sequence since I need to upgrade Ambari as well along with HDF ? Note, HDF version 3.1.0 is NOT compatible with Ambari version 2.7.3. Can you recommend which of the below sequence will be good to follow ? Sequence#1: Step 1) Upgrade first HDF to 3.1.0 from 3.0.2 This is because HDF 3.1.0 is compatible with Ambari 2.6.1 At the end of this upgrade you will have HDF version 3.1.0 and Ambari version 2.6.1 The first step in upgrading the HDF 3.1 is to upgrade to Ambari 2.6.1. Step 2) Upgrade Ambari alone to 2.7.3 from 2.6.1 At the end of this upgrade you will have HDF version 3.1.0 and Ambari version 2.7.3 Ref - https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.0/ambari-managed-hdf-upgrade/content/hdf-upgrade-paths.html Note: HDF version 3.1.0 is NOT compatible with Ambari version 2.7.3 Step 3) Upgrade “HDF” component to 3.3.0 from 3.1.0 At the end of this upgrade you will have HDF version 3.3.0 and Ambari version 2.7.3 HDF version 3.3.0 is compatible with Ambari version 2.7.3 Step 4) Upgrade “HDF” component to 3.4.0 from 3.3.0 At the end of this upgrade you will have HDF version 3.4.0 and Ambari version 2.7.3 HDF version 3.4.0 is compatible with Ambari version 2.7.3 OR Sequence#2: Step 1) Upgrade Ambari alone to 2.7.3 from 2.6.1 At the end of this upgrade you will have HDF version 3.0.2 and Ambari version 2.7.3 Note: HDF version 3.0.2 is NOT compatible with Ambari version 2.7.3 Step 2) Upgrade HDF to 3.1.0 from 3.0.2 At the end of this upgrade you will have HDF version 3.1.0 and Ambari version 2.7.3 Note: HDF version 3.1.0 is NOT compatible with Ambari version 2.7.3 Step 3) Upgrade “HDF” component to 3.3.0 from 3.1.0 At the end of this upgrade you will have HDF version 3.3.0 and Ambari version 2.7.3 HDF version 3.3.0 is compatible with Ambari version 2.7.3 Step 4) Upgrade “HDF” component to 3.4.0 from 3.3.0 At the end of this upgrade you will have HDF version 3.4.0 and Ambari version 2.7.3 HDF version 3.4.0 is compatible with Ambari version 2.7.3 Thanks Chethan
... View more
06-24-2019
05:25 PM
My requirement is to upgrade HDF (not HDP) . Basically my environment have two separate clusters - HDP and HDF I want to upgrade only HDF to 3.4.0 Version from HDF 3.0.2 . Ambari version currently installed is 2.6.1 and HDF version is 3.0.2 . Can you please help me to know if I can directly upgrade to 3.4.0 from 3.0.2 ? I tried looking into this but no luck https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.4.0/ambari-managed-hdf-upgrade/content/hdf-upgrade-paths.html Can i directly perform HDF upgrade to 3.4.0 from 3.0.2.5 ? Or does it requires first to upgrade to some version first and then upgrade it to higher version hdf 3.4.0 ?
... View more
