Support Questions

Find answers, ask questions, and share your expertise

Migration of current Hadoop to Isilon ( HDP 2.6 with ambari 2.5) to EMC isilon


Could you please guide me how to migrate present current Hadoop to EMC Isilon.


Well this is complex because its a NAS migration that needs to maintain user and group information between the two platforms. If you are using AD or LDAP for Hadoop access on the source side it will be greatly simplified, because you can add Isilon to that same authentication provider. As a first step determine how users are going to be mapped between the two systems. In all cases you need to provide UID/GID parity between the two systems. Isilon has many options there that can help so investigate what is going to work for your environment. Next look at your network and take care to provide a physical network topology that will give the best b/w between the servers and Isilon. After you have these pieces in place configure you Access zone and HDFS setting on isilon to correspond to your design. There are scripts available to help creation of users and directories at these are useful for new installs and putting users in Isilons "Local Provider" so if your case differs from that manual creation should be used...

Test using simple hdfs command:

hdfs dfs -touchz hdfs://<isilon_access_zone_fqdn>:8020/user/<username>test.file

and check its creation:

hdfs dfs -ls hdfs://<isilon_access_zone_fqdn>:8020/user/<username>test.file

Test it for several users to make sure. Once this is working to your satisfaction leverage the DistCp utility to copy the files using the Isilon URI as the destination. i.e. hdfs://<isilon_access_zone_fqdn>:8020/

then over the course of several days do incrementals to keep the two systems in sync, then pick a day and do the cutover. Since you can run several instances of Hadoop against the same isilon cluster, I would recommend creating a test Access zone and using a few VMs validate that file permissions and ownership are as expected. This is really the hardest piece because Isilon does strict authentication and Hadoop basically does minimal. Always feel free to reach out to Dell/EMC directly for Professional Services for assistance.