About SuryaNarayana

SuryaNarayana · ‎03-01-2018

Hi @Dominika Bialek and @stevel, Thank you for your valuable inputs. The following set of configurations worked for me even for 1 TB file size each. It depends on the infrastructure, network bandwidth between nodes and upload speed (bandwidth) from data node to S3 etc. It took several iterations to do stress tests with many files of small file sizes, small number of big files etc and tune fast buffer size, multipart size and thresholds based on my cluster infrastructure speeds. -Dmapreduce.task.timeout=0 \ -Dfs.s3a.fast.upload=true \ -Dfs.s3a.fast.buffer.size=157286400 \ -Dfs.s3a.multipart.size=314572800 \ -Dfs.s3a.multipart.threshold=1073741824 \ -Dmapreduce.map.memory.mb=8192 \ -Dmapreduce.map.java.opts=-Xmx7290m \ -Dfs.s3a.max.total.tasks=1 \ -Dfs.s3a.threads.max=10 \ -bandwidth 1024 Thanks Surya

SuryaNarayana · ‎02-21-2018

Hello, I have a cluster with HDP 2.5.0.2 (HDFS version 2.7.1.2.5) and I'm trying to distcp large file (200G) from on-premise cluster to Netapp S3, using fast upload. a map task is launched for each file and each map task reads something like this Task: task_1518158406102_3329_m_000000 Status: (84.5% Copying hdfs://HDP1/user/backup/backup-test/2tb_of_200M_each/test_io_0 to s3a://hadoop-acceptance-tests/user/backup/backup-test/test_io_0 [169.1G/200.0G] > map) Once the progress reaches 100% (200.0G/200.0G), then it starts again from 0% (0%/200.0G). This repeats for few times. Here is the command that I used to trigger distcp hadoop distcp -Dfs.s3a.endpoint=s3.in.myhost:8082 -Dfs.s3a.access.key=XXXXXXXXX -Dfs.s3a.secret.key=YYYYYYYYYYY -Dfs.s3a.signing-algorithm=S3SignerType \ -Dfs.s3a.buffer.dir=<Local_path>/tmp-hadoop_backup \ -Dfs.s3a.fast.upload=true \ -Dfs.s3a.fast.buffer.size=1048576 \ -Dfs.s3a.multipart.size=10485760 \ -Dfs.s3a.multipart.threshold=10485760 \ -Dmapreduce.map.memory.mb=8192 \ -Dmapreduce.map.java.opts=-Xmx7360m \ -log /backup/log/distcp_2tb \ -m=300 -bandwidth 1024 \ -skipcrccheck \ -update hdfs:/user/backup/load-test/2tb_of_200M_each s3a://hadoop-acceptance-tests/user/backup/load-test/ Q1. What is the meaning of running each task several times? Q2. After running the job for several hours, job fails with not even copying a single file. It worked (copied) for 10G, 50G and 100G file size each. Q3. Any recommendations/ thumb rules to change/tune the configs based on input file size and upload speeds to s3? Thanks Surya

SuryaNarayana · ‎07-17-2017

Dear All, I need a read-only user (say data_backup) on HDFS for management, data backup purposes. The application backup needs to be taken from UAT cluster to DEV cluster. The application owners wants to do it by themselves rather than requesting admin team. My cluster (HDP-2.5.0.2, ambari Version 2.4.0.1) is kerberized and users are created on linux hosts (Not LDAP). Ranger security is not enabled. Setting ACLs is an approach.. but, is there a better way? Adding the user to some group (or) any property on hdfs-site xml etc ? Please advise. Thanks

SuryaNarayana · ‎03-08-2017

Dear Team, Presently an Ambari managed HDP cluster is running perfectly alright. Due to a change of ownership of the infrastructure, there is a change in the IP ranges, hostnames and domain names of the hosts. So, I have to apply the changes while the cluster is running as usual (or may be with a minimal outages.) Someone might have done this exercise before. Please share any reference links and/or steps you might have documented. Do we have any scripts already come with the installation? Thanks in advance. The following information may help: Ambari version 2.4.x HDP version 2.5 Kerberos - Not Enabled AD integrated - Yes SSL certificates - None Thanks Surya

SuryaNarayana · ‎07-07-2016

@Deepak Sharma, Thank you for the quick reply.. I was trying to bring the github code to easily readable format in excel spreadsheet. For instance, for HDFS service the service checks are as follows.. Trying to prepare the same for rest of the services.. if anyone already did this exercise, please save my time. 🙂 Thanks Surya

SuryaNarayana · ‎07-04-2016

Hi All, I was running the "Run Service Checks" from service actions and getting a overall status as success. I'm interested to know what all the checks covered under this for each service. Is there a documentation on this? (or) anyone listed out this? Please share the information Thanks, Surya

SuryaNarayana · ‎04-21-2016

Hive didn't work after enabling Ranger. HiverServer2 failed to restart.

SuryaNarayana · ‎04-20-2016

Hi All, I'm Surya. I'm from development and support background. I started my career with Visual basic and classic ASP programming and then moved to Rapid development using Power Builder tool. Then worked on .NET 1.1 through 4.5 Until 2012. With the change in software market needs, I learnt Microsoft Azure, Linux, Weblogic and WebSphere Administration and Hadoop 1.x. Out of the new skills, I found hadoop is more interesting with the features of commodity hardware, distributed computing etc. and choose that as new career path. Presently I'm working as Hadoop Technical Consultant @ Teradata India.

Online	Offline
Last Visited	‎11-19-2018 05:16 AM

Date Registered	‎04-20-2016 10:50 PM
Last Visited	‎11-19-2018 05:16 AM
Total Messages Posted	10
Total Kudos Received	1

Re: Each map task of a distcp job to s3 is running...

Each map task of a distcp job to s3 is running sev...

Read-only user on HDFS

How to change the host name, IP and domain name of...

Re: what all the checks covered under service chec...

what all the checks covered under service checks

Hive didn't work after enabling Ranger. HiverServe...

Re: INTRODUCE YOURSELF: Please Reply here to tell ...