Data offload from SAP HANA to Hadoop

r_ranjana · ‎04-13-2017

The requirement is to offload the data from SAP HANA to Hadoop on daily basis. There is two level of firewall between my Hadoop cluster ( 1300 + nodes) and remote SAP HANA database. I am exploring options like Sqoop/SPARK/Apache NiFi.Can anyone suggest the best and reliable option? I believe all DataNodes in the cluster should have access to SAP HANA to fetch the data through Sqoop... so it looks like the firewall needs to open across all DataNodes to SAP HANA. Is there any best way to fetch the data without giving access to all DataNodes? like routing the SAP HANA DB connection to only a few DataNodes... Node Labelling....

r_ranjana · ‎04-29-2016

@Bryan King Regarding username conflict.Alternative approach could be customizing the username specific to the cluster(say for eg <cluster-name>-hdfs,<cluster-name>yarn ) and maintain all of them in single KDC.

r_ranjana · ‎04-29-2016

@Pradeep kumar Do you have smartsense running ?

r_ranjana · ‎04-21-2016

@ARUNKUMAR RAMASAMY You would choose option 1. Hope this helps.

r_ranjana · ‎04-18-2016

@shimrit hoori You can download putty and use the ssh option. The port is 2222 and host is 127.0.0.1 or from the command line. ssh root@127.0.0.1 -p 2222 should prompt for username and password . root/hadoop is the password. Then you are in.! Hope this helps.

r_ranjana · ‎04-13-2016

Paul Tamburello You can manage these instances by provisioning tool (eg Chef/Puppet) . Setup the Nodename as you wish for example , master , slave, edge and use one of the recipes like the below one to set the nodename to hostname. This way whenever you reboot the AWS instances receipes will automatically kicks in if there is some discrepency. https://supermarket.chef.io/cookbooks/hostnames/versions/0.3.1 Also make sure both reverse and forward lookup works. Hope this helps.

Online	Offline
Last Visited	‎04-13-2017 07:08 AM

Date Registered	‎03-09-2016 05:44 AM
Last Visited	‎04-13-2017 07:08 AM
Total Messages Posted	6
Total Kudos Received	1

Data offload from SAP HANA to Hadoop

Re: How does Kerberos work?

Re: heartbeat lost for all the services (HDP 2.4)

Re: Type of KDC

Re: Connect to VM VNC server

Re: HDP Hostnames on EC2