question Re: Hadoop services high availability in Archives of Support Questions (Read Only)

Hadoop services high availability

TAZIMehdi — Thu, 07 Jan 2016 18:07:54 GMT

Hello all ! I have to make some hadoop services highly available. so i was wondering about the best way to achieve that, here is the services list : HBase Lily indexer, Sqoop, HiveServer2 thanks a lot

Re: Hadoop services high availability

gbraccialli3 — Thu, 07 Jan 2016 18:52:19 GMT

@Mehdi TAZI

There is a High Availability section in: http://docs.hortonworks.com (choose you version)

For lastest HDP version (2.3.4), see this:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-...

Re: Hadoop services high availability

TAZIMehdi — Thu, 07 Jan 2016 19:16:31 GMT

Hello ! first of all thanks a lot for you answer, otherwise I already checked the high availability section on the official documentation, but there is no information about how to make lilly indexer, sqoop highly available.

Re: Hadoop services high availability

gbraccialli3 — Thu, 07 Jan 2016 19:35:44 GMT

sqoop is a client only, so you can have sqoop installed in multiple nodes behind a IP load balancer.

I don't know about Lilly indexer (part of HDP Search Connector). Documentation is here: https://doc.lucidworks.com/lucidworks-hdpsearch/2.3/Guide-Jobs.html#_hbase-indexer, but I'm not sure if it has HA out-of-the-box with Solr Cloud.

Re: Hadoop services high availability

aervits — Thu, 07 Jan 2016 22:06:24 GMT

@Mehdi TAZI

Lily is not an HDP product, it was contributed by Ngdata. Perhaps you can reach out to their mailing list for any HA advice? You may have to come up with your own HA for it, then you should certainly share your docs here for the best of the community! As far as HBase, make sure you run multiple master servers.

Re: Hadoop services high availability

TAZIMehdi — Fri, 08 Jan 2016 00:15:48 GMT

Are you sur that sqoop is only a client ? i'm agree that the processing is already HA, cause it uses yarn.but what about the Sqoop metastore and job tool ?

Re: Hadoop services high availability

TAZIMehdi — Fri, 08 Jan 2016 00:16:13 GMT

Thanks :), i'll check for Lily.

Re: Hadoop services high availability

gbraccialli3 — Mon, 19 Aug 2019 12:18:46 GMT

See this screenshot from sqoop in Ambari. It's only client.

There no sqoop metastore service, by default sqoop uses derby database, but if you want, you can use external mysql or postgres database for sqoop, then, you can configure this database in HA mode.

Re: Hadoop services high availability

TAZIMehdi — Fri, 08 Jan 2016 01:04:08 GMT

i didn't mention that i'm using sqoop2 with the sqoop server.

Re: Hadoop services high availability

gbraccialli3 — Fri, 08 Jan 2016 01:13:23 GMT

Hortonworks does not support sqoop2 now. Sqoop supported in lastest hdp 2.3.4 is:

Apache Sqoop 1.4.6

Re: Hadoop services high availability

aervits — Sat, 06 Feb 2016 03:48:53 GMT

@Mehdi TAZI please accept best answer or provide your own solution.