Hadoop services high availability

[#1]

Hello all ! I have to make some hadoop services highly available. so i was wondering about the best way to achieve that, here is the services list : HBase Lily indexer, Sqoop, HiveServer2 thanks a lot

[#2]

@Mehdi TAZI

There is a High Availability section in: http://docs.hortonworks.com (choose you version)

For lastest HDP version (2.3.4), see this:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-...

[#3]

Hello ! first of all thanks a lot for you answer, otherwise I already checked the high availability section on the official documentation, but there is no information about how to make lilly indexer, sqoop highly available.

[#4 · Accepted solution]

sqoop is a client only, so you can have sqoop installed in multiple nodes behind a IP load balancer.

I don't know about Lilly indexer (part of HDP Search Connector). Documentation is here: https://doc.lucidworks.com/lucidworks-hdpsearch/2.3/Guide-Jobs.html#_hbase-indexer, but I'm not sure if it has HA out-of-the-box with Solr Cloud.

[#5]

Are you sur that sqoop is only a client ? i'm agree that the processing is already HA, cause it uses yarn.but what about the Sqoop metastore and job tool ?

[#6]

See this screenshot from sqoop in Ambari. It's only client.

There no sqoop metastore service, by default sqoop uses derby database, but if you want, you can use external mysql or postgres database for sqoop, then, you can configure this database in HA mode.

[#7]

i didn't mention that i'm using sqoop2 with the sqoop server.

[#8]

Hortonworks does not support sqoop2 now. Sqoop supported in lastest hdp 2.3.4 is:

• Apache Sqoop 1.4.6

[#9]

@Mehdi TAZI

Lily is not an HDP product, it was contributed by Ngdata. Perhaps you can reach out to their mailing list for any HA advice? You may have to come up with your own HA for it, then you should certainly share your docs here for the best of the community! As far as HBase, make sure you run multiple master servers.

[#10]

Thanks :), i'll check for Lily.