question Re: Sqoop Metastore - Usage, Supported databases, backup in Archives of Support Questions (Read Only)

Sqoop Metastore - Usage, Supported databases, backup

learninghuman — Wed, 09 Mar 2016 23:58:24 GMT

1. Its not very clear from the HDP documentation whether Sqoop supports Mysql or any DB other than HSQLDB. Has any one tried using MySQL or Postgres for the sqoop metastore?

2. I think metastore is required only for the sqoop jobs and not required otherwise. Is this correct?

3. If only HSQLDB is supported, what are the backup, DR strategies for the same?

Re: Sqoop Metastore - Usage, Supported databases, backup

aervits — Thu, 10 Mar 2016 00:05:24 GMT

Sqoop metastore on MySQL is not out but there are jiras open. There are tutorials outthere but i was never successful to make it work.

Yes metastore is for metadata about jobs, storing last record procesaed, job name, etc.

There really aren't any backup strategies for hsqldb. It's a file on disk, while some in memory data is not flushed to disk you have potential Dara loss. Maybe putting it on RAID 1? Use ECC memory chips, etc.

Re: Sqoop Metastore - Usage, Supported databases, backup

aervits — Thu, 10 Mar 2016 00:20:00 GMT

I should mention that you can store job definition in source control and write out the last record procesaed to an additional destination like hdfs as precaution. I would do that with Java hdfs api at some point to make sure not losing the row sequence.

Re: Sqoop Metastore - Usage, Supported databases, backup

learninghuman — Thu, 10 Mar 2016 00:26:22 GMT

sounds good, thanks. I am looking to store last record info at enterprise level metadata store in DB2.

Re: Sqoop Metastore - Usage, Supported databases, backup

aervits — Thu, 10 Mar 2016 00:31:20 GMT

Sure whatever works, you can show info on completed Sqoop job in metastore and parse that, then store it in whatever means convenient for you.

Re: Sqoop Metastore - Usage, Supported databases, backup

rene_sluiter — Wed, 14 Dec 2016 19:58:14 GMT

We are currently using a mysql metastore in our test environment so it is possible (we run hpd 2.5). Only thing is that oozie requires the sqoop-site.xml file to be placed somewhere in hdfs to access the metastore. We don't really like the idea that passwords are just stored like that..