sqoop job -list command

[#1]

sqoop job -list

command lists no availables jobs. I have created the jobs successfully though.

I changed my metastore to mysql and then created the jobs. Is it possible that this has something to do with metastore change? When I try to create a job with the same name that already exists, it confirms that there is a job with that name already.

1. Where are these jobs stored?

2. What are the possible reasons that it is not able to list the jobs ?

3. How do I fix this?

[#2 · Accepted solution]

You need to make sure you're connecting to the correct metastore. Are you connecting to the shared or your personal metastore? Try to pass --meta-connect and url of metastore.

[#3]

Awesome. Got it 🙂

[#4]

Awesome. Got it 🙂 Can I make the changed metastore the default one ? so that I do not have to mention it each time? I made the changes to sqoop-site.xml though.

[#5]

Here's user guide https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_literal_sqoop_metastore_literal

[#6]

@Artem Ervits: Awesome. Got it 🙂 Can I make the changed metastore the default one ? so that I do not have to mention it each time? I made the changes to sqoop-site.xml though.

[#7]

Yes you can, read the instructions in user guide, it needs to be a global policy.

[#8]

Also make sure you have a way to persist metastore service as your clients will expect your metastore to be up. In Oozie job for example, you can't tell metastore is up so jobs will fail when you execute wf.

[#9]

@Artem Ervits : Thanks . But what does it exactly mean to persist a metastore service? What steps do I need to follow to configure that. I am using sqoop command in oozie so I shall have it that way? Do I need to configure it this way only when I am creating this as teh default metastore or whenever I have my jobs stored in a custom metastore?

[#10]

Make sure sqoop metastore service is up. If it goes down, you need to have a way to bring it up or at least alert you. In my days, I wrote a script that would check whether mestore service was up every few minutes or so and alert if down.