Support Questions

Find answers, ask questions, and share your expertise

How to create/update Sqoop Jobs in Java?

avatar
Contributor

I am developing a Java program that uses the Apache Sqoop Metastore API to complete full and incremental loads. I would like to store job details such as the check column in the job. Is there an example leveraging this API to illustrate?

1 ACCEPTED SOLUTION

avatar

This should do it:

/** 
 * If this is an incremental import then we save the user's state back to the metastore
*/

private void saveIncrementalState(SqoopOptions options) throws IOException {
  if (!isIncremental(options)) {
    return;
  }
  Map<String,String> descriptor=options.getStorageDescriptor();
  String jobName=options.getJobName();
  if (null != jobName && null != descriptor) {
    LOG.info("Saving incremental import state to the metastore");
    JobStorageFactory ssf=new JobStorageFactory(options.getConf());
    JobStorage storage=ssf.getJobStorage(descriptor);
    storage.open(descriptor);
    try {
      JobData data=new JobData(options.getParent(),this);
      storage.update(jobName,data);
      LOG.info("Updated data for job: " + jobName);
    }
  finally {
      storage.close();
    }
  }

}

View solution in original post

1 REPLY 1

avatar

This should do it:

/** 
 * If this is an incremental import then we save the user's state back to the metastore
*/

private void saveIncrementalState(SqoopOptions options) throws IOException {
  if (!isIncremental(options)) {
    return;
  }
  Map<String,String> descriptor=options.getStorageDescriptor();
  String jobName=options.getJobName();
  if (null != jobName && null != descriptor) {
    LOG.info("Saving incremental import state to the metastore");
    JobStorageFactory ssf=new JobStorageFactory(options.getConf());
    JobStorage storage=ssf.getJobStorage(descriptor);
    storage.open(descriptor);
    try {
      JobData data=new JobData(options.getParent(),this);
      storage.update(jobName,data);
      LOG.info("Updated data for job: " + jobName);
    }
  finally {
      storage.close();
    }
  }

}