Cloudera Labs
Provide feedback on Cloudera Labs
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

"Table does not exist" error running Envelope job

Solved Go to solution

"Table does not exist" error running Envelope job

Explorer

I'm attempting to run an upsert planner in envelope to populate a "fact" table. The fact table needs to get surrogate keys from 3 different dimensions which I've defined as input steps. The literal.sql uses joins between the source tables for the fact, and the 3 tables reference in the input steps. When I run the job it gives the following error: 

Exception in thread "main" java.util.concurrent.ExecutionException: 
org.apache.kudu.client.NonRecoverableException:
The table does not exist:
table_name: "impala::kudu_poc_envelope.practitioner_dimension"

I have tried various permutations to get this to work, with no success. For instance, I tried removing the inputs, and just referencing the dimension tables directly in the literal.sql statement. Any thoughts on what is going wrong? I'm using envelope-0.6.0. The conf files, error log, and create table statements are in the attached file.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: "Table does not exist" error running Envelope job

Explorer

The table was in kudu, it just was listed as "impala::kudu_poc_envelope.PRACTITIONER_DIMENSION". Changing my conf file to reference the table using the same case fixed the problem:

  practitioner_dim {
    input {
      type = kudu
      connection = ${kudu_connection}
      table.name = "impala::kudu_poc_envelope.PRACTITIONER_DIMENSION"
    }
  }  
5 REPLIES 5

Re: "Table does not exist" error running Envelope job

Explorer

Attaching the files isn't working, so adding them here...

 

conf:

application {
  name = "kudu poc - patient encounter fact"
  executor.memory = 4g
}

kudu_connection = "edpdev-68b88f09.corp.ad.sentaratest.lab:7051,edpdev-d4e02e23.corp.ad.sentaratest.lab:7051,edpdev-3d02f541.corp.ad.sentaratest.lab:7051"

steps {
  patient_dim {
    input {
      type = kudu
      connection = ${kudu_connection}
      table.name = "impala::kudu_poc_envelope.patient_dimension"
    }
  }
  encounter_dim {
    input {
      type = kudu
      connection = ${kudu_connection}
      table.name = "impala::kudu_poc_envelope.encounter_dimension"
    }
  }
  practitioner_dim {
    input {
      type = kudu
      connection = ${kudu_connection}
      table.name = "impala::kudu_poc_envelope.practitioner_dimension"
    }
  }
  merge {
    dependencies = [patient_dim, encounter_dim, practitioner_dim]
    deriver {
      type = sql
      query.literal = """
  select ed.encounter_dk, cast(peh.pat_enc_csn_id as string) as pat_enc_csn_id, pd.patient_dk as patient_sk,
    prd.prov_id as practitioner_sk,peh.department_id, cast(peh.hsp_account_id as string) as hsp_account_id,
    peh.hosp_admsn_time,  peh.hosp_disch_time,  cast(peh.adt_serv_area_id as string) as adt_serv_area_id,
    peh.bill_num, cast(peh.adm_event_id as string) as adm_event_id, peh.inpatient_data_id, cast(peh.ed_departure_time as string) as ed_departure_time,
    peh.eddisp_edit_user_id, peh.emer_adm_date, peh.eddisp_edit_inst, cast(peh.op_adm_event_id as string) as op_adm_event_id,
    peh2.ex_dis_dt_entr_dttm, peh2.mu_adv_dir_flo_yn, cast(peh2.hsp_adm_event_id as string) as hsp_adm_event_id,
    cast(peh2.hsp_dis_event_id as string) as hsp_dis_event_id, peh2.ed_depart_utc_dttm, peh2.hosp_admsn_utc_dttm
  from clarity_parquet.pat_enc_hsp peh
   left outer join clarity_parquet.pat_enc_hsp_2 peh2 on peh.pat_enc_csn_id=peh2.pat_enc_csn_id
   left outer join encounter_dim ed on cast(peh.pat_enc_csn_id as string)=ed.pat_enc_csn_id
   left outer join patient_dim pd on peh.pat_id=pd.pat_id
   left outer join practitioner_dim prd on peh.admission_prov_id=prd.prov_id
  where peh.contact_date between pd.effective_from_dt and pd.effective_to_dt
   and peh.etl_batch_id = """${etl_batch_log_id}""" and peh2.etl_batch_id = """${etl_batch_log_id_2}
    }
    planner {
      type = upsert
    }
    output {
      type = kudu
      connection = ${kudu_connection}
      table.name = "impala::kudu_poc_envelope.encounter_fact"
    }
  }
}

 

Re: "Table does not exist" error running Envelope job

Explorer

Log:

19/01/08 22:39:44 INFO envelope.EnvelopeMain: Envelope application started
19/01/08 22:39:44 INFO envelope.EnvelopeMain: Configuration loaded
19/01/08 22:39:44 INFO run.Runner: Validating provided Envelope configuration
19/01/08 22:39:44 INFO run.Runner: Provided Envelope configuration is valid (246 checks passed)
19/01/08 22:39:44 INFO run.Runner: Steps instantiated
19/01/08 22:39:44 INFO run.Runner: Security manager created
19/01/08 22:39:44 INFO security.TokenStoreManager: Adding security provider for [envelope.kudu.edpdev-68b88f09.corp.ad.sentaratest.lab:7051,edpdev-d4e02e23.corp.ad.sentaratest.lab:7051,edpdev-3d02f541.corp.ad.sentaratest.lab:7051]
19/01/08 22:39:45 INFO spark.SparkContext: Running Spark version 2.3.0.cloudera3
19/01/08 22:39:45 INFO spark.SparkContext: Submitted application: kudu poc - patient encounter fact
19/01/08 22:39:45 INFO spark.SecurityManager: Changing view acls to: excrutc1
19/01/08 22:39:45 INFO spark.SecurityManager: Changing modify acls to: excrutc1
19/01/08 22:39:45 INFO spark.SecurityManager: Changing view acls groups to:
19/01/08 22:39:45 INFO spark.SecurityManager: Changing modify acls groups to:
19/01/08 22:39:45 INFO spark.SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users  with view permissions: Set(excrutc1); groups with view permissions: Set(); users  with modify permissions: Set(excrutc1); groups with modify permissions: Set()
19/01/08 22:39:46 INFO util.Utils: Successfully started service 'sparkDriver' on port 40525.
19/01/08 22:39:46 INFO spark.SparkEnv: Registering MapOutputTracker
19/01/08 22:39:46 INFO spark.SparkEnv: Registering BlockManagerMaster
19/01/08 22:39:46 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/01/08 22:39:46 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/01/08 22:39:46 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-4af17466-1cec-4c38-a6b2-fe4a2687c741
19/01/08 22:39:46 INFO memory.MemoryStore: MemoryStore started with capacity 912.3 MB
19/01/08 22:39:46 INFO spark.SparkEnv: Registering OutputCommitCoordinator
19/01/08 22:39:46 INFO util.log: Logging initialized @3449ms
19/01/08 22:39:46 INFO server.Server: jetty-9.3.z-SNAPSHOT
19/01/08 22:39:46 INFO server.Server: Started @3523ms
19/01/08 22:39:46 INFO server.AbstractConnector: Started ServerConnector@4c4f4365{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
19/01/08 22:39:46 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@151db587{/jobs,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@8bffb8b{/jobs/json,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@10ee04df{/jobs/job,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@36f1046f{/jobs/job/json,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1686f0b4{/stages,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3900fa71{/stages/json,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7d0614f{/stages/stage,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6f38a289{/stages/stage/json,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3cec79d3{/stages/pool,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4e31c3ec{/stages/pool/json,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6eaa21d8{/storage,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@72e789cb{/storage/json,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@43034809{/storage/rdd,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@77010a30{/storage/rdd/json,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@12aa4996{/environment,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@67c119b7{/environment/json,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6c03fb16{/executors,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6de0f580{/executors/json,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1d61c6dc{/executors/threadDump,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6f9ad11c{/executors/threadDump/json,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@58e92c23{/static,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@184dbacc{/,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@359ff4d9{/api,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3701e6e4{/jobs/job/kill,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@65f58c6e{/stages/stage/kill,null,AVAILABLE,@Spark}
19/01/08 22:39:46 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://edpdev-3d02f541.corp.ad.sentaratest.lab:4040
19/01/08 22:39:46 INFO spark.SparkContext: Added JAR file:/home/excrutc1/kudu_poc/./envelope-0.6.0.jar at spark://edpdev-3d02f541.corp.ad.sentaratest.lab:40525/jars/envelope-0.6.0.jar with timestamp 1547005186650
19/01/08 22:39:46 INFO util.Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
19/01/08 22:39:47 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm26
19/01/08 22:39:48 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
19/01/08 22:39:48 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (12288 MB per container)
19/01/08 22:39:48 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
19/01/08 22:39:48 INFO yarn.Client: Setting up container launch context for our AM
19/01/08 22:39:48 INFO yarn.Client: Setting up the launch environment for our AM container
19/01/08 22:39:48 INFO yarn.Client: Preparing resources for our AM container
19/01/08 22:39:48 INFO yarn.Client: Uploading resource file:/tmp/spark-f84621c9-ef84-4f51-bb00-956e4926784d/__spark_conf__3745643135438003745.zip -> hdfs://edpprd/user/excrutc1/.sparkStaging/application_1546096108651_0212/__spark_conf__.zip
19/01/08 22:39:49 INFO spark.SecurityManager: Changing view acls to: excrutc1
19/01/08 22:39:49 INFO spark.SecurityManager: Changing modify acls to: excrutc1
19/01/08 22:39:49 INFO spark.SecurityManager: Changing view acls groups to:
19/01/08 22:39:49 INFO spark.SecurityManager: Changing modify acls groups to:
19/01/08 22:39:49 INFO spark.SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users  with view permissions: Set(excrutc1); groups with view permissions: Set(); users  with modify permissions: Set(excrutc1); groups with modify permissions: Set()
19/01/08 22:39:49 INFO security.HadoopFSDelegationTokenProvider: getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_414180898_1, ugi=excrutc1@CORP.AD.SENTARATEST.LAB (auth:KERBEROS)]]
19/01/08 22:39:49 INFO hdfs.DFSClient: Created token for excrutc1: HDFS_DELEGATION_TOKEN owner=excrutc1@CORP.AD.SENTARATEST.LAB, renewer=yarn, realUser=, issueDate=1547005189483, maxDate=1547609989483, sequenceNumber=3362, masterKeyId=468 on ha-hdfs:edpprd
19/01/08 22:39:49 INFO security.HadoopFSDelegationTokenProvider: getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_414180898_1, ugi=excrutc1@CORP.AD.SENTARATEST.LAB (auth:KERBEROS)]]
19/01/08 22:39:50 INFO yarn.Client: Submitting application application_1546096108651_0212 to ResourceManager
19/01/08 22:39:51 INFO impl.YarnClientImpl: Submitted application application_1546096108651_0212
19/01/08 22:39:51 INFO cluster.SchedulerExtensionServices: Starting Yarn extension services with app application_1546096108651_0212 and attemptId None
19/01/08 22:39:52 INFO yarn.Client: Application report for application_1546096108651_0212 (state: ACCEPTED)
19/01/08 22:39:52 INFO yarn.Client:
  client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
  diagnostics: N/A
  ApplicationMaster host: N/A
  ApplicationMaster RPC port: -1
  queue: root.users.excrutc1
  start time: 1547005190928
  final status: UNDEFINED
  tracking URL: https://edpdev-d4e02e23.corp.ad.sentaratest.lab:8090/proxy/application_1546096108651_0212/
  user: excrutc1
19/01/08 22:39:53 INFO yarn.Client: Application report for application_1546096108651_0212 (state: ACCEPTED)
19/01/08 22:39:58 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
19/01/08 22:39:59 INFO yarn.Client: Application report for application_1546096108651_0212 (state: RUNNING)
19/01/08 22:39:59 INFO yarn.Client:
  client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
  diagnostics: N/A
  ApplicationMaster host: 10.11.5.12
  ApplicationMaster RPC port: 0
  queue: root.users.excrutc1
  start time: 1547005190928
  final status: UNDEFINED
  tracking URL: https://edpdev-d4e02e23.corp.ad.sentaratest.lab:8090/proxy/application_1546096108651_0212/
  user: excrutc1
19/01/08 22:39:59 INFO cluster.YarnClientSchedulerBackend: Application application_1546096108651_0212 has started running.
19/01/08 22:39:59 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37105.
19/01/08 22:39:59 INFO netty.NettyBlockTransferService: Server created on edpdev-3d02f541.corp.ad.sentaratest.lab:37105
19/01/08 22:39:59 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/01/08 22:39:59 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, edpdev-3d02f541.corp.ad.sentaratest.lab, 37105, None)
19/01/08 22:39:59 INFO storage.BlockManagerMasterEndpoint: Registering block manager edpdev-3d02f541.corp.ad.sentaratest.lab:37105 with 912.3 MB RAM, BlockManagerId(driver, edpdev-3d02f541.corp.ad.sentaratest.lab, 37105, None)
19/01/08 22:39:59 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, edpdev-3d02f541.corp.ad.sentaratest.lab, 37105, None)
19/01/08 22:39:59 INFO storage.BlockManager: external shuffle service port = 7337
19/01/08 22:39:59 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, edpdev-3d02f541.corp.ad.sentaratest.lab, 37105, None)
19/01/08 22:39:59 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /metrics/json.
19/01/08 22:39:59 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@14b5752f{/metrics/json,null,AVAILABLE,@Spark}
19/01/08 22:39:59 INFO scheduler.EventLoggingListener: Logging events to hdfs://edpprd/user/spark/spark2ApplicationHistory/application_1546096108651_0212
19/01/08 22:39:59 INFO util.Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
19/01/08 22:39:59 INFO spark.SparkContext: Registered listener com.cloudera.spark.lineage.NavigatorAppListener
19/01/08 22:39:59 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
19/01/08 22:39:59 INFO security.TokenStoreManager: Written new token store file to /user/excrutc1/.sparkStaging/application_1546096108651_0212/envelope_tokens.1547005199627
19/01/08 22:39:59 INFO security.TokenStoreManager: Started token renewal thread
19/01/08 22:39:59 INFO kudu.KuduOutput: Kudu output requesting accumulators
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Seconds spent random planning
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Seconds spent extracting keys
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Seconds spent applying random mutations
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Seconds spent extracting keys
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Seconds spent getting existing
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Seconds spent extracting keys
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Seconds spent getting existing
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Seconds spent random planning
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Seconds spent getting existing
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Seconds spent random planning
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Seconds spent scanning Kudu
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Seconds spent applying random mutations
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Seconds spent getting existing
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Seconds spent extracting keys
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Seconds spent applying random mutations
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Number of filters scanned in Kudu
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Seconds spent random planning
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Seconds spent applying random mutations
19/01/08 22:39:59 INFO spark.Accumulators: Processed accumulator request: Number of Kudu scanners
19/01/08 22:39:59 INFO kudu.KuduOutput: Kudu output received accumulators
19/01/08 22:39:59 INFO internal.SharedState: loading hive config file: file:/etc/spark2/conf.cloudera.CD-SPARK2_ON_YARN-kFdAMExr/yarn-conf/hive-site.xml
19/01/08 22:39:59 INFO internal.SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/user/hive/warehouse').
19/01/08 22:39:59 INFO internal.SharedState: Warehouse path is '/user/hive/warehouse'.
19/01/08 22:39:59 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL.
19/01/08 22:39:59 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6ea717fa{/SQL,null,AVAILABLE,@Spark}
19/01/08 22:39:59 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/json.
19/01/08 22:39:59 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2db900aa{/SQL/json,null,AVAILABLE,@Spark}
19/01/08 22:39:59 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution.
19/01/08 22:39:59 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@73544298{/SQL/execution,null,AVAILABLE,@Spark}
19/01/08 22:39:59 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution/json.
19/01/08 22:39:59 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@597b39c9{/SQL/execution/json,null,AVAILABLE,@Spark}
19/01/08 22:39:59 INFO ui.JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /static/sql.
19/01/08 22:39:59 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@144ac986{/static/sql,null,AVAILABLE,@Spark}
19/01/08 22:40:00 INFO state.StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
19/01/08 22:40:00 INFO spark.ContextCleaner: Cleaned accumulator 14
19/01/08 22:40:00 INFO spark.ContextCleaner: Cleaned accumulator 2
19/01/08 22:40:00 INFO spark.ContextCleaner: Cleaned accumulator 7
19/01/08 22:40:00 INFO spark.ContextCleaner: Cleaned accumulator 1
19/01/08 22:40:00 INFO spark.ContextCleaner: Cleaned accumulator 4
19/01/08 22:40:00 INFO spark.ContextCleaner: Cleaned accumulator 9
19/01/08 22:40:00 INFO spark.ContextCleaner: Cleaned accumulator 5
19/01/08 22:40:00 INFO spark.ContextCleaner: Cleaned accumulator 0
19/01/08 22:40:00 INFO spark.ContextCleaner: Cleaned accumulator 8
19/01/08 22:40:00 INFO spark.ContextCleaner: Cleaned accumulator 6
19/01/08 22:40:00 INFO spark.ContextCleaner: Cleaned accumulator 11
19/01/08 22:40:00 INFO spark.ContextCleaner: Cleaned accumulator 3
19/01/08 22:40:00 INFO hive.HiveUtils: Initializing HiveMetastoreConnection version 1.1.0 using file:[removed]
19/01/08 22:40:01 INFO session.SessionState: Created local directory: /tmp/29628236-b2a3-400c-8d21-f7dbe0bbe042_resources
19/01/08 22:40:01 INFO session.SessionState: Created HDFS directory: /tmp/hive/excrutc1/29628236-b2a3-400c-8d21-f7dbe0bbe042
19/01/08 22:40:01 INFO session.SessionState: Created local directory: /tmp/excrutc1/29628236-b2a3-400c-8d21-f7dbe0bbe042
19/01/08 22:40:01 INFO session.SessionState: Created HDFS directory: /tmp/hive/excrutc1/29628236-b2a3-400c-8d21-f7dbe0bbe042/_tmp_space.db
19/01/08 22:40:01 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
19/01/08 22:40:01 INFO client.HiveClientImpl: Warehouse location for Hive client (version 1.1.0) is /user/hive/warehouse
19/01/08 22:40:02 INFO hive.metastore: Trying to connect to metastore with URI thrift://edpdev-68b88f09.corp.ad.sentaratest.lab:9083
19/01/08 22:40:02 INFO hive.metastore: Opened a connection to metastore, current connections: 1
19/01/08 22:40:02 INFO hive.metastore: Connected to metastore.
19/01/08 22:40:03 WARN util.Utils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf.
19/01/08 22:40:03 INFO security.TokenStoreManager: Stopped token renewal thread
Exception in thread "main" java.util.concurrent.ExecutionException: org.apache.kudu.client.NonRecoverableException: The table does not exist: table_name: "impala::kudu_poc_envelope.practitioner_dimension"
 at java.util.concurrent.FutureTask.report(FutureTask.java:122)
 at java.util.concurrent.FutureTask.get(FutureTask.java:192)
 at com.cloudera.labs.envelope.run.Runner.awaitAllOffMainThreadsFinished(Runner.java:379)
 at com.cloudera.labs.envelope.run.Runner.runBatch(Runner.java:346)
 at com.cloudera.labs.envelope.run.Runner.run(Runner.java:115)
 at com.cloudera.labs.envelope.EnvelopeMain.main(EnvelopeMain.java:57)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
 at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
 at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
 at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.kudu.client.NonRecoverableException: The table does not exist: table_name: "impala::kudu_poc_envelope.practitioner_dimension"
 at org.apache.kudu.client.TabletClient.dispatchMasterErrorOrReturnException(TabletClient.java:557)
 at org.apache.kudu.client.TabletClient.messageReceived(TabletClient.java:467)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
 at org.apache.kudu.client.TabletClient.handleUpstream(TabletClient.java:603)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
 at org.apache.kudu.client.shaded.org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:184)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
 at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
 at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
 at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
 at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
 at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
 at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
 at org.apache.kudu.client.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
 at org.apache.kudu.client.shaded.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
 at org.apache.kudu.client.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
 at org.apache.kudu.client.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

Re: "Table does not exist" error running Envelope job

Rising Star
The first thing would be to check that the table does exist in Kudu.

In Cloudera Manager if you go to the Kudu service, and then Kudu Master Web UI, and then to the Tables tab, do you see table "impala::kudu_poc_envelope.practitioner_dimension"?

If it is not there and you also can't see which table should match the Impala table "kudu_poc_envelope.practitioner_dimension" then you can run the Impala query "DESCRIBE EXTENDED kudu_poc_envelope.practitioner_dimension" and the "kudu.table_name" entry should show you which Kudu table it has mapped to.

Re: "Table does not exist" error running Envelope job

Explorer

Thanks Jeremy. I checked the list of tables and it's listed there as "impala::kudu_poc_envelope.PRACTITIONER_DIMENSION". The other tables that are part of my conf file are listed with lowercase names. I bet that's the problem.

 

Cheers,

Eric

Highlighted

Re: "Table does not exist" error running Envelope job

Explorer

The table was in kudu, it just was listed as "impala::kudu_poc_envelope.PRACTITIONER_DIMENSION". Changing my conf file to reference the table using the same case fixed the problem:

  practitioner_dim {
    input {
      type = kudu
      connection = ${kudu_connection}
      table.name = "impala::kudu_poc_envelope.PRACTITIONER_DIMENSION"
    }
  }