Reply
Highlighted
New Contributor
Posts: 1
Registered: ‎05-14-2017

Design architecture for trigerring spark job from UI using queueing mechanism

Requirement:
Trigger a spark job from UI by user action (say submit button click).
Once the spark job is finished, the summary of the status has to be displayed in UI.

Design approach:
1. Once the user initiates a job run by clicking the submit button from UI, we will make an insert into a Impala queue table using Impala JDBC.
   The simplified structure of the queue table is as follows:
   
   JOB_RUN_QUEUE (REQUEST_ID, STATUS, INPUT_PARAM_1, INPUT_PARAM_2, SUMMARY)

   The initial request will have STATUS='SUBMIT'

2. Oozie will be configured to orchestrate the request handling and job spark execution.
   Once Oozie finds entry into the queue table JOB_RUN_QUEUE with status='SUBMIT' it will pull the arguments from the queue table and trigger the spark job.
   It will update the status in the queue table to 'IN PROGRESS'. Upon successfull completion it will update the summary and status in the queue table.
   On failure it will update the status to FAILURE.

3. UI will read the data from the queue table and display on UI.   

Questions:
1. Is there any alternative ad better design approach.
2. D I need to have a queue mechanism to the initial request or can I leverage some inbuilt functionality?

Announcements