Hi,
We are in process of setting up a CDP 7.1.7 SP1 cluster. We have got Knox enabled and configured across all services which works fine for all except Livy.
When we attempt to submit a spark session via Livy service (Knox URL), it does not recognise the user session is getting submitted from and so returns "unauthorised error). There is a log entry in Knox gateway which clearly shows that the user name is not detected.
However, when we submit a spark session directly via Livy URL with the same user, it passes through. This confirms definitely something is not correct in the Knox / Livy configuration. In this attempt, at the same place in Knox gateway logs, it has a POST entry which clearly displays the user name who has submitted this job.
Digging it further, found that there are is no entry in simplified topology configuration for Livy. Not sure if this is an issue? Attached is the configuration.
Secondly, we also found that in KNOX_DATA_DIR/services, we have a folder called Livy and it has 3 version folders. What sense this makes, not sure? Honestly, I do not understand what is the actual significance of these folders. It has rewrite and services.xml in it.
Further referring to below article, did mentioned about creating these files in services folder, however, we are not sure how exactly that would help.
Add custom service to existing descriptor in Apache Knox Proxy | CDP Private Cloud (cloudera.com)
Is there a documentation that actually helps to understand all the steps that are involved in configuring Livy to work with Knox and explains the communications / interactions within these services? Also, any help in getting this setup correct would be really great.
Additionally, how exactly topologies in Knox have an impact in this.
Thanks
snm1523
