Member since
04-08-2019
4
Posts
1
Kudos Received
0
Solutions
04-12-2019
02:04 AM
Dear Romainr: I follow your advise to modify the srcCode and made the hue3.9 notebook worked on my CDH5.7.1 Kerberized cluster with livy0.5.0.Now i can using spark-shell on notebook and its run well. but on yarn i saw the spark job user was always livy(i set livy be a haddop.proxyuser), seems its bind on the user which the keytab of livy-server-luancher. so i can't control the notebook authority by hue. i can see the hue set proxyuser on a the token, but the spark job's user was not be reset. livy-conf livy.impersonation.enabled=true
livy.repl.enable-hive-context=true
livy.spark.deploy-mode=client
livy.spark.master=yarn
livy.superusers=hue
livy.server.auth.type=kerberos
livy.server.auth.kerberos.keytab=/etc/security/keytabs/spnego.keytab
livy.server.auth.kerberos.principal=HTTP/xxx.com@xxx.COM
livy.server.launch.kerberos.keytab=/etc/security/keytabs/livy.keytab
livy.server.launch.kerberos.principal=livy/xxx.com@xxx.COM livy-log 19/04/12 14:48:14 INFO InteractiveSession$: Creating Interactive session 3: [owner: hue, request: [kind: pyspark, proxyUser: Some(baoyong), heartbeatTimeoutInSecond: 0]]
19/04/12 14:48:25 INFO LineBufferedStream: stdout: client token: Token { kind: YARN_CLIENT_TOKEN, service: }
19/04/12 14:48:25 INFO LineBufferedStream: stdout: diagnostics: N/A
19/04/12 14:48:25 INFO LineBufferedStream: stdout: ApplicationMaster host: 192.168.103.166
19/04/12 14:48:25 INFO LineBufferedStream: stdout: ApplicationMaster RPC port: 0
19/04/12 14:48:25 INFO LineBufferedStream: stdout: queue: root.livy
19/04/12 14:48:25 INFO LineBufferedStream: stdout: start time: 1555051701595
19/04/12 14:48:25 INFO LineBufferedStream: stdout: final status: UNDEFINED
19/04/12 14:48:25 INFO LineBufferedStream: stdout: tracking URL: http://bigdata166.xxx.com:8088/proxy/application_1555044848792_0063/
19/04/12 14:48:25 INFO LineBufferedStream: stdout: user: livy
19/04/12 14:48:28 INFO InteractiveSession: Interactive session 3 created [appid: application_1555044848792_0063, owner: hue, proxyUser: None, state: idle, kind: pyspark, info: {driverLogUrl=null, sparkUiUrl=null}]
... View more