Member since
05-08-2019
18
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1730 | 02-12-2020 02:25 AM |
02-12-2020
02:25 AM
Figured this out internally . Turns out the spark optimization routine can be affected by the configuration setting spark.sql.codegen.Maxfields which can have implications in how spark will optimize the read from 'fat' tables . In my case the setting was set low which means DAG stages of the read from the right side of the join (the "fat" table) were performed without being assigned to a wholestage codegen . Important to note that the read of the hive data in either case returned the same results just with a different optimization applied to the physical plan .
... View more
12-20-2019
01:15 PM
I have the following issue with this setup . .I define Livy service on a Knox topology with authentication provider enabled . When I request the Livy session over Knox url Knox requests the Livy session with doAs = myuser . So far so good. .. Livy sessions is started with owner=Knox and proxyuser =myuser.. Problem is when we attempt to post to Livy statements API over the Knox url. If we use the Knox url for posting to the running Livy session Knox will add the doAs=myuser . But now we get a forbidden response . Basically because the Livy session is owned by Knox we cannot post statement into the session over the Knox url with doAs=myuser . in my setup at least only the Knox user may post a statement to a Livy session owned by Knox .
... View more
05-10-2019
10:00 AM
Hi @Ryanp , You are very welcome and glad to hear the issue is resolved! Thanks, Li
... View more
02-14-2018
02:40 PM
1 Kudo
@Bryan Bende & @Greg Keys the error seems to persist with HDF 3.0.2. Are there any news on a workaround or a fix?
... View more
08-21-2018
12:22 PM
According to this spark jira this is only available (or planned) in spark 2.4. @jzhang could you confirm?
... View more