Member since
08-10-2016
170
Posts
14
Kudos Received
6
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
13239 | 01-31-2018 04:55 PM | |
2708 | 11-29-2017 03:28 PM | |
901 | 09-27-2017 02:43 PM | |
818 | 09-12-2016 06:36 PM | |
1044 | 09-02-2016 01:58 PM |
08-12-2019
05:03 PM
your blog link is a dead link.
... View more
04-08-2019
02:38 PM
Super helpful to know it can be done, but is this the ambari-server conf file you edit or the ambari-agent file? I assume it's ambari-server but it would be nice to have that as a clear call out in the article.
... View more
02-14-2019
03:03 PM
Did you ever resolve this issue? I"m facing the same thing
... View more
10-26-2018
02:04 PM
Althought this isn't supported by ambari, the fact that his runs as the same users as ambari, means you may have tools you need. The ambari users does run as a user that has passwordless ssh access to all nodes, this can be used to scp files to nodes. (Something I abuse a lot to move file to all servers.)
... View more
09-27-2018
07:57 PM
Actually, You could also probably just add them as config in your interpreter. I'm not sure you need do use the above parameter setting.
... View more
09-27-2018
07:53 PM
Try adding the options to livy env in dep env LIVY_REPL_JAVA_OPTS="-Dhdp.version=2.3.2.0-2950" This should give you what you want. For reference: https://github.com/cloudera/hue/commit/7289799f5c314aeae1b636bc32b0bcdf67def7f3
... View more
09-15-2018
08:10 PM
I setup knox to use LDAP [r00t@clu-s6 ~]$ sudo /usr/hdp/current/knox-server/bin/knoxcli.sh user-auth-test --cluster default --u [user] --p [password] --d
LDAP authentication successful!
But it's still letting anonymous users in: 18/09/15 19:49:22 |||audit|172.17.159.226|ZEPPELINUI|anonymous|||access|uri|/gateway/default/zeppelin/extensions/MathZoom.js?V=2.7.0|success|Response status: 200
How do I get knox to only allow authenticated users? I must have missed something simple Advanced topology: <topology>
<gateway>
<provider>
<role>authentication</role>
<name>ShiroProvider</name>
<enabled>true</enabled>
<param>
<name>sessionTimeout</name>
<value>30</value>
</param>
<param>
<name>main.ldapRealm</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
</param>
<param>
<name>main.ldapRealm</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
</param>
<param>
<name>main.ldapRealm.userDnTemplate</name>
<value>uid={0},ou=People,dc=cloud,dc=res,dc=bngf,dc=local</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.systemUsername</name>
<value>cn=ldapadm,dc=cloud,dc=res,dc=bngf,dc=local</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.systemPassword</name>
<value>********</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.url</name>
<value>ldap://fqdn.host.com:389</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.authenticationMechanism</name>
<value>simple</value>
</param>
<param>
<name>urls./**</name>
<value>authcBasic</value>
</param>
</provider>
<provider>
<role>identity-assertion</role>
<name>Default</name>
<enabled>true</enabled>
</provider>
<provider>
<role>authorization</role>
<name>XASecurePDPKnox</name>
<enabled>true</enabled>
</provider>
</gateway>
<service>
<role>NAMENODE</role>
<url>hdfs://{{namenode_host}}:{{namenode_rpc_port}}</url>
</service>
<service>
<role>ATLAS</role>
<url>http://{{ambari_server_hostname}}:8080</url>
</service>
<service>
<role>RANGER</role>
<url>http://this.host.com:6080</url>
</service>
<service>
<role>RANGERUI</role>
<url>http://clu-hgm3:6080</url>
</service>
<service>
<role>AMBARI</role>
<url>http://{{ambari_server_hostname}}:8080</url>
</service>
<service>
<role>AMBARIUI</role>
<url>http://{{ambari_server_hostname}}:8080</url>
</service>
<service>
<role>JOBTRACKER</role>
<url>rpc://{{rm_host}}:{{jt_rpc_port}}</url>
</service>
<service>
<role>WEBHDFS</role>
{{webhdfs_service_urls}}
</service>
<service>
<role>WEBHCAT</role>
<url>http://{{webhcat_server_host}}:{{templeton_port}}/templeton</url>
</service>
<service>
<role>OOZIE</role>
<url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url>
</service>
<service>
<role>WEBHBASE</role>
<url>http://{{hbase_master_host}}:{{hbase_master_port}}</url>
</service>
<service>
<role>HIVE</role>
<url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url>
</service>
<service>
<role>RESOURCEMANAGER</role>
<url>http://{{rm_host}}:{{rm_port}}/ws</url>
</service>
<service>
<role>DRUID-COORDINATOR-UI</role>
{{druid_coordinator_urls}}
</service>
<service>
<role>DRUID-COORDINATOR</role>
{{druid_coordinator_urls}}
</service>
<service>
<role>DRUID-OVERLORD-UI</role>
{{druid_overlord_urls}}
</service>
<service>
<role>DRUID-OVERLORD</role>
{{druid_overlord_urls}}
</service>
<service>
<role>DRUID-ROUTER</role>
{{druid_router_urls}}
</service>
<service>
<role>DRUID-BROKER</role>
{{druid_broker_urls}}
</service>
<service>
<role>ZEPPELINUI</role>
{{zeppelin_ui_urls}}
</service>
<service>
<role>ZEPPELINWS</role>
{{zeppelin_ws_urls}}
</service>
</topology>
... View more
Labels:
- Labels:
-
Apache Knox
05-10-2018
08:19 PM
Look like the hook-bin is actually only present on the server where hive is installed. So you can't run it from just any node.
... View more
05-10-2018
08:14 PM
hdp 2.6.3 is missing this script. Any hints to where it was moved?
... View more
05-10-2018
08:12 PM
@Rama Erigindindla did you ever figure this out? I'm facing the same issue.
... View more
05-05-2018
06:19 PM
@Liana Napalkova Click on hosts, and try to start the nodemanagers. If you can provide the output of the error message.(If there is one) that would be really helpful. To do this, either click on the operation and click through the the output/error message. (once you click start the number of pending operations should change to 1, click on this and it will enable you to get more info.. just keep clicking through to the log.) If that's not clear you can alway grab the log, ssh into the server and look in the following directory /var/log/[path to log] (I dont' recall the exact path off the top of my head but all logs for hdp are in /var/log/ so you should be able to find it with a little bit of looking.) Hope this helps. Matt
... View more
05-02-2018
05:02 PM
Did you ever figure it out. i"m facing the same issue.
... View more
05-02-2018
01:31 PM
I also from the output of the cleanup command the old version of hdp. Hopefully this will also fix my issue.
... View more
05-02-2018
01:30 PM
I used the output from the cleanup-script to reinstall all the packages... everything seems to be running ok now.
... View more
05-01-2018
09:49 PM
I was having an issue with old libraries being used instead of the new ones. I was having this issue but its solution didn't work for me. https://community.hortonworks.com/content/supportkb/154247/error-message-javaioinvalidclassexception-orgapach.html So I looked for a way to clean up the old files and found this: https://community.hortonworks.com/idea/138/cleanup-scripts-to-remove-old-hdp-bits.html Well it actually got things a little to clean and now my 1 node has everything uninstalled on it... so what the best path forward from here? I still have the /usr/hdp directory? Is that any good or am i screwed and should just decommission it and then bring it back as a "new node"? (Yes the node still appears to be up as the scripts to stop the services have been deleted)
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
05-01-2018
02:40 AM
Did you ever figure this out? I'm running into the same issue.
... View more
04-18-2018
11:04 AM
1 Kudo
It's my understanding that the authentication for Jupyter is pluggable. Here's their documentation that explaines their security model. It's a little light on details. This seems to be the integration point you are looking for https://github.com/jupyterhub/ldapauthenticator
... View more
04-10-2018
10:12 AM
@Boualem SAOULA I agree with what @Miles Yao. If you wanted a quick method to test or just add some jars quickly there is also a spark-submit parameter --jars that takes a comma separated list of (full path to) Jars. But it ships the jars every time so that's why the method @Miles Yao suggested has some extra benefit as you save on network traffic.
... View more
04-08-2018
08:55 PM
You are mixing ` and ' in your post. (Is that really what you are using for the definition? or just a copy paste thing?) I'd start with a simple table definition and keep adding features until you reproduce the error with LLAP. As @Sindhu showed, the simplest case does work, so it's likely a specific feature that you are using that 's triggering the error.
... View more
04-08-2018
07:28 PM
@Boualem SAOULA Here's how you can add it so you can work with it: https://github.com/databricks/spark-csv The --packages will also work with 1.6.
... View more
04-08-2018
07:12 PM
@Ryan Redmond I think my guess was kinda correct... According to the docs: Difference between Sort By and Order By
Hive supports SORT BY which sorts the data per reducer. The difference between "order by" and "sort by" is that the former guarantees total order in the output while the latter only guarantees ordering of the rows within a reducer. If there are more than one reducer, "sort by" may give partially ordered final results.
Hive uses the columns in Distribute By to distribute the rows among reducers. All rows with the same Distribute By columns will go to the same reducer. However, Distribute By does not guarantee clustering or sorting properties on the distributed keys. So because you are using "Sort by" with "Distribute by" you are going to continue to get a funky order. By reducer your are getting guaranteed order... I have a feeling Even if you change to using order by you still may get weird results if you keep using "Distribubte by" with columns that don't contain the "sort by" column.
... View more
04-08-2018
04:58 PM
I'm just going to be that the "distrubute by" is actually throwing things off. Just a guess. No evidence to back it up.
... View more
04-08-2018
04:56 PM
Maybe you could format your code for more answers: set hive.enforce.sorting=true;
insert into database2.filtered_rn
select cust_id, timeOf, keys, cd, score, collect_list(weight), concat_WS(',',collect_list(ui)), comp, ROW_NUMBER() over(partition by cust_id order by timeOf desc) as row_num
FROM (
select ld.cust_id, ld.timeOf, ld.keys, lf.cd, ld.score, t.weight, t.ui, COALESCE(ld.compI, 0L) as comp
from database1.xxxx ld
inner join database2.xxxx lf on ld.lof_id = lf.lof_id
left join database1.xxxx r on r.lof_id = ld.lof_id
left join datbase1.xxxx t on r.score_id = t.score_id)
distribute by (cust_id,timeOf)
sort by abs(weight) desc)
sub group by cust_id, timeOf, keys, cd, score, comp;
... View more
04-08-2018
04:50 PM
Interestingly I did try "use" but zeppelin's SQL interpreter doesn't seem to track 'use database' between statements. Maybe that's an issue I should report. @bkosaraju I really appreciate the thorough answer.
... View more
04-05-2018
02:56 PM
My question still stands... but I did fine a ranger work around. Ranger>Hive>default database policy. default database Allow create/select on table "tmp_*"
... View more
04-05-2018
02:21 PM
Using hdp 2.6.3 with LLAP in Zeppelin. Hive impersonation = true LLAP doas = false insert into table myDatabase.test_sql values "test" I get the following error message shadehive.org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [test] does not have [CREATE] privilege on [default/tmp_0218b93f51bf49afb291f47ca315ee57] I do not have permissions to create tables in default.. only in myDatabase. How can i change where this internal temp table is being created so that it's not trying to write to default, but instead myDatabase which I do have full permissions on?
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Ranger
03-29-2018
09:12 PM
No idea why but livy stopped working today. It spits the following out non stop. No idea why anyone got an idea of where to look. org.apache.zeppelin.livy.LivyException: Session 349 is finished, appId: application_1522011783190_0196, log: [ at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866), at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66), at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:766), at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala), 18/03/29 17:06:54 INFO ApplicationMaster: Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: java.lang.ClassNotFoundException: org.apache.livy.rsc.driver.RSCDriverBootstrapper), 18/03/29 17:06:54 INFO ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: Uncaught exception: java.lang.ClassNotFoundException: org.apache.livy.rsc.driver.RSCDriverBootstrapper), 18/03/29 17:06:54 INFO ApplicationMaster: Deleting staging directory hdfs://lrdccdhm01.cloud.res.bngf.local:8020/user/andm013/.sparkStaging/application_1522011783190_0196, 18/03/29 17:06:55 INFO ShutdownHookManager: Shutdown hook called, , Failing this attempt. Failing the application.] Livy logs: 18/03/29 17:07:35 INFO RSCClient: Failing pending job 94534a77-1855-45c9-bc18-8e53c8e42165 due to shutdown.
18/03/29 17:07:35 INFO InteractiveSession: Failed to ping RSC driver for session 348. Killing application.
18/03/29 17:07:35 INFO InteractiveSession: Stopping InteractiveSession 348...
18/03/29 17:07:35 WARN InteractiveSession: (Fail to get rsc uri,java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for context to start.)
18/03/29 17:07:38 INFO InteractiveSession: Stopped InteractiveSession 348.
18/03/29 17:08:09 ERROR RSCCli
ent: Failed to connect to context.
java.util.concurrent.TimeoutException: Timed out waiting for context to start.
at org.apache.livy.rsc.ContextLauncher.connectTimeout(ContextLauncher.java:134)
at org.apache.livy.rsc.ContextLauncher.access$300(ContextLauncher.java:63)
at org.apache.livy.rsc.ContextLauncher$2.run(ContextLauncher.java:122)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
18/03/29 17:08:09 INFO RSCClient: Failing pending job 04982890-5fc8-4015-98a9-ad888b4e6b45 due to shutdown.
18/03/29 17:08:09 INFO InteractiveSession: Failed to ping RSC driver for session 349. Killing application.
18/03/29 17:08:09 INFO InteractiveSession: Stopping InteractiveSession 349...
18/03/29 17:08:09 WARN InteractiveSession: (Fail to get rsc uri,java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for context to start.)
18/03/29 17:08:09 INFO InteractiveSession: Stopped InteractiveSession 349.
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache Zeppelin
03-27-2018
07:55 PM
I ran into this issue too.... THeir clearly are daemons running just they aren't able to do anything...
... View more