Member since
06-02-2017
39
Posts
4
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
774 | 09-04-2018 09:07 PM | |
8375 | 08-30-2018 04:57 PM | |
2290 | 08-14-2018 03:49 PM |
04-13-2019
05:13 PM
I read https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/running-spark-applications/content/automating_spark_jobs_with_oozie_spark_action.html However, I don't know where should I set "oozie.action.sharelib.for.spark.exclude=oozie/jackson.*=". Should I set in job.properties? or oozie-site? If oozie-site, I need restart oozie (it is not mentioned in the document) before running sharelibupdate. Thanks for any hints on this. Also, I deleted all files under /user/oozie/share/lib/lib_{TS}/oozie/ (except oozie-hadoop-utils-hadoop-2-4.3.1.3.0.0.0-1634.jar and oozie-sharelib-oozie-4.3.1.3.0.0.0-1634.jar) which should have the same effect of oozie.action.sharelib.for.spark.exclude, but I still got below exception: Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.0
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:751)
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
Looks like the conflict is not coming from oozie share lib. Where this conflict could come from? Thanks again!
... View more
04-11-2019
07:38 AM
Hi, I am using hortonwork HDP3.0 which has oozie 4.3.1 and spark 2.3.1. My spark job throws intermittent dependency error: 2019-04-11 06:29:07,178 [Driver] ERROR org.apache.spark.deploy.yarn.ApplicationMaster - User class threw exception: java.lang.NoClassDefFoundError: Could not initialize class com.fasterxml.jackson.databind.SerializationConfig java.lang.NoClassDefFoundError: Could not initialize class com.fasterxml.jackson.databind.SerializationConfig at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:565) at com.fasterxml.jackson.databind.ObjectMapper.<init>(ObjectMapper.java:480) Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.0 at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64) at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19) at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:751) at org.apache.spark.util.JsonProtocol$.<init>(JsonProtocol.scala:59) at org.apache.spark.util.JsonProtocol$.<clinit>(JsonProtocol.scala) The same spark job running via spark-submit always pass. I have tried below settings but no luck: <configuration>
<property>
<name>oozie.launcher.mapreduce.user.classpath.first</name>
<value>true</value>
</property>
<property>
<name>spark.yarn.user.classpath.first</name>
<value>true</value>
</property>
<property>
<name>oozie.launcher.mapreduce.task.classpath.user.precedence</name>
<value>true</value>
</property>
</configuration> Why spark action has such issue? Thanks for any clue.
... View more
Labels:
- Labels:
-
Apache Oozie
03-22-2019
04:24 PM
Hi,
I am using Horton work HDP3.0 which has zeppelin 0.8.0. I followed https://zeppelin.apache.org/docs/0.8.0/development/helium/writing_visualization_basic.html to install helium viz packages. Since my environment does not have internet access, I have to install packages into local registry. Here is what I did:
step 1. added zeppelin.helium.localregistry.default in custom zeppelin-site settings, pointing to a local helium folder on the zeppelin master host. Restart zeppelin in ambari.
step 2. download https://s3.amazonaws.com/helium-package/helium.json to the helium folder.
step 3. create a npm package tarball, upload the zeppelin master host, unzip into the helium folder. I also updated the artifact to point to local npm folder (e.g. from sogou-map-vis@1.0.0 to /u01/helium/sogou-map-vis)
However, zeppelin's helium page does not list any viz packages.
Below post all indicate it is possible to use helium packages offline but it does not work for me:
https://community.hortonworks.com/questions/223589/installing-helium-modules-in-zeppelin.html
I am not sure if using helium offline is officially supported by zeppelin.
Any clue is highly appreciated! Thanks.
... View more
- Tags:
- zeppelin
Labels:
02-02-2019
07:37 PM
Thanks @Geoffrey Shelton Okot. We are using HDP3 and ambari 2.7. The process of ambari automation is a little different from that from your reference. For example, your reference does not use VDF which is required by hortonwork HDP3. With VDF, the process is: 1. I use ambari api to register the vdf json which tells ambari the location of an xml containing stack and repo info. In my case, it is a file using http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.1.0/HDP-3.0.1.0-187.xml as the template. 2. ambari extracts the repo info (e.g. base url, repo id) and generates /etc/yum.repo.d/ambari-hdp-1.repo which is used by ambari to download hadoop yum packages. As you can see, this process does not need to explicitly register the local repo using HTTP PUT mentioned by your reference. Unfortunately, neither HDP-3.0.1.0-187.xml or ambari source code related to VDF seems to support any option related to skipping base url verification. Also, verifyRepository and verifyRepositories in AmbariManagementControllerImpl.java does not skip SSL cert validation. Not sure how verify_base_url=false can help ambari access the HTTPS yum repo. Any clarification is appreciated
... View more
02-02-2019
04:18 PM
Thanks @Geoffrey Shelton Okot. "Skip
Repository Base URL Validation" seems to only exist in manual install
via ambari UI. I am using ambari blueprint to automate the installation.
Is this option available for automation? Thanks.
... View more
02-01-2019
06:39 PM
When using ambari to install hadoop, ambari will generate a yum conf under /etc/yum.repo.d. We cannot use this yum conf because: 1. Since our environment cannot access internet, we put hadoop yum packages on our yum repo server using HTTPS. We cannot make our yum repo server use HTTP since it is against company policy. However, the yum conf generated by ambari seems not to support HTTPS and ambari cannot access this HTTPS yum server. 2. When ambari's yum conf is generated under /etc/yum.repo.d, this faulty yum conf will mask other valid yum configs and Ambari cannot download hadoop packages by using other valid yum conf. Is there a way to disable Ambari generate a yum conf under /etc/yum.repo.d so that ambari can use the valid yum configs to download packages from our yum repo server? Thanks any clue.
... View more
Labels:
- Labels:
-
Apache Ambari
11-02-2018
07:03 PM
Below command failed on HDP3. Any idea? Thanks a lot. sudo -u hadoop hadoop distcp hdfs://namenode.mycluster.com:8020/tmp/txt hdfs://namenode.mycluster.com:8020/tmp/txt2
java.lang.NullPointerException
at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$TimelineEntityDispatcher.stop(TimelineV2ClientImpl.java:563)
at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl.serviceStop(TimelineV2ClientImpl.java:141)
at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:495)
at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102)
at org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158)
at org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1868)
at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1309)
at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:202)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1761)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
2018-11-02 18:57:15,753 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:178)
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:979)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1761)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort(MRClientService.java:177)
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:159)
... 14 more
2018-11-02 18:57:15,754 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException
... View more
Labels:
09-25-2018
08:45 PM
1 Kudo
@Hayati İbiş Please copy all jar files instead of only spark*.jar. Hope this helps. Thanks.
... View more
09-04-2018
09:07 PM
It is a knox bug https://issues.apache.org/jira/browse/KNOX-1424. After patching the fix, SQL interpreter show results.
... View more
08-31-2018
04:40 PM
I am using HDP3.0 (having zeppelin 0.8.0). Below
queries do not show any result, even though downloading the csv show the
data correctly (e.g. if there is no tables, show the header). %livy2.sqlshow tables %spark2.sql show tables Infrequently I saw the table show a short time and then disappear. I guess it is html rendering issue. This issue does not happen when I ssh to the zeppelin host and use
zeppelin via port forwarding. It happens when I use zeppelin via knox. Any idea? Any setting is related to this? Sql interpreters worked fine in HDP2.6 using zeppelin 0.7.3. javascript error: Error: [ngRepeat:iidexp] '_item_' in '_item_ in _collection_'
should be an identifier or '(_key_, _value_)' expression, but got
'/gateway/ui/zeppelin/app'.
http://errors.angularjs.org/1.5.7/ngRepeat/iidexp?p0=%2Fgateway%2Fui%2Fzeppelin%2Fapp
b/<@https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:376
Kg</<.compile@https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:40:30580
Z@https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:37:4755
S@https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:30480
S@https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:30611
S@https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:30611
N@https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:29413
X/<@https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:37:440
d@https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:30833
m@https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:37:909
mg</<.link/<@https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:40:17582
xc/this.$get</o.prototype.$digest@https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:38:10966
a/n.prototype.safeDigest@https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:76:1460
b@https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:76:3748
a/n.prototype._onMessageHandler@https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:76:3960
R/<@https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:5633
" <!-- ngRepeat: /gateway/ui/zeppelin/app in apps --> Appreciate any clue.
... View more
Labels:
08-30-2018
04:57 PM
2 Kudos
Per https://spark.apache.org/docs/latest/building-spark.html, spark 2.3.1 is built with hadoop 2.6.X by default. This is why I see my fat jar includes hadoop 2.6.5 (instead of 3.1.0) jars. HftpFileSystem has been removed in hadoop 3. I need spark 2.3.1 jars that built with hadoop 3.1. On https://spark.apache.org/downloads.html, I only see spark 2.3.1 built with hadoop 2.7. Where can I get spark 2.3.1 built with hadoop 3? Does spark 2.3.1 support hadoop 3?
Appreciate your help. [UPDATE] I solved this issue by using spark 2.3.1 jars under /usr/hdp/current/spark2-client/ from the HDP3.0 cluster. Thanks.
... View more
08-30-2018
03:42 PM
Hi, My spark structured streaming jobs working in HDP2.6 failed in HDP3.0: java.lang.IllegalAccessError: class org.apache.hadoop.hdfs.web.HftpFileSystem cannot access its superinterface org.apache.hadoop.hdfs.web.TokenAspect$TokenManagementDelegator
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3268)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3313)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3352)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)
at org.apache.spark.sql.execution.datasources.HadoopFileLinesReader.<init>(HadoopFileLinesReader.scala:46)
at org.apache.spark.sql.execution.datasources.json.TextInputJsonDataSource$.readFile(JsonDataSource.scala:125)
at org.apache.spark.sql.execution.datasources.json.JsonFileFormat$$anonfun$buildReader$2.apply(JsonFileFormat.scala:132)
at org.apache.spark.sql.execution.datasources.json.JsonFileFormat$$anonfun$buildReader$2.apply(JsonFileFormat.scala:130)
at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:148)
at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:132)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:128)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:182)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:109)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:216)
at org.apache.spark.sql.execution.SortExec$$anonfun$1.apply(SortExec.scala:108)
at org.apache.spark.sql.execution.SortExec$$anonfun$1.apply(SortExec.scala:101)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) I did not find useful info online. Any clue is appreciated.
... View more
Labels:
08-28-2018
04:21 PM
Thanks guys. Rolling back to python 2.7 made livy server succeed to start.
... View more
08-24-2018
08:58 PM
1 Kudo
The domain name used by hadoop hosts and the one used by the load balancer are different. DEFAULT setting will use the load balancer's domain to construct whitelist filter. I need to update the whitelist filter to use hadoop hosts' domain name instead. Hope this helps.
... View more
08-24-2018
08:54 PM
Hi, I am using HDP3.0 and cannot start the livy server: 18/08/24 20:24:39 WARN LivyConf: The configuration key livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.repl.enable-hive-context instead.
18/08/24 20:24:39 WARN LivyConf: The configuration key livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.server.csrf-protection.enabled instead.
18/08/24 20:24:39 INFO AccessManager: AccessControlManager acls disabled;users with view permission: ;users with modify permission: ;users with super permission: ;other allowed users: *
18/08/24 20:24:39 INFO LineBufferedStream: stdout: File "/usr/bin/hdp-select", line 251
18/08/24 20:24:39 INFO LineBufferedStream: stdout: print "ERROR: Invalid package - " + name
18/08/24 20:24:39 INFO LineBufferedStream: stdout: ^
18/08/24 20:24:39 INFO LineBufferedStream: stdout: SyntaxError: Missing parentheses in call to 'print'. Did you mean print("ERROR: Invalid package - " + name)?
18/08/24 20:24:39 INFO LineBufferedStream: stdout: ls: cannot access /usr/hdp//hadoop/lib: No such file or directory
18/08/24 20:24:39 INFO LineBufferedStream: stdout: Exception in thread "main" java.lang.IllegalStateException: hdp.version is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file in conf with -Dhdp.version=xxx
18/08/24 20:24:39 INFO LineBufferedStream: stdout: at org.apache.spark.launcher.Main.main(Main.java:118)
Exception in thread "main" java.lang.IllegalArgumentException: Fail to parse Spark version from is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file in conf with -Dhdp.version=xxx
at org.apache.livy.utils.LivySparkUtils$.formatSparkVersion(LivySparkUtils.scala:155)
at org.apache.livy.utils.LivySparkUtils$.testSparkVersion(LivySparkUtils.scala:82)
at org.apache.livy.server.LivyServer.start(LivyServer.scala:74)
at org.apache.livy.server.LivyServer$.main(LivyServer.scala:339)
at org.apache.livy.server.LivyServer.main(LivyServer.scala) The error looks like livy tried to use python3 to parse the python2 syntax. Any idea why this happens? Thanks.
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
08-23-2018
11:51 PM
Thanks guys. I got the whitelist filter as mentioned by @Phil Zampino and updated it as my need. Then knox allowed my requests.
... View more
08-21-2018
07:36 PM
Hi, I am using HDP3.0 and ambari 2.7 blueprint. webhdfs via knox failed due to: 2018-08-21 19:26:33,035 ERROR knox.gateway (GatewayDispatchFilter.java:isDispatchAllowed(155)) - The dispatch to http://myhost.com:50070/webhdfs/v1/user was disallowed because it fails the dispatch whitelist validation. See documentation for dispatch whitelisting. I have verified webhdfs without knox works: curl -vvv http://myhost.com:50070/webhdfs/v1/user/?op=LISTSTATUS Also, ambari, zeppelin and ranger UI work fine via knox. The knox settings are: gateway.dispatch.whitelist: DEFAULT
gateway.dispatch.whitelist.services: DATANODE,HBASEUI,HDFSUI,JOBHISTORYUI,NODEUI,RESOURCEMANAGER,WEBHBASE,WEBHDFS,YARNUI webhdfs via knox worked for me on HDP2.6. Any idea? Appreciate any help.
... View more
Labels:
08-16-2018
03:49 AM
It turned out hbase_java_io_tmpdir is enough. The hbase regional server not starting issue is because our security setting disabled the yarn-ats user whose uid is out of a range. After creating this user in the desired range, ts reader v2 worked.
... View more
08-15-2018
05:10 PM
Thanks @Jay Kumar SenSharma This setting worked: "yarn-hbase-env": {
"properties": {
"hbase_java_io_tmpdir": "/u01/tmp"
}
} These setting made TS reader v2 start. Progress! However, after some time, it stopped again due to the issue described in this post. The reason is that the yarn hbase regional server cannot start (though the yarn hbase master starts). I guess it is still related to the fact that regional server is using /tmp. In ambari UI, I cannot find a place to set yarn_hbase_java_io_tmpdir. Any idea?
... View more
08-14-2018
11:47 PM
Looks like this is also caused by noexec /tmp like this post:
Suppressed: java.lang.UnsatisfiedLinkError: /tmp/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_644869269367588546881.so: /tmp/liborg_apache_hbase_thirdparty_netty_transport_
native_epoll_x86_644869269367588546881.so: failed to map segment from shared object: Operation not permitted
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:263)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255)
at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)
... 27 more
No sure what setting can be used to make yarn not use netty.
... View more
08-14-2018
07:06 PM
Hi, I am using HDP3.0 and ambari 2.7 blueprint to install my cluster. Yarn's timeline service V2 reader cannot create zookeeper node: at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:168)
at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:323)
at java.lang.Thread.run(Thread.java:745)
2018-08-14 17:59:55,827 INFO [main] client.RpcRetryingCallerImpl: Call exception, tries=6, retries=6, started=4142 ms ago, cancelled=false, msg=org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-unsecure/meta-region-server, details=row 'prod.timelineservice.entity' on table 'hbase:meta' at null I observed yarn is using timeline service 1.5 and timeline service reader v2. I am not sure if this is expected. My blueprint is using: {
"name": "APP_TIMELINE_SERVER"
},
Adding hbase client per this post did not help. Any idea? Thanks.
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
08-14-2018
03:49 PM
This problem is resolved by adding hbase.netty.nativetransport = false. Do you foresee any issue using this fix? @Akhil S Naik Thanks. [UPDATE 8/15/2018] This setting works and may be better than the above one: {
"hbase-env": {
"properties": {
"hbase_java_io_tmpdir": "/u01/tmp"
}
}
},
... View more
08-14-2018
04:08 AM
Thanks Akhil S Naik OS: oracle linux 7, memory: 314g deployment tool: ambari 2.7 blueprint. Is HDP3.0 using the netty having bug 6678 fixed? 6678 seems to be just a warning. But it failed hbase in my case. Any solution? Thanks.
... View more
08-13-2018
11:33 PM
Hi,I installed hadoop cluster using HDP3.0 but hbase does not work due to this error: Caused by: java.lang.UnsatisfiedLinkError: failed to load the required native library
at org.apache.hbase.thirdparty.io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:81)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.<clinit>(EpollEventLoop.java:55)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:134)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
at org.apache.hbase.thirdparty.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
at org.apache.hbase.thirdparty.io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:104)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:91)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:68)
at org.apache.hadoop.hbase.util.NettyEventLoopGroupConfig.<init>(NettyEventLoopGroupConfig.java:61)
at org.apache.hadoop.hbase.regionserver.HRegionServer.setupNetty(HRegionServer.java:673)
at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:532)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:472)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2923)
... 5 more
Caused by: java.lang.UnsatisfiedLinkError: /tmp/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_644257058602762792223.so: /tmp/liborg_apache_hbase_thirdparty_netty_transport_native_ep
oll_x86_644257058602762792223.so: failed to map segment from shared object: Operation not permitted
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:243)
at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:187)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:207)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.<clinit>(Native.java:65)
at org.apache.hbase.thirdparty.io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:33)
... 24 more
Looks it is caused by nonexec permission of /tmp. To make hbase not use /tmp, I added: { "hbase-site": { "properties": { "hbase.tmp.dir": "/u01/tmp" } } } But this does not work and hbase still uses /tmp. Any idea? Appreciate any help.
... View more
Labels:
08-08-2018
08:42 PM
By understanding ambari source code, the problem is solved. I need to specify below in blueprint:
"cluster-env": {
"properties": {
"dfs_ha_initial_namenode_active": "%HOSTGROUP::master_host_group%",
"dfs_ha_initial_namenode_standby": "%HOSTGROUP::master2_host_group%"
}
}
... View more
08-01-2018
03:51 PM
Geoffrey, blueprint for HDP3.0 Thanks.
... View more
07-31-2018
11:16 PM
"see here" link is unavailable now. Could you please have an updated link? @Jonas Straub I am installing HDP3.0 HA using ambari 2.7 blueprint. The namenodes failed to start due to error "NameNode is not formatted." This did not happen to HDP 2.6 using almost the same blueprint. Look like ambari rollout failed to distribute the metadata. Is there way to get the root cause? The log did not help much. Thanks.
... View more
07-05-2018
06:28 AM
Hi, Happy July 4th! I am using HDP2.6 and knox + ranger. I have tried multiple solutions for setting up knox authentication but got no luck. This email is to get some clue to get me unblocked. Solution 1: LDAP. This does not work because our IT disables LDAP in production hosts by following cis control (https://www.cisecurity.org/controls/). AD is not applicable since I am not using Microsoft technology. Solution 2: OS auth. Ranger can sync the local users from a specified host. However, knox causes trouble. For knox, I did: a) use pam_unix.so to authenticate input username/password. Unfortunately, pam_unix.so uses unix_chkpw to get password from /etc/shadow. Our production hosts use 000 as /etc/shadow which cannot be changed. This means unix_chkpw need to be run as root to be able to access /etc/shadow. However, gateway.sh cannot be run as root. Setting setuid and setgid bits for gateway.sh did not help either. b) use SSSD. Among the supported id providers: ldap, ipa, ad, proxy, local, only local is close to OS auth used by Ranger. Still, it is not appropriate because: reason 1: it is not designed for production. See https://serverfault.com/questions/826848/sss-useradd-vs-useradd-with-sssd. reason 2: local provider does not use /etc/passwd and /etc/shadow. To sync local users on a host with SSSD, I need to use sssd tools to explicitly copy over the local users to SSSD's database, which is not ideal and those synced users in SSSD database may not work for Ranger.
I did not go deep for proxy provider since the document says it is for a legacy NSS provider which is a red flag. Any clue for a working knox authentication in our production hosts? Appreciate very much!
... View more
- Tags:
- knox-gateway
- Security
Labels:
- Labels:
-
Apache Knox
05-04-2018
11:09 PM
Got reply in another thread: To allow end users setting ‘udp_preference_limit’ to 1 we implemented the following change: https://github.com/apache/ambari/pull/129 This will be available in 2.7.
... View more
05-04-2018
10:26 PM
Hi, According to https://community.hortonworks.com/content/supportkb/149955/errorcould-not-configure-server-because-sasl-confi.html, I need "udp_preference_limit = 1" for zookeeper to work with kerberos.How can I set this property using krb5-conf section in ambari blueprint? Thanks for any hint.
... View more
Labels:
- Labels:
-
Apache Ambari