1971
Posts
1224
Kudos Received
124
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 64 | 04-03-2024 06:39 AM | |
| 424 | 01-12-2024 08:19 AM | |
| 272 | 12-07-2023 01:49 PM | |
| 623 | 08-02-2023 07:30 AM | |
| 1058 | 03-29-2023 01:22 PM |
02-17-2017
04:13 PM
Installed the Hadoop client on Windows! see here: https://github.com/steveloughran/winutils
https://github.com/jaceklaskowski/mastering-apache-spark-book/blob/master/spark-tips-and-tricks-running-spark-windows.adoc
... View more
02-17-2017
01:55 PM
When I do a PutHDFS I get The directory in HDFS is chmod 777 2017-02-17 08:00:29,703 INFO [FileSystemRepository Workers Thread-3] o.a.n.c.repository.FileSystemRepository Successfully archived 1 Resource Claims for Container default in 8 millis
2017-02-17 08:00:36,859 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@193710c3 checkpointed with 8 Records and 0 Swap Files in 53 milliseconds (Stop-the-world time = 25 milliseconds, Clear Edit Logs time = 10 millis), max Transaction ID 612570
2017-02-17 08:00:45,483 INFO [Timer-Driven Process Thread-3] o.a.hadoop.hive.ql.io.orc.WriterImpl Padding ORC by 909609 bytes (<= 0.01 * 67108864)
2017-02-17 08:01:20,966 ERROR [Timer-Driven Process Thread-3] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=44443db5-015a-1000-c206-8210d8a75e26] Failed to write to HDFS due to org.apache.nifi.processor.exception.ProcessException: IOException thrown from PutHDFS[id=44443db5-015a-1000-c206-8210d8a75e26]: java.io.IOException: (null) entry in command string: null chmod 0644 E:\import\requestid\.182390590411603.orc: org.apache.nifi.processor.exception.ProcessException: IOException thrown from PutHDFS[id=44443db5-015a-1000-c206-8210d8a75e26]: java.io.IOException: (null) entry in command string: null chmod 0644 E:\import\requestid\.182390590411603.orc
2017-02-17 08:01:20,981 ERROR [Timer-Driven Process Thread-3] o.apache.nifi.processors.hadoop.PutHDFS
org.apache.nifi.processor.exception.ProcessException: IOException thrown from PutHDFS[id=44443db5-015a-1000-c206-8210d8a75e26]: java.io.IOException: (null) entry in command string: null chmod 0644 E:\import\requestid\.182390590411603.orc
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2105) ~[na:na]
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2053) ~[na:na]
at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:300) ~[nifi-hdfs-processors-1.1.1.jar:1.1.1]
at java.security.AccessController.doPrivileged(Native Method) [na:1.8.0_121]
at javax.security.auth.Subject.doAs(Unknown Source) [na:1.8.0_121]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678) [hadoop-common-2.7.3.jar:na]
at org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:230) [nifi-hdfs-processors-1.1.1.jar:1.1.1]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.1.jar:1.1.1]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.1.jar:1.1.1]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.1.jar:1.1.1]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.1.jar:1.1.1]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.1.jar:1.1.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_121]
at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [na:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) [na:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_121]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_121]
Caused by: java.io.IOException: (null) entry in command string: null chmod 0644 E:\import\requestid\.182390590411603.orc
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770) ~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.util.Shell.execCommand(Shell.java:866) ~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.util.Shell.execCommand(Shell.java:849) ~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733) ~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225) ~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209) ~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307) ~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296) ~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328) ~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398) ~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461) ~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440) ~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911) ~[hadoop-common-2.7.3.jar:na]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892) ~[hadoop-common-2.7.3.jar:na]
at org.apache.nifi.processors.hadoop.PutHDFS$1$1.process(PutHDFS.java:310) ~[nifi-hdfs-processors-1.1.1.jar:1.1.1]
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2082) ~[na:na]
... 18 common frames omitted
... View more
Labels:
02-16-2017
07:48 PM
HDP 2.4 is the last windows edition. HDP on Linux or HDInsights are your upgrade paths
... View more
02-16-2017
02:07 PM
Thanks for the config tip, really helpful. It eventually came up by the morning. I think the issue is that my Hadoop cluster is down and I have at least 50 connections to it. I imagine a few minute time out per connection, 3-6 connections each, that's a big chunk of time.
... View more
02-16-2017
06:07 AM
My NiFi won't restart after it was shutdown. During the shutdown I lost access to my HBase and Hadoop clusters and they are not coming back. I would like to restart NiFi and remove components accessing those no longer available services. But it won't start up even after waiting a few hours. 2017-02-16 01:04:48,660 ERROR [StandardProcessScheduler Thread-3] o.a.n.c.s.StandardControllerServiceNode HBase_1_1_2_ClientService[id=1a5f679f-015a-1000-dd13-6c12daef3685] Failed to invoke @OnEnabled method due to java.io.IOException: java.lang.reflect.InvocationTargetException
2017-02-16 01:04:48,664 ERROR [StandardProcessScheduler Thread-3] o.a.n.c.s.StandardControllerServiceNode
java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240) ~[na:na]
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218) ~[na:na]
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119) ~[na:na]
at org.apache.nifi.hbase.HBase_1_1_2_ClientService.createConnection(HBase_1_1_2_ClientService.java:241) ~[na:na]
at org.apache.nifi.hbase.HBase_1_1_2_ClientService.onEnabled(HBase_1_1_2_ClientService.java:181) ~[na:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_91]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_91]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) ~[na:na]
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) ~[na:na]
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) ~[na:na]
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) ~[na:na]
at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:345) ~[na:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_91]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_91]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_91]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_91]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_91]
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238) ~[na:na]
... 20 common frames omitted
Caused by: java.lang.IllegalArgumentException: A HostProvider may not be empty!
at org.apache.phoenix.shaded.org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:63) ~[na:na]
at org.apache.phoenix.shaded.org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445) ~[na:na]
at org.apache.phoenix.shaded.org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380) ~[na:na]
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.checkZk(RecoverableZooKeeper.java:141) ~[na:na]
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:221) ~[na:na]
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:418) ~[na:na]
at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65) ~[na:na]
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105) ~[na:na]
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:886) ~[na:na]
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:642) ~[na:na]
... 25 common frames omitted
2017-02-16 01:04:48,664 ERROR [StandardProcessScheduler Thread-3] o.a.n.c.s.StandardControllerServiceNode Failed to invoke @OnEnabled method of HBase_1_1_2_ClientService[id=1a5f679f-015a-1000-dd13-6c12daef3685] due to java.io.IOException: java.lang.reflect.InvocationTargetException
2017-02-16 01:04:55,143 INFO [Finalizer] o.a.h.h.c.ConnectionManager$HConnectionImplementation Closing zookeeper sessionid=0xffffffffffffffff
^C
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache NiFi
02-14-2017
04:54 PM
1 Kudo
Tensorflow on Spark by Yahoo
http://yahoohadoop.tumblr.com/post/157196317141/open-sourcing-tensorflowonspark-distributed-deep
https://github.com/yahoo/TensorFlowOnSpark
https://github.com/yahoo/TensorFlowOnSpark/wiki/GetStarted_YARN
https://github.com/yahoo/TensorFlowOnSpark/wiki/Conversion
https://github.com/yahoo/TensorFlowOnSpark/tree/master/examples/slim
https://github.com/tensorflow/models/tree/master/slim
... View more
02-14-2017
04:49 PM
make Spark 1.6.2 and not all the sub version #s. Make Kafka, kafka in the reference just in case. What JDK are you using to compile? Is this Linux, Windows or OSX you are compiling from? Make sure you are building with 1.6.2 and pushing to 1.6.2, having both versions may confuse things. Make sure you have no firewall issues. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_spark-component-guide/content/spark-choose-version.html if you run from the command line on your PC (spark submit) does that work, is it only an issue in IntelliJ? See: https://community.hortonworks.com/articles/15030/spark-remote-debugging.html http://www.datasciencebytes.com/bytes/2016/04/18/getting-started-with-spark-running-a-simple-spark-job-in-java/
... View more
02-14-2017
04:39 PM
If you have any questions on testing spark let me know. Holden's testing is very good way to do this. Also, going line by line in Zeppelin is a great way to debug Spark code.
... View more
02-14-2017
03:13 PM
That Livy is only for Zeppelin, it's not safe to use that In HDP 2.6, there will be a Livy available for general usage.
... View more