Member since
09-23-2016
35
Posts
20
Kudos Received
12
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
960 | 06-01-2017 11:21 AM | |
2640 | 05-15-2017 12:20 PM | |
3215 | 05-03-2017 08:53 AM | |
5297 | 05-03-2017 07:53 AM | |
3014 | 02-21-2017 08:27 AM |
02-06-2017
08:41 AM
Ah, sorry:) Yes, here you can't specify driver related parameters using <spark-opts>--driver-memory 10g</spark-opts> because your driver (oozie launcher job) is already launched before that point. It's a oozie launcher (which is a mapreduce job) launches your actual spark job and so spark-opts is not relevant. But the Oozie spark action doc says: The configuration element, if present, contains configuration properties that are passed to the Spark job. This is shouldn't be spark configuration. It should be mapreduce configuration for launcher job. So, please try to add the following <configuration> <property> <name>oozie.launcher.mapreduce.map.memory.mb</name> <value>4096</value> </property> <property> <name>oozie.launcher.mapreduce.map.java.opts</name> <value>-Xmx3072m</value>
</property> </configuration>
... View more
02-06-2017
08:12 AM
1 Kudo
It seems your spark driver is running with very small heap size, please try increasing the java.driver memory and see if it helps. Use this parameter (e.g.) when submitting the job: --driver-memory 1g
... View more
02-06-2017
07:57 AM
1 Kudo
Do you have another application that reads from the Event Hub using EventProcessorHost? EventProcessorHost sets an epoch on the receiver to ensure that only one active reader for a given consumer group and an event hub partition is active. You can try it with a different consumer group. Other scenario where this could happen is if you turn on checkpointing on EventProcessorHost. Here is some guidelines from MS how to use the Epoch settings for asynch receivers: https://blogs.msdn.microsoft.com/gyan/2014/09/02/event-hubs-receiver-epoch/
... View more
01-03-2017
08:59 AM
This error message indicates that Hive can not find the file under the given path. I assume you are using the Sandbox, right? So no permission issues for user Admin for Hive and HDFS... - are you using the Sandbox? Can you please check that the path you entered does not contain a leading or tailing blankspace '/tmp/data/geolocation.csv'
... View more
11-30-2016
09:53 AM
Very good, glad to help. I transformed the comment that helped finally to an answer and would be happy if you accept it 😉 Thanks.
... View more
11-30-2016
09:13 AM
You can also use the commandline: https://community.hortonworks.com/questions/49338/using-the-nifi-rest-api-to-execute-a-data-flow-cre.html
... View more
11-30-2016
09:08 AM
Thanks Bhanu, the error comes from Hive. Can you please also have a look at the hive metastore log file and share any related error message?
... View more
11-30-2016
09:08 AM
1 Kudo
Ah, Hive has reached the maximum number of open transactions. There is the parameter hive.max.open.txns that limits the number (compare to https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions ). Default Value is 100000 - this should be high enough, and I guess you did not change it, but maybe you want to check it. Not sure what is going on in your environment but maybe you want to have a look Chapter 6 of http://hortonworks.com/hadoop-tutorial/using-hive-acid-transactions-insert-update-delete-data/ to see how to manage your open transactions. Hope this helps.
... View more
11-30-2016
08:35 AM
Avijeet, yes this is possible. You will have to use the "Update Processor" API from NIFI. There is a great description written by Andrew Grande about How to update Nifi Flows on the fly: https://community.hortonworks.com/articles/3160/update-nifi-flow-on-the-fly-via-api.html
... View more
- « Previous
-
- 1
- 2
- Next »