Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.
priyal
Explorer

Re: Fetch stock exchange data from Alpha vantage A...

by Explorer in Support Questions
‎07-05-2018 09:51 AM
‎07-05-2018 09:51 AM
Hi, @Felix Albani Thanks. ... View more

Fetch stock exchange data from Alpha vantage API u...

by Explorer in Support Questions
‎06-28-2018 10:20 AM
‎06-28-2018 10:20 AM
Hi, I want to fetch stock exchange data from Alpha vantage API using Spark Streaming. I used below API which return data in JSON format : https://www.alphavantage.co/query?function=TIME_SERIES_INTRADAY&symbol=TCS&interval=1min&apikey=apikey How to fetch continuous streaming of stock exchange data using Spark Streaming Java API. ... View more
Labels:

Re: Spark job aborted due to java.lang.OutOfMemory...

by Explorer in Support Questions
‎06-14-2018 07:44 AM
‎06-14-2018 07:44 AM
Hi, @Felix Albani I set driver memory to 20 GB.I tried using below spark-submit parameters : ./bin/spark-submit --driver-memory 20g --executor-cores 3 --num-executors 20 --executor-memory 2g --conf spark.yarn.executor.memoryOverhead=1024 --conf spark.yarn.driver.memoryOverhead=1024 --class org.apache.TransformationOper --master yarn-cluster /home/hdfs/priyal/spark/TransformationOper.jar Cluster configuration is : 1 Master node(r3.xlarge) and 1 worker node(r3.xlarge) : 4 vCPUs, 30GB memory,40 GB storage Still getting the same issue spark job is in running state and YARN memory is 95% used. ... View more

Re: Spark job aborted due to java.lang.OutOfMemory...

by Explorer in Support Questions
‎06-13-2018 01:39 PM
‎06-13-2018 01:39 PM
Hi, @Vinicius Higa Murakami , @Felix Albani I have set spark.yarn.driver.memoryOverhead=1 GB,spark.yarn.executor.memoryOverhead=1 GB and spark_driver_memory=12 GB. I have set storage level to MEMORY_AND_DISK_SER(). Hadoop Cluster configuration is : 1 Master node(r3.xlarge) and 1 worker node (m4.xlarge). Here is the spark-submit parameter : ./bin/spark-submit --driver-memory 12g --executor-cores 2 --num-executors 3 --executor-memory 3g --class org.apache.TransformationOper --master yarn-cluster /spark/TransformationOper.jar Spark job entered into running state but it has been executing for last one hour still execution not completed. ... View more

Re: Spark job aborted due to java.lang.OutOfMemory...

by Explorer in Support Questions
‎06-11-2018 07:07 AM
‎06-11-2018 07:07 AM
Hi, @Vinicius Higa Murakami I want to process 4 GB file so I have configured executor memory to 10 gb and number of executors to 10 in spark-env.sh file.Here is the spark-submit parameters : ./bin/spark-submit --class org.apache.TransformationOper --master local[2] /root/spark/TransformationOper.jar /Input/error.log I tried to set configuration manually using below spark-submit parameters : ./bin/spark-submit --driver-memory 5g --num-executors 10 --executor-memory 10g --class org.apache.TransformationOper --master local[2] /root/spark/TransformationOper.jar And set master as a yarn-cluster still got the OutOfMemoryError error. ... View more

Re: How to refresh the cluster configuration for r...

by Explorer in Support Questions
‎06-08-2018 11:48 AM
‎06-08-2018 11:48 AM
@Jay Kumar SenSharma Thanks ... View more

How to refresh the cluster configuration for refle...

by Explorer in Support Questions
‎06-08-2018 11:17 AM
‎06-08-2018 11:17 AM
Hi, I have created HDP 2.6 on AWS with 1 master node and 4 worker nodes.I am using cluster management tool Ambari. I have configured spark-env.sh file on master node now i want to apply all those setting to all worker nodes on cluster. How to refresh the cluster configuration for reflecting the latest configs to all nodes in the cluster. ... View more
Labels:

Spark job aborted due to java.lang.OutOfMemoryErro...

by Explorer in Support Questions
‎06-08-2018 11:11 AM
‎06-08-2018 11:11 AM
Hi, I have Created HDP 2.6 on AWS with master node(m4.2xlarge) and 4 worker nodes(m4.xlarge). I want to process 4GB log file using Spark job but i am getting below error while executing Spark Job : Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3236) I have configured spark-env.sh file for master node : SPARK_EXECUTOR_MEMORY="5G" SPARK_DRIVER_MEMORY="5G" but it throws the same error. I also configured worker nodes with those settings and increase Java heap size for hadoop client,Resource Manager,Node Manager and for YARN still spark job aborted. Thanks, ... View more
Labels:

Re: Oozie Hive script not working

by Explorer in Support Questions
‎04-17-2018 07:58 AM
‎04-17-2018 07:58 AM
Hi, @Marcos Jimenez Rodriguez Have you tried to execute oozie job as a admin user from Ambari? Check admin user have permissions to write : hdfs dfs -chown -R admin:hadoop /user/admin ... View more

Re: Call external service from Pig

by Explorer in Support Questions
‎04-05-2018 09:22 AM
‎04-05-2018 09:22 AM
@Aditya Sirna I want to post pig relation output to external service using curl. How to pass pig relation values in a below curl command : curl -X POST http://xxx.xx.xxx.xx/services//api/data -H "accept: application/json" -H "authorization: authorization-token value" -H "cache-control: no-cache" -H "content-type: multipart/form-data;boundary=----xxxxxxxxxxxxxxxxx" -H "postman-token:postman-token value" -F "title=the value which i want to fetch from pig relation " -F "description=descirption" i want to fetch value for title from pig relation. ... View more
Contact Me
Online
Offline
Last Visited
‎04-03-2019 05:51 AM
Public Statistics
Date Registered ‎03-31-2017 01:37 PM
Last Visited ‎04-03-2019 05:51 AM
Total Messages Posted 57
Total Kudos Received 1