Member since
07-31-2013
1924
Posts
462
Kudos Received
311
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1543 | 07-09-2019 12:53 AM | |
9298 | 06-23-2019 08:37 PM | |
8052 | 06-18-2019 11:28 PM | |
8677 | 05-23-2019 08:46 PM | |
3477 | 05-20-2019 01:14 AM |
02-18-2019
01:07 PM
try giving the heap size when you start the flume agent flume-ng agent -n a1 -f flume_config.conf -Xms256m -Xmx1024m
... View more
02-15-2019
08:03 PM
@Harsh J you are a genius! Thanks a lot!
... View more
02-15-2019
07:13 PM
1 Kudo
The share-lib in Oozie is modular, so it only adds necessary jars for each action type. The java action is the most generic of all action types, and therefore receives none of the other action type dependencies (such as hive, pig, distcp, spark, etc.). The article you've linked to carries an answer to the question of 'how do I further include jars from action type X into my action type Y', which I've quoted below for convenience: """ For example, if you want all Pig actions in one of your Workflows to include the HCatalog ShareLib, you would add oozie.action.sharelib.for.pig=pig,hcatalog to your job.properties. """ So in your case, you may want to try and add: oozie.action.sharelib.for.java=java,hive
... View more
02-14-2019
12:19 AM
Great! Thank you very much!
... View more
02-02-2019
01:07 AM
Thank you for answering the question. I am able to get a response back. But how can I HTTP PUT the below object to create a new resource pool? Would you happen to have a CURL example? { 'minSharePreemptionTimeout': None, 'name': 'username', 'schedulingPolicy': 'fair', 'fairSharePreemptionTimeout': None, 'queues': [], 'aclAdministerApps': None, 'fairSharePreemptionThreshold': None, 'allowPreemptionFrom': None, 'aclSubmitApps': 'scrubbed', 'schedulablePropertiesList': [ { 'maxAMShare': None, 'weight': 1.0, 'impalaDefaultQueryMemLimit': None, 'maxRunningApps': None, 'impalaQueueTimeout': None, 'maxChildResources': None, 'impalaMaxQueuedQueries': None, 'impalaMaxRunningQueries': None, 'scheduleName': 'default', 'maxResources': {'vcores': 60, 'memory': 131072}, 'impalaDefaultQueryOptions': None, 'impalaMaxMemory': None, 'minResources': {'vcores': 25, 'memory': 32768} } ], 'type': None } When I insert this into the response and PUT it back to the endpoint, I get a 200 status code. However the resource pool is not actually created nor visible in Web UI.
... View more
02-01-2019
02:39 PM
@Harsh J How would I do this just for one job ?. I tried using below setting but it is not working. The issue is that I want to use a version of jersey which I bundled into my fat jar,however gateway node has an older version of that jar and it loads a class from there resulting in a NosSuchMethodException .My application is not a map reduce job and I run it by using hadoop jar and running on 5.14.4 export HADOOP_USER_CLASSPATH_FIRST=true export HADOOP_CLASSPATH=/projects/poc/test/config:$HADOOP_CLASSPATH
... View more
01-29-2019
08:45 AM
Hello, I am facing the same problem, could you help me, and give me more detail, I will appreciate it. Thanks in advance.
... View more
01-04-2019
04:19 AM
i have used the below command to copy 36TB to blob using snaoshot. HADOOP_CLIENT_OPTS="-Xmx40G" hadoop distcp -update -delete $SNAPSHOT_PATH wasbs://buclusterbackup@blobplatformdataxe265ecb.blob.core.windows.net/sep_backup/application_data getting Azure exception errors and Java IO error. i re ran with -skipcrccheck still the same error.
... View more
12-12-2018
10:20 PM
I am having the same problem. Can you please expain about 'hosts file' and how can i add IP and hostname? Are we still using IP and hostname of Namenode?
... View more