Member since
10-24-2015
207
Posts
18
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
4436 | 03-04-2018 08:18 PM | |
4330 | 09-19-2017 04:01 PM | |
1809 | 01-28-2017 10:31 PM | |
976 | 12-08-2016 03:04 PM |
04-01-2018
02:26 PM
@Aishwarya Sudhakar nn is namenode, you will find it in the core-site.xml properties under fs.defaultFS. but i think your issue as i mentioned earlier is you have saved your file without '/' in the beginning of "demo" directory, it got saved into the user home, look at the output of "hdfs dfs -ls demo/data.csv and it will display the user home it is in, use either that or mv it to the root like this: hdfs dfs -mv demo/dataset.csv /demo/dataset.csv make sure your /demo directory exists, if not create it: hdfs dfs -mkdir /demo Hope this helps.
... View more
03-30-2018
09:28 PM
@Andrew Watson @Dave Russell Have you tried to install python 3 on hdp? what did you use to install? virtualenv? conda env? which works better? do you have some instructions? is this advisable to do on production environment?
... View more
03-30-2018
03:11 PM
@Aishwarya Sudhakar
use the whole absolute path and try: sc.textFile("hdfs://nn:8020/demo/dataset.csv") you can find the absolute path core-site.xml and look for fs.defaultFS Also make sure your file in in root path because you mentioned "demo/dataset.csv" and not "/demo/dataset/csv", if it is not then it hsould be in the user home directory like "/user/yourusername/demo/dataset.csv".
... View more
03-08-2018
03:31 PM
Hi, I am using sqoop to import data from netezza, everything works fine but issue is it uses queue 'A'(which is mentioned in the sqoop import) for pulling data from nz and then while loading data into hive, it is using "default" queue. It is not a huge issue but i am having issues with yarn.scheduler.capacity.maximum-am-resource-percent where the queue is overloaded with different jobs. There seems to be 2 solutions here, one being increase the max am resource percent for the default queue or change the load to hive in the sqoop job to use queue 'A'. how can i change it?
... View more
Labels:
- Labels:
-
Apache Sqoop
03-06-2018
09:32 PM
1 Kudo
@Aymen Rahal if you are doing manual installation, you need to install ambari-agent first on each of your servers. use repos to install and edit the /etc/ambari-agent/conf/ambari-agent.ini to use the ambari server hostname and start ambari-agent. when you are adding your hosts, use manual installation .. you dont need ssh key in this case.
... View more
03-05-2018
02:33 PM
@Vikas Srivastava yeah .. but i thought i have to increase the value so there can be more resources available to launch more AM's...
... View more
03-04-2018
08:18 PM
Is this smth related to yarn.scheduler.capacity.maximum-am-resource-percent <value>0.2</value> which is Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running applications. at the time this happened, i had about 5-6 jobs using same queue... i guess this is the reason AM for this job didnt get resources allocated until rest of them finished. so is the 0.2 mean 20% per queue or altogether all queues ?
... View more
03-04-2018
04:15 PM
@Vikas Srivastava Thanks for reply. My table is huge, msck just hangs. Also, i see that although job started at 20:28 pm, the container dint launch until 20:55 and i dont see any logs. what does explain exended do? how to use debug mode for a single query without actually changing in the configurations? FinalStatus Reported by AM: SUCCEEDED Started: Sat Mar 03 20:21:14 -0500 2018 Elapsed: 34mins, 44sec Log Type: launch_container.sh Log Upload Time: Sat Mar 03 20:55:59 -0500 2018| Log Length: 9051 Showing 4096 bytes of 9051 total. Click here for the full log. I dont see anythign in logs. this happened twice during the last 2 days.
... View more
03-03-2018
09:09 PM
i have a job that runs every hour, i put a csv file into hdfs location and do an alter table to add that new location to the partition. Weirdly it took more than 50 min when it just takes 5-10 seconds. I am not sure why? how to start root cause analysis on this?
... View more
Labels:
- Labels:
-
Apache Hive
02-14-2018
04:13 PM
1 Kudo
never mind, its gone automatically... probably yesterday deletion was still happening after going to trash
... View more