Member since
05-05-2016
65
Posts
117
Kudos Received
7
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
5860 | 06-20-2016 09:08 AM | |
1248 | 05-26-2016 01:25 PM | |
9301 | 05-26-2016 01:14 PM | |
1242 | 05-25-2016 09:14 AM | |
1976 | 05-25-2016 09:03 AM |
05-19-2016
12:24 PM
1 Kudo
@Thomas Larsson From the above list output on mount point , it seems that whole volume is not inaccessible , some of its subdir are inaccessible . what is the output of this command : sudo ls -la /mnt/data21/
... View more
05-19-2016
10:01 AM
3 Kudos
Adding to above , In this scenario datanode service may fail with "Incompatible ID" . To resolve you need to re-register the datanode as datanode will have the old namespaceid.
... View more
05-19-2016
09:55 AM
4 Kudos
@kavitha velaga 1. Number of mappers depends on InputSplit of the file and hadoop launches mappers as much as reqired. User do not have direct control to set number of mapper via property. 2. To control the number of mapper, user has to control the number of inputsplit which is not necessary until there is requirement of custom logic. 3. User can control the number of reducer for a MR job by setting this property : job.setNumReduceTasks(numOfReducer); numOfReducer can have value from 0 to any positive integer. if you choose 0 then MR job will be mapper only job(no reducer means no aggregation) There are some usecases where Reducer is not necessary so putting numOfReducer=0 will make MR job to finish quickly (as job avoid shuffle and sorting). 4. Container size depends on how much memory your program would require in general. 5. Distcp - This ticket https://issues.apache.org/jira/browse/HDFS-7535 has improved distcp performance. To make distcp run quicker we might disable post copy check like checksum but then we trade-off with reliability. Hope this helps
... View more
05-18-2016
12:35 PM
1 Kudo
@sivasaravanakumar k If you are using any IDE to develop program like Eclipse then you can Add all required JAR file to program path and then build JAR using Eclipse. (http://tutoringcenter.cs.usfca.edu/resources/adding-user-libraries-in-eclipse.html) If you want to run your program on remote job where these jar might not be on classpath then build FAT/Executable jar using Eclipse. FAT JAR will ensure that all the required dependency is packaged with the program. (http://stackoverflow.com/questions/502960/eclipse-how-to-build-an-executable-jar-with-external-jar) You can also use build-tool like Maven to handle dependency for you.
... View more
05-18-2016
12:32 PM
3 Kudos
@John Garrigan You can filter out the non-relevant terms in Mapper phase. Doing this will also increase your program performance as you are dropping unwanted data at early stage. You can use Regex to identify all words start with hashtag. Regex : ^# Psuedocode in above comment looks good .To do more optimization , you can introduce combiner class which will reduce network traffic between mapper and reducer phase. Combiner example code : https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html Combiner concept : http://www.tutorialspoint.com/map_reduce/map_reduce_combiners.htm Hope this helps you
... View more
05-13-2016
10:29 AM
We are trying to abort one request from ambari-server with this command but it is not working it seems. Getting following exception in ambari-server: 11:28:25,579 WARN [Thread-20] ActionScheduler:189 - Exception received
javax.persistence.RollbackException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.4.0.v20120608-r11652): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.BatchUpdateException: Batch entry 0 UPDATE request SET end_time = 1463135305576 WHERE (request_id = 433) was aborted. Call getNextException to see the cause.
Error Code: 0
Call: UPDATE request SET end_time = ? WHERE (request_id = ?)
bind => [2 parameters bound]
at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commitInternal(EntityTransactionImpl.java:102)
at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:63)
at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:91)
at org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.endRequest(ActionDBAccessorImpl.java:302)
at org.apache.ambari.server.actionmanager.ActionDBAccessorImpl.abortOperation(ActionDBAccessorImpl.java:150)
at org.apache.ambari.server.actionmanager.ActionScheduler.abortOperationsForStage(ActionScheduler.java:636)
at org.apache.ambari.server.actionmanager.ActionScheduler.processCancelledRequestsList(ActionScheduler.java:829)
at org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:203)
at org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:184)
at java.lang.Thread.run(Thread.java:745)
Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.4.0.v20120608-r11652): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.BatchUpdateException: Batch entry 0 UPDATE request SET end_time = 1463135305576 WHERE (request_id = 433) was aborted. Call getNextException to see the cause.
Error Code: 0
... View more
05-13-2016
08:56 AM
6 Kudos
This is unix filesystem permission issue. Make sure to provide user running ambari-server has Read/Write permission on /var/run/ambari-server and its all subdirectories. You can run command : chown -R <ambari-server-user> /var/run/ambari-server If you are not sure which user is running ambari-server , go to ambariserver logs (/var/log/ambari-server/ambari-server.log) and you can find the username in detailed error stack trace or the point when you start running ambari-server. Hope this helps.
... View more
05-12-2016
08:24 AM
3 Kudos
It seems that ambari-server has lost connection with ambari-agent somehow. Try these steps : 1. Stop ambari-server 2. Stop ambari-agent service on all nodes 3. Start ambari-agent service on all nodes 4. Start ambari-server server View logs of ambari-server and ambari-agent and see if it throws any error other than component in UNKOWN State.
... View more
- « Previous
- Next »