Hi, I'm facing same issue. Have there any luck? ... View more

@pavan p 1. Open a terminal and run the following command to open the crontab editor (it's like "vi" editor) # crontab -e 2. Now add the following entries inside it: (here 30 17 means 5:30 PM) and similarly (30 06 means 6:30 AM) 30 17 * * * curl -u admin:admin -H 'X-Requested-By:ambari' -X PUT -d '{"RequestInfo":{"context":"_PARSE_.STOP.ALL_SERVICES","operation_level":{"level":"CLUSTER","cluster_name":"BlueprintCluster"}},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}' http://blueprint.example.com:8080/api/v1/clusters/BlueprintCluster/services? 30 06 * * * curl -u admin:admin -H 'X-Requested-By:ambari' -X PUT -d '{"RequestInfo":{"context":"_PARSE_.START.ALL_SERVICES","operation_level":{"level":"CLUSTER","cluster_name":"BlueprintCluster"}},"Body":{"ServiceInfo":{"state":"STARTED"}}}' http://blueprint.example.com:8080/api/v1/clusters/BlueprintCluster/services? . Reference: https://www.drupal.org/docs/7/setting-up-cron-for-drupal/configuring-cron-jobs-using-the-cron-command ... View more

@Sonu Sahi i am also facing same problem. Activity-analyzer is automatically stopping. i checked the activity-analyzer logs i got the below error in my 10 node cluster. com.hortonworks.smartsense.activity.ActivityException: Invalid smartsense ID unspecified. Please configure a valid SmartSense ID to proceed. Is there any work around / fix to solve this issue? Thanks in advance, Rambabu Chamakuri ... View more

@pavan p Its a vast question. Apart from Structured/ Un-structured you may have to look on other parameters like frequency of ingestion, size of file, Is it a event or batch load, from where you are picking the data. In general, Sqoop(Structured)--> Used to import the RDBMS data into HDFS/Hive Flume/Kafka/NiFi(Unstructured) --> Can be used to capture unstructured data into HDFS. Choosing the client depends on the many other parameters apart from what I have mentioned above. Each tool has its own pros & cons. You may have to dig deep if it is other than learning purpose. Hope it helps!! ... View more

@pavan p If it answers you question please choose it as a best answer! ... View more

Hi Pavan, Did you manage to change the Instance Type? Even we are upgrading it from m4.4xlarge to m4.10xlarge, let me know if you have faced any issues post doing this. If it was successful can you post the brief steps. Thanks. -Ram Charan ... View more

@pavan p Yarn logs are aggregated into HDFS directory '/app-logs/<user_name>/logs'. Refer link for more details on log aggregation. ... View more

@pavan p From command line you can check -Dmapreduce.job.maps, -Dmapreduce.job.reduces and -Dmapreduce.job.running.map.limit, -Dmapreduce.job.running.reduce.limit. ... View more

How does yarn or mapreduce gets to know how much resources is given by particular job? I have a dev team and test team.and i have allocated 20% resources to test team and 40 % resources to dev team. when test team launch 1 job and dev team launch 1 job. How does yarn or mapreduce gets to know test team having 20% resources. dev team having 40% resources?. and how to see which team has launched which job? ... View more

@pavan p If you are using Ambari version earlier than 2.4 you will have new future in Map Reduce and Tez Dashboard in Zeppelin Dashboard Activity Explorer(SmartSense) where it shows "Top N Resource Wasting Jobs". Links: https://docs.hortonworks.com/HDPDocuments/SS1/SmartSense-1.4.0/bk_user-guide/content/mr_notebook.html https://hortonworks.com/blog/whats-new-smartsense-1-3/ Hope this helps you in further identifying your jobs. ... View more

@pavan p 1. First we need to check which component is responding slow (DataNode, NameNode ....etc). 2. We will need to get the PID of that process and collect 5-6 thread dumps during slowness. Please refer to the following article to know how to collect Thread dump along with the CPU data for the slowness analysis. https://community.hortonworks.com/articles/72319/how-to-collect-threaddump-using-jcmd-and-analyse-i.html 3. We will also need to see the Memory utilization/GC pause of those HDFS components. For that we should take a look at the GC log of those components. Also in the logs of NameNode/DataNode we will find th JVMUtil logging if Garbage Collection Pause is too high. . ... View more

how can we identify a job taking more resources than allocation? ... View more

I am not sure what the issue could be. I just walked through this on a local cluster and it worked smoothly for me. Maybe try to disable Kerberos, make sure all starts up ok, then re-enabled Kerberos. ... View more