Hi Bart,   By design Altus Data Engineering and Data Warehouse clusters intend for their long-term storage to reside in the cloud provider object store, such as AWS S3 or Microsoft's ADLS.  If youi're looking for the functionality of using a cloud-based cluster with the ability for traditional HDFS usage, Altus Director may be better suited for your use-case, as Director allows for creation of a full-fleged CDH cluster residing within the cloud that has the capability to utilize both S3/ADLS as well as cluster-local HDFS.   Listed below are a few links regarding Altus Director in the event you would like to know more about it's features and functionality, as well as how to install and configure it:   https://www.cloudera.com/products/product-components/cloudera-director.html https://www.cloudera.com/documentation/director/latest/topics/director_intro.html     ... View more

Hi @Vik, As a follow-up to my colleagues' posts, please note that if you are trying to install 3rd party software managed by Cloudera Manager (via CSDs [Custom Service Descriptors]), please note that this is not currently supported through Altus Data Engineering, Altus SDX, or Altus Data Warehouse clusters-- It is currently supported within the Altus Director portion of our Altus offerings. If you have a particular use-case that applies to deployment of 3rd party software onto the Altus platform apart from Altus Director, please reach out to Cloudera Support or your Account manager and we can work with you to understand your use-case and help identify solutions to best fit your needs. Kind Regards, Anthony ... View more

Glad to hear, have a wonderful day! ... View more

Hi Kostas,   As a follow up from my colleague's post, were you able to create a cluster on Azure successfully?  ... View more

Hi TldnSysadmin,   Thanks for reaching out to the Cloudera Community and raising this to our attention!   Regarding the instance types reported within the Altus Web UI where the various instance types report specs that aren't aligned to the corresponding Azure instance types is a bug.  I'll go ahead and file this on your behalf to get this resolved in an upcoming update.    In the meantime, regarding the reported symptom of "compute resource(s) not available", this is a separate issue and we can certainly work with you on resolving that.  I will reach out via PM (private message) so that we can further assist with troubleshooting this issue.   Kind Regards,   Anthony   ... View more

Hi @uzubair,   Thanks for raising this to our attention.   You may be able to check the S3 bucket (if configured) for log output that may help with determining why the described symptoms occurred.    Given the nature of this issue, are you able to create a support case through the Altus Web UI (Altus Web UI -> Support -> Support Center -> Technical Support;  Component = Altus Data Engineering;  Sub-Component = Clusters)?   Would you be able to provide some additional details within the support case as well, in particular:   The output of the following command as well (if you have the Altus CLI installed): $ altus dataeng describe-clusters --cluster-name <cluster_name> Cluster Creation time # of Workers and Computer workers created,  and if spot instances were used Was there an Instance Bootstrap Script used?  If so can you attach that to the Support Case as well? Has any other Cluster creations failed in this same manner recently, or has this only been a single manifestation?   Kind Regards,   Anthony ... View more

Hi Nickk,   If you are looking for what features that are available for YARN resource accounting, we do have two metrics available within the YARN API, as well as a more robust reporting capability within Cloudera Manager 5.7 onward.   The following are the definitions of memorySeconds and vcoreSeconds  which are used to provide a very basic measurement of utilization in YARN[1]: memorySeconds = The aggregated amount of memory (in megabytes) the application has allocated times the number of seconds the application has been running. vcoreSeconds = The aggregated number of vcores that the application has allocated times the number of seconds the application has been running. The memorySeconds value can be used loosely for generically measuring the amount of resource that the job consumed; For example, job 1 used X amount of memorySeconds as compared to job 2 which used Y amount of memorySeconds. Any further calculations attempting to extrapolate further insight from this measure isn't recommended. There are some additional reporting efforts that are being worked on, one is now available with CM. Starting in CM 5.7 on, CM offers cluster utilization reporting which can help provide per tenant/user cluster usage reporting.  Further details regarding Cluster Utilization reporting in CM is available here[2].  References: [1] Link to ApplicationResourceUsageReport.java (part of the YARN API) in the Apache source code for Hadoop:   https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport.java [2] Link to Cloudera Documentation regarding CM's Cluster Utilization Reporting functionality: http://www.cloudera.com/documentation/enterprise/5-7-x/topics/admin_cluster_util_report.html     Hope this helps!     ... View more

Hi @acer,   It appears that you're running an Ubuntu / Debian OS;  Have you had a chance to run the CM upgrade wizard which upgrades the CM Agents?   You can access the upgrade wizard by navigating to the following URL (please modify the hostname to suit your CM installation): http://cloudera-manager-node.yourcompany.com:7180/cmf/upgrade-wizard/welcome   Please try to go through the CM Wizard to upgrade the CM Agents.  If this doesn't resolve the issue, you can also try to manually upgrade the CM Agents on each node.  On each node containing an outdated CM Agent, please perform the following steps:     1. Verify the current version of the CM Agent $ dpkg -s cloudera-manager-agent | grep Version   2. Update the apt-get cache $ sudo apt-get update   3. Upgrade the CM Agent package $ sudo apt-get install --only-upgrade cloudera-manager-agent   4. Verify that the CM Agent was upgraded, confirming the version of the package installed: $ dpkg -s cloudera-manager-agent | grep Version     Kind Regards,   Anthony ... View more

Hi  Eyal,   Regarding your questions: >   But how does those resources split? Assuming there are no granular limitations being hit (i.e. max # of jobs per pool/queue/user) , the resources would be calculated by dividing by the sum up all of the weights from the queues/pools with running jobs, then reattributing the weighing.  So in your example, two queues are running development and production, which sums to 6.  Divided by 100, each weight is allocated 16.66667.  So the development pool would get 16.6667% of all of the cluster resources, and the production pool would get 83.3333% of all the cluster resources.  If the default pool receives a job submission, then the weighting is recalculated with 7 weights instead of 6;  Depending on how you have the various other Fair Scheduler properties setup, then various fairness actions would be taken (i.e. preemption, preemption timeouts, etc).   > Is it going to any user based on the fail scheduler? No, as mentioned above, if no jobs are running in the default pool, then it is not calculated within the resources being used;  Only when there's a job submitted/running in that pool (default pool) is when it gets counted/calculated.   > Or is it being split 5:1 (production:development, respectively) between the active jobs in the other resource pools? Correct 🙂     Hope this helps!   -- Anthony   ... View more

Hi Doni,   That's a great question!  Currently there isn't a straightforward way at the moment to verify the actual runtime value of the Capacity Scheduler's specific properties via Web UI, however it is possible to confirm what the value of the property each specific process was given at the time of YARN services/role startup.  Since the Capacity Scheduler resource calculator is run on the Resource Manager and all the Node Managers, you can spot-check the property given to these roles by doing the following:   For CM-managed clusters: 1) Navigate to CM -> Clusters -> YARN -> Configuration -> Search for  yarn.resourcemanager.scheduler.class   2) Confirm that the yarn.resourcemanager.scheduler.class has been set to the Capacity Scheduler using the value org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.   Save changes if needed.   3) Navigate to Instances -> (Click on Resource Manager or Node Manager) -> Processes -> Click on capacity-scheduler.xml under Configuration Files.  NOTE:  This property needs to be set on all Resource Manager(s) and  all Node Managers to be effective.    4)  Search for the property yarn.scheduler.capacity.resource-calculator. If the property is not present here, it will be using the default value of org.apache.hadoop.yarn.util.resource. DefaultResourceCalculator.   5) Confirm that the intended the value that is set, i.e. org.apache.hadoop.yarn.util.resource.DominantResourceCalculator   6) If any changes were made, please Save changes and restart YARN services accordingly and reconfirm the above starting from Step 1.   For CDH-only (no CM) clusters: 1) Confirm that /etc/hadoop/conf/yarn-site.xml for all Resource Manager(s) and Node Managers  contain the entry: <property>    <name>yarn.resourcemanager.scheduler.class</name>    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value> </property>   2)  Verify that the   yarn.scheduler.capacity.resource-calculator property sets the intended value (for example:   org.apache.hadoop.yarn.util.resource.DominantResourceCalculator) in  /etc/hadoop/conf/capacity-scheduler.xml on each of the Resource Manager(s) and all Node Managers.     Hope this helps!   -- Anthony   ... View more

Hi JimAcxiom,   Some interesting details about the counters provided: The reduce shuffle bytes difference is roughly 102GB between the two clusters, which yielded an 885 min difference as well (Newer cluster taking much longer) The number of spilled records was much higher on the old cluster around 243 million records were spilled more on the old cluster Theres a different of about 50GB for physical memory snapshot bytes, consumed more by the old (prod) cluster (which includes the spills) The old cluster used 36GB more virtual memory   Description Diff Old  (prod) cluster New Cluster Launched Map Tasks   171 171 Launched Red Tasks   37 37 Data local map tasks 10 150 160 Rack local map tasks 10 20 10 Time spent all maps in occupied slots (ms) 39 mins. 62483634 60096803 Time spent all reds in occupied slots (ms) 767 mins 55419788 101496628 MR Framework CPU time spent (ms) 885 mins 147935890 201093460 MR Framework Input Split bytes 2 KB 104451 102405 MR Framework Phy memory snapshot bytes 49 GB 563,320,909,824 513,454,596,096 MR Framework Red Shuffle Bytes 102 GB 50,991,891,912 153,054,745,975 MR Framework Spilled Records 243392028 731615420 488223392 MR Framework Total committed heap usage 69 GB 557672038400 626664927232 MR Framework Virt Mem bytes snapshot 36 GB 1070805831680 1034891923456 Given that the job run on the prod cluster consumed more physical memory and spilled much more records, it may be worth looking to ruling out hardware and existing settings as part of the culprit.   Have you had a chance to review (or would you be able to provide some addiitonal details) of both of the clusters, in particular:   The hardware differences (Number of HDD spindles, RAM, CPU type and number cores) between both cluster's worker nodes The various compression properties used in both clusters MR-specific properties, such as:  mapred.reduce.slowstart.completed.maps mapred.tasktracker.map.tasks.maximum mapred.tasktracker.reduce.tasks.maximum mapred.map.child.ulimit The job.xmls of both jobs for comparison   Kind Regards,   Anthony   ... View more

In CM 5.0, the YARN Client configuration had a bug that did not propagate the topology.py file within the client configuration deployment or download, which was subsequently fixed in CM 5.1 onward.  For those still running CM 5.0 who are hitting this issue, we recommend to copy the topology.* from a known good location (DataNode for example) to the existing node.     In general, for any nodes that will be submitting Spark-On-YARN jobs, we recommend that these nodes contain Gateway roles for both Spark and Yarn, and that the Client configuration has been deployed (either by Deploy Client Configuration from the Cluster Actions drop-down, or by each individual component, ensuring that all compoents have their Client configurations deployed).   ... View more

Hi enelso and hkumar449,   That stacktrace is typically indicative of a potential version mismatch between the CM Agents and the CM Server, or by using an unsupported JDK.  Can you confirm that the CM Agent versions are equal to the version of the CM Server and the JDK type and version used on the nodes?       ... View more

Hi Grenoble,   Thanks for your reply!  It looks like even the simple Pi job is also stalling, so let's take Pig out of the equation and focus on YARN configuration settings to get at least a successful Pi job to run first.   Judging by the information provided so far, it seems that no MR jobs are able to properly run with the current settings and will need to be reviewed for establishing a basic starting point.   There's a few pieces still missing to help establish a starting point-- Can you kindly provide the following info? - How many NodeManagers are in the cluster? - How much physical memory is available on each NodeManager? - How many CPU cores are available on each NodeManager? - What are all the services configured on each NodeManager (i.e. HBase Regionserver, DataNode, etc).     ... View more

Thanks for the additional feedback Fewcents!  To clarify, the steps provided ( CM -> YARN -> Instances -> Add Role Instances -> Gateway) added a Node 1 as a YARN Gateway, which effectively pushes the YARN client configurations to Node 1, which will then allow Node 1 to properly submit YARN jobs to the cluster.     Unless Node 1 was added also as a NodeManager service, adding Node 1 as a NodeManager isn't required to submit jobs from it, only that it is specified as a YARN Gateway.       ... View more

 Hi Grenloble,   Can you provide some additional details regarding the behavior on your cluster, in particular:   - How are all the services distributed on your cluster, and how much memory is allocated for each?    - What are the values set for the following properties in YARN? ApplicationMaster Java Maximum Heap Size mapreduce.map.java.opts.max.heap mapreduce.map.memory.mb mapreduce.reduce.java.opts.max.heap mapreduce.reduce.memory.mb yarn.app.mapreduce.am.resource.mb yarn.nodemanager.resource.memory-mb yarn.scheduler.minimum-allocation-mb yarn.scheduler.increment-allocation-mb yarn.scheduler.maximum-allocation-mb   - Are you able to run a simple Pi job successfully?   For Parcel installs: $ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 5 5   For package-based installs: $ hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 5 5   - Do all Pig jobs fail?   - Can you provide the Resource Manager service log snippet along with the Application Master log containing the Pig job that is reportedly hanging?   ... View more

Hi Grenoble, Alxrud,   Thanks for bringing this to our attention.  Given the details of the reported problem, I attempted to reproduce the same issue spinning up a cluster, and was able to successfully run the Pig script in Hue (as shown below, including cluster configuration steps):   Repro details: 1) Setup test cluster with Cloudera Express CM 5.4.1, CDH 5.3.0 (Parcels) 2) Configured the Core Hadoop services in the following fashion (for testing):     Master (16GB RAM): CM, NN, SNN, Hue, Sqoop, RM, JHS, Hive Gateway     Worker 1 (8GB RAM): DN, NM, ZK, Hive Gateway, HMS     Worker 2 (8GB RAM): DN, NM, ZK, Hive Gateway, Oozie     Worker 3 (8GB RAM): DN, NM, ZK, Hive Gateway, HS2 3) Installed All Hue application examples as Hue admin user 4) Created regular user account in Hue 5) Logged in as regular (non-admin) user in Hue 6) Ran the test query via Hue -> Query Editors -> Pig -> Pasted the following output:    data = LOAD '/user/hue/pig/examples/data/midsummer.txt' as (text:CHARARRAY);    upper_case = FOREACH data GENERATE UPPER(text);    STORE upper_case INTO '$output' ;   7) Clicked on Submit 😎 Output filename = test3 9) Confirmed workflow output was successful:   2016-02-12 09:02:21,226 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher  - 0% complete 2016-02-12 09:02:43,143 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher  - 50% complete Heart beat 2016-02-12 09:02:46,386 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation  - mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces 2016-02-12 09:02:46,435 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher  - 100% complete     More Information Needed:  To better understand the cause of the reported behavior, kindly provide responses to the following:   1) How are all the services distributed on your cluster, and how much memory is allocated for each?    2) Are you able to run a simple Pi job successfully?     For Parcel installs: $ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 5 5   For package-based installs: $ hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 5 5   3) Can you provide the Resource Manager service log along with the Application Master log of the Pig job that is reportedly hanging?   Gathering the above information will help help narrow down components where the culprit resides.  If a simple Pi job does not work, then further attention is needed on the YARN configuration and ensuring that the AM, map (and reduce if applicable) containers are properly launched.      ... View more