Alter the timezone in the Hue configuration in Cloudera Manager 1.  Navigate to  Hue > Configuration > Service-Wide > Time Zone 2.  Change the time zone based on the instructions from your specific distribution of Linux.   3.  Restart the Hue service for the changes to take affect.  ... View more

Question  Invalidate metadata is a costly operation and in the scenario of adding new data files to an existing table, we can do a REFRESH rather than INVALIDATE METADATA.   What is considered Best Practice? Answer I'll refer you to the documentation on this topic here: http://www.cloudera.com/content/cloudera/en/documentation/cloudera-impala/latest/topics/impala_refresh.html and http://www.cloudera.com/content/cloudera/en/documentation/cloudera-impala/latest/topics/impala_invalidate_metadata.html   In terms of "best practice":   Use the REFRESH statement to load the latest metastore metadata and block location data for a particular table in these scenarios: After loading new data files into the HDFS data directory for the table. (Once you have set up an ETL pipeline to bring data into Impala on a regular basis, this is typically the most frequent reason why metadata needs to be refreshed.) After issuing ALTER TABLE, INSERT, LOAD DATA, or other table-modifying SQL statement in Hive. INVALIDATE METADATA and REFRESH are counterparts: INVALIDATE METADATA waits to reload the metadata when needed for a subsequent query, but reloads all the metadata for the table, which can be an expensive operation, especially for large tables with many partitions. REFRESH reloads the metadata immediately, but only loads the block location data for newly added data files, making it a less expensive operation overall. If data was altered in some more extensive way, such as being reorganized by the HDFS balancer, use INVALIDATE METADATA to avoid a performance penalty from reduced local reads. If you used Impala version 1.0, the INVALIDATE METADATA statement works just like the Impala 1.0 REFRESH statement did, while the Impala 1.1 REFRESH is optimized for the common use case of adding new data files to an existing table, thus the table name argument is now required.       ... View more

Question I am unable to find a table in Impala when running: show tables but it appears when running the same command in Hive. My other tables from Hive appear as expected. How do I find it?  Answer The Impala metadata may be out of date on the specific Impalad that the client is connecting to, so a recently created table will not show up. You will need to run the "invalidate metadata" command to cause that specific Impalad instance to fetch the updated metadata from the Hive MetaStore.   From the impala-shell, run the " invalidate metadata " command, followed by "show tables" again and verify that the expected table now appears in the output from the "show tables" command.   impala-shell -i <impalad-node> [<impalad-node>:21000] > invalidate metadata Successfully refreshed catalog [<impalad-node>:21000] > show tables ... This fetches the up to date table metadata from the Hive MetaStore for Impala to find the latest tables. This " invalidate metadata " needs to be done only on the Impalad node that your client connects to / submits it's query to.   ... View more

Question   By default the Cloudera Manager Web UI listens on port 7180.   If our environment or company policy requires Cloudera Manager to use a different port, how can I change it?     Answer   Preferred Method:   Modify the Cloudera Manager HTTP Port   H ave an admin log into CM Edit the "HTTP Port for Admin Console" property under Administration->Settings->Ports and Addresses to a port that's not restricted.   Alternate Method:   Create an SSH Tunnel to Cloudera Manager   ssh into the host running Cloudera Manager, as any user,  and  set up an SSH tunnel. For example, if your Cloudera Manager runs on cmbox.example.com. Then you run ssh on your local machine as:   $ ssh -L 9999:localhost:7180 any-user@cmbox.example.com   3.  Once you are logged on, visit http://locahost:9999/ on your local browser and it will pipe traffic to/from cmbox.example.com:7180   Note: While the change can also be completed via direct database manipulation it is error-prone and tedious, therefore it is not a recommended method.  ... View more

Although Hadoop is designed to run on industry-standard hardware, recommending an ideal cluster configuration is not as easy as delivering a list of hardware specifications. Selecting hardware that provides the best balance of performance and economy for a given workload requires testing and validation. (For example, users with IO-intensive workloads will invest in more spindles per core.)   The first step in choosing a machine configuration is to understand the type of hardware your operations team already manages. Operations teams often have opinions or hard requirements about new machine purchases, and will prefer to work with hardware with which they’re already familiar. Hadoop is not the only system that benefits from efficiencies of scale. Again, as a general suggestion, if the cluster is new or you can’t accurately predict your ultimate workload, we advise that you use balanced hardware.   There are four types of roles in a basic Hadoop cluster: NameNode (and Standby NameNode), JobTracker, TaskTracker, and DataNode. (A node is a machine performing a particular task.) Most machines in your cluster will perform two of these roles, functioning as both DataNode (for data storage) and TaskTracker (for data processing).   Here are the recommended specifications for DataNode/TaskTrackers in a balanced Hadoop cluster: 12-24 1-4TB hard disks in a JBOD (Just a Bunch Of Disks) configuration 2 quad-/hex-/octo-core CPUs, running at least 2-2.5GHz 64-512GB of RAM Bonded Gigabit Ethernet or 10Gigabit Ethernet (the more storage density, the higher the network throughput needed) The NameNode role is responsible for coordinating data storage on the cluster, and the JobTracker for coordinating data processing. (The Standby NameNode should not be co-located on the NameNode machine for clusters and will run on hardware identical to that of the NameNode.) Cloudera recommends that customers purchase enterprise-class machines for running the NameNode and JobTracker, with redundant power and enterprise-grade disks in RAID 1 or 10 configurations.   The NameNode will also require RAM directly proportional to the number of data blocks in the cluster. A good rule of thumb is to assume 1GB of NameNode memory for every 1 million blocks stored in the distributed file system. With 100 DataNodes in a cluster, 64GB of RAM on the NameNode provides plenty of room to grow the cluster. We also recommend having HA configured on both the NameNode and JobTracker, features that have been available in the CDH4 line for some time.   Here are the recommended specifications for NameNode/JobTracker/Standby NameNode nodes. The drive count will fluctuate depending on the amount of redundancy: 4–6 1TB hard disks in a JBOD configuration (1 for the OS, 2 for the FS image [RAID 1], 1 for Apache ZooKeeper, and 1 for Journal node) 2 quad-/hex-/octo-core CPUs, running at least 2-2.5GHz 64-128GB of RAM Bonded Gigabit Ethernet or 10Gigabit Ethernet If you expect your Hadoop cluster to grow beyond 20 machines, we recommend that the initial cluster be configured as if it were to span two racks, where each rack has a top-of-rack 10 GigE switch. As the cluster grows to multiple racks, you will want to add redundant core switches to connect the top-of-rack switches with 40GigE. Having two logical racks gives the operations team a better understanding of the network requirements for intra-rack and cross-rack communication.   With a Hadoop cluster in place, the team can start identifying workloads and prepare to benchmark those workloads to identify hardware bottlenecks. After some time benchmarking and monitoring, the team will understand how additional machines should be configured. Heterogeneous Hadoop clusters are common, especially as they grow in size and number of use cases – so starting with a set of machines that are not “ideal” for your workload will not be a waste of time. Cloudera Manager offers templates that allow different hardware profiles to be managed in groups, making it simple to manage heterogeneous clusters.   Below is a list of various hardware configurations for different workloads, including our original “balanced” recommendation: Light Processing Configuration (1U/machine): Two hex-core CPUs, 24-64GB memory, and 8 disk drives (1TB or 2TB) Balanced Compute Configuration (1U/machine): Two hex-core CPUs, 48-128GB memory, and 12 – 16 disk drives (1TB or 2TB) directly attached using the motherboard controller. These are often available as twins with two motherboards and 24 drives in a single 2U cabinet. Storage Heavy Configuration (2U/machine): Two hex-core CPUs, 48-96GB memory, and 16-24 disk drives (2TB – 4TB). This configuration will cause high network traffic in case of multiple node/rack failures. Compute Intensive Configuration (2U/machine): Two hex-core CPUs, 64-512GB memory, and 4-8 disk drives (1TB or 2TB) (Note that Cloudera expects to adopt 2×8, 2×10, and 2×12 core configurations as they arrive.)   The following diagram shows how a machine should be configured according to workload:     For more related information such as why workloads matter, going beyond MapReduce as well as other considerations, please see the original article from our blog.   How-to: Select the Right Hardware for Your New Hadoop Cluster ... View more