What's New @ Cloudera

Version 0.15.0 contains the following notable changes:   Kite artifacts are built against Apache Hadoop 2 and related projects, and are now available in Maven Central. Added new introduction and concepts documentation. Added a new Datasets convenience class for opening and working with Datasets, superseding DatasetRepositories. Deprecated partition related methods in Dataset in favor of the views API. Added a CLI copy task for copying datasets and also for dataset format conversion and data compaction. Added an application parent POM that makes it easy to use Kite in a Maven project. The examples now use this parent POM. Updated to Crunch 0.10.0 Morphlines Library Added morphline command that parses an InputStream that contains protobuf data: readProtobuf (Rober Fiser via whoschek) Added morphline command that extracts specific values from a protobuf object, akin to a simple form of XPath: extractProtobufPath

Dear CDH and Cloudera Manager users,   We are pleased to announce the release of Cloudera Enterprise 5.0.3 (CDH 5.0.3 and Cloudera Manager 5.0.2).   This release is focused on fixing key bugs and includes the following: CDH Fixes FLUME-2245  - HDFS files with errors unable to close FLUME-2416 - Use CodecPool in compressed stream to prevent leak of direct buffers HBASE-10871  Indefinite OPEN/CLOSE wait on busy RegionServers HDFS-5891 - WebHDFS should not try connecting the DataNode during redirection HDFS-6021 - NPE in FSImageFormatProtobuf upgrading from layout -52 to -53 HDFS-6077 - running slive with WebHDFS on secure HA cluster fails with unknown host exception HDFS-6340 - DataNode can't finalize upgrade HDFS-6475 - WebHDFS clients fail without retry because incorrect handling of StandbyException HDFS-6510 - WebHDFS clients clear the delegation token on retry (for HA), thus failing retry requests HDFS-6527 - Edit log corruption due to defer

Dear CDH and Cloudera Manager users,   We are pleased to announce the release of Cloudera Enterprise 5.0.2 (CDH 5.0.2 and Cloudera Manager 5.0.2). This release is focused on fixing key bugs and includes the following:   Cloudera Manager Cloudera Manager Impala Query Monitoring works with Impala 1.3.1 CDH Fixes HADOOP-10556: Add toLowerCase support to auth_to_local rules for service name HADOOP-10638: Updating hadoop-daemon.sh to work as expected when nfs is started as a privileged user. HADOOP-10639: FileBasedKeyStoresFactory initialization is not using default for SSL_REQUIRE_CLIENT_CERT_KEY HADOOP-10658: SSLFactory expects truststores being configured HBASE-6990: Pretty print TTL HBASE-10312: Flooding the cluster with administrative actions leads to collapse HBASE-10371: Compaction creates empty hfile, then selects this file for compaction and creates empty hfile and over again HDFS-6326: WebHdfs ACL compatibility is broken HIVE-6913: Hive u

Dear CDH, Cloudera Manager, and Search Users,   We are pleased to announce the release of CDH 4.7 and Search 1.3.  These are both minor releases that contain multiple bug fixes.   CDH 4.7 Bug fixes in CDH 4.7 include: HBase HBASE-10514  - Forward port HBASE-10466, possible data loss when failed flushes HBASE-10257 - Master aborts due to assignment race HBASE-8912 - AssignmentManager throws IllegalStateException from PENDING_OPEN to OFFLINE HDFS HDFS-6289 - HA failover can fail if there are pending DataNode messages for DataNodes which no longer exist HDFS-5944 - LeaseManager:findLeaseWithPrefixPath can't handle path like /a/b/ right and cause SecondaryNameNode failed do checkpoint HDFS-6191 - Disable quota checks when replaying edit log HDFS-4943 - WebHdfsFileSystem does not work when original file path has encoded chars HDFS-5064 - Standby checkpoints should not block concurrent readers HDFS-6160 - TestSafeMode occasionally fails HDFS-549

Dear CDH, Cloudera Manager, and Cloudera Impala users,   We are pleased to announce the release of Cloudera Enterprise 5.0.1 (CDH 5.0.1 and Cloudera Manager 5.0.1) and Cloudera Manager 4.8.3.   Cloudera Enterprise 5.0.1 This release is focused on fixing key bugs and includes the following:   Component updates Impala 1.3.1 Cloudera Manager HDFS NFS gateway does not work on all Cloudera-supported platforms Replace YARN_HOME with HADOOP_YARN_HOME during upgrade Insufficient password hashing in Cloudera Manager Upgrade to Cloudera Manager 5.0.0 from SLES older than Service Pack 3 with PostgreSQL older than 8.4 fails MR1 to MR2 import fails on a secure cluster After upgrade from CDH 4 to CDH 5, Oozie is missing workflow extension schemas CDH Fixes SOLR-5608 - Frequently reproducible failures in CollectionsAPIDistributedZkTest#testDistribSearch HIVE-6648 - Fixed permission inheritance for multi-partitioned tables HIVE-6740 - Fixed addition of Av

Dear CDH, Cloudera Manager, and Cloudera Impala users, We are pleased to announce the general availability of Cloudera Impala 1.3.1 for use with CDH 4.   Impala 1.3.1 contains stability enhancements and the following new features: Improved memory usage under contention including higher concurrency, high numbers of Parquet columns, or low memory limits Additional SQL built-in functions Note that Impala 1.3.1 does not work with Cloudera Manager 4.7 or earlier. If your environment is managed by Cloudera Manager 4.7 or before, you should upgrade to Impala 1.3.1 and Cloudera Manager 4.8 together. For more details on new Impala features, please see “New Features in Impala”: http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Cloudera-Impala-Release-Notes/cirn_new_features.html “Installing and Using Cloudera Impala” guide: http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/Installing-and-Using-Impala.

We are thrilled to announce the GA release of Cloudera Enterprise 5 (comprising CDH 5.0 and Cloudera Manager 5.0).    When it became generally available 18 months ago, Cloudera Enterprise 4 was widely recognized to be more flexible, more scalable, and less expensive than traditional data management platforms; with Cloudera Enterprise 5, we can also say it is faster (HDFS caching FTW!), more functional (includes new components such as Apache Spark and Parquet), and better integrated (for example, Cloudera Impala and Cloudera Search are now both inside CDH). Furthermore, in an impressive demonstration of ecosystem growth, 100 Cloudera partners invested technical resources to certify on Cloudera Enterprise 5 before its release.   Cloudera Enterprise contains many new components and features, with the major ones listed below. (See CDH 5 Release Notes and Cloudera Manager 5 Release Notes for full details.) Thanks to everyone who partici

Dear Cloudera users,  We are pleased to announce the general availability of the Cloudera Connector Powered by Teradata and the Cloudera Connector for Netezza version 1.2. This release has many updates including: For Cloudera Connector Powered by Teradata: Upgraded TDCH to version 1.2.1 Added support for Avro file format Added support for Incremental import Added support for --where argument Added support for Hive import Added support for Importing all tables using import-all-tables Added support for Query Bands Added new import method split.by.amp (supported only on Teradata 14.10 and higher) For Cloudera Connector for Netezza: Added --nz-uploaddir for uploading logs to HDFS directory Added --nz-ctrlchars to enable CTRLCHARS option for the Netezza external table Both connectors are released in editions compatible with CDH 4 and CDH 5. See download page for further details.   Please note that the Cloudera Connector for Teradata is not supported on CDH 5.  Y

We're happy to announce the release of Kite 0.12.0.This release includes a new MapReduce InputFormat and OutputFormat for reading from and writing to Kite Datasets. There are also several Morphlines updates including a new set of examples, and bug fixes and minor improvements.For more details, see the release notes, and the documentation.

Dear Cloudera Impala and CDH users,     We are pleased to announce the general availability of Cloudera Impala 1.2.4. This maintenance release fixes issues with the Catalog Service as well as a few other bugs, and enables metadata refreshes without full server permissions.   Note that Impala 1.2.4 does not work with Cloudera Manager 4.7 or earlier. If your environment is managed by Cloudera Manager, you should upgrade to Impala 1.2.4 and Cloudera Manager 4.8 together.   For more details on new Impala features, please see “New Features in Impala”.   “Installing and Using Cloudera Impala” guide: http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/Installing-and-Using-Impala.html   As always, we welcome your feedback. Please send your comments and suggestions through our new community forums.  You can also file bugs in the Impala project at issues.cloudera.org.  

We are pleased to announce the general availability of CDH 4.6, Cloudera Manager 4.8.2 and Cloudera Search 1.2.  Details on each release are below.   CDH 4.6 This is an update to the CDH 4 platform and provides a considerable amount of bug-fixes and stability enhancements. Along with these fixes, we have also included some new features, most notable of which are: Rebase to HBase .94.15 HUE-1664 - URL Redirection Whitelist For more details on new features, see “New Features in CDH 4”.   Refer to the “CDH 4 Quick Start Guide” or the “CDH 4 Installation Guide” for more information on how to install this update release.   As always, we are happy to hear your feedback. Please send your comments and suggestions through our new community forums. You can also file bugs through the Distro project on issues.cloudera.org.     Cloudera Manager 4.8.2 Cloudera Manager 4.8.2 is a maintenance release and has a number of bug-fixes and usability improvements.&nbs

Cloudera has released the Beta 2 version of Cloudera Enterprise 5 (comprises CDH 5.0.0 and Cloudera Manager 5.0.0).  This release (download) contains a number of new features and component versions including the ones below: New Components Apache Spark 0.9 Apache Crunch 0.9 Parquet 1.2.5 Kite SDK  0.10 Apace Avro 1.7.5 Apache Hadoop Rebase to Hadoop 2.2.0+ HDFS HDFS Caching NFS gateway Cloudera Manager Support for Apache Spark Oozie and YARN Resource Manager High Availability Extensibility to support add-on services Enhancements to Monitoring & Charts Apache HBase Rebase to HBase 0.96.1.1 Impala Rebase to Impala 1.2.3 Apache Flume New Kite Dataset Sink Apache Hive Hive 0.12 Rebase Improved JDBC spec coverage SSL encryption support on non-kerberos authentication Native Parquet support Apache Sqoop Rebase to Sqoop 1.99.3 Apache Oozie Support for CRON like scheduling Hue Single Sign On (SSO) support Graphical facets in Search application Result graphing for

We are pleased to announce the release of Kite 0.11.0.The main new feature in this release is the views API for working with a subset of a dataset using logical constraints such as field matching or ranges. There are also assorted Morphlines updates, and several bug fixes and minor improvements.For more information, see the release notes, and the documentation.

Dear CDH and Cloudera Manager users,     We are pleased to announce the general availability of Cloudera Manager 4.8.1.   Cloudera Manager 4.8.1 is a maintenance release and has a number of bug-fixes and usability improvements.   For more details, see the release notes: http://www.cloudera.com/content/cloudera-content/cloudera-docs/Cloudera Manager4Ent/latest/Cloudera-Manager-Release-Notes/Cloudera-Manager-Release-Notes.html   All of the Cloudera Manager documentation can be found here: http://www.cloudera.com/content/support/en/documentation/manager/cloudera-manager-v4-latest.html   You can download this latest Cloudera Manager here or you can upgrade from an existing installation of Cloudera Manager.   Installation and upgrade documentation can be found here.

Cloudera is pleased to announce the immediate availability of its first release of Apache Spark for Cloudera Enterprise (comprising CDH and Cloudera Manager).   Spark was created and contributed to the Apache Software Foundation by UC Berkeley, and it has quickly gained adoption for machine learning, interactive analytics, and streaming analytics over large datasets. It features a general programming model for writing applications by composing arbitrary operators, such as mappers, reducers, joins, group-bys, and filters. Spark keeps track of the data that each of the operators produces, enabling applications to reliably store this data in memory, which makes it ideal for low-latency computations and efficient iterative algorithms. Spark applications can be up to 100x faster and require writing 2x to 10x less code than equivalent MapReduce applications.    Cloudera provides enterprise support for Spark through Cloudera Enterprise Flex Edition (as an optiona

Dear Cloudera Impala users,   We would like to announce the general availability of Cloudera Impala 1.2.3. This release fixes a bug where Impala couldn’t read Parquet files written by MapReduce.   Note that Impala 1.2.3 does not work with Cloudera Manager 4.7 or earlier. If your environment is managed by Cloudera Manager, you should upgrade to Impala 1.2.3 and Cloudera Manager 4.8 together.   For more details on new Impala features, please see “New Features in Impala”: http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Cloudera-Impala-Release-Notes/cirn_new_features.html   “Installing and Using Cloudera Impala” guide: http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/Installing-and-Using-Impala.html   As always, we are happy to hear your feedback. Please send your comments and suggestions to impalala-user@cloudera.org or through our new community forums.  You can also file bu

Dear CDH, Cloudera Manager, and Cloudera Impala users,   We are pleased to announce the general availability of Cloudera Impala 1.2.2 and version 2.5.13 of the Cloudera ODBC Driver for Cloudera Impala.      Impala 1.2.2 provides key enhancements including a number of the most requested features: Cost-based join order optimizations Compute statistics for tables and columns directly from Impala Cross join Preliminary support for clients to securely and more easily authenticate with their username and password via LDAP or Active Directory Cloudera ODBC Driver for Impala version 2.5.13 includes: Ability to authenticate with a username and password Ability to secure client traffic via encrypted SSL Ability for multi-tiered applications to proxy via delegation IDs Note that Impala 1.2.2 does not work with Cloudera Manager 4.7 or earlier. If your environment is managed by Cloudera Manager, you should upgrade to Impala 1.2.2 and Cloudera Manager 4.8 toge

We are pleased to announce a new name for the CDK: Kite. We've just released Kite version 0.10.0, which is purely a rename of CDK 0.9.0.The new repository and documentation are: Kite repository: https://github.com/kite-sdk/kite Kite documentation: http://kitesdk.org/ Kite examples: https://github.com/kite-sdk/kite-examples Why rename?The goal of Kite, and CDK, is to increase the accessibility of the platform. That isn't Cloudera-specific, and we want the name to correctly represent the project as an open, community-driven set of tools.What will this break?The rename mainly affects dependencies and package names. Once imports and dependencies are updated, almost everything should work the same. However, there are a couple of configuration changes to make for anyone using flume or morphlines. The changes are detailed on our migration page.Again, this 0.10.0 release is a rename only. There are no feature changes, and 0.9.0 will be supported as long as 0.10.0, so that

We are pleased to announce version 0.9.0 of the Cloudera Development Kit.The major highlights of 0.9.0 include a new experimental API for working with random access datasets stored in HBase, Crunch Parquet integration, improved support for existing datasets (including those in CSV format), and several new Morphlines commands.Further details are available in the release notes.To get started with the CDK please consult the documentation, source, and the examples.

Hello CDH Users,   We are pleased to announce the release of version 2.5.5 of the Cloudera ODBC Driver for Apache Hive.  This version adds: Ability for clients to securely and more easily authenticate with their username and password Ability to secure client traffic via encrypted SSL Ability for multi-tiered applications to proxy via delegation IDs These drivers work for previous versions of HiveServer2 in CDH 4.1 or higher but the new functionality above is only available in CDH 4.5.   Getting started with the Cloudera ODBC Drivers: Read the the Hive installation guide Download the connector from the Cloudera Components and Connectors page As always, we are happy to hear your feedback. Please send your comments and suggestions to cdh-user@cloudera.org or post to our new Community Forums.   Kind regards, The Cloudera Team

Dear CDH, Cloudera Manager, Cloudera Impala, and Cloudera Search users,   We are pleased to announce the general availability of CDH 4.5, Cloudera Manager 4.8, Cloudera Impala 1.2.1, and Cloudera Search 1.1. Details on each release are below. Please note that if you use Impala and plan to upgrade to either Cloudera Manager 4.8 or Impala 1.2.1, you should plan on upgrading to both versions together as they have a mutual dependency.   CDH 4.5This is an update to the CDH 4 platform and provides a considerable amount of bug-fixes and stability enhancements. Along with these fixes, we have also included some new features, most notable of which are: Hive - ability for clients to securely and more easily authenticate with their username and password Hive - improved security for the metastore with metastore authentication Hue - secure fine-grained, Sentry authorization for end-users of Hue with Impala Parquet 1.x integrated into CDH 4.5 and ready for use across MapReduce, Pig, Impal

We are pleased to announce the beta release of Cloudera Enterprise 5 (CDH 5 and Cloudera Manager 5).  This release has both Impala and Search integrated into CDH. It also includes many new features and updated component versions including the ones below: Hadoop Rebase on Hadoop 2.2 GA HDFS HDFS Snapshots HDFS Caching Preview MR MR2 is production ready MR2 Management by Cloudera Manager MR1 and MR2 are both supported Resource Management/YARN Ability to run MR and Impala with resource sharing Resource Management High Availability Cloudera Manager Platform coverage/support for CDH 5 Resource/Workload management Extensibility support for Partner applications Advanced Impala monitoring HBase Significant reductions to Mean time to Recovery (MTTR) Rebase on HBase 0.95.2+ Impala Language extensibility with native UDFs and UDAF plus support for existing Hive UDFs Automatic and coordinated metadata refresh Preview of YARN-integrated resource management Support for HDFS cachi

We are pleased to announce version 0.8.0 of the Cloudera Development Kit.The major highlights of 0.8.0 include Dataset Repository URIs, Dataset API improvements, support for Parquet 1.2.0, and new metrics commands in Morphlines.Further details are available in the release notes.To get started with the CDK have a look at the documentation, source, and the examples.

To receive email notifications about new releases, simply register and then "Subscribe" to this forum. You'll get all release notices by email.

Dear CDH Users,   We are pleased to announce the general availability of CDH 4.3.2. This is a maintenance update to the CDH4 platform that includes the following fix: HADOOP-9150 - Unnecessary DNS resolution attempts for logical URIs For more information on this release, please see the Release Notes.   Refer to the CDH4 Quick Start Guide or CDH4 Installation Guide for more information on how to install this update release.   As always, we are happy to hear your feedback. Please send your comments and suggestions to cdh-user@cloudera.org or through our new community forums http://community.cloudera.com/.   Thanks,   The Cloudera Team