What's New @ Cloudera

We are pleased to announce the release of Cloudera Enterprise 5.4.10 (CDH 5.4.10, Cloudera Manager 5.4.10, and Cloudera Navigator 2.3.10).  This release fixes key bugs and includes the following:CDH fixes for the following issues:FLUME-2712 - Optional channel errors slows down the Source to Main channel event rateFLUME-2886 - Optional Channels can cause OOMsHDFS-7553 - Fix the TestDFSUpgradeWithHA due to BindExceptionHDFS-9083 - Replication violates block placement policyHDFS-9092 - Nfs silently drops overlapping write requests and causes data copying to failHDFS-9406 - FSImage may get corrupted after deleting snapshotHDFS-9445 - Datanode may deadlock while handling a bad volumeMAPREDUCE-6302 - Incorrect headroom can lead to a deadlock between map and reduce allocationsMAPREDUCE-6460 - TestRMContainerAllocator.testAttemptNotFoundCausesRMCommunicatorException failsYARN-2902 - Killing a container that is localizing can orphan resources in the DOWNLOADING stateYARN-4204 - ConcurrentM

The organizers of HBaseCon, the conference for the Apache HBase community, have published the agenda for the conference (May 24, 2016, in San Francisco)—and once again, the impressive geographical and use-case diversity of HBase are on full display.   See full agenda/register here:   http://hbasecon.com

A new release of the RecordService beta (0.3.0) is now available; see details here:   http://community.cloudera.com/t5/Beta-Releases-Apache-Kudu/ANNOUNCE-RecordService-0-3-0-Released/m-p/39488#M190

Cloudera Enterprise 5.7 is now generally available (comprising CDH 5.7, Cloudera Manager 5.7, and Cloudera Navigator 2.6).   Cloudera is excited to announce the general availability of Cloudera Enterprise 5.7! Main highlights of this release include production-ready Hive-on-Spark functionality, which will help users accelerate their use of Apache Spark as a data processing standard; 2x performance gains for Apache Impala (incubating); easier cluster configuration and utilization reporting; and end-to-end encryption for Apache Spark data.   The release also contains a long list of incremental improvements across the stack, in addition to the usual hundreds of bug fixes (some of which were uncovered during our multi-dimensional hardening/QA process). Here is a partial list of those improvements (see the Release Notes for a full list):   Performance & Scale Hive-on-Spark GA (graduates from Cloudera Labs) 2x performance gains for Impala: Better join ordering and cardina

We are pleased to announce the release of the Cloudera Distribution of Apache Kafka 2.0.1 for CDH 5.  Apache Kafka is a highly scalable, distributed, publish-subscribe messaging system. Apache Kafka 2.0.1 is maintenance release for Kafka 2.0 with bug fixes listed below.Notable Issues Fixed in Cloudera Distribution of Apache Kafka 2.0.1Notable fixes backported into Kafka 2.0.1:KAFKA-3409: MirrorMaker hangs indefinitely due to commit.KAFKA-3378: Client blocks forever if SocketChannel connects instantly.KAFKA-3426: Improve protocol type errors when invalid sizes are received.KAFKA-3330: Truncate log cleaner offset checkpoint if the log is truncated.KAFKA-3463: Change default receive buffer size for consumer to 64K.KAFKA-1148: Delayed fetch/producer requests should be satisfied on a leader change.KAFKA-3352: Avoid DNS reverse lookups.KAFKA-3341: Improve error handling on invalid requests.KAFKA-3310: Fix for NPEs observed when throttling clients.KAFKA-2784: swallow exceptions when Mirr

We are pleased to announce the release of Hive ODBC v2.5.18 and Impala ODBC v2.5.32 drivers. This release has the following fixes and enhancements: Hive ODBC v2.5.18Enhancements & New FeaturesUpdated handling of result set metadataWhen connected to Hive 0.14 or later, the driver now reports result set columns as being updatable, improving compatibility with the DotNet Odbc library and OdbcDataAdapter.Updated default socket timeout thresholdThe default value of the Socket Timeout option (the SocketTimeout key) has been changed from 30 to 60.Resolved IssuesWhen attempting to connect to the server using a non-Windows version of the driver with Service Discovery Mode enabled, the client stops workingWhen executing a parameterized INSERT statement on a DATE, DECIMAL, or TIMESTAMP column, an error occursWhen executing a query that contains a large number of filters using OR operators, an error occurs Impala ODBC v2.5.32Advanced Kerberos authentication mechanism implementedThe driver

Dear Cloudera Users,   We are pleased to announce the general availability of the Cloudera Connector Powered by Teradata 1.5. This release fixes a compatibility issue with CDH 5.5.0 and later. See the download page for more details.   For more details on new features and usage of Cloudera Connector Powered by Teradata, see:   Release Notes Cloudera Connector Powered by Teradata version 1.5 Cloudera Connector Powered by Teradata User Guide, version 1.5 As always, we welcome your feedback. Please send your comments and suggestions through our new community forums.  You can also file bugs in the CDH project at issues.cloudera.org.

The Kudu team is happy to announce the availability of Kudu 0.7.1. Kudu is currently undergoing Incubation at the Apache Software Foundation. Kudu 0.7.1 is a bug fix release only for Kudu 0.7.0. Highlights:Notable bug fixes:KUDU-1325 fixes a tablet server crash that could occur during table deletion. In some cases, while a table was being deleted, other replicas would attempt to re-replicate tablets to servers that had already processed the deletion. This could trigger a race condition that caused a crash.KUDU-1341 fixes a potential data corruption and crash that could happen shortly after tablet server restarts in workloads that repeatedly delete and re-insert rows with the same primary key. In most cases, this corruption affected only a single replica and could be repaired by re-replicating from another.KUDU-1343 fixes a bug in the Java client that occurs when a scanner has to scan multiple batches from one tablet and then start scanning from another. In particular, this would a

The Kudu team is happy to announce the availability of Kudu 0.7.0. Kudu is currently undergoing Incubation at the Apache Software Foundation, with a beta available from Cloudera.   Kudu 0.7.0 includes improvements to client interfaces and metrics, and fixes for several key issues. We’re also releasing a refresh of the Impala Kudu parcel.   Highlights:   New features and improvements: The Python client has been completely rewritten, with a focus on improving code quality and testing. The read path (scanners) has been improved by adding many of the features already supported by the C++ and Java clients. The Python client is no longer considered experimental. KUDU-1321 Spark integration: a new API, kuduRDD,has been added, which wraps newAPIHadoopRDD and includes a default source for Spark SQL. KUDU-1250 The Java client includes new methods countPendingErrors() and getPendingErrors() on KuduSession. These methods allow you to count and retrieve outstanding row errors when

We are pleased to announce the release of Cloudera Enterprise 5.6. This release introduces a pluggable storage interface to allow support and management of data on shared storage appliances instead of on traditional direct attached storage.  EMC's newly released all-flash storage appliance, the DSSD D5, has been fully tested and qualified with this release and is now supported for CDH and Cloudera Manager.   Cloudera Enterprise 5.6.0 is identical to Cloudera Enterprise 5.5.2 except for the support for shared storage as described above. If you do not need DSSD support, you do not need to upgrade if you are already using the latest 5.5.x release (currently 5.5.2).   We look forward to you trying it, using the information below: Download Cloudera Enterprise View the documentation   For more information about the DSSD D5, see this Cloudera VISION Blog post by our Chief Strategy Officer Mike Olson.   As always, we are happy to hear your feedback. Please send y

Dear CDH Users,   We are pleased to announce the release of the Cloudera Distribution of Apache Kafka 2.0 for CDH 5.     Apache Kafka is a highly scalable, distributed, publish-subscribe messaging system. This release is based on Apache Kafka 0.9, and adds security features such as Kerberos authentication, wire encryption, secure mirroring, a new consumer API, per-user throttling, and many other features and bug fixes that solidify Kafka as an enterprise production-grade component of the Hadoop ecosystem. Kafka 2.0 also ships with new management tooling in Cloudera Manager, for point-and-click configuration of each new capability.   New Features in Cloudera Distribution of Apache Kafka 2.0 Kafka is rebased on Apache Kafka 0.9: http://archive.apache.org/dist/kafka/0.9.0.0/RELEASE_NOTES.html. Kerberos authentication of connections from clients and other brokers, including to ZooKeeper. Wire encryption of communications from clients and other brokers using SSL. A new

On February 9th, 2016,  we announced the release of Cloudera Enterprise 5.5.2. Shortly after that, we discovered a severe bug in Cloudera Manager 5.5.2, so we decided to remove the release. CDH 5.5.2 was not affected by this bug.  Today we are releasing CDH 5.5.2, Cloudera Manager 5.5.2 and Cloudera Manager 5.5.3. All improvements, fixes, and changes in Cloudera Manager 5.5.2, as well as a change related to LDAP authentication, are included in Cloudera Manager 5.5.3. If you are upgrading from a previous version of Cloudera Manager, Cloudera recommends that you upgrade to version 5.5.3 or higher.   This release fixes key bugs and includes the following:CDH fixes for the following issues:(HBase) RegionCoprocessorHost System.nanoTime() performance bottleneck.(Hadoop) Exception may be swallowed in KMSClientProvider. (HDFS) Fix the TestDFSUpgradeWithHA due to BindException.(HDFS) Replication violates block placement policy.(HDFS) HttpFS does not parse ACL syntax correctl

On February 9th, 2016,  we announced the release of Cloudera Enterprise 5.5.2. Shortly after that, we discovered a severe bug in Cloudera Manager 5.5.2, so we decided to remove the release. CDH 5.5.2 is not affected by this bug. We will release Cloudera Manager 5.5.3 and CDH 5.5.2 soon.  Cloudera Manager 5.5.3 will include all the improvements, fixes, and changes that were in Cloudera Manager 5.5.2, plus the fix for the severe bug that was discovered.

We are pleased to announce the release of Cloudera Director 2.0.0.Cloudera Director provides a simple, reliable, enterprise-grade way to deploy, scale, and manage Apache Hadoop in the cloud of your choice. Cloudera Director is available as a free download for use with Cloudera Express and is fully supported with Cloudera Enterprise. Cloudera Director 2.0.0 includes the following new features:Support for AWS Spot Instances and Google Cloud Platform's Preemptible InstancesAutomated setup of clusters that are highly available and authenticated via KerberosAbility to automate submission of jobs to clusters with dynamic creation and termination of clustersAbility to run custom scripts after cluster setup and before cluster terminationEnhanced user interface with deeper insights into health of clustersIncreased reliability of cluster modifications, including rollback in some failure scenariosRHEL 7.1 support Links to the product and documentation are listed below:Download Cloudera Direc

We are pleased to announce the release of Hive ODBC v2.5.17 and Impala ODBC v2.5.31 drivers. This release has the following fixes and enhancements: Hive ODBC v2.5.17Support for Debian 6Data encryption and integrity support for the SASL SSPI pluginOptimized retrieval of result set schemasSupport for ODBC 3.80Improved implementation of SQLCancelUpgraded OpenSSL libraryUpgraded ICU libraryIssues resolved:The driver returns incorrect ODBC error code for authentication-related errors.When using the Windows version of the driver, a SideBySide error occurs due to a conflict in the OpenSSL and libcurl libraries being loaded.When attempting to execute a non-parameterized write-back statement multiple times, the driver only executes the statement against Hive once.The driver returns incorrect results for SET <property name> queries.When connected to Hive 0.13 or earlier, the driver cannot insert result sets from queries into tables.The driver does not correctly translate "INSERT OVERWRITE

Dear CDH, Cloudera Manager, and Cloudera Navigator users,We are pleased to announce the release of Cloudera Enterprise 5.3.9 (CDH 5.3.9, Cloudera Manager 5.3.9, Cloudera Navigator 2.2.9).  This release fixes key bugs and includes the following.CDH fixes for the following issues:Apache Commons Collections deserialization vulnerability, described in TSB-98Improve diagnostics information for HttpPutFailedExceptionCheckpointing failure caused by shared KerberosAuthenticatorCopying from the root to a subdirectory should be forbiddenEncryption zone on root not loaded from fsimage after NameNode restartKerberos credentials can expire if the KDC is slow to respondFor a full list of upstream JIRAs fixed in CDH 5.3.9, see the issues fixed section of the Release Notes.Cloudera Manager fixes for the following issues:Apache Commons Collections deserialization vulnerability, described in TSB-98For full list of issues fixed in Cloudera Manager 5.3.9, see the issues fixed section of the Release N

We are pleased to announce the availability of Cloudera Navigator Encrypt 3.8.0.   Navigator Encrypt is a key component of Cloudera's Security and Governance solution.  It complements HDFS encryption to provide encryption at rest for sensitive data in the hadoop cluster in temp/spill files, metadata databases and ingest volumes.      It works together with Navigator Key Trustee, so that encryption keys are managed following security best-practices required by a number of compliance regulations, including PCI and HIPAA.    Navigator Encrypt is available to all Cloudera customers licensed for Navigator use. Here's what's new in this release:Support for Red Hat Enterprise Linux (RHEL) 7.x - starting with 7.1Easier deployment on existing server volumes, through automatic mounting of encrypted loop devices on boot (see Block-Level Encryption with a Loop Device for details)The use of loop devices simplifies adding Navigator Encrypt to ex

Dear CDH, Cloudera Manager, and Cloudera Navigator users,We are pleased to announce the release of Cloudera Enterprise 5.4.9 (CDH 5.4.9, Cloudera Manager 5.4.9, Cloudera Navigator 2.3.9, and Key Trustee Server 5.4.9).  This release fixes key bugs and includes the following.CDH fixes for the following issues:Apache Commons Collections Deserialization Vulnerability, described in TSB - 98Enable using a proxy server to connect to S3aImprove diagnostics information for HttpPutFailedExceptionFix precondition failures from NameNodeEditLogRoller while saving namespaceImprove clearing stale state of Java serializationCheckpointing failure caused by shared KerberosAuthenticatorEncryption zone on root not loaded from fsimage after NN restartException on HBaseSerDe.serialize binary fieldAnalysis exception when a binary operator contains an IN operator with valuesConnection idle time 1 second is too short and the connection is closed too quickly by the ChoreServiceAdded resource path is sent t

Cloudera Enterprise 5.5.1 and Director 1.5.2 provide fixes for critical issues found in the Apache Commons library. For a description of issues fixed in Cloudera Enterprise 5.5.1, see the issues fixed section of the Release Notes for CDH and Cloudera Manager. For a description of the issues fixed in Director 1.5.2, see the issues fixed section of the Release Notes. We look forward to you trying it out, using the information below:Download Cloudera EnterpriseDownload Cloudera Director View the documentation for Cloudera Enterprise and Cloudera Director As always, we are happy to hear your feedback. Please send your comments and suggestions to the user group or through our community forums. You can also file bugs through our external JIRA projects on issues.cloudera.org.

We are pleased to announce the release of the Cloudera Distribution of Apache Kafka 1.4.0 for CDH 5.  Apache Kafka is a distributed publish-subscribe messaging system. This release is based on Apache Kafka 0.8.2, adds support for distribution as a package as well as a parcel, and includes fixes for key issues.   New Features:   Cloudera Distribution of Apache Kafka 1.4 is now distributed via native packages as well as a parcel Notable Fixes:   KAFKA-2633: Default logging from tools to Stderr. KAFKA-1664: Kafka does not properly parse multiple ZK nodes with non-root chroot. KAFKA-2477: Fix a race condition between log append and fetch that causes OffsetOutOfRangeException. KAFKA-2024: Cleaner can generate unindexable log segments. KAFKA-2118: Cleaner can not clean after shutdown during replaceSegments. We look forward to you trying it out, using the information below:   Install or upgrade Kafka Review the Documentation Review the Release No

Today, we’re pleased to announce the availability of a Cloudera QuickStart Docker image!   If you or your organization is using Docker, this image may provide the ideal lightweight, disposable environment for learning and exploring new technology, playing with new ideas, and for doing continuous integration before testing at scale. (However, Cloudera recommends using a more realistic test environment before moving to production.)   More details/docs here:   http://blog.cloudera.com/blog/2015/12/docker-is-the-new-quickstart-option-for-apache-hadoop-and-cloudera/

We are happy to announce Kudu 0.6.0, the second beta release of Kudu, the industry’s first columnar storage manager developed for the Hadoop ecosystem. This release includes the following fixes and enhancements: The Java client's CreateTableBuilder and AlterTableBuilder classes have been renamed to CreateTableOptions and AlterTableOptions. Their methods now also return these objects, allowing them to be used as builders. The Java client's AbstractKuduScannerBuilder#maxNumBytes() setter is now called batchSizeBytes, as is the corresponding property in AsyncKuduScanner. This makes it consistent with the C++ client. The kudu-admin tool can now list and delete tables: kudu-admin listkudu-admin delete <table_name> OS X is now supported for single-node development For a full list of issues addressed in Kudu 0.6.0, see this JIRA query. For more details on Kudu, see http://getkudu.io/ Kudu already has an extensive set of capabilities, but there’s still work to be don

Cloudera Enterprise 5.5 is now generally available (comprising CDH 5.5, Cloudera Manager 5.5, and Cloudera Navigator 2.4). Cloudera is excited to bring you news of Cloudera Enterprise 5.5 (Release Notes). Our persistent emphasis on quality is especially pronounced in this release, with more than 500 issues identified and triaged during its development. Here are some of the highlights (see the Release Notes for full lists of features and fixes):SecurityColumn-level security is now provided in Impala and Apache Hive (via Apache Sentry [incubating]).In Cloudera Manager-managed clusters, cleartext passwords can now be redacted from configuration files.Cloudera Manager includes a new wizard for setting up HDFS encryption, KMS, and Navigator Key Trustee.In Cloudera Navigator Encrypt, dmcrypt+loopfile replaces eCryptfs (deprecated) for file encryption.LDAP/AD auth is now supported for Apache Solr clients.Apache HBase replication is now encrypted.Performance, Scale, and OperationsHDF

We are pleased to announce the release of Impala JDBC v2.5.29 driver. This release has the following fixes and enhancements: Issues resolved:When executing a query with parameter in Native query mode,the setObject operation fails. This issue has been resolvedexecuteBatch() operation does not work properly when multiple parameter sets are added to the batch. Only the first insert is working. This issue has been resolved. Please see Release Notes for JDBC for details on all of the fixes. Getting started with the Cloudera Drivers:Read the Cloudera JDBC 2.5.29  Driver for Impala Release Notes and Installation Guide.Download the connector from the Cloudera Connectors page.As always, we welcome your feedback. Please send your comments and suggestions to the user group or through our community forums.  You can also file bugs through our external JIRA projects on issues.cloudera.org

We are pleased to announce the release of Cloudera Enterprise 5.4.8 (CDH 5.4.8, Cloudera Manager 5.4.8, and Cloudera Navigator 2.3.8).     Cloudera Enterprise 5.4.8 This release fixes key bugs and includes the following. CDH fixes for the following issues: Handling NullPointer in ZKProcedureMemberRpcs while getting ZNode data. "Alter index rebuild" statements submitted through HiveServer2 fail when Sentry is enabled. Duplicate groupby keys cause ClassCastException. IndexOutOfBounds compiling query with duplicated groupby keys. Indexing threads stuck during network partition can put leader into down state. For a full list of upstream JIRAs fixed in CDH 5.4.8, see the issues fixed section of the Release Notes.   Cloudera Manager fixes for the following issues: Clusters can fail if Java 1.6 is installed. OutOfMemory error when running HDFS replication. Cloudera Management Service can fail with a large Flume configuration file. Client configuration