What's New @ Cloudera

Dear CDH, Cloudera Manager, and Cloudera Navigator users, We are pleased to announce the release of Cloudera Enterprise 5.9.2 (CDH 5.9.2, and Cloudera Manager 5.9.2.)This release fixes key bugs and includes the following: CDH fixes for the following issues:FLUME-2812 - Fix semaphore leak causing java.lang.Error: Maximum permit count exceeded in MemoryChannelHBASE-15378 - Scanner cannot handle heartbeat message with no resultsHBASE-15931 - Add log for long-running tasks in AsyncProcess HBASE-16289 AsyncProcess stuck messages need to print region/serverHBASE-16304 - HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock when trying to obtain write lock on updatesLockHDFS-10763 - Open files can leak permanently due to inconsistent lease updateHIVE-12465 - Hive might produce wrong results when (outer) joins are mergedHIVE-12768 - Thread safety: binary sortable serde decimal deserializationHIVE-13539 - HiveHFileOutputFormat searching the wrong directory for HFi

We are pleased to announce the release of Cloudera Director 2.4.0. Cloudera Director provides a simple, reliable, enterprise-grade way to deploy, scale, and manage Apache Hadoop in the cloud of your choice. Cloudera Director is available as a free download for use with Cloudera Express and is fully supported with Cloudera Enterprise. Cloudera Director 2.4.0 includes the following new features: Improved support for long-running clusters: Cloudera Director 2.4 will support upgrades, change of services, and configurations from Cloudera Manager 5.11.Spark 2 supportKudu supportFor Azure clusters, support for Azure Data Lake Store (ADLS) via Cloudera Manager and CDH 5.11 deployments.Support for the following Microsoft Azure Clouds and Regions:Azure Germany: Germany Central and Germany NortheastAzure Government: US DoD East and US DoD Central Use the following links to download the product and view the documentation: Download:Download Cloudera DirectorDocumentation:Clouder

Cloudera is pleased to announce that Cloudera Enterprise 5.11 is now generally available (GA). The highlights of this release include lineage support for Apache Spark, Apache Kudu security integration, embedded data discovery for self-service BI, and new cloud capabilities for Microsoft ADLS and Amazon S3. As usual, there are also a number of quality enhancements, bug fixes, and other improvements across the stack. Here is a partial list of what’s included (see the Release Notes for a full list): Core Platform and CloudAmazon S3 Consistency: S3Guard ensures that operations on Amazon S3 are immediately visible to other clients, making it easier to migrate workloads from consistent file systems like HDFS to Amazon S3.Support for Azure Data Lake Store (ADLS): Microsoft introduced ADLS to provide a cost-effective, persistent storage layer for big data applications. With  C5.11, Hive, Spark, and MapReduce can all directly operate on data stored in ADLS, enabling separation of

We are happy to announce Apache Spark 2.1 release 1. You can download the parcel and apply it directly to provisioned clusters without disrupting your currently running Spark workloads.Cloudera Distribution of Apache Spark 2.1 release 1 is compatible with the following CDH versions: CDH 5.7, CDH 5.8, CDH 5.9, CDH 5.10. What's New in Cloudera Distribution of Apache Spark 2.1 Release 1 New direct connector to Kafka that uses the new Kafka consumer API. See Spark 2 Kafka Integration for details. Issues Fixed in Cloudera Distribution of Apache Spark 2.1 - Release 1[SPARK-19554][UI,YARN] Allow SHS URL to be used for tracking in YARN RM.[SPARK-16554][CORE] Automatically Kill Executors and Nodes when they are Blacklisted[SPARK-16654][CORE] Add UI coverage for Application Level Blacklisting[SPARK-8425][CORE] Application Level Blacklisting[SPARK-18117][CORE] Add test for TaskSetBlacklist[SPARK-18949][SQL][BACKPORT-2.1] Add recoverPartitions API to Catalog[SPARK-19459][SQL][BRANCH

Cloudera is happy to announce the availability of parcels and packages for Apache Kudu 1.2.0/ CDH 5.10.1. Apache Kudu 1.2.0/ CDH 5.10.1 is a bug fix release, with no new features or backwards incompatible changes. To upgrade Kudu to 1.2.0/ CDH 5.10.1, see Upgrade Parcels or Upgrade Packages. Issues Fixed in Kudu 1.2.0/ CDH 5.10.1 KUDU-1904 - Fixed a bug where RLE columns with only NULL values would crash on scan.KUDU-1899 - Fixed an issue where tablet servers would crash after inserting an empty string primary key ("").KUDU-1851 - Fixed an issue with the Python client which would crash whenever a TableAlterer is instantiated directly.KUDU-1852 - KuduTableAlterer will no longer crash when given nullptr range bound arguments.KUDU-1821 - Improved warnings when the catalog manager startsFor a complete list of changes, bug fixes, and known issues, see the Kudu 1.2.0/ CDH 5.10.1 Release Notes. As always, your feedback is appreciated. For general Kudu questions, visit the

We are pleased to announce the release of the Cloudera Distribution of Apache Kafka 2.1.1 for CDH 5.  Apache Kafka is a highly scalable, distributed, publish-subscribe messaging system. Apache Kafka 2.1.1 is maintenance release for Kafka 2.1.0. Notable Issues Fixed in Cloudera Distribution of Apache Kafka 2.1.1 KAFKA-3965: MirrorMaker should not commit offset when exception is thrown from producer.sendKAFKA-4158: Reset quota to default value if quota override is deletedKAFKA-2720: Expire group metadata when all offsets have expiredKAFKA-3810: replication of internal topics should not be limited by replica.fetch.max.bytesAll backported fixes can be viewed in the git release notes here or on our website under the Issues fixed section.  We look forward to you trying Kafka 2.1.1. For more information, please use the links below:Install or upgrade KafkaReview the documentationReview the Release Notes As always, we welcome your feedback. Please send your co

Dear CDH, Cloudera Manager, and Cloudera Navigator users, We are pleased to announce the release of Cloudera Enterprise 5.10.1 (CDH 5.10.1, Cloudera Manager 5.10.1, Cloudera Navigator 2.9.1, and Key Trustee Server 5.10.1). This release fixes key bugs and includes the following: CDH fixes for the following issues:FLUME-2812 - Fix semaphore leak causing java.lang.Error: Maximum permit count exceeded in MemoryChannel.HBASE-15378 - Scanner cannot handle heartbeat message with no results.HBASE-15931 - Add log for long-running tasks in AsyncProcess HBASE-16289 HDFS-11379 - DFSInputStream may infinite loop requesting block locations. AsyncProcess stuck messages need to print region/server.HIVE-12465 - Hive might produce wrong results when (outer) joins are merged.HIVE-15338 - Wrong result from non-vectorized DATEDIFF with scalar parameter of type DATE/TIMESTAMP.IMPALA-4263 - Fix wrong omission of agg/analytic hash exchanges.IMPALA-4282 - Remove max length check for type strings

We are pleased to announce the release of Impala JDBC v2.5.37 and Impala ODBC v2.5.37 drivers. This release has the following fixes and enhancements:  Cloudera JDBC Driver for Impala 2.5.37Enhancements & New Features Specify asynchronous exec poll interval. You can now specify the time in milliseconds between each poll that the driver makes for the query execution status. Specify the number of milliseconds in the Advanced Options dialog box in the Async Exec Poll Interval field, or in the AsyncExecPollInterval configuration option.Support for Impala 2.8. The driver now supports Impala versions 1.0.1 through 2.8.Resolved IssuesUpdate/Delete statements require a semicolon at the end.Conflicting information in documentation regarding CDH. Cloudera ODBC Driver for Impala 2.5.37Enhancements & New FeaturesSpecify asynchronous exec poll interval. You can now specify the time in milliseconds between each poll that the driver makes for the query execution status. Specify

We are pleased to announce the general availability of the Cloudera Connector for Netezza 1.5c5. New Features in Cloudera Connector for Netezza 1.5c5 Fixed import in direct mode with multi-schema by removing unnecessary object owner check.Added fail fast scenario for import with --as-parquetfile in direct mode.Added fail fast scenario for import with --as-avrodatafile in direct mode.For more details on new features and usage of Cloudera Connector for Netezza, see: Release Notes Cloudera Connector for Netezza Version 1.5c5Cloudera Connector for Netezza User Guide, version 1.5c5 As always, we welcome your feedback. Please send your comments and suggestions through our new community forums.  You can also file bugs in the CDH project at issues.cloudera.org.

Dear CDH, Cloudera Manager, and Cloudera Navigator users, We are pleased to announce the release of Cloudera Enterprise 5.7.6 (CDH 5.7.6, Cloudera Manager 5.7.6, and Cloudera Navigator 2.6.6). This release fixes key bugs and includes the following: CDH fixes for the following issues: HADOOP-13838 - KMSTokenRenewer should close providersHBASE-15430 - Failed taking snapshot - Manifest proto-message too largeHDFS-10271 - Extra bytes are getting released from reservedSpace for appendHDFS-10763 - Open files can leak permanently due to inconsistent lease updateHDFS-11229 - HDFS-11056 failed to close meta fileHIVE-12465 - Hive might produce wrong results when (outer) joins are mergedHIVE-13539 - HiveHFileOutputFormat searching the wrong directory for HFilesHIVE-15338 - Wrong result from non-vectorized DATEDIFF with scalar parameter of type DATE/TIMESTAMPHUE-4466 - [security] Deliver csrftoken cookie with secure bit set if possibleIMPALA-3875 - Thrift threaded server hangs

We are happy to announce Spark 2.0 release 2. You can download the parcel and apply it directly to provisioned clusters without disrupting your currently running Spark workloads. Release 2 addresses a Hive compatibility issue that affects CDH 5.10.1 and higher, CDH 5.9.2 and higher, CDH 5.8.5 and higher, and CDH 5.7.6 and higher. If you are using one of these CDH versions, you must upgrade to the Spark 2.0 release 2 parcel to avoid Spark 2 job failures when using Hive. Release 2 is based on Apache Spark 2.0.2. Issues Fixed in Cloudera Distribution of Apache Spark 2.0 Release 2[SPARK-4563] [CORE] Allow driver to advertise a different network address[SPARK-18993] Unable to build/compile Spark in IntelliJ due to missing Scala deps in spark-tags[SPARK-19314] Do not allow sort before aggregation in Structured Streaming plan[SPARK-18762] Web UI should be http:4040 instead of https:4040[SPARK-18745] java.lang.IndexOutOfBoundsException running query 68 Spark SQL on (100TB)[SPARK

Dear CDH, Cloudera Manager, and Cloudera Navigator users, We are pleased to announce the release of Cloudera Enterprise 5.8.4 (CDH 5.8.4 and Cloudera Manager 5.8.4).This release fixes key bugs and includes the following: CDH fixes for the following issues: FLUME-2812 - Fix semaphore leak causing java.lang.Error: Maximum permit count exceeded in MemoryChannelHADOOP-12483 - Maintain wrapped SASL ordering for postponed IPC responsesHADOOP-13838 - KMSTokenRenewer should close providersHBASE-15430 - Failed taking snapshot - Manifest proto-message too largeHBASE-17072 - CPU usage starts to climb up to 90-100% when using G1GCHDFS-6962 - ACLs inheritance conflict with umaskmodeHDFS-10763 - Open files can leak permanently due to inconsistent lease updateHIVE-12465 - Hive might produce wrong results when (outer) joins are mergedHIVE-13539 - HiveHFileOutputFormat searching the wrong directory for HFilesHUE-4546 - Auto-strip invalid characters from name field of converted docsIMPALA

We are pleased to announce the release of Cloudera Director 2.3.0. Cloudera Director provides a simple, reliable, enterprise-grade way to deploy, scale, and manage Apache Hadoop in the cloud of your choice. Cloudera Director is available as a free download for use with Cloudera Express and is fully supported with Cloudera Enterprise. Cloudera Director 2.3.0 includes the following new features: Support for AWS Spot Blocks. Instances in AWS Spot blocks are guaranteed to remain available during cluster bootstrap, so that the process is resilient to fluctuations in AWS Spot Instance availability.More control of cluster configuration with the introduction of instance-level post-creation scripts, in addition to existing cluster-level post-creation scripts.Updated look and feel of the user interface.AWSNew regions: ap-south-1, ca-central-1, eu-west-1Many new instance typesGCPNew regions: asia-northeast1, us-west1, us-east1AzureNew instance types: DS15_v2, GS4, GS5New OS image: CentO

Cloudera is proud to announce that Cloudera Enterprise 5.10 is now generally available (GA).The highlights of this release include the GA of the new columnar storage engine Apache Kudu, improved cloud performance and cost-optimizations, and cloud-native data governance for Amazon S3.As usual, there are also a number of quality enhancements and bug fixes (learn more about our multi-dimensional hardening/QA process) and other improvements across the stack. Here is a partial list of what’s included (see the Release Notes for a full list): GA of Apache Kudu - Unleash Cloudera’s new storage engine to enable fast analytics on fast changing data with the first generally available release of Kudu. Kudu is purpose-built to enable use cases for time series data, machine data analytics, and online reporting—as part of a complete analytic or operational database.Improved Cloud Performance - Run Cloudera workloads on public cloud infrastructure more efficiently and cost-effectively than ever b

Cloudera takes cluster security very seriously, and provides guidelines for securing CDH environments:   http://www.cloudera.com/documentation/enterprise/latest/topics/sg_edh_overview.html   Security measures become especially important when clusters are exposed to the internet. For example, hackers using network scanning tools are actively looking for WebHDFS ports on clusters, and when they find an open port, they can wreak havoc on the cluster (delete data, steal data, corrupt data).   There are many other services and access considerations that should be protected as well.   It’s imperative that you design clusters in a secure fashion which will not leave the services interfaces exposed to the Internet this way. It’s our strong recommendation that you secure your cluster with kerberos, TLS, proper firewall or proxy access, and use the guidance from our security guide to protect your deployment.   Users affected: All unsecured clusters exposed to the interne

Dear CDH, Cloudera Manager, and Cloudera Navigator users, We are pleased to announce the release of Cloudera Enterprise 5.9.1 (CDH 5.9.1, and Cloudera Manager 5.9.1, and Cloudera Navigator 2.8.1). This release fixes key bugs and includes the following: CDH fixes for the following issues: HADOOP-12483 - Maintain wrapped SASL ordering for postponed IPC responsesHADOOP-13353 - LdapGroupsMapping getPassward should not return null when IOException throwsHBASE-15430 - Failed taking snapshot - Manifest proto-message too largeHBASE-15856 - Do not cache unresolved addresses for connectionsHDFS-6962 - ACLs inheritance conflict with umaskmodeHDFS-8269 - getBlockLocations() does not resolve the .reserved path and generates incorrect edit logs when updating the atimeHUE-5028 - Share Oozie workflow with modify permission; however, the user can't edit the shared WFHUE-5163 - Speed up initial page renderingHUE-5041 - Hue export large file to HDFS doesn't work on non-default databas

Apache Spark is a core component of the Cloudera Enterprise platform. It is the de facto processing engine for Hadoop and the modern analytics engine for an increasing number of workloads. Organizations leverage Apache Spark to reduce churn, implement predictive maintenance, and perform complex risk modeling and analysis. IT professionals leverage Spark to accelerate data processing, train large-scale machine learning models, and perform exploratory data science. Taneja reports that for the most critical Spark workloads, 57% of users choose to partner with Cloudera because of the quality of support and breadth of training and services.  The Apache Spark ecosystem continues to grow at a fast pace, and Cloudera delivers the newest, most desired features with  reliability and performance at scale. We are happy to announce support for Apache Spark version 2.0. CDH users can download the parcel and apply it directly to provisioned clusters. You can leverage Spark 2.0 wit

Dear CDH and Cloudera Manager users, We are pleased to announce the release of Cloudera Enterprise 5.5.6 (CDH 5.5.6 and Cloudera Manager 5.5.6). This release fixes key bugs and includes the following:     CDH fixes for the following issues: HADOOP-12483 - Maintain wrapped SASL ordering for postponed IPC responsesHDFS-9290 - DFSClient#callAppend() is not backward compatible for slightly older NameNodesHDFS-10178 - Permanent write failures can happen if pipeline recoveries occur for the first packetHBASE-14241- Fix deadlock during cluster shutdown due to concurrent connection closeHBASE-14474 - DeadLock in RpcClientImpl.Connection.close()HBASE-15430 - Failed taking snapshot - Manifest proto-message too largeIMPALA-3378 - HiveUdfCall::Open() produces unsynchronized access to JniUtil::global_refs_ vectorIMPALA-3379 - HBaseTableWriter::CreatePutList() produces unsynchronized access to JniUtil::global_refs_ vectorIMPALA-3698 - Fix Isilon permissions testIMPALA-3

We are pleased to announce the release of the Cloudera JDBC Driver for Apache Hive v2.5.18. This release includes the following fixes and enhancements: When using Kerberos authentication, you can now configure the KrbAuthType connection property to specify how the driver obtains the Kerberos Subject. For more information, see the Installation and Configuration Guide.In some cases, when the heap size is restricted, the driver retrieves fewer rows than expected but does not return any exceptions or errors. This issue has been resolved. The driver now retrieves the correct number of rows when the heap size is restricted. See Release Notes for details on all of the fixes and enhancements.  Getting started with the Cloudera Drivers:Read the Cloudera JDBC v2.5.18 Driver for Hive Release Notes and Installation Guide.Download the connector from the Cloudera Connectors page. As always, we welcome your feedback. Please send your comments and suggestions to the user group or throug

We are pleased to announce the release of Hive ODBC v2.5.21, Impala ODBC v2.5.36, and Impala JDBC v2.5.35. Resolved issues in Impala JDBC 2.5.35The driver does not properly detect a TOP statement and inserts a TOP 0 before it to generates the limit 0 query. This happens if there is an additional space in the query before the original TOP. This issue has been resolved. Queries with YEAR and ISO_WEEK functions failing due to a data type mismatch. This issue has been resolved.Please see below links for details on all of the fixes and enhancements Release NotesInstallation GuideEnhancements and new features in Impala ODBC 2.5.36Support added for Kerberos delegation You now have the option for the driver to forward the user credentials to the server when using Kerberos authentication. Support for Windows Trust Store You now have the option to use the CA certificates in the Windows Trust Store for server verification when using SSL. Auto Reconnect You can configure the driver to automaticall

Cloudera is excited to announce the general availability of Cloudera Enterprise 5.9! Cloudera Enterprise 5.9 contains a long list of new features, quality enhancements, bug fixes, and other improvements across the stack. Here is a partial list of those improvements; see the Release Notes for a full list: What's New In CDH 5.9.0 Apache HadoopYou can use temporary credentials to log in to Amazon S3 and obtain temporary credentials from Amazon's Security Token Service (STS).Apache HBaseA tool has been added to dump existing replication peers, configurations, and queues when using HBase replication. For more information, see Class DumpReplicationQueues.Metrics have been added that expose the amount of replayed work occurring in the HBase replication system. For more information, see Replication Metrics in the Apache HBase Reference Guide.Apache HiveHIVE-14270 : Added parameters to optimize write performance for Hive tables and partitions that are stored on Amazon S3. See Optimizing Hi

Dear CDH, Cloudera Manager, and Cloudera Navigator users, We are pleased to announce the release of Cloudera Enterprise 5.7.5 (CDH 5.7.5, Cloudera Manager 5.7.5, and Cloudera Navigator 2.6.5). This release fixes key bugs and includes the following: CDH fixes for the following issues: HBASE-16699 - Overflows in AverageIntervalRateLimiter's refill() and getWaitInterval()HDFS-6962 - ACLs inheritance conflict with umaskmodeHDFS-8224 - Schedule a block for scanning if its metadata file is corruptHDFS-9038 - DFS reserved space is erroneously counted towards non-DFS usedHDFS-10178 - Permanent write failures can happen if pipeline recoveries occur for the first packetHIVE-12077 - MSCK Repair table should fix partitions in batchesHIVE-13997 - Insert overwrite directory does not overwrite existing filesHIVE-14799 - Query operations are not thread safe during cancellationHUE-5042 - Unable to kill jobs after ResourceManager failoverIMPALA-1619 - Support 64-bit allocationsIMPALA

We are pleased to announce the general availability of the Cloudera Connector for Netezza 1.4c5. New Features in Cloudera Connector for Netezza 1.4c5Added the escapeColName method to handle, for example, column names with lowercase characters.For more details on new features and usage of Cloudera Connector for Netezza, see:Release Notes Cloudera Connector for Netezza Version 1.4c5Cloudera Connector for Netezza User Guide, version 1.4c5As always, we welcome your feedback. Please send your comments and suggestions through our new community forums.  You can also file bugs in the CDH project at issues.cloudera.org.

We are pleased to announce the release of Cloudera Director 2.2.0. Cloudera Director provides a simple, reliable, enterprise-grade way to deploy, scale, and manage Apache Hadoop in the cloud of your choice. Cloudera Director is available as a free download for use with Cloudera Express and is fully supported with Cloudera Enterprise. Cloudera Director 2.2.0 includes the following new features: AWS EBS volume support Azure Standard Storage supportSupport for Cloudera 5.9 out of the boxCollection of diagnostic logsImproved validation and usability of configuration files Use the following links to download the product and view the documentation:Download Cloudera Director from http://www.cloudera.com/content/support/en/downloads.htmlView the documentation:Cloudera Director Release NotesGetting StartedInstalling Cloudera Director ClientInstalling Cloudera Director Server As always, we are happy to hear your feedback. Please post your questions and comments on our communi

Dear CDH, Cloudera Manager, and Cloudera Navigator users,   We are pleased to announce the release of Cloudera Enterprise 5.8.3 (CDH 5.8.3, Cloudera Manager 5.8.3, and Cloudera Navigator 2.7.3).   This release fixes key bugs and includes the following:   CDH fixes for the following issues:   HADOOP-13558 - UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket HBASE-16699 - Overflows in AverageIntervalRateLimiter's refill() and getWaitInterval() HDFS-8224 - Schedule a block for scanning if its metadata file is corrupt HIVE-10965 - Direct SQL for stats fails in 0-column case HUE-4891 - An unprivileged user can list document items IMPALA-4135 - Thrift threaded server times out connections during high load MAPREDUCE-6628 - Potential memory leak in CryptoOutputStream SPARK-12941 - Spark-SQL JDBC Oracle dialect fails to map string datatypes to Oracle VARCHAR datatype SQOOP-3021 - ClassWriter fails if a column name contains a backsl