Announcements

Apache Spark is a core component of the Cloudera Enterprise platform. It is the de facto processing engine for Hadoop and the modern analytics engine for an increasing number of workloads. Organizations leverage Apache Spark to reduce churn, implement predictive maintenance, and perform complex risk modeling and analysis. IT professionals leverage Spark to accelerate data processing, train large-scale machine learning models, and perform exploratory data science. Taneja reports that for the most critical Spark workloads, 57% of users choose to partner with Cloudera because of the quality of support and breadth of training and services.  The Apache Spark ecosystem continues to grow at a fast pace, and Cloudera delivers the newest, most desired features with  reliability and performance at scale. We are happy to announce support for Apache Spark version 2.0. CDH users can download the parcel and apply it directly to provisioned clusters. You can leverage Spark 2.0 wit

Dear CDH and Cloudera Manager users, We are pleased to announce the release of Cloudera Enterprise 5.5.6 (CDH 5.5.6 and Cloudera Manager 5.5.6). This release fixes key bugs and includes the following:     CDH fixes for the following issues: HADOOP-12483 - Maintain wrapped SASL ordering for postponed IPC responsesHDFS-9290 - DFSClient#callAppend() is not backward compatible for slightly older NameNodesHDFS-10178 - Permanent write failures can happen if pipeline recoveries occur for the first packetHBASE-14241- Fix deadlock during cluster shutdown due to concurrent connection closeHBASE-14474 - DeadLock in RpcClientImpl.Connection.close()HBASE-15430 - Failed taking snapshot - Manifest proto-message too largeIMPALA-3378 - HiveUdfCall::Open() produces unsynchronized access to JniUtil::global_refs_ vectorIMPALA-3379 - HBaseTableWriter::CreatePutList() produces unsynchronized access to JniUtil::global_refs_ vectorIMPALA-3698 - Fix Isilon permissions testIMPALA-3

We are pleased to announce the release of the Cloudera JDBC Driver for Apache Hive v2.5.18. This release includes the following fixes and enhancements: When using Kerberos authentication, you can now configure the KrbAuthType connection property to specify how the driver obtains the Kerberos Subject. For more information, see the Installation and Configuration Guide.In some cases, when the heap size is restricted, the driver retrieves fewer rows than expected but does not return any exceptions or errors. This issue has been resolved. The driver now retrieves the correct number of rows when the heap size is restricted. See Release Notes for details on all of the fixes and enhancements.  Getting started with the Cloudera Drivers:Read the Cloudera JDBC v2.5.18 Driver for Hive Release Notes and Installation Guide.Download the connector from the Cloudera Connectors page. As always, we welcome your feedback. Please send your comments and suggestions to the user group or throug

We are pleased to announce the release of Hive ODBC v2.5.21, Impala ODBC v2.5.36, and Impala JDBC v2.5.35. Resolved issues in Impala JDBC 2.5.35The driver does not properly detect a TOP statement and inserts a TOP 0 before it to generates the limit 0 query. This happens if there is an additional space in the query before the original TOP. This issue has been resolved. Queries with YEAR and ISO_WEEK functions failing due to a data type mismatch. This issue has been resolved.Please see below links for details on all of the fixes and enhancements Release NotesInstallation GuideEnhancements and new features in Impala ODBC 2.5.36Support added for Kerberos delegation You now have the option for the driver to forward the user credentials to the server when using Kerberos authentication. Support for Windows Trust Store You now have the option to use the CA certificates in the Windows Trust Store for server verification when using SSL. Auto Reconnect You can configure the driver to automaticall

Cloudera is excited to announce the general availability of Cloudera Enterprise 5.9! Cloudera Enterprise 5.9 contains a long list of new features, quality enhancements, bug fixes, and other improvements across the stack. Here is a partial list of those improvements; see the Release Notes for a full list: What's New In CDH 5.9.0 Apache HadoopYou can use temporary credentials to log in to Amazon S3 and obtain temporary credentials from Amazon's Security Token Service (STS).Apache HBaseA tool has been added to dump existing replication peers, configurations, and queues when using HBase replication. For more information, see Class DumpReplicationQueues.Metrics have been added that expose the amount of replayed work occurring in the HBase replication system. For more information, see Replication Metrics in the Apache HBase Reference Guide.Apache HiveHIVE-14270 : Added parameters to optimize write performance for Hive tables and partitions that are stored on Amazon S3. See Optimizing Hi

Dear CDH, Cloudera Manager, and Cloudera Navigator users, We are pleased to announce the release of Cloudera Enterprise 5.7.5 (CDH 5.7.5, Cloudera Manager 5.7.5, and Cloudera Navigator 2.6.5). This release fixes key bugs and includes the following: CDH fixes for the following issues: HBASE-16699 - Overflows in AverageIntervalRateLimiter's refill() and getWaitInterval()HDFS-6962 - ACLs inheritance conflict with umaskmodeHDFS-8224 - Schedule a block for scanning if its metadata file is corruptHDFS-9038 - DFS reserved space is erroneously counted towards non-DFS usedHDFS-10178 - Permanent write failures can happen if pipeline recoveries occur for the first packetHIVE-12077 - MSCK Repair table should fix partitions in batchesHIVE-13997 - Insert overwrite directory does not overwrite existing filesHIVE-14799 - Query operations are not thread safe during cancellationHUE-5042 - Unable to kill jobs after ResourceManager failoverIMPALA-1619 - Support 64-bit allocationsIMPALA

We are pleased to announce the general availability of the Cloudera Connector for Netezza 1.4c5. New Features in Cloudera Connector for Netezza 1.4c5Added the escapeColName method to handle, for example, column names with lowercase characters.For more details on new features and usage of Cloudera Connector for Netezza, see:Release Notes Cloudera Connector for Netezza Version 1.4c5Cloudera Connector for Netezza User Guide, version 1.4c5As always, we welcome your feedback. Please send your comments and suggestions through our new community forums.  You can also file bugs in the CDH project at issues.cloudera.org.

We are pleased to announce the release of Cloudera Director 2.2.0. Cloudera Director provides a simple, reliable, enterprise-grade way to deploy, scale, and manage Apache Hadoop in the cloud of your choice. Cloudera Director is available as a free download for use with Cloudera Express and is fully supported with Cloudera Enterprise. Cloudera Director 2.2.0 includes the following new features: AWS EBS volume support Azure Standard Storage supportSupport for Cloudera 5.9 out of the boxCollection of diagnostic logsImproved validation and usability of configuration files Use the following links to download the product and view the documentation:Download Cloudera Director from http://www.cloudera.com/content/support/en/downloads.htmlView the documentation:Cloudera Director Release NotesGetting StartedInstalling Cloudera Director ClientInstalling Cloudera Director Server As always, we are happy to hear your feedback. Please post your questions and comments on our communi

Dear CDH, Cloudera Manager, and Cloudera Navigator users,   We are pleased to announce the release of Cloudera Enterprise 5.8.3 (CDH 5.8.3, Cloudera Manager 5.8.3, and Cloudera Navigator 2.7.3).   This release fixes key bugs and includes the following:   CDH fixes for the following issues:   HADOOP-13558 - UserGroupInformation created from a Subject incorrectly tries to renew the Kerberos ticket HBASE-16699 - Overflows in AverageIntervalRateLimiter's refill() and getWaitInterval() HDFS-8224 - Schedule a block for scanning if its metadata file is corrupt HIVE-10965 - Direct SQL for stats fails in 0-column case HUE-4891 - An unprivileged user can list document items IMPALA-4135 - Thrift threaded server times out connections during high load MAPREDUCE-6628 - Potential memory leak in CryptoOutputStream SPARK-12941 - Spark-SQL JDBC Oracle dialect fails to map string datatypes to Oracle VARCHAR datatype SQOOP-3021 - ClassWriter fails if a column name contains a backsl

We are pleased to announce the general availability of the Cloudera Connector Powered by Teradata 1.6c5.  New Features in Cloudera Connector Powered by Teradata Version 1.6c5Upgrades the JDBC driver to version 15.10.00.22 and the TDCH library to version 1.5.0. These libraries contain several bug fixes and improvements.Adds the --schema argument, used to override the <td-instance> value in the connection string of the Sqoop command. For example, if the connection string in the Sqoop command is jdbc:teradata:// <td-host>/DATABASE=database1, but you specify --schema database2, your data is imported from database2 and not database1. If the connection string does not contain the DATABASE parameter — for example jdbc:teradata://<td-host>/CHARSET=UTF8) — you can also use the --schema databasename argument to have Sqoop behave as if you specified the jdbc:teradata://<td-host>/DATABASE=databasename,CHARSET=UTF8 connection string. For more details on n

Cloudera is pleased to announce a new minor release of our supported packaging of Apache Accumulo for use on CDH 5.This release adds support for:Kerberos authentication for Accumulo ClientsRHEL 7 / CentOS 7 supportRunning Accumulo on top of Apache HDFS Transparent EncryptionAccumuloStorgeHandler for Apache Hive access to data stored in AccumuloApache Spark processing of data stored in Accumulo through AccumuloInputFormat and AccumuloOutputFormatUsers are cautioned to read the installation guide for known issues and unsupported upstream features.Existing users should see a new parcel available for Accumulo 1.7.2-cdh5.5.0.This release is supported with Cloudera Manager and CDH versions 5.5.0 or later. Cloudera recommends using the latest version of Cloudera Manager available.Docs: http://tiny.cloudera.com/accumulo-docsCloudera Manager 5.8.2 Download: http://tiny.cloudera.com/cm-5.8.2Accumulo 1.7.2 on CDH 5 (CDH 5.5.0 or later): http://archive.cloudera.com/accumulo-c5/List of JIRAs includ

We are pleased to announce the release of Cloudera Enterprise 5.7.4 (CDH 5.7.4, Cloudera Manager 5.7.4, Cloudera Navigator 2.6.4, and Key Trustee KMS 5.7.4). This release fixes key bugs and includes the following:CDH fixes for the following issues:HADOOP-8436 - NPE In getLocalPathForWrite ( path, conf ) when the required context item is not configuredHDFS-8269 - getBlockLocations() does not resolve the .reserved path and generates incorrect edit logs when updating the atimeHDFS-9781 - FsDatasetImpl#getBlockReports can occasionally throw NullPointerExceptionMAPREDUCE-6628 - Potential memory leak in CryptoOutputStreamHBASE-16284 - Unauthorized client can shutdown the clusterHIVE-14436 - Hive 1.2.1/Hitting "ql.Driver: FAILED: IllegalArgumentException Error"HIVE-14743 - ArrayIndexOutOfBoundsException - HBASE-backed views query with JOINsIMPALA-3682 - Do not retry unrecoverable socket creation errorsIMPALA-4020 - Handle external conflicting changes to HMS gracefullySPARK-8428 -Fix integer o

We are pleased to announce the release of Impala JDBC v2.5.34 driver. This release has the following fixes and enhancements: Enhancements and new features:Authentication method and transport protocol. You can now specify if SASL should be used in conjunction with the 'Username and Password' Authentication Mechanism (AuthMech=3) by using the Use Sasl connection property. Previously this was done through the transport Mode connection property, which has now been removed. See Release Notes for JDBC for details on all of the fixes. Getting Started with the Cloudera DriversRead the Cloudera JDBC 2.5.34 Driver for Impala Release Notes and Installation Guide.Download the connector from the Cloudera Connectors page. As always, we welcome your feedback. Please send your comments and suggestions to the user group or through our community forums.  You can also file bugs through our external JIRA projects on issues.cloudera.org.

Dear CDH, Cloudera Manager, and Cloudera Navigator users,We are pleased to announce the release of Cloudera Enterprise 5.8.2 (CDH 5.8.2, Cloudera Manager 5.8.2, and Cloudera Navigator 2.7.2). This release fixes key bugs and includes the following:CDH fixes for the following issues:HADOOP-8436 - NPE In getLocalPathForWrite ( path, conf ) when the required context item is not configuredHBASE-15496 - Throw RowTooBigException only for user scan/getHDFS-9137 - DeadLock between DataNode#refreshVolumes and BPOfferService#registrationSucceededHDFS-10525 - Fix NPE in CacheReplicationMonitor#rescanCachedBlockMapHIVE-14006 - Hive query with UNION ALL fails with ArrayIndexOutOfBoundsExceptionIMPALA-3646 - Handle corrupt RLE literal or repeat counts of 0IMPALA-3820 - Handle linkage errors while loading Java UDFs in CatalogSPARK-16106 - [CORE] TaskSchedulerImpl should properly track executors added to existing hostsSQOOP-2999 - Sqoop ClassNotFoundException (org.apache.commons.lang3.StringUtils) is t

Cloudera is happy to announce the availability of parcels and packages for Kudu 1.0.0. After approximately a year of beta releases, Apache Kudu has reached version 1.0.0. This version number signifies that the development team feels that Kudu is stable enough for usage in production environments.This is the first non-beta release of the Apache Kudu project. (Although because Kudu is not currently integrated into CDH, it is not yet an officially supported CDH component.)Kudu 1.0.0 delivers a number of new features, bug fixes, and optimizations. We are also releasing a refresh of the Impala Kudu parcel.To upgrade Kudu to 1.0.0, see Upgrade Parcels or Upgrade Packages.In addition, you can read more about Kudu 1.0.0 in the vision blog post we just published. New Features in Kudu 1.0.0Removal of multiversion concurrency control (MVCC) history is now supported. This is known as tablet history GC. This allows Kudu to reclaim disk space, where previously Kudu would keep a full history of

We are pleased to announce the release of CDH 5.4.11. This release fixes key bugs and includes the following: HADOOP-8436 - NPE In getLocalPathForWrite ( path, conf ) when the required context item is not configuredHADOOP-10048 - LocalDirAllocator should avoid holding locks while accessing the filesystemHADOOP-11901 - BytesWritable fails to support 2G chunks due to integer overflowHBASE-14359 - HTable#close will hang forever if unchecked error/exception thrown in AsyncProcess#sendMultiActionHBASE-16207 - can't restore snapshot without "Admin" permissionHDFS-4660 - Block corruption can happen during pipeline recoveryHDFS-10360 - DataNode might format directory and lose blocks if current/VERSION is missing.HIVE-13251 - Hive can't read the decimal in AVRO file generated from previous versionIMPALA-2711 - Fix memory leak in Rand().YARN-5048 - DelegationTokenRenewer#skipTokenRenewal might throw NPEFor a full list of upstream JIRAs fixed in CDH 5.4.11, see the issues fixed section of th

We are pleased to announce the release of CDH 5.5.5. This release fixes key bugs and includes the following: FLUME-2918 - Speed up TaildirSource on directories with many filesHADOOP-12963 - Allow using path style addressing for accessing the s3 endpointHBASE-15811 - Batch Get after batch Put does not fetch all CellsHDFS-9805 - Add server-side configuration for enabling TCP_NODELAY for DataTransferProtocol and default it to trueHDFS-10396 - Using -diff option with DistCp may get "Comparison method violates its general contract" exceptionHIVE-11827 - STORED AS AVRO fails SELECT COUNT(*) when emptyHIVE-13527 - Using deprecated APIs in HBase client causes ZooKeeper connection leaksHUE-3866 - [core] Hue CPU reaches ~100% usage while uploading files with SSL to HTTPFS/WebHDFSMAPREDUCE-6473 - Job submission can take a long time during Cluster initializationOOZIE-2345 - Parallel job submission for forked actionsSENTRY-1265 - Sentry service should not require a TGT as it is not talking to

We are pleased to announce the release of CDH 5.7.3 This release fixes key bugs and includes the following:CDH fixes for the following issues:FLUME-2821 - KafkaSourceUtil Can Log Passwords at Info remove logging of security related data in older releases.HADOOP-11361 - Fix a race condition in MetricsSourceAdapter.updateJmxCache.HDFS-8581 - ContentSummary on / skips further counts on yielding lock.MAPREDUCE-6675 - TestJobImpl.testUnusableNode failed.YARN-5048 - DelegationTokenRenewer#skipTokenRenewal may throw NPE.HBASE-14818 - user_permission does not list namespace permissions.HIVE-11980 - Follow up on HIVE-11696, exception is thrown from CTAS from the table with table-level serde is Parquet while partition-level serde is JSON.HIVE-13749 - Memory leak in Hive Metastore.HUE-4477 - [security] Select All is not filtering out the non visible roles from the selection .IMPALA-3441 - , IMPALA-3659: check for malformed Avro data.SENTRY-1252 - grantServerPrivilege and revokeServerPrivilege sho

We are pleased to announce the release of Cloudera Director 2.1.1. Cloudera Director provides a simple, reliable, enterprise-grade way to deploy, scale, and manage Apache Hadoop in the cloud of your choice. Cloudera Director is available as a free download for use with Cloudera Express and is fully supported with Cloudera Enterprise. Cloudera Director 2.1.1 provides bug fixes and and product improvements, as outlined in the Cloudera Director 2.1.1 release notes. Cloudera Director Release NotesDownload Cloudera DirectorGetting Started with Cloudera DirectorInstalling Cloudera Director ClientInstalling Cloudera Director Server As always, we are happy to hear your feedback. Please post your questions and comments on our community forum.

Hello CDH Users,We are pleased to announce the release of Hive ODBC v2.5.20 driver. This release has the following fixes and enhancements:Enhancements & New FeaturesDelegate Kerberos credentials. You can now have the driver forward your Kerberos user credentials to the server to simplify the authentication process.Optimized Fast SQLPrepare behavior. The Fast SQLPrepare driver configuration option (the FastSQLPrepare key) is now disabled for non-SELECT queries. This ensures that the driver retrieves the necessary result set metadata at prepare time.Resolved IssuesUnicode characters in parameter values causing errors. Returning errors for some queries with dates in them. Unable to create new tables when Unicode character types option set.Getting Started with the Cloudera DriverRead the Cloudera ODBC 2.5.20 Driver for Hive Release Notes and Installation Guide.Download the connector from the Cloudera Connectors page.As always, we welcome your feedback. Please send your comments and sug

Dear CDH, Cloudera Manager, and Cloudera Navigator users,   We are pleased to announce the release of Cloudera Enterprise 5.7.2 (CDH 5.7.2, Cloudera Manager 5.7.2, and Cloudera Navigator 2.6.2).   This release fixes key bugs and includes the following: CDH fixes for the following issues: HADOOP-11409 - FileContext.getFileContext can stack overflow if default fs misconfigured HADOOP-12787 - KMS SPNEGO sequence does not work with WebHDFS HADOOP-13251 - Authenticate with Kerberos credentials when renewing KMS delegation token HDFS-10360 - DataNode may format directory and lose blocks if current/VERSION is missing HBASE-11625 - Reading data block throws "Invalid HFile block magic" and can not switch to hdfs checksum HIVE-9499 - hive.limit.query.max.table.partition makes queries fail on non-partitioned tables HIVE-10685 - Alter table concatenate operator will cause duplicate data IMPALA-1928 - Fix Thrift client transport wrapping order HUE-4113 - [Pig] Hue bre

We are pleased to announce the release of the Cloudera Distribution of Apache Kafka 2.0.2 for CDH 5.  Apache Kafka is a highly scalable, distributed, publish-subscribe messaging system. Apache Kafka 2.0.2 is maintenance release for Kafka 2.0. Notable Issues Fixed in Cloudera Distribution of Apache Kafka 2.0.2: KAFKA-3495: NetworkClient.blocking Send And Receive should rely on requestTimeout.KAFKA-2998: Log warnings when client is disconnected from bootstrap brokers.KAFKA-3488: Avoid failing of unsent requests in consumer where possible.KAFKA-3528: Handle wakeups while rebalancing more gracefully.KAFKA-3594: After calling MemoryRecords.close() method, hasRoomFor() method should return false.KAFKA-3602: Rename RecordAccumulator dequeFor() and ensure proper usage.KAFKA-3789: Upgrade Snappy to fix Snappy decompression errors.KAFKA-3830: getTGT() debug logging exposes confidential information.KAFKA-3840: Allow clients default OS buffer sizes.KAFKA-3691: Confusing logging during metadat

Cloudera is excited to announce the general availability of Cloudera Enterprise 5.8! Main highlights of this release include Impala read/write support on Amazon S3, a redesigned SQL query editor GUI, the expansion of role-based access control functionality to Cloudera Search, and the GA of Cloudera Navigator Optimizer to facilitate and optimize workload migrations.   For those new to it, Cloudera Navigator Optimizer (previously in beta) is a cloud-based service that helps with offload planning and active data optimization for Apache Hadoop. For example, it provides workload visibility and assessments for building a predictable offload plan, adapting to evolving Hadoop data and workload demands, and optimizing query performance in Impala and Apache Hive. (For more details about Cloudera Navigator Optimizer, check out the release blog post.)   As usual, Cloudera Enterprise 5.8 contains a long list of quality enhancements and bug fixes (learn more about our multi-dimensiona

We are pleased to announce the release of Cloudera ODBC 2.5.34 Driver for Impala. This release has the following fixes and enhancements:  Enhancements and New FeaturesSupport for queries that use the EXISTS syntax.Additional SQL Column optimization for catalog functions that include schema restrictions. Resolved IssuesQuery fails if the column name contains a character from the Latin-1 Supplement character set.Warning dialog when non-Admin user clicks OK on the DSN dialog should be suppressed.Driver reports statement length limit as 16 KB. Getting Started with the Cloudera DriversRead the Cloudera ODBC 2.5.34 Driver for Impala Release Notes and Installation Guide.Download the connector from the Cloudera Connectors page. As always, we welcome your feedback. Please send your comments and suggestions to the user group or through our community forums.  You can also file bugs through our external JIRA projects on issues.cloudera.org. 

We are pleased to announce the release of Cloudera Director 2.1.0.Cloudera Director provides a simple, reliable, enterprise-grade way to deploy, scale, and manage Apache Hadoop in the cloud of your choice. Cloudera Director is available as a free download for use with Cloudera Express and is fully supported with Cloudera Enterprise. Cloudera Director 2.1.0 includes the following new features: Usage-based billing for Cloudera servicesSupport for running Cloudera Manager and CDH clusters on Microsoft AzureSupport for deploying Cloudera Manager and CDH on a different cloud provider or region than Cloudera Director with a simple network setupSupports Cloudera 5.7 out of the boxVarious bug fixes and improvements Links to the product and documentation are listed below:Download Cloudera Director from: http://www.cloudera.com/content/support/en/downloads.htmlView the documentation:Cloudera Director Release NotesGetting StartedInstalling Cloudera Director ClientInstalling Cloudera Director