This article explains how to configure Spark Connections in Cloudera Machine Learning.   CML enables easy Spark data connections to the data stored in the Data Lake by abstracting the SparkSession connection details. CML users don't need to set complicated endpoint and configuration parameters or load HWC or Iceberg libraries. They can use the cml.data library to get preconfigured connections.   CML Data Connection Snippets     The spark SparkSession object has all the necessary options set to make connections work. Users have two options to set custom or job-specific configurations like executor CPU and memory parameters.   1. Specify the configuration inline With SparkContext's setSystemProperty method, you can set Spark properties that will be picked up while building the SparkSession object. You can set any Spark property like below before calling cmldata's get_spark_session method.     Python script   import cml.data_v1 as cmldata from pyspark import SparkContext SparkContext.setSystemProperty('spark.executor.cores', '4') SparkContext.setSystemProperty('spark.executor.memory', '8g') CONNECTION_NAME = "go01-aw-dl" conn = cmldata.get_connection(CONNECTION_NAME) spark = conn.get_spark_session()      2. Use spark-defaults.conf By placing a file called spark-defaults.conf in your project root (/home/cdsw/), you can set Spark properties for your SparkSession. The properties configured in this file will be automatically appended to the global Spark defaults.   spark-defaults.conf   spark.executor.cores=4 spark.executor.memory=8g     Python script   import cml.data_v1 as cmldata CONNECTION_NAME = "go01-aw-dl" conn = cmldata.get_connection(CONNECTION_NAME) spark = conn.get_spark_session()     Conclusion CML supports flexibility by offering two ways to configure its Spark connection snippets. The inline option helps users who want to fine-tune their Spark applications and want to configure different properties per Spark job. The spark-defaults.conf option helps users who want to set Spark properties that are applied for the whole Project.    To read more about the Data Connections feature, read the following blog post: https://blog.cloudera.com/one-line-away-from-your-data/ ... View more

This article explains how to use the Snowflake Connector for Spark in Cloudera Machine Learning. Save your Snowflake password for your account. Go to Account Settings > Environment Variables and create a new entry with Name as "SNOW_PASSWORD" and Value as your <password>. Create a new session in your CML project. Use any of the editors, and the Python 3.7 kernel. You also need to enable Spark for this session and select the Spark 2.4.7 version. Download required dependencies. To initiate the connection, you need to have a Snowflake Connector for Spark and a Snowflake JDBC Driver that's compatible with CML's Spark version. You can get these from the Maven Central Repository: spark-connector and jdbc-driver. Place them in a ./jars folder in your CML Project. cdsw@ysezv3fm94edq4pb:~$ ls -l ./jars/ total 28512 -rw-r--r-- 1 cdsw cdsw 28167364 Mar 17 21:03 snowflake-jdbc-3.13.16.jar -rw-r--r-- 1 cdsw cdsw 1027154 Jan 27 01:18 spark-snowflake_2.11-2.9.3-spark_2.4.jar Initiate the Snowflake connection.  You need to set your Snowflake Account ID and your username for the connection. Your Snowflake password is retrieved from the environment variable that you configured in the Account Settings.  I'm using the default Snowflake Warehouse and a sample database/schema. import os from pyspark.sql import SparkSession sfOptions = { "sfURL" : "<Account ID>.snowflakecomputing.com", "sfUser" : '<Username>', "sfPassword" : os.environ['SNOW_PASSWORD'], "sfDatabase" : 'SNOWFLAKE_SAMPLE_DATA', "sfSchema" : 'TPCH_SF1', "sfWarehouse" : 'COMPUTE_WH' } snowJars = [os.path.join('/home/cdsw/jars', x) for x in os.listdir('/home/cdsw/jars')] spark = SparkSession.builder \ .appName("cml-spark-snowflake-test") \ .config("spark.jars", ",".join(snowJars)) \ .getOrCreate() SNOWFLAKE_SOURCE_NAME = "net.snowflake.spark.snowflake" query = ''' select * from SNOWFLAKE_SAMPLE_DATA.TPCH_SF1.CUSTOMER limit 100 ''' df = spark.read.format(SNOWFLAKE_SOURCE_NAME) \ .options(**sfOptions) \ .option("query", query) \ .load() df.show() Execute the code; you can see the results as follows: ... View more

This article explains how to connect to Snowflake from Cloudera Machine Learning.   Save your Snowflake password for your account. Go to Account Settings > Environment Variables and create a new entry with Name: "SNOW_PASSWORD" and Value: <your password> Create a new session in your CML Project. You can use any of the editors, and a Python kernel: Install required Python packages pip install pandas snowflake-connector-python snowflake-connector-python[pandas] Initiate the Snowflake connection. You need to set your Snowflake Account ID and your username for the connection. Your Snowflake password is retrieved from the environment variable that you configured in the Account Settings. I'm using the default Snowflake warehouse and a sample database/schema. import os import pandas as pd import snowflake.connector conn = snowflake.connector.connect( account='<Account ID>', user='<Username>', password=os.environ['SNOW_PASSWORD'], warehouse='COMPUTE_WH', database='SNOWFLAKE_SAMPLE_DATA', schema='TPCH_SF1' ) query = ''' select * from SNOWFLAKE_SAMPLE_DATA.TPCH_SF1.CUSTOMER limit 100 ''' pd.read_sql(query, conn) Once you execute the code, you can see the results: ... View more

At Cloudera, we believe that data can make what is impossible today, possible tomorrow.   There are many good uses of data. With data, we can monitor our business, the overall business, or specific business units. We can segment based on the customer verticals or whether they run in the public or private cloud. We can understand customers better, see usage patterns and main consumption drivers. We can find customer pain points, see where they get stuck, and understand how different bugs affect them. With data, we can discover new market opportunities, and review where we stand compared to the global market. We can track feature adoption, see how new features are picked up and what usage/consumption they generate. With data, we can set better goals, know where we are and where we want to go. And in the end, we can make better decisions.    At Cloudera, we practice what we preach. As the Cloudera Data Platform (CDP) gains popularity and more and more customers make it a critical piece of their infrastructure, we set out to create the best data platform in the enterprise. Today, we will highlight a new feature that showcases one great example of using data in the service of our customers.    I’m excited to share this feature because this is a success story! Late last year, we saw our customers struggling to get CML Workspaces up. The elevated escalation count put a strain on our engineering team, trials were slowed down, and even worse our customers had a very bad experience with our product. We needed to figure this out. We tried to understand “is this a systemic issue?” or “how widespread is this problem?”, and the results were alarming. Customers experienced issues more than half, 57% of the time.   There are two phases of CML Workspace creation; first, we create a K8s Cluster via the liftie APIs - this is the ‘Provision’ step; second, we install the CML service. The above chart shows the workspace provisioning results broken down for CML releases between June ‘20 and Jan ‘21.   Once we saw the results, we dug in and analyzed dozens of failure modes. We discovered that actual product bugs caused only a small portion of the failures. The most common failures we found were instance types requested in unsupported regions, failures due to conflicts between the admin-provided CIDR address ranges, and environments where CML Workspaces were failing due to an unhealthy DataLake. Okay, we identified the problem: we attempt to create CML Workspaces when we know they will surely fail.  Preflight checks to the rescue.  Liftie and CML engineers teamed up to solve this problem. They built a framework and released a series of checks over the course of the last few quarters to catch issues early. The results are astonishing.    For the most recent - Aug ‘21 - release, customers experienced issues with the workspace creation just 7% of the time. For 39% of the attempts, we caught issues early and showed a meaningful error message, this saved hours and hours of work for support, engineering, and our customers.   This was a data-driven project. We used data to qualify the problem, to understand the issues, and to measure our progress and the outcome. The result is a significantly more stable platform and a new framework that all other CDP Data Services will benefit from.    Get started with Cloudera Machine Learning in CDP Now, you can start here. ... View more