I have a CML project using a JupyterLab Runtime with Python 3.10 and I want to start a Spark cluster with my CDP Datalake. I'm using the predefined Spark Data Lake Connection in CML which looks like this: ``` import cml.data_v1 as cmldata # Sample in-code customization of spark configurations #from pyspark import SparkContext #SparkContext.setSystemProperty('spark.executor.cores', '1') #SparkContext.setSystemProperty('spark.executor.memory', '2g') CONNECTION_NAME = "hiaa-dl" conn = cmldata.get_connection(CONNECTION_NAME) spark = conn.get_spark_session() # Sample usage to run query through spark EXAMPLE_SQL_QUERY = "show databases" spark.sql(EXAMPLE_SQL_QUERY).show() ```  When I execute this I get the error: IllegalArgumentException: The value of property spark.app.name must not be null I'm using the predefined spark-defaults.conf which looks like this: ``` spark.executor.memory=1g spark.executor.cores=1 spark.yarn.access.hadoopFileSystems=abfs://[container]@[storage-account].dfs.core.windows.net ``` Is there something else I need to configure in the CML session or at the data lake level? ... View more

I have two hive tables and I want to manually create a lineage relationship between them using the Atlas API. I'm trying to run this POST request: ``` curl --location 'https://[url]/atlas/api/atlas/v2/entity' \ --header 'Content-Type: application/json' \ --header 'Authorization: ••••••' \ --data-raw '{ "entity": { "typeName": "hive_process", "qualifiedName": "my_etl_process@cluster", "name": "my_etl_process", "description": "ETL process from input_table to output_table", "attributes": { "inputs": [ { "guid": "9768381b-8783-49c3-850d-39bf1f14b73f" } ], "outputs": [ { "guid": "998de7ba-2254-418d-b405-656eba428643" } ] } } } ' ``` I'm getting this 404 response: ``` { "errorCode": "ATLAS-404-00-007", "errorMessage": "Invalid instance creation/updation parameters passed : hive_process.qualifiedName: mandatory attribute value missing in type Referenceable" } ```  Any suggestions for how to manually create lineage using the API would be appreciated. ... View more

I want to use the Atlas API to add/edit user-defined properties to a particular entity. I've previously had success using the API to edit properties that are already defined, such as an entity description: ``` curl --location --request PUT 'https://[host]/api/atlas/v2/entity/guid/[guid]?name=description' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic' \ --data '"a test table"' ``` However, it's not clear how to use the API to create and edit user-defined properties. When I try changing the url query to a user-defined property name, it complains that that property is not defined for the entity type. For reference, in the UI, this is done fairly easily.   ... View more