Member since
08-05-2024
21
Posts
12
Kudos Received
0
Solutions
09-11-2024
08:47 AM
@ggangadharan thanks for your reply. Yes, as soon spark sees NUMBER data type in oralce it convert the df datatype to decimal(38,10) then when precision value in oracle column contains >30 spark cant accommodate it as it only allows 28 max digits if decimal(38,10) hence getting this issue. yeah as you said the probable solution is to cast it as string Type.
... View more
08-30-2024
01:28 AM
1 Kudo
I am reading oracle table using pySpark. my oracle table contains data type NUMBER and it contains 35 digits long value. when I read this column using spark, it seems spark assumes more precision than original (+10 precision)and end up in throwing following error: java.lang.ArithmeticException: Decimal precision 45 exceeds max precision 38` please note: I am inferring schema from spark data frame as I don't want to explicitly pass schema. code: df_rdbms_data = spark.read.jdbc(url=source_jdbc_url, table=source_table_name, properties=source_properties)
df_rdbms_data.printSchema()
df_rdbms_data.show() Is there any way to resolve this issue apart from not passing explicit schema or explicit casting? Thanks in advance.
... View more
Labels:
- Labels:
-
Apache Spark
08-26-2024
02:15 AM
2 Kudos
yes, it works with PUT command in place not get command.
... View more
08-26-2024
01:09 AM
1 Kudo
I have tried with partition statement (dynamic) and still staging directory has been creatde at table level. it seems this works for only static partition
... View more
08-23-2024
01:54 AM
1 Kudo
Thank you. it seems hive warehouse connector creating these tmp/staging directories in table level rather than partition level.
... View more
08-22-2024
12:58 AM
Documentation saying syntax as follows. PUT /v2/entity/guid/{guid}
... View more
08-22-2024
12:49 AM
ok. Thanks for your reply @vats I am getting all my entities updates in real time like create/update/alter of hive tables. so it might be configuration issue? By any chance do yo know where to look for delete hive operation configuration in atlas/hive?
... View more
08-21-2024
12:15 PM
an we know why are you using GET command to update the entity ? is it typo ?
... View more
08-21-2024
12:12 PM
@mayank_gupta - thanks attaching my json file and CURL command i tried below. curl --negotiate -u : -X PUT -H "Content-Type: application/json" https://URL/api/atlas/v2/entity/guid/7g5678h9-4003-407a-ac00-791c7c53e6d5 -d @test_oracle_tbl.json error I am getting - {"errorCode":"ATLAS-400-00-023","errorMessage":"Attribute null not found for type oracle_table_test"} my json : { "referredEntities": { "a4ff269b-566f-40f6-b7a3-8e0678c100fa": { "typeName": "oracle_column", "attributes": { "owner": null, "replicatedTo": null, "userDescription": null, "replicatedFrom": null, "qualifiedName": "sales@oracle_column_fc_opo_test", "displayName": null, "dataType": "double", "name": "sales", "description": null, "comment": "Sales", "position": 0, "table": { "guid": "7g5678h9-4003-407a-ac00-791c7c53e6d5", "typeName": "oracle_table_test" } }, "guid": "a4ff269b-566f-40f6-b7a3-8e0678c100fa", "isIncomplete": false, "status": "ACTIVE", "createdBy": "ownerTest", "updatedBy": "ownerTest", "createTime": 1722463451536, "updateTime": 1722463451536, "version": 0, "relationshipAttributes": { "inputToProcesses": [], "pipeline": null, "schema": [], "model": null, "meanings": [], "table": { "guid": "7g5678h9-4003-407a-ac00-791c7c53e6d5", "typeName": "oracle_table_test", "entityStatus": "ACTIVE", "displayText": "fc_opo_test", "relationshipType": "oracle_table_test_columns", "relationshipGuid": "4581a28b-73fb-4743-9069-649ff1812a80", "relationshipStatus": "ACTIVE", "relationshipAttributes": { "typeName": "oracle_table_test_columns" } }, "outputFromProcesses": [] }, "labels": [] } }, "entity": { "typeName": "oracle_table_test", "attributes": { "owner": "ownerTest", "temporary": false, "lastAccessTime": 0, "aliases": null, "replicatedTo": null, "userDescription": null, "replicatedFrom": null, "qualifiedName": "fc_opo_test@oracle_table_test", "displayName": null, "columns": [ { "guid": "a4ff269b-566f-40f6-b7a3-8e0678c100fa", "typeName": "oracle_column" } ], "description": "fc_opo_test oracle Table Test", "viewExpandedText": null, "tableType": null, "rewriteEnable": null, "createTime": 0, "name": "fc_opo_test", "comment": null, "parameters": null, "retention": 0, "viewOriginalText": null, "db": null }, "guid": "7g5678h9-4003-407a-ac00-791c7c53e6d5", "isIncomplete": false, "status": "ACTIVE", "createdBy": "ownerTest", "updatedBy": "ownerTest", "createTime": 1722463451536, "updateTime": 1722463451536, "version": 0, "relationshipAttributes": { "inputToProcesses": [], "pipeline": null, "schema": [], "columns": [ { "guid": "a4ff269b-566f-40f6-b7a3-8e0678c100fa", "typeName": "oracle_column", "entityStatus": "ACTIVE", "displayText": "sales", "relationshipType": "oracle_table_test_columns", "relationshipGuid": "b63f4847-14ef-4e70-a384-8fa40e422b39", "relationshipStatus": "ACTIVE", "relationshipAttributes": { "typeName": "oracle_table_test_columns" } } ], "model": null, "meanings": [], "db": null, "outputFromProcesses": [] }, "labels": [] } }
... View more
08-20-2024
06:25 AM
I just downloaded same entity json and just added 'dummy description' under description of entity and tried with same PUT command and I got same error.
... View more