Find answers, ask questions, and share your expertise
Recently active
table is `HiveFileFormat`. It doesn't match the specified format `ParquetFileFormat` I have this issue when I try to write using pyspark with the following command:df.write.mode("append").format("parquet").saveAsTable("schema.table") Before you say change from parquet to hive i know it works. But the thing is the table is partitioned in parquet and I really don't know why not its not working any more. It worked fine until now. The same command ran correctly for 1 month and 5 times so far. But today it does not want to write like this any more. If i check the metadata it also points to everything being in parquet:``103SerDe Library: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDeNULL104InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormatNULL105OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutput
Hi All, I am currently building a pipeline that fetches all changes from 2 mysql DBs into a final mysql warehouse. I am using changecapturemysql processor to identify inserts,updates and deletes and then sending them to putdatabaserecord processor. All inserts work good. My source has a PK (primary key) column. My destination has the same column + another column as PK . I am able to replicate inserts. But when it comes to updates and deletes, updates on non primary keys were getting captured, but if i update primary key column in source, it is unchanged in destination. same with deletes. a delete with where clause on PK, doesnot deleting the record in destination. As i read the documentation this is expected behaviour - "MPORTANT: If the Statement Type is UPDATE, then the incoming records must not alter the value(s) of the primary keys (or user-specified Update Keys). If such records are encountered, the UPDATE statement issued to the database may do nothing (if no exis
We have a SQL that normally runs for 2 mins including a few stages, including 8 mappers and 1000 reducers. One day, among the 8 mappers, 7 finished in <1 min but 1 stuck for 2.5 hours. The logs in that container stopped after 1-2 mins (the task was killed so I could not produce the logs but there were nothing suspicious). The counters show that there were no activities at all for a very long time, CPU usage low, network, IO, all low. The 1000 reducers were up and waiting, using all the resources and kept the rest of the tasks waiting. Re-running the SQL with the same data (no skew) multiple times and it always completes within 1-2mins. What could cause a mapper to stuck silently? Is there a way to debug this (the logs were not very helpful)? This happens occasionally for different SQLs, is there a way or tool to detect such situation and restart / kill the job? Thanks for the help!
How to upload files to cloudera virtual box? I'm taking SimpliLearn's Big Data Hadoop and Spark Developer Training course, which is a free course. That's why I don't have access to the labs. So what I did was download cloudera virtual box to practice spark and hadoop codes. However, some exercises require uploading files such as datasets using FTP.I tried using FileZilla to connect to the VM and upload the files to the VM, but it didn't work connecting using the VM's IP address.I'm a beginner with these tools, could anyone help me with how to put these files in the VM? See below what I need to do:1. Download the Hadoop-mapreduce-example.jar file and wordcount.txt file2. Log in to the FTP using the username and password from the lab and upload the file3. Log in to the Webconsole using the username and password from the lab and create a new directory demo in HDFS using the mkdir command4. Push the wordcount.txt file into the directory using the put command5. Execute the command
How to run selected processor group in apache nifi while restarting nifi?
How to set the partition life cycle of the hive table? Error: Error while compiling statement: FAILED: ParseException line 1:270 missing EOF at 'lifecycle' near ')' (state=42000,code=40000)
Hi Team,We have CDP cluster integrated with OpenShift cluster, private cloud setup with all the service like hive, atlas, ranger running. When we are creating the Datawarehouse CatLog in management console we are getting the below error in console, can you please help us to find out the cause.some Errors from management console: 1. Driver failed to create DB Catalog: IP address is not expected for default warheouse RDS endpoint. Check if RDS is enabled in the env? Error Code : 99992. timeout exceeded transitioning from state 'Starting'.Last event: {InProgress checking if metastore statefulset is ready with at least 1 ready replica(s) (config-id: 5afc7144-9409-47e2-a9b9-2c7822adafbf version: 2022.0.7.1-2) 2023-02-09 10:08:44.366274685 +0000 UTC} Error Code : 9999OpenShift pods log:/usr/local/bin/config-merger /tmp/hive/conf/atlas-application.properties /mnt/config/current/atlas-application.properties /etc/hive/conf/atlas-application.properties -wf2 atlas.rest.address -wf2
To get started with CDP public cloud we have to create a env that is named as datalake and if we wanna run any jobs/workload on top of it we have to create datahubs based on the services we need. Inside the datahub cluster, there is an hdfs service that serves as intermediate place to store data while working on to be eventually stored/moved to cloud object storage. But there is another hdfs service inside the dataleke? what is usage of it? it's total capacity of 1.4TB SSD, how can we utilize it?
is there anyway to change username and credential for Nifi in Windows ? there is a command in Documentation to change it but I guess it works only in Linux
Beginner here...after installing the Hortonworks Sandbox HDP 3.0 on VirtualBox (Mac), I uploaded couple of csv files through ambari under the user name maria_dev by following the tutorial, was expecting to find in /users/maria_dev but logging in as maria_dev into the sandbox, i could not locate these and get an error "No such file or directory". I managed to upload files via scp into the sandbox and complete the tutorial, however still confused where the upload to ambari UI might have gone. Not sure where the files went or i am looking at the wrong place. Any help will be appreciated.
we are testing the trial version of the cloudera data platform (CDP7). But when we are running to the Step7 of the Parcel installation. It is stuck in the status of the distribution. Any suggestion for this step? Thanks.
I am looking to upgrade the cluster from CR 7.1.7 to CR 7.1.7 SP2. Can it be done directly or do I have to upgrade to CR 7.1.7 SP1 and then to SP2?
Hi, I'm new to Nifi and currently trying to understand how to improve its performance. In this use case I am getting large zip files (30 - 700 GB) from a network share using the FetchFile processor and then unpacking with UnpackContent. Files that are in the 500GB+ range are taking several hours to move onto the unpack stage. Files are eventually make their way through but I'm not sure on the best way to improve performance. The server is beefy with 1TB RAM, plenty of disk space, 40 core CPU. Utilisation across CPU/Mem/Disk/Network I/O all seem under used whilst the processor is running. I've upped the max timer/event thread counts to 80/5 respectively but that doesn't seem to have made any difference. Is tackling the jvm min/max limits a good option to go for next? thanks for any guidance.
My use-case is: 1) Have API credentials2) Use UpdateAttribute to update (1) schema, (2) s3 bucket/location (my list of reports)3) Query API endpoint for report4) API endpoint paginates and gets more records5) Call MergeRecord6) Save to s3 Since 3, 4, 5, 6 are all the same, I'm re-using the processors like below (screenshot). My problem is (5) MergeRecord will try to merge different schemas together, which is obviously a problem. How can I restructure this? I'd like to re-use processors as much as possible, but still be able to add more schemas as my needs evolve.
Hi,we would like install Cloudera CDP 7.1.7 SP1 on linux host with Linux Centos Stream 8.x.The information in Cloudera's support matrix is generically "Centos". Can we consider them valid for Linux Centos Stream 8.x? Thank you.
Error while processing statement: FAILED: Error in acquiring locks: Error communicating with the metastore (state=42000,code=10) Unable to create Hive Table, while creating a table getting error in acquiring locks: Error communicating with the metastore, ERROR : FAILED: Error in acquiring locks: Error communicating with the metastoreorg.apache.hadoop.hive.ql.lockmgr.LockException: Error communicating with the metastoreat org.apache.hadoop.hive.ql.lockmgr.DbLockManager.lock(DbLockManager.java:178)at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:447)at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocksWithHeartbeatDelay(DbTxnManager.java:463)at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:278)at org.apache.hadoop.hive.ql.lockmgr.HiveTxnManagerImpl.acquireLocks(HiveTxnManagerImpl.java:76)at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:95)at org.apache.hadoop.hive.ql.Dr
Please help design data flow in nifi, I need to insert the incoming JSON data into table which having single column. The condition are I need to concat all the keys and values and insert that data.
Hi Team, I'm new to CDP and am currently looking at myaccount page "CDP Public Cloud Usage summary" staring to get a feel for the CDP product, but wanted to know if anyone has had experience in Billing and Metering of CDP subscriptions. I'd like to know if it is at all possible to create rules around subscription spend eg. If estimated cost went above $20, can an alert generated advising me of this.Also is there any other info that might help with managing cost that this team might know about, that might be useful? Thanks
I have an avro schema that I can get from a registry: { "name": "bar_tbl", "fields": [ { "name": "foo1", "type": "string" }, { "name": "foo2", "type": "string" } ]} I know how to use `EvaluateJsonPath` and get `name` and `fields`.I'd like to set an attribute with `UpdateAttribute` to equal: select foo1, foo2 from bar_tbl to use it later downstream in another processor.How can I achieve this? Or, more directly, I have a REST api that takes an url-encoded sql statement. How can I feed the sql statement to `InvokeHttp`?
We are encountering some issues when starting Nifi 1.16.3 after calling encrypt-config.shFor background, Nifi is deployed and managed using an in-house deployment product, and is used as a component within our application stack. In general everything works fine when using Nifi 1.15.1, however, we are encountering issues when using Nifi 1.16.3. See the following in nifi-bootstrap.log 2022-11-13 06:32:06,488 INFO [main] org.apache.nifi.bootstrap.Command Launched Apache NiFi with Process ID 828732022-11-13 06:32:14,269 INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for Bootstrap requests on port 329082022-11-13 06:34:06,217 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: Decryption Failed with Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]2022-11-13 06:34:06,218 ERROR [NiFi logging handler] org.apache.nifi.StdErr Shutting down...2022-11-13 06:34:06,945 INFO [main] org.apac
So guys,Need your help with something with which I have been struggling in the last couple of days. I just installed a NiFi cluster, composed of 5 machines. I have used NiFI 1.19.1 as it is the latest version.Now, when trying to create a DBCP Connection Pool, I keep on receiving the below error messages. Beside these two, I receive plenty more with other classes. Right now, I am using "openjdk version "11.0.8" 2020-07-14 LTS" so thereafter I have uploaded ojdbc11.jar in my lib folder, on all servers. Afterwards I have restarted each server and tried to create a new DBCP Connection Pool, but kept on receiving the these error messages.Database Connection URL = jdbc:oracle:thin:@HOSTNAME:PORT/SERVICEDatabase Driver Class Name = oracle.jdbc.driver.OracleDriverDatabase Driver Location(s) = /opt/nifi-1.19.1/lib/ojdbc11.jarDo you know what might be the cause of this problem? On a cluster of servers with java 8 I do not encounter such issues.2023-01-27 18:00:08,841 ERROR [Timer-Driven Process
It seems this question gets asked a lot in this forum, but there aren't any good responses. Hopefully somebody can help me out here.I am attempting to use the InvokeHTTP processor to POST a query to an REST endpoint. I currently use the InvokeHTTP processor to access data on this data service using the GET method, and that always works fine. However, there is a specific service endpoint that only accepts POST requests. I've verified this service is functional using curl. But when I attempt to use the NiFi InvokeHTTP processor, I get no response. Hopefully somebody here knows what's going on. Below are some screen shots of my processor configuration. If I understand the documentation correctly, I need to specify in the "Attributes to Send" field the flowfile attributes I want to send as parameters. Since this processor is the first processor in the flow, it does't have any attributes yet, so I add a dynamic attribute named "data" (sho
I have a apache/nifi:latest instance spun inside an Amazon Linux 2 EC2. For reference, see this guide: hereI have a QuerySalesforceObject ver. 1.18.0 that makes use of StandardOauth2AccessTokenProvider. The oauth2 provider url is configured at https://test.salesforce.com/services/oauth2/tokenI can curl this url from the box and from inside the docker container just fine (I don’t get a timeout).[root@ip-10-229-18-107 \~\]# docker exec -it nifi_container_persistent /bin/shprintenv | grep -i proxyHTTPS_PROXY=http://proxy.MY_DOMAIN.com:3128no_proxy=localhost,127.0.0.1,MY_DOMAIN.com,.amazonaws.comNO_PROXY=localhost,127.0.0.1, MY_DOMAIN.com,.amazonaws.comhttps_proxy=http://proxy.MY_DOMAIN.com:3128http_proxy=http://proxy.MY_DOMAIN.com:3128HTTP_PROXY=http://proxy.MY_DOMAIN.com:3128curl https://test.salesforce.com/services/oauth2/token{"error":"unsupported_grant_type","error_description":"grant type not supported"}#But when I run the task, oauth2 fails with an errorjava.io.UncheckedI
Hi,Currently we are using CDP 7.1.7 with phoenix version 5.0.We were told by platform admins that cloudera is dropping support for phoenix from the next cdp version onwards. Is this true? If so, what are the alternative to phoenix are suggested?What is the strategy on embodying phoenix with CDP package? ThanksSagar
Other API calls work such as to /nifi-api/tenants/user-groups. I do not see an entry for /resources in authorizations.xml nor does there seem to be an entry VIA the gui policy editing. NIFI 19.1...
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.