Find answers, ask questions, and share your expertise
Recently active
Hello, I have a simple process getting data from sql server and send them by email. Short emails, nothing heavy and not that many during the day. But few days ago, it suddenly started to be frozen and it happened again today. It does not show any error and I do not see any error in logs either. I checked this thread, maybe it's the same, but there is no real solution and in my logs I can't see anything with SMTP or that it's "waiting", so would like to ask you for help or to point me to the right direction.Maybe I'm not checking everything... ThanksRgdsLudvik
I need to load the data from Azure Cosmos DB. I am using Nifi 1.21.0 Version. what is the way to do that?
Hi.I'm using NiFi 1.11.2. I have a set of Json-files which contain a base64-coded field (Jsonpath to this field is $.data.content). Decoding the field works as expected and I'm able to write the content from this field to a file in a bucket in S3. But what I would like to do is to be able to replace the coded value for this field with the decoded value in stead of writing the decoded value to file. And after replacing the value write the updated Json-file to a new S3 bucket.I'm thinking taking the output from RouteOnAttribute and ConvertRecord in the figure below and perhaps use UpdateRecord or ReplaceText. But I'm not sure how to do this.What is the best way to "re-connect" the data flow here from these to objects in the process?Any tips? Thanksbjornmy
When creating a container using hue, I mount volume with the hive configuration. The configuration is taken from the cluster host. In the hive-site.xml has the hive.server 2.authentication parameter, which has the ldap value. With this value, hue does not connect to hive. If I replace it with a KERBEROS value, then the value will change on the host (the host is part of a hadoop cluster). If you copy the hive configurations to the container, they will not be updated when updating in the cluster. When trying to define a variable for the container HIVE_OPTS='--hiveconf hive.server2.authentication=KERBEROS' nothing happens. Is it possible to somehow redefine the variable without changing the original configurations
Hi, I have a nifi flow which has 1000's of flow files. Each of the flow file has an impala/hive statement and each of them have different execution times. How do i use control rate processor to trigger next flow file only if the previous flow file has been processed (success/failure of the flow file doesn't matter)?
Hi,I am unable to create a support case for this, as my support license does not appear to cover this ML Experiments.I am evaluating Cloudera CML on my company's CDP instance. I created several Experiments on a project to test out the feature (it was a simple script generating some random numbers for metrics) and the experiment created and ran ok. However, I tried to create the same experiment later and the UI just timed out. I used the developer's tools in the browser to see the request and "runs" failed with 504 timeout error. I am unable to troubleshoot the issue further in the UI, and am uncertain if ECS would have any useful information. I could get it to run successfully a few hours later. To my knowledge, nobody touched the CDP cluster during in between this period. During the period I cannot create experiments, I looked at my kubernetes namespace <CML workspace name - userXX> inside ECS, and did not see a pod starting. There was also no events being created. We would like
Hi all, I have a Hadoop cluster built on Cloudera. Now I am trying to increase concurrent with impala and I have no ideal for change the config files. Do you have experiences to how to resolve this, please help me. Thank you all,
Hello @All, I am totally new in apache NiFi i am going to integrate Polarion ALM to NiFi using Rest Apis. But i am not able to understand where i need to give API link and where i need to give logIn credential i am using invokeHTTP process so anyone having any idea on this please help me. Thanks and Regards,Digambar Dhere
Hi Team , As we have more then 14 million small file as per Cloudera Navigator as below : 14.8M Small files created in the last 30 days 14.8M / 21.4M 69.3% small files We are doing partition of data per day wise can we increase it or any other suggestion is there to overcome the small file problem. Thank You in Advance
I created a managed table called "questiuon".However, a path called "/warehouse/tablespace/external/hive/question.db" was created on the hdfs path.Conversely, creating an external table also creates a path called "/warehouse/tablespace/managed/hive/question.db".What is the reason?
Hello! I am creating a view `view_business` with IMPALA: since the source table `table_a` is a parquet formatted table with complex data types such as array and struct, I have to always run an query option statement "SET PARQUET_ARRAY_RESOLUTION=TWO_LEVEL_THEN_THREE_LEVEL" to load the arrays; otherwise those array columns are showing NULL value. DROP VIEW IF EXISTS discovery_database.view_business;SET PARQUET_ARRAY_RESOLUTION=TWO_LEVEL_THEN_THREE_LEVEL;CREATE VIEW discovery_database.view_business AS ( SELECT * FROM table_a)Once the view is created, if I want to query the view, I still need to run the query option statement first as follows: SET PARQUET_ARRAY_RESOLUTION=TWO_LEVEL_THEN_THREE_LEVEL; SELECT * FROM discovery_database.view_businessHowever, the query option statement above is not supported in ODBC connector when I tried to use the statements above to link the curated table `view_business` as a data source in a power BI report. My question is: is it possi
Hi Team, We have 2 node managers of each having cpu 16 . we restrict CPU limit of YARN by set yarn.scheduler.maximum-allocation-vcores & yarn.nodemanager.resource.cpu-vcores to 12. But CPU utilization shows 100% utilization. May i know how to troubleshoot this issue. Though we limit cpu to 12 why it is taking 100% of cpu i.e..,16. Immediate suggestions much appreciated. Regards,Hanu
Hi Team, When we are running Msck Repair Table command it is hanging , can anyone please help me on this issue?? Regards,Hanu
Good afternoon all, and thank you in advance.I have a Apache NIFI solution that is pulling from API a pair of dates, i've grabbed out the relevant bit from the json:"arrival" : "2023-07-24T14:33:00.263Z", "departure" : "2023-07-24T14:21:15.263Z"And i'm successfully getting those values down to attributes via extract attributes.However when i perform a toNumber() on them to convert to miliseconds since EPOCH to determine the time between the dates, i return an empty condition upon review of my flow in the queue.I've gone so far as to arbitarily grab "NOW()" and attempt to toNumber() it but with the same result as trying to modify the above attributes. incidentally NOW() returns a format like: Mon Jul 24 15:39:21 UTC 2023 - its different that my UTC formatted items, but it also doesn't want to toNumber() - returning again an empty valueShows me attempting to convert the attributes:and this shows the flow in the queue after hitting that updateattrs. The datetimestamps are
Hello experts, I am working on a project where I have two Gitlab instances available in two different networks i.e., A and B. I need to design a solution using NiFi to read data from certain branches of different Gitlab repos available in Network A and transfer the data in real time to the corresponding branches of the Gitlab instance in Network B. I will greatly appreciate if you could give me some high level idea to develop the solution using NiFi. If relevant, please mention some standard NiFi processor which can help read in this regard. Thank you
Need help if anyone is aware: Platform Ambari (HDP 3.1) with Kerberos Enabled, I am trying to push command after that I did KINIT -V -kt kafka.keytab kafka-principal@realm and got "Authenticated to Kerberos v5": ./kafka-console-producer.sh --broker-list host.kafka:6667 --topic cleanCsv --producer-property security.protocol=SASL_PLAINTEXT < /tmp/clean_csv_full.csv Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>&g
HelloI want to set up a local environment of Hive with Iceberg similar to my working environment - CDP public cloud 7.2.15 so I can test locally.The problem is that on my local environment insert into iceberg tables (via Hive) does not work with Tez execution engine (works only with MapReduce). And according to the Iceberg documentation it should not work with Tez for Hive 3. But the thing is that it does work on CDP - I am able to insert data into Iceberg tables with Tez on Hive 3.Is there a way to make it work locally? Thank you Here is my local config:Hadoop: 3.2.1Hive: 3.1.3 (on CDP 3.1.3000.2023.0.14.0-155 is used)Iceberg-hive-runtime: 0.13.1.7.2.15.0-147 (the same version is used on CDP)Tez: 0.9.1.7.2.15.0-147 (the same version is used on CDP)
Hello All,I'm trouble-shooting the following issue with our Cloudera Nutch cluster and would appreciate any help the community can offer:We have two NameNode roles and three JournalNode roles running, however both NameNode roles are failing to start and reporting the error below (IP addresses obfuscated). This occurred following a restart of the underlying hosts.Any recommendations for a recovery path from this error would be greatly appreciated. Error: recoverUnfinalizedSegments failed for required journal (JournalAndStream(mgr=QJM to [x.x.x.95:8485, x.x.x.86:8485, x.x.x.130:8485], stream=null))org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size 2/3. 1 successful responses:x.x.x.130:8485: null [success]2 exceptions thrown:10.103.28.95:8485: tried to access method com.google.common.collect.Range.<init>(Lcom/google/common/collect/Cut;Lcom/google/common/collect/Cut;)V from class com.google.common.collect.Ranges at
Hi everyone,I'm trying to use NiFis Rest API to start and stop a Processor, but evertime I get the response The desired state is not setHere is my curl-Command which I use:curl -k --header "Authorization: Bearer $token" -i -H 'Content-Type: application/json' -XPUT -d'{ "revision": { "version": 18, "clientId": "01891026-da00-199f-9dc9-ae3a37dce067" }, "status": { "runStatus": "RUNNING" }, "component": { "id": "0189100e-1e29-115c-ec96-b857cd155f4a", "state": "RUNNING" }, "id": "0189100e-1e29-115c-ec96-b857cd155f4a" }' https://localhost:8443/nifi-api/processors/0189100e-1e29-115c-ec96-b857cd155f4a/run-statusHas anyone an idea what's the problem?Best regardsMaik
Hi , i've a flow where there two flows one for xml file and other for corresponding Pdf file. Currently i'm merging those using Mergecontent and generating the zip file out of it. by making sure minimum entries 2.if input files load are heavy the merge is not matching corresponding file means (same invoice file should be merged) but sometimes other invoice file merging due to big load from input at a time. i already implemented 10 secs delay for each flow and merging two files. is there a mechanism to merge files based on common attribute before mergecontent. Thanks in advance
Hello everyone ! for 24 hours i was trying to send data flow from apache nifi to minio s3. i have 2 processors getflowfile and puts3object. i'm putting everything correc like acces key secret key etc... but it show me always the sdame error in nifi that they don'tv recognize the acces key could you help please
I'm using the NIFI ExtractText Processor and I'm trying to come up with the regular expression to extract values from a JSON String that is in the flowfile-content coming from a response of an API (of course I cannot change how the API responds). JSON Data returned from the REST API is: { "Result" : { "SessionID" : null, "ResultData" : { "DATAENTITYNAME" : "List", "CURRENTCURSORPOSITION" : 2, "DATARECORD" : [ { "DATAFIELD" : [ { "FIELDNAME" : "name", "FIELDLABEL" : "Name", "DATATYPE" : "VARCHAR", "FIELDVALUE" : "JSMITH", "ISKEYFIELD" : true }, { "FIELDNAME" : "namedesc", "FIELDLABEL" : "Name Desc.",
Hello. I am trying to run explorative Pyspark code within a Databricks ApacheSpark environs. I am pretty sure this syntax is correct, but the subject-referenced apache.spark error keeps throwing. Any insights, please? databaseName = "database"desiredColumn = "variable"database = spark.sql(f"show tables in {databaseName} ").collect()display(database) tablenames = []for row in database: cols = spark.table(row.tableName).columns listColumns= spark.table(row.tableName).columns if desiredColumn in listColumns: tablenames.append(row.tableName)
I need to convert the the request with Apache-nifi, can you help me how to do that? curl --location --request POST 'https://localhost:8080/token' \--header 'Content-Type: application/x-www-form-urlencoded' \--data-urlencode 'client_id=xxxxxxxx' \--data-urlencode 'client_secret=yyyyyyyy' \--data-urlencode 'type=EEEEEEEEEEE' \--data-urlencode 'scope=ZZZZZZ' Thank for your help!!
Hi Team CM UI Down post OEL , But CM server is running in backend, getting below error in log...Could you please advise WARN avro-servlet-hb-processor-4:com.cloudera.server.cmf.AgentProtocolImpl: Received an optimized heartbeat for a host with ID 'bef21682-baaf-462d-92f8-e43464fc4152' that is not recognized Thankssam
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.