Find answers, ask questions, and share your expertise
Recently active
After successfully running a 3-node NiFi cluster without any issues for the past three years, I am considering setting up an additional cluster. Upon checking the resource utilization of the 3-node ZooKeeper cluster, I noticed that the nodes have very low compute utilization. I would like to know if it is recommended to stick with a setup of 3 NiFi nodes and 3 ZooKeeper nodes, totaling 6 nodes, or if it would be more beneficial to use embedded ZooKeeper.
Having issues flattening the nested array in an array of objects back to its parent. Given this JSON : [ { "id": "13", "uom": "EA", "name": "pOne", "display_name": "displayOne", "brand": "brandOne", "categories": [ { "level1": "cLevel1", "level2": "cLevel2" } ] }, { "id": "87", "uom": "EA", "name": "pTwo", "display_name": "displayTwo", "brand": "brandTwo", "categories": [ { "level1": "cLevel1", "level2": "cLevel2" } ] }] I want to transform to this: [ { "pName" : "pOne", "pBrand" : "brandOne", "pVariant" : "displayOne", "pSku" : "13-EA-000", "categoryHierarchy" : "cLevel1/cLevel2"}, { "pName" : "pTwo", "pBrand" : "brandTwo", "pVariant" : "displayTwo", "pSku" : "87-EA-000", "categoryHierarchy" : "cLevel1/cLevel2"} ] So far I have these specs: [ { "operation": "modify-default-beta", "spec": { "*": { "productSku": "=conc
driver class: com.cloudera.hive.jdbc.HS2Driverjar : HiveJDBC42-2.6.11.1014the kerberos needs a keytab file, i wonder if this can be set by 'org.apache.hadoop.security.UserGroupInformation' class thks.
I'm new to NiFi, as you'll realize in this post. I'm trying to troubleshoot this issue where after I'm logging in with the provided password in the log file, I don't get past the loading screen. Here are things I've noticed: in nifi.app log file, I get the error:Archive cleanup completed for container default; will now allow writing to this container.in cmd, I have this error: Command Failed to set permissions so that only the owner can read status file [path]; this may allows others to have access to the key needed to communicate with NiFi. Permissions should be changed so that only the owner can read this file when opening any .bat other than run-nifi.bat as administrator, the window immediately closes.This is the process that's looping over and over in the nifi-app.log file My feeling is that these are having to do with some sort of permission issue. I've tried taking the folder off read-only but that potential solution alone has been a rabbit hole of issue
I had setup Secure Nifi Registry using Docker compose.Everything worked and I am able to open UI at https://localhost:18443/nifi-registryBut when i try to add buckets, I don't even see the settings icon. As I checked the logs i can see below error: nifi_registry_container_persistent | 2023-06-29 15:23:01,075 INFO [NiFi Registry Web Server-23] o.a.n.r.w.m.AccessDeniedExceptionMapper identity[CN=admin, OU=NiFi], groups[] does not have permission to access the requested resource. Unknown user with identity 'CN=admin, OU=NiFi'. Returning Forbidden response.nifi_registry_container_persistent | 2023-06-29 15:23:01,129 INFO [NiFi Registry Web Server-20] o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos service ticket login not supported by this NiFi Registry. Returning Conflict response.nifi_registry_container_persistent | 2023-06-29 15:23:01,190 INFO [NiFi Registry Web Server-16] o.a.n.r.w.m.IllegalStateExceptionMapper java.lang.Illegal
Hi, I am running a query on Hive through my Spark application using HiveWarehouseConnector.I want to use a particular YARN queue for Tez job launched by HiveWarehouseConnector (custom queue configuration at application level).I have tried following two ways:1. By using Spark conf and setting spark.hive.tez.queue.name = <queue name>2. By setting tez.queue.name parameter in the hiveserver2 URL as suggested in the following thread https://community.cloudera.com/t5/Support-Questions/Setting-yarn-queue-for-hive-with-beeline/td-p/161499 I am able to set the queue for beeline using the URL option. However, any of the options is not working for HiveWarehouseConnector. Can anyone please help in this regard?
I have NiFi installed on an EC2 instance NiFi version - 1.22.0 I have a requirement to pull files text files from s3 bucket and write them locally in NiFi as PoC. I have configured an input_port and FetchS3Object and PutFile processor as shown in the screenshot. The Nifi ec2 intance already has access to the s3 bucket and I have checked that by using "aws s3 cp s3_uri ./" command to copy files from s3 bucket to ec2 instance and it worked, However, for some reason, it seems NiFi FetchS3Object processor is running into some issue. The Nifi-app.log havent been very helpful even though I enabled "debug" level on the processor. It showed no error or warn messages. The only messages the processor showed is on the gui as shown in the screenshot. I have shown the values I configured in configuration. Only specified bucket name, object key, which I initially set to *.evtx but then changed to * to get any file and region option. Did not specify access key or se
I want to execute export command to take the PSQL dump of the database and transfer it using NIFI and the execute the import command at the destination db.can anyone provide a flow to do so ?
Hi guysI'm new to nifi and I need some help if possible.I am consuming from a Kafka Consumer that, when paused for some time, returns the json with several records, and I would like to divide them into separate files 1:1ConsumeKafka ResultSet: here is my json from 1 record { "before":null, "after":{ "uid":"80faad95-3383-4093-b4c1-e1840f076eef", "nome":"yuri", "sobrenome":"feminino", "email":"yuri.feminino@garupa.co", "celular":"53999122007", "cpf":"3", "avaliacao":"AfQ=", "created_at":1687173349000, "updated_at":1687423717000, "promocode":null, "fcm_registration_id":"edzz9eK3QDOUx-jsmpix4K:APA91bFPjXEsD7h-fYzih-pbr-TMfnF21Qk6sPDQYoa52etUwuIn3jCv1nM7ts4QYQ6f2Wcg-S88KdZsAqcEk98ZUAle-xnvFreavHUjB9_JBjj5wKDfjIYx6Z0en65Po_9knVGpz1u9", "paypal_client_id":null, "paypal_refresh_token":null, "foto":"https://garupa-dev.s3.amazonaws.com/207e9b35-c4b2-4660-af4f-cc20a629b581.jpg", "stripe_cust
Hi all, Is it possible to install Nifi in the Cloudera CDP Private Cloud Base trial? Its not available from the list of services and when I try to add the Parcel (https://archive.cloudera.com/p/cfm2/2.1.5.1000/redhat7/yum/tars/parcel/CFM-2.1.5.1000-56-el7.parcel) it requires authentication. Its strange as just about every other service is there. Thanks.
Hi Team, I require to pass the json created via JoltTranformJSON into the InvokeHTTP call as the body of the rest api post request, require to authenticate the api call via api key (need to send in the header) and need to get the api call response into a variable in Nifi 1.21.0. Can someone please help on this ? Thank you
Hi..... We are using following commands to get the updated records count in our table after Kerberos Authentication implementation and Ranger Installation. We get count1 variable before update the table count1=$(beeline -u "jdbc:hive2://dev-lisa.realm.com:10000/default;principal=hive/dev-lisa.intranet.slt.com.lk@REALM.COM;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks" -n hadoopuser02 -e 'SELECT count(*) from table_name; ') We get count2 variable after update the table count2=$(beeline -u "jdbc:hive2://dev-lisa.realm.com:10000/default;principal=hive/dev-lisa.intranet.slt.com.lk@REALM.COM;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks" -n hadoopuser02 -e 'SELECT count(*) from table_name; ') entries=$(echo "$count2-$count1") we unable to get the "entries" value into variable. please support to fix this.
Hello, @MattWho I upgraded my local Nifi instance from 1.21 to 1.22 using brew once its upgraded all my flows/processor groups were removed and I tried to recover by coping the flow.xml.gz from the archive the config folder but still I'm not able to get my flows is there any other way that I can get my flows back into the new instance? ThanksVas
Dear Community! I have a simple flow in Apache NiFi 1.20:1. QueryDatabaseTableRecord -> 2. SplitAvro -> 3. ConvertAvroToJson -> 4. LogAttribute -> 5. PutDatabaseRecord Flow descrition:1. Read records from a table order by "updated" column (a datetime with millisecs).2. put each of records into single flowfile3. Convert a record into JSON4. Log JSON into log file.5. insert a record to different databaseEach of processors have only 1 single concurrent thread on SETTING page! Everything works fine, except sometime the order of the records will be changed according to "updated" column. Could you please find me the cause? How can be ensured the order of records right?
Hello, I have gone through Joining collection in SOLR - Part-1 i.e. join with local shardWe all are waiting for Part - 2. i.e. Joining collection with multi shard. Part-2 of below link.https://community.cloudera.com/t5/Community-Articles/Joining-Collections-in-SOLR-Part-I/ta-p/246952As, now SOLR supports multiple collection joining with little bit config in solrconfig.xml file. We love to have an example like in Part-1 to get an idea how we can achieve joining of multiple shard collection. Please look at below link of SOLR. With 8.6.0 version, it allows multiple shard joininghttps://solr.apache.org/news.htmlAlso please have a look and search for below string in the link"Join Query Parser Definition in solrconfig.xml"https://solr.apache.org/guide/8_6/other-parsers.htmlEagerly waiting for part-2 with an example to join between multiple shards
Hi Everyone, I want to set up an outbound Apache Nifi SMTP email server with Outlook so that It could send the outbound email. Does anybody have any suggestions for this? I have tried with the ListenSMTP processor but not sure how to set up the "Listening Port", and what other components it would require for that I have tried to follow this page: https://medium.com/@tspann/using-nifi-as-an-smtp-email-test-server-713677ac5b67But not so sure how to configure that processor with Outlook, or even need outlook at all for that email server. Thank you, everyone, for reading.
when i use hdfs comands it display errors: hadoop fs -mkdir /user/,,myfile,, or "hadoop fs -put myfile.txt /user/,,/,," hadoop will display"Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:dr,,,"and so on,why?who can help me?
I'm trying to add users after setting up LDAP in Grafana on CDSW.Currently, only the default Admin account exists and is required to be added with email approval.We manage users using LDAP of AD (Acrive Directory).While attempting to add LDAP user settings in Grafana, we found that the current physical path of Grafana, "/var/lib/grafana/", does not exist.I tried to add LDAP settings to grafana.ini and ldap.toml in "/var/lib/grafana", but the file does not exist even if I search the entire server.When I accessed the Grafana pod directly and checked, path and files existed.Where can I find the grafana.ini and ldap.toml files?Is there any other way to solve this problem?Thank you for any help.
the example is the next:GenerateFlowFile content:I want to clean the ETX character with a python script (Nifi wasn't able to clean it with replaceText), so I decided to pass the list to a python script to clean it, but am not sure how to read the Flowfile content. The list (my_list) should be passed as Flowfile to the python script.
I want to use RouteOnAttribute to filter on a regex. To do so, I added two attributes in the Properties: PropertyValueprivate${ip:find('^(10|127|169\.254|172\.1[6-9]|172\.2[0-9]|172\.3[0-1]|192\.168)\.')}public${ip:find('^([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(?<!172\.(16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31))(?<!127)(?<!^10)(?<!^0)\.([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(?<!192\.168)(?<!172\.(16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31))\.([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$')} This processor shows two errors, both the same: `Unexpected token ':' at line 1 column 4`Why is this an error?
Hi I get this error when i try to create a new cluster or add a new host to an exists cluster..It's the basic way to do it through the cloudera manager, but I can't figure out how to fix it, all nodes can access the internet and have been working before in another cluster that was removedThis is an screenshot the agent log in the node show me this: Traceback (most recent call last):File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1390, in _send_heartbeatself.cfg.master_port)File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 469, in __init__self.conn.connect()File "/usr/lib64/python2.7/httplib.py", line 833, in connectself.timeout, self.source_address)File "/usr/lib64/python2.7/socket.py", line 571, in create_connectionraise errtimeout: timed out[07/Jul/2021 18:21:00 +0000] 1352 MainThread agent WARNING Long HB processing time: 45.0577571392[07/Jul/2021 18:21:00 +0000] 1352 MainThread agent WARNING Dela
Is it possible to reference parameters within the ExecuteScript processor? My current solution is to create an attribute which contains the parameter value, but it would be a much cleaner solution to be able to directly reference the parameter from the parameter context in my script. For reference, I am using Python.
HI , Could you please help me How to merge two pdf files which are coming from two flow files into one pdf file in Nifi. I'm trying to merge two files (invoice file and QR File) from two flowfiles and used mergecontent processor but only single file(first file from 2 input files but not merged file) coming as merged file, with the combined file size. Files flow:for the first flow file updateattribute maintained fragment.index=0 and second flowfile udpateattribute maintained fragment.index=1 and for combined flowfile updateattribute Before merge processor given fragment.count=2 Mergecontent Config: Can anyone help on this? Regards,Pradeep
Hi Team, I have below requirement, my flow file contains below data:[ {{"ID" : "100000","Date" : "2022-09-22","Start Time" : "08:00","End Time" : "14:00",}, {"ID" : "100001","Date" : "2022-09-02","Start Time" : "08:00","End Time" : "14:00",},{"ID" : "100002","Date" : "2022-09-02","Start Time" : "08:00","End Time" : "14:00",}} ] [ {"res_data" : {"Record" : "Record 2","Description" : "Invalid values for ID or Date"},{"Record" : "Record 3","Description" : "Invalid values for ID or Date"}} ] And the requirement is:if concatenate ('Record' with index of array of json object == res_data.Record then it should return{{"ID" : "100001","Date" : "2022-09-02","Start Time" : "08:00","End Time" : "14:00","Record" : "Record 2","Description" : "Invalid values for ID or Date"}} and so on else{"ID" : "100000","Date" : "2022-09-02","Start Time" : "08:00","Time Out" : "14:00","End Time" : "14:00","Description": null} and the final output looks like[{"ID" : "100001",
Hi all, I have an requirement where I need to parse the data into the required formatInput: {"Message" : "\nRecord 1:\nRequired data is missing. \n\nRecord 2:\nprocessing failed\n"} Here the content and delimiters are not fixed. The fixed part is only /nRecord keyword on which I am writing the Script. But I am not getting desired Output using Groovy. desired Output: [{"Record 1": "nRequired data is missing"}, {"Record 2": "processing failed"}] I have written Groovy Script for the same but I am getting empty array. import org.apache.commons.io.IOUtilsimport groovy.json.*import java.util.ArrayListimport java.nio.charset.*import java.nio.charset.StandardCharsetsimport groovy.json.JsonSlurperimport groovy.json.JsonBuilderdef flowFile = session.get()if(!flowFile) returntry {flowFile = session.write(flowFile,{ inputStream, outputStream ->def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)splitted = text.split('\nRecord')int j = splitted.size
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.