Q&A

Hi all, Is it possible to install Nifi in the Cloudera CDP Private Cloud Base trial? Its not available from the list of services and when I try to add the Parcel (https://archive.cloudera.com/p/cfm2/2.1.5.1000/redhat7/yum/tars/parcel/CFM-2.1.5.1000-56-el7.parcel) it requires authentication. Its strange as just about every other service is there. Thanks. 

Hi Team, I require to pass the json created via JoltTranformJSON into the InvokeHTTP call as the body of the rest api post request, require to authenticate the api call via api key (need to send in the header) and need to get the api call response into a variable in Nifi 1.21.0. Can someone please help on this ?   Thank you

Hi.....   We are using following commands to get the updated records count in our table after Kerberos Authentication implementation and Ranger Installation.   We get count1 variable before update the table count1=$(beeline -u "jdbc:hive2://dev-lisa.realm.com:10000/default;principal=hive/dev-lisa.intranet.slt.com.lk@REALM.COM;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks" -n hadoopuser02 -e 'SELECT count(*) from table_name; ')   We get count2 variable after update the table count2=$(beeline -u "jdbc:hive2://dev-lisa.realm.com:10000/default;principal=hive/dev-lisa.intranet.slt.com.lk@REALM.COM;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks" -n hadoopuser02 -e 'SELECT count(*) from table_name; ')   entries=$(echo "$count2-$count1") we unable to get the "entries" value into variable. please support to fix this.

Hello, @MattWho I upgraded my local Nifi instance from 1.21 to 1.22 using brew once its upgraded all my flows/processor groups were removed and I tried to recover by coping the flow.xml.gz  from the archive the config folder but still I'm not able to get my flows is there any other way that I can get my flows back into the new instance? ThanksVas 

Dear Community! I have a simple flow in Apache NiFi 1.20:1. QueryDatabaseTableRecord -> 2. SplitAvro -> 3. ConvertAvroToJson -> 4. LogAttribute -> 5. PutDatabaseRecord Flow descrition:1. Read records from a table order by "updated" column (a datetime with millisecs).2. put each of records into single flowfile3. Convert a record into JSON4. Log JSON into log file.5. insert a record to different databaseEach of processors have only 1 single concurrent thread on SETTING page! Everything works fine, except sometime the order of the records will be changed according to "updated" column. Could you please find me the cause? How can be ensured the order of records right? 

Hello, I have gone through Joining collection in SOLR - Part-1 i.e. join with local shardWe all are waiting for Part - 2. i.e. Joining collection with multi shard. Part-2 of below link.https://community.cloudera.com/t5/Community-Articles/Joining-Collections-in-SOLR-Part-I/ta-p/246952As, now SOLR supports multiple collection joining with little bit config in solrconfig.xml file. We love to have an example like in Part-1 to get an idea how we can achieve joining of multiple shard collection. Please look at below link of SOLR. With 8.6.0 version, it allows multiple shard joininghttps://solr.apache.org/news.htmlAlso please have a look and search for below string in the link"Join Query Parser Definition in solrconfig.xml"https://solr.apache.org/guide/8_6/other-parsers.htmlEagerly waiting for part-2 with an example to join between multiple shards

Hi Everyone, I want to set up an outbound  Apache Nifi SMTP email server with Outlook so that It could send the outbound email. Does anybody have any suggestions for this? I have tried with the ListenSMTP processor but not sure how to set up the "Listening Port", and what other components it would require for that I have tried to follow this page: https://medium.com/@tspann/using-nifi-as-an-smtp-email-test-server-713677ac5b67But not so sure how to configure that processor with Outlook, or even need outlook at all for that email server. Thank you, everyone, for reading. 

when i use hdfs comands it display errors:   hadoop fs -mkdir /user/,,myfile,, or   "hadoop fs -put myfile.txt /user/,,/,,"   hadoop will display"Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:dr,,,"and so on,why?who can help me?

I'm trying to add users after setting up LDAP in Grafana on CDSW.Currently, only the default Admin account exists and is required to be added with email approval.We manage users using LDAP of AD (Acrive Directory).While attempting to add LDAP user settings in Grafana, we found that the current physical path of Grafana, "/var/lib/grafana/", does not exist.I tried to add LDAP settings to grafana.ini and ldap.toml in "/var/lib/grafana", but the file does not exist even if I search the entire server.When I accessed the Grafana pod directly and checked, path and files existed.Where can I find the grafana.ini and ldap.toml files?Is there any other way to solve this problem?Thank you for any help.

the example is the next:GenerateFlowFile content:I want to clean the ETX character with a python script (Nifi wasn't able to clean it with replaceText), so I decided to pass the list to a python script to clean it, but am not sure how to read the Flowfile content. The list (my_list) should be passed as Flowfile to the python script.

I want to use RouteOnAttribute to filter on a regex. To do so, I added two attributes in the Properties: PropertyValueprivate${ip:find('^(10|127|169\.254|172\.1[6-9]|172\.2[0-9]|172\.3[0-1]|192\.168)\.')}public${ip:find('^([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(?<!172\.(16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31))(?<!127)(?<!^10)(?<!^0)\.([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(?<!192\.168)(?<!172\.(16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31))\.([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$')} This processor shows two errors, both the same: `Unexpected token ':' at line 1 column 4`Why is this an error? 

Hi I get this error when i try to create a new cluster or add a new host to an exists cluster..It's the basic way to do it through the cloudera manager, but I can't figure out how to fix it, all nodes can access the internet and have been working before in another cluster that was removedThis is an screenshot   the agent log in the node show me this: Traceback (most recent call last):File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1390, in _send_heartbeatself.cfg.master_port)File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 469, in __init__self.conn.connect()File "/usr/lib64/python2.7/httplib.py", line 833, in connectself.timeout, self.source_address)File "/usr/lib64/python2.7/socket.py", line 571, in create_connectionraise errtimeout: timed out[07/Jul/2021 18:21:00 +0000] 1352 MainThread agent WARNING Long HB processing time: 45.0577571392[07/Jul/2021 18:21:00 +0000] 1352 MainThread agent WARNING Dela

Is it possible to reference parameters within the ExecuteScript processor? My current solution is to create an attribute which contains the parameter value, but it would be a much cleaner solution to be able to directly reference the parameter from the parameter context in my script. For reference, I am using Python.

HI ,     Could you please help me How to merge two pdf files  which are coming from two flow files into one pdf file in Nifi. I'm trying to merge two files (invoice file and QR File) from two flowfiles and used mergecontent   processor but only single file(first file from 2 input files but not merged file) coming as merged file, with the combined file size. Files flow:for the first flow file updateattribute maintained fragment.index=0 and second flowfile udpateattribute maintained fragment.index=1 and for combined flowfile updateattribute Before merge processor  given fragment.count=2 Mergecontent Config:  Can anyone help on this? Regards,Pradeep

Hi Team, I have below requirement, my flow file contains below data:[ {{"ID" : "100000","Date" : "2022-09-22","Start Time" : "08:00","End Time" : "14:00",}, {"ID" : "100001","Date" : "2022-09-02","Start Time" : "08:00","End Time" : "14:00",},{"ID" : "100002","Date" : "2022-09-02","Start Time" : "08:00","End Time" : "14:00",}} ] [ {"res_data" : {"Record" : "Record 2","Description" : "Invalid values for ID or Date"},{"Record" : "Record 3","Description" : "Invalid values for ID or Date"}} ] And the requirement is:if concatenate ('Record' with index of array of json object == res_data.Record then it should return{{"ID" : "100001","Date" : "2022-09-02","Start Time" : "08:00","End Time" : "14:00","Record" : "Record 2","Description" : "Invalid values for ID or Date"}} and so on else{"ID" : "100000","Date" : "2022-09-02","Start Time" : "08:00","Time Out" : "14:00","End Time" : "14:00","Description": null} and the final output looks like[{"ID" : "100001",

Hi all, I have an requirement where I need to parse the data into the required formatInput: {"Message" : "\nRecord 1:\nRequired data is missing. \n\nRecord 2:\nprocessing failed\n"} Here the content and delimiters are not fixed. The fixed part is only /nRecord keyword on which I am writing the Script. But I am not getting desired Output using Groovy. desired Output: [{"Record 1": "nRequired data is missing"}, {"Record 2": "processing failed"}] I have written Groovy Script for the same but I am getting empty array. import org.apache.commons.io.IOUtilsimport groovy.json.*import java.util.ArrayListimport java.nio.charset.*import java.nio.charset.StandardCharsetsimport groovy.json.JsonSlurperimport groovy.json.JsonBuilderdef flowFile = session.get()if(!flowFile) returntry {flowFile = session.write(flowFile,{ inputStream, outputStream ->def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)splitted = text.split('\nRecord')int j = splitted.size

I run NiFi on a docker engine (official image). It works fine but now I would like as part of a flow to connect to a MySQL instance.This requires me to configure a Database Driver Class Name. The value I found in an example on Internet (com.mysql.jdbc.Driver) requires a .jar (sorry, I know nothing about Java). What is the proper way to make this file available to NiFi? (not on a docker level - I have a volume mounted and can put is over there, but on a NiFi level, i.e. how to make NiFi aware of this file)

I have NiFi pipeline which get data from a csv and transform into a JSON. Then I want to send a post API request from the JSIN created.I need help setting up the InvokeHTTP processor in NiFi. API authorization is using an API key. I need assitance of how to define the api keys, url and JSON body inside InvokeHTTP processor 

Hi,I am hitting a kryoserializer buffer issue with the following simple line in PySpark (Spark version 2.4):df=ss.read.parquet(data_dir).limit(how_many).toPandas() Thus I am reading a partitioned parquet file in, limit it to 800k rows (still huge as it has 2500 columns) and try to convert toPandas. Kryoserializer.buffer.max is already at the maximum possible value:.config("spark.kryoserializer.buffer.max", "2047m")  What other ways are there to make it run (except of reducing the amount of rows even further down)? While in stage1 he is making many steps (approx 157), in stage2 he has only one step - and thus tries to juggle with a very large object.  The following in an excerpt of the error message: Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 158, localhost, executor driver): org.apache.spark.SparkException: Kryo serialization fai

I am getting following error in NiFi `ExecuteScript` processor. ```java18:07:09 UTCERROR9e1d9c5e-0188-1000-ffff-fffffbc2581710.42.62.250:8080ExecuteScript[id=9e1d9c5e-0188-1000-ffff-fffffbc25817] ExecuteScript[id=9e1d9c5e-0188-1000-ffff-fffffbc25817] failed to process due to org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: java.lang.IllegalStateException: java.lang.IllegalStateException: StandardFlowFileRecord[uuid=40d42213-2c7a-49a2-8c5d-5a7fc58bad2d,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1686706804350-73290, container=default, section=586], offset=6556, length=39642],offset=0,name=40d42213-2c7a-49a2-8c5d-5a7fc58bad2d,size=39642] already in use for an active callback or an InputStream created by ProcessSession.read(FlowFile) has not been closed in <script> at line number 37- Caused by: javax.script.ScriptException: java.lang.IllegalStateException: java.lang.IllegalStateException: StandardFlowFileRecord[uuid=40d4

Hello,I wanted to install Impala-shell for remote querying my CDP cluster, but looking with no much success I found the latest impala-shell client is in: archive.cloudera.com/p/cdh7/7.1.2.1/redhat7/yumnothing about 8 version, nothing about my CDP version: 7.1.8... Finally going with this "old" version but now I'm facing these issues:I already have python2 and 3But not sure what more to try... no information at all in cloudera docs, even in apache impala web page 😞  

we are planning to separate cluster hosts into 2 physical racks, 10 hosts for each rack. our doubt is: as we only have two physical racks, is it still recommended to use the "rack" option in cloudera manager to separate the hosts?  

联系请登陆zoom：560 656 3972其中之一的报错如下：2023-06-21 07:57:03,547 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@136] - Setting authorizedID: rangerkms 2023-06-21 07:57:03,547 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@1030] - adding SASL authorization for authorizationID: rangerkms 2023-06-21 07:57:03,612 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x488dce1cd620027, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203) at java.lang.Thread.run(Thread.java:745) 2023-06-21 07:57:03,612 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1035] - Closed socket connection for client /10.233.68.145:49378 which had sessionid 0x488dce1cd620027 2023-06-21 07:57:03,668 - ERR

We are having escaping issues in some expressions. Just an example (using Nifi 1.21.0):ExpressionValue{"$${key}": "value"}{"${key}": "value"}{"$${key}": "${literal('value')}"}{"$${key}": "value"}As one can see, the second expression is not escaping the dollar sign as it should (from the docs, see "Escaping Expression Language"), and just stays as "$$".Any comments/hints would be much appreciated.

we are having trouble starting the "navigator metadata server" service, in the "stderr" log we have the error below: Conflicting collector combinations in option list; please refer to the release notes for the combinations allowedError: Could not create the Java Virtual Machine.Error: A fatal exception has occurred. Program will exit.can help us with this problem?