Q&A

I'm creating a pipeline in NiFi and I have a csv file to be extracted and I want to skip the first 10 rows and read the csv. Need some assistance on how to achieve this in Apache NiFi

Hi. I'm using the ConnectWebSocket processor to receive data from a remote server.The connection is established successfully, but eventually something happens and, even though the processor keeps running normally, the remote server indicates the connection is broken and I stop receiving data.Is there a way to monitor this connection using the ping/pong packets and some other processor or the nifi logs?

Hello,Is there an online resource available outlining a working example for Site-to-Site communication between two NiFi instances?

Suppose I have below code:```pythonimport pandas as pddf = pd.read_csv(SOME_PATH)df.to_csv(OUTPATH,index=False)``` How do I configure OUTPATH to be HDFS? 

Hi Team, Looking for some inputs on the below error. Basically we have a sample Spark Java application which reads an input file that is existing in HDFS, perform transformations on the data and rewrites it back to another file into HDFS. On spark-submit like below : spark-submit --master yarn --deploy-mode cluster --class com.sample.SparkJavaApiTest /tmp/sample-spark-java.jar <HDFS File Path : /user/user1/test.txt> <HDFS Output File : /user/user1/output.txt>The sample code works perfectly. However when the same job is submitted via spark-submit with files in Ozone File system getting the below error : spark-submit --master yarn --deploy-mode cluster --keytab /tmp/test.keytab --principal user1@EXAMPLE.COM --class com.sample.SparkJavaApiTest /tmp/sample-spark-java.jar 'ofs://sk-ozone-test1/user1vol/user1bucket/test.txt' 'ofs://sk-ozone-test1/user1vol/user1bucket/output'orspark-submit --master yarn --deploy-mode cluster --keytab /tmp/test.keytab --principal use

Hi, I want to read only a finite sample from an Apache Phoenix table into Spark dataframe without using primary keys in where condition. I tried using 'limit' clause in query as shown in the code below. Map<String, String> map = new HashMap<>();map .put("url", url );map .put("driver", "org.apache.phoenix.jdbc.PhoenixDriver");map .put("query", "select * from table1 limit 100"); Dataset<Row> df= spark.read().format("jdbc").options(map).load(); Following exception has occurred.java.sql.SQLFeatureNotSupportedException: Wildcard in subqueries not supported. at org.apache.phoenix.compile.FromCompiler Then if I use limit() method of dataframe as shown below.map .put("query", "select * from table1");Dataset<Row> df = spark.read().format("jdbc").options(map).load().limit(100); In this case, spark is first reading all the data into dataframe then it is trying to filter the data. The mentioned 'table1' has millions of rows. I am getting t

failed during installing spark 3 for CDP718, I didn't find any log on the installed nodethen I checked https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/cds-3/topics/spark-install-spark-3-parcel.html, try to install CDS3,follow the offical doc, put parcal under /opt/cloudera/csdafter restart cloudera manager service, I still can't see this parcel under Parcelsso could you please give a solution for it, thanks a lot 

I am using Spark 2.3.2.3.1.0.0-78. I tried to use: spark_session.sparkContext._conf.get('spark.executor.memory') but I only received 'None'.Can someone help me, please?

Hi ,Is there a way to have more than one Admin user in NIFI ?How do you manage users/nodes,only with one user ?I want additional NIFI user that will add users and authorize them. Thanks Edi

Hello,I have upgraded nifi from 1.16.3 to 1.21.0Now, the processors ListFTP, FetchFTP, PutFTP won't work with filenames that contains french accents like à é etc..It worked like a charm on Nifi 1.16.3Could you please help me on this topic ? I will happy to provide all the information that you need to debug the problem. Thank you.   

Hello, I'm new to using labels on YARN nodes. I have successfully set up the labels but the scheduler is allocating all resources to the DEFAULT_PARTITION under "Effective Capacity" and 0 resources to the labeled partition.  As the screenshots illustrate, the NodeManager is launching with the correct label and has the correct resources assigned to that label, however, applications will not start when assigned to that label because although the partition has resources assigned to it, the queue under the partition does not. Here's my capacity-scheduler.xml:    <configuration> <property> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name> <value>1.0</value> </property> <property> <name>yarn.scheduler.capacity.resource-calculator</name> <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value> </property>

Hello, I'm trying to upload the tez tar files using a rest api command, but I'm not able to find any in the swagger documentation (maybe I just missed it..)I've tried replicating the url that is pointed at when using the button through the UI, but I require the Tez service ID for it, and I couldn't find an API call to get that neither.. Is there an easier way to do this perhaps? Or if you know about one of the api calls I'm missing from the swagger docs mentioned above, that would help me as well. Thanks!

Hi there,sorry for my silly question. I have a InvokeHTTP processor which returns plaintext access token in the response body, since it is larger than 256 bytes I cannot assign it to an attribute. I need to use the token in another InvokeHTTP as a header to retrieve data. Token is a plain text (not a json). Thanks!

I am getting below errors while trying to trigger a python script using ExecuteStreamCommand Processor in NiFi. Attaching the screenshot of the properties of the processor. Please help   Below is the error in bulletin board 23:21:24 CST ERROR 3bc05a73-0170-1000-6f23-80001cc130eb   ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Failed to process session due to java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application: org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application   23:21:24 CST ERROR 3bc05a73-0170-1000-6f23-80001cc130eb   ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Could

I have a template to do a post method using InvokeHTTP, there can be many number of post requests so, I want to pause the execution for 5 minutes after every 1000 flowfiles. Is it possible to get the count of flowfiles processed?I tried creating a counter, but sadly I couldn't come up with a logic to increase the count.  I do not have any release signal identifier to use Wait processor. please advise. Thanks in advance

We use Hadoop and Hive 3.1, and vendor is Cloudera and is licensed. Now there is a requirement to connect to that Hive environment from another application server to extract data and we are planning to use ODBC drivers. We would like to know if we need to purchase the license to use Cloudera Hive ODBC driver, or can it be downloaded and used to connect to Cloudera Hadoop and Hive v3.1 without purchasing a license.

Hi,My task is to move data from my locally installed SQL server (16.0) to Azure SQL (12.0) hosted on my Azure subscription. So far, I have used ExecuteSQL processor to run my select query from SQL Server and it returns me the rows successfully. I have created a Database Polling Connection Service which connect to my SQL Server.After fetching the rows, I am using PutDatabaseRecord processor to insert the fetched rows into Azure SQL but its showing me error that "use statement is not supported to switch between databases". Note that the Database Polling Connection Service is different for PutDatabaseRecord where I have mentioned connection string of my Azure SQL.Please if anyone can give feedback on this issue and suggest any workaround. 

Hi Team, Looking for inputs on defaulting HDFS storage to Ozone FS. As part of HDFS configuration from Cloudera Manager, in core-site.xml tried setting the following properties mentioned in : https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/ozone-storing-data/topics/ozone-setting-up-ozonefs.html https://ozone.apache.org/docs/1.0.0/interface/ofs.html Post restarting the service, SecondaryNameNode fails with below exception : Failed to start secondary namenodejava.io.IOException: This is not a DFSat org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.getInfoServer(SecondaryNameNode.java:456)at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:239)at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:194)at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:690) However other role types i.e DataNode and NamedNode start up fine.&

Hi ,I have an issue integrating NIFI with LDAP groups, I created two groups in LDAP and added users to those groups but the NIFI shows those groups empty without users. I started DEBUG level, and I can see from the debug the following message devops-nifi-general-users-viewer contains member CN=Edi,OU=XX,OU=XX,OU=XXX,OU=XXX,OU=XXX,OU=XXX,OU=XXXX,DC=XXXX,DC=local but that user was not found while searching users. This may be due to a misconfiguration or it's possible the user is not a NiFi user. Ignoring group membership. Any suggestion on what can be the issue ? why can't I see the users in the groups?(attaching the configuration files )full log messages 2023-08-14 14:52:11,537 DEBUG [main] o.a.n.ldap.tenants.LdapUserGroupProvider devops-nifi-general-users-viewer contains member CN=Edi,OU=XX,OU=XX,OU=XXX,OU=XXX,OU=XXX,OU=XXX,OU=XXXX,DC=XXXX,DC=local but that user was not found while searching users. This may be due to a misconfiguration or it's possible the user is not a

I am trying to convert the following date format -> 8/6/2023 12:46 am to this format -> 2023-08-06 00:46:00.I want to do this using NiFi. I have tried using the UpdateRecod processor for the same and used toDate() and format() for the above conversion. But it didn't give the desired results.How can I achieve this using NiFi?

Hello everyone,I apologize for my limited English proficiency. I am currently using Nifi version 1.14.My issue is that Nifi frequently stops, and the processors become unresponsive and inactive. However, the UI operations continue to work normally. After a few minutes, or sometimes after I restart, these flows start working again. I have checked the logs, but I haven't found any errors. I have configured the Maximum Timer Driven Thread Count to be 32 (as my system has only 8 cores). Unfortunately, due to certain constraints, I am unable to upgrade to a newer version at the moment.I would greatly appreciate any suggestions or assistance you could provide. Your help would mean a lot to me. Thank you in advance for your response.Best regards, 

Is there any known issue for CDP Private Base 7.1.8 and CM 7.7.1 on RHEL 8.8??Our company regulations are pushing hard to upgrade to RHEL 8.8 and after many rounds of negotiations we cannot delay it anymore. We saw that RHEL 8.7 was tested and recently added to the Cloudera Support Matrix but the RedHat support of this version is already over.When would we have some news about 8.8?Thanks in advance   

Hello I am new to NIFI and I have requirement like  use Apache Tika in NIFI to extract metadata of file . any help would be much appreciated .

Hello, how do I properly use the nifi-api to create flowfile from a file? I tried to do what is described here: https://nifi.apache.org/docs/nifi-docs/rest-api/index.html under 'Data transfer' I've tried to use a simple curl command to transfer a file as flowfile to an input port as follows:curl -X POST -F 'file=@Documents/Nifi_Docs/example.txt' http://10.0.1.111:8080/nifi-api/data-transfer/input-ports/dac743ae-0189-1000-c6a6-efd211a162d1/transactions/1010/flow-files As response I get a 415 error Unsupported Media Type. What is the supported media type then? Anyone tried this before? What is the transaction id good for? I just put 1010 as a random number. I also tried sending from postman

Getting above error while adding cloud management service to the hadoop cluster cluster.Attached server log for reference.Cluster DetailsIt's a five node cluster. Installed Cloudera server version 7.7.1. Installed services as below