Solved

Hi. I'm using the ConnectWebSocket processor to receive data from a remote server.The connection is established successfully, but eventually something happens and, even though the processor keeps running normally, the remote server indicates the connection is broken and I stop receiving data.Is there a way to monitor this connection using the ping/pong packets and some other processor or the nifi logs?

Suppose I have below code:```pythonimport pandas as pddf = pd.read_csv(SOME_PATH)df.to_csv(OUTPATH,index=False)``` How do I configure OUTPATH to be HDFS? 

Hi Team, Looking for some inputs on the below error. Basically we have a sample Spark Java application which reads an input file that is existing in HDFS, perform transformations on the data and rewrites it back to another file into HDFS. On spark-submit like below : spark-submit --master yarn --deploy-mode cluster --class com.sample.SparkJavaApiTest /tmp/sample-spark-java.jar <HDFS File Path : /user/user1/test.txt> <HDFS Output File : /user/user1/output.txt>The sample code works perfectly. However when the same job is submitted via spark-submit with files in Ozone File system getting the below error : spark-submit --master yarn --deploy-mode cluster --keytab /tmp/test.keytab --principal user1@EXAMPLE.COM --class com.sample.SparkJavaApiTest /tmp/sample-spark-java.jar 'ofs://sk-ozone-test1/user1vol/user1bucket/test.txt' 'ofs://sk-ozone-test1/user1vol/user1bucket/output'orspark-submit --master yarn --deploy-mode cluster --keytab /tmp/test.keytab --principal use

Hi, I want to read only a finite sample from an Apache Phoenix table into Spark dataframe without using primary keys in where condition. I tried using 'limit' clause in query as shown in the code below. Map<String, String> map = new HashMap<>();map .put("url", url );map .put("driver", "org.apache.phoenix.jdbc.PhoenixDriver");map .put("query", "select * from table1 limit 100"); Dataset<Row> df= spark.read().format("jdbc").options(map).load(); Following exception has occurred.java.sql.SQLFeatureNotSupportedException: Wildcard in subqueries not supported. at org.apache.phoenix.compile.FromCompiler Then if I use limit() method of dataframe as shown below.map .put("query", "select * from table1");Dataset<Row> df = spark.read().format("jdbc").options(map).load().limit(100); In this case, spark is first reading all the data into dataframe then it is trying to filter the data. The mentioned 'table1' has millions of rows. I am getting t

failed during installing spark 3 for CDP718, I didn't find any log on the installed nodethen I checked https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/cds-3/topics/spark-install-spark-3-parcel.html, try to install CDS3,follow the offical doc, put parcal under /opt/cloudera/csdafter restart cloudera manager service, I still can't see this parcel under Parcelsso could you please give a solution for it, thanks a lot 

I am using Spark 2.3.2.3.1.0.0-78. I tried to use: spark_session.sparkContext._conf.get('spark.executor.memory') but I only received 'None'.Can someone help me, please?

Hi ,Is there a way to have more than one Admin user in NIFI ?How do you manage users/nodes,only with one user ?I want additional NIFI user that will add users and authorize them. Thanks Edi

Hello,I have upgraded nifi from 1.16.3 to 1.21.0Now, the processors ListFTP, FetchFTP, PutFTP won't work with filenames that contains french accents like à é etc..It worked like a charm on Nifi 1.16.3Could you please help me on this topic ? I will happy to provide all the information that you need to debug the problem. Thank you.   

Hello, I'm new to using labels on YARN nodes. I have successfully set up the labels but the scheduler is allocating all resources to the DEFAULT_PARTITION under "Effective Capacity" and 0 resources to the labeled partition.  As the screenshots illustrate, the NodeManager is launching with the correct label and has the correct resources assigned to that label, however, applications will not start when assigned to that label because although the partition has resources assigned to it, the queue under the partition does not. Here's my capacity-scheduler.xml:    <configuration> <property> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name> <value>1.0</value> </property> <property> <name>yarn.scheduler.capacity.resource-calculator</name> <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value> </property>

Hello, I'm trying to upload the tez tar files using a rest api command, but I'm not able to find any in the swagger documentation (maybe I just missed it..)I've tried replicating the url that is pointed at when using the button through the UI, but I require the Tez service ID for it, and I couldn't find an API call to get that neither.. Is there an easier way to do this perhaps? Or if you know about one of the api calls I'm missing from the swagger docs mentioned above, that would help me as well. Thanks!

Hi there,sorry for my silly question. I have a InvokeHTTP processor which returns plaintext access token in the response body, since it is larger than 256 bytes I cannot assign it to an attribute. I need to use the token in another InvokeHTTP as a header to retrieve data. Token is a plain text (not a json). Thanks!

I am getting below errors while trying to trigger a python script using ExecuteStreamCommand Processor in NiFi. Attaching the screenshot of the properties of the processor. Please help   Below is the error in bulletin board 23:21:24 CST ERROR 3bc05a73-0170-1000-6f23-80001cc130eb   ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Failed to process session due to java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application: org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Cannot run program "C:\Users\503067805\Desktop\IDP_Project\POC_Well_Path\Trigger_Python_POC\Python\csv_convertion_final.py": CreateProcess error=193, %1 is not a valid Win32 application   23:21:24 CST ERROR 3bc05a73-0170-1000-6f23-80001cc130eb   ExecuteStreamCommand[id=3bc05a73-0170-1000-6f23-80001cc130eb] Could

I have a template to do a post method using InvokeHTTP, there can be many number of post requests so, I want to pause the execution for 5 minutes after every 1000 flowfiles. Is it possible to get the count of flowfiles processed?I tried creating a counter, but sadly I couldn't come up with a logic to increase the count.  I do not have any release signal identifier to use Wait processor. please advise. Thanks in advance

We use Hadoop and Hive 3.1, and vendor is Cloudera and is licensed. Now there is a requirement to connect to that Hive environment from another application server to extract data and we are planning to use ODBC drivers. We would like to know if we need to purchase the license to use Cloudera Hive ODBC driver, or can it be downloaded and used to connect to Cloudera Hadoop and Hive v3.1 without purchasing a license.

Hi,My task is to move data from my locally installed SQL server (16.0) to Azure SQL (12.0) hosted on my Azure subscription. So far, I have used ExecuteSQL processor to run my select query from SQL Server and it returns me the rows successfully. I have created a Database Polling Connection Service which connect to my SQL Server.After fetching the rows, I am using PutDatabaseRecord processor to insert the fetched rows into Azure SQL but its showing me error that "use statement is not supported to switch between databases". Note that the Database Polling Connection Service is different for PutDatabaseRecord where I have mentioned connection string of my Azure SQL.Please if anyone can give feedback on this issue and suggest any workaround. 

Hi Team, Looking for inputs on defaulting HDFS storage to Ozone FS. As part of HDFS configuration from Cloudera Manager, in core-site.xml tried setting the following properties mentioned in : https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/ozone-storing-data/topics/ozone-setting-up-ozonefs.html https://ozone.apache.org/docs/1.0.0/interface/ofs.html Post restarting the service, SecondaryNameNode fails with below exception : Failed to start secondary namenodejava.io.IOException: This is not a DFSat org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.getInfoServer(SecondaryNameNode.java:456)at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:239)at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:194)at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:690) However other role types i.e DataNode and NamedNode start up fine.&

Hi ,I have an issue integrating NIFI with LDAP groups, I created two groups in LDAP and added users to those groups but the NIFI shows those groups empty without users. I started DEBUG level, and I can see from the debug the following message devops-nifi-general-users-viewer contains member CN=Edi,OU=XX,OU=XX,OU=XXX,OU=XXX,OU=XXX,OU=XXX,OU=XXXX,DC=XXXX,DC=local but that user was not found while searching users. This may be due to a misconfiguration or it's possible the user is not a NiFi user. Ignoring group membership. Any suggestion on what can be the issue ? why can't I see the users in the groups?(attaching the configuration files )full log messages 2023-08-14 14:52:11,537 DEBUG [main] o.a.n.ldap.tenants.LdapUserGroupProvider devops-nifi-general-users-viewer contains member CN=Edi,OU=XX,OU=XX,OU=XXX,OU=XXX,OU=XXX,OU=XXX,OU=XXXX,DC=XXXX,DC=local but that user was not found while searching users. This may be due to a misconfiguration or it's possible the user is not a

Hello everyone,I apologize for my limited English proficiency. I am currently using Nifi version 1.14.My issue is that Nifi frequently stops, and the processors become unresponsive and inactive. However, the UI operations continue to work normally. After a few minutes, or sometimes after I restart, these flows start working again. I have checked the logs, but I haven't found any errors. I have configured the Maximum Timer Driven Thread Count to be 32 (as my system has only 8 cores). Unfortunately, due to certain constraints, I am unable to upgrade to a newer version at the moment.I would greatly appreciate any suggestions or assistance you could provide. Your help would mean a lot to me. Thank you in advance for your response.Best regards, 

Hello, how do I properly use the nifi-api to create flowfile from a file? I tried to do what is described here: https://nifi.apache.org/docs/nifi-docs/rest-api/index.html under 'Data transfer' I've tried to use a simple curl command to transfer a file as flowfile to an input port as follows:curl -X POST -F 'file=@Documents/Nifi_Docs/example.txt' http://10.0.1.111:8080/nifi-api/data-transfer/input-ports/dac743ae-0189-1000-c6a6-efd211a162d1/transactions/1010/flow-files As response I get a 415 error Unsupported Media Type. What is the supported media type then? Anyone tried this before? What is the transaction id good for? I just put 1010 as a random number. I also tried sending from postman

Hi NIFI Team, We have currently installed NIFI with OIDC configuration which is working perfectly fine using the UI without any issues. But when using NIFI toolkit commands and providing Bearer token we are receiving ''Unauthorized error". Does NiFI provide any endpoint to get the OIDC Token separately which can then be used in the Toolkit commands? or please suggest if there is any other approach by which we can make a successful call to NIFI from Nifi Toolkit. Thanks in advance    

hello cloudera community, I have a problem in a datanode only in hdfs this datanode has 10 disks and in the hdfs settings (dfs.datanode.data.dir) is configured the 10 disks but for some time this datanode is not recognizing one of the disks configured only was already done the restart of the datanode and even so did not back to recognize looking at the datanode logs, it is not really recognizing the disk, only the 9 disks these disks are also being used by other services in the cluster and these other services are able to recognize the 10 disks without any problem, only hdfs is having this problem could you help us to solve this problem?

PutSQL is penalizing the queue and  no exceptions are found in logs2023-08-09 08:24:07,045 DEBUG [Timer-Driven Process Thread-5] o.apache.nifi.processors.standard.PutSQL PutSQL[id=8e5c441f-696c-3145-8ba0-67499ed29990] Not enough FlowFiles for transaction. Returning all FlowFiles to queue2023-08-09 08:24:07,046 DEBUG [Timer-Driven Process Thread-5] o.apache.nifi.processors.standard.PutSQL PutSQL[id=8e5c441f-696c-3145-8ba0-67499ed29990] No incoming FlowFiles.2023-08-09 08:24:07,047 DEBUG [Timer-Driven Process Thread-5] o.apache.nifi.processors.standard.PutSQL PutSQL[id=8e5c441f-696c-3145-8ba0-67499ed29990] Not enough FlowFiles for transaction. Returning all FlowFiles to queue2023-08-09 08:24:07,047 DEBUG [Timer-Driven Process Thread-5] o.apache.nifi.processors.standard.PutSQL PutSQL[id=8e5c441f-696c-3145-8ba0-67499ed29990] No incoming FlowFiles.I know if there is SQLException, Flowfile will be forwarded via failure else the flowfile will be added back to the queue for any other exce

Quoting directly from the official docu: FetchSMB: "Fetches files from a SMB Share. Designed to be used in tandem with ListSmb."My workflow is pretty simple, I have ListSMB which reads from a shared network directory, connected to FetchSMB which should get 1 file (the processor forces you to put the path of a specific file or it won't work) and finally PutHDFS to write all the files in a distributed file system (hadoop).I don't / can't understand why whenever I use List and Fetch in tandem the pipeline will just take all the files it can find on the directory specified in the List processor but won't write to the HDFS the single file I specified on the Fetch, instead it will write on the HDFS all the files from the List. The fetch, directly connected to the HDFS, should pass only a single file. What am I missing here ? What is the purpose of the fetch if it is "by-passed" by the list processor ?On top of that, all the file written to the HDFS will pick the same weight (KB or MB) of the

Good afternoon!  In planning out how we reset an environment when we have a refresh of one of our source systems, I need to ensure that any stateful processors (like QueryDatabaseTableRecord) have their state cleared so that they pick up all new records. Is there a way to find all processors storing state?  Or, even better, to just destroy the state of all processors on the NiFi instance? Thanks.

Hi,We have a solution in our flow which is currently running on 1.16.3 and we are referencing parameters dynamically for some schema validations. Now we have upgraded NIFI to version 1.21.0 and this functionality of function "evaluateELString()" not working as expected. Example -  Using above NIFI expression language, we used to reference parameters dynamically but now we are getting null values although parameter is not sensitive.  I find out this was changed in NIFI version 1.19.0 to not expose sensitive parameters but why have they change it for all the parameters - NIFI-10674I can see a JIRA raised for similar issue here - NIFI-11627Can someone please help us over here ? I do not want to hardcode this thing as we have 10's of schemas in parameters and I do not want to hardcode for each one.