Q&A

is there anyway to change username and credential for Nifi in Windows ? there is a command in Documentation to change it but I guess it works only in Linux

Beginner here...after installing the Hortonworks Sandbox HDP 3.0 on VirtualBox (Mac), I uploaded couple of csv files through ambari under the user name maria_dev by following the tutorial, was expecting to find in /users/maria_dev but logging in as maria_dev into the sandbox, i could not locate these and get an error "No such file or directory".  I managed to upload files via scp into the sandbox and complete the tutorial, however still confused where the upload to ambari UI might have gone. Not sure where the files went or i am looking at the wrong place. Any help will be appreciated.  

we are testing the trial version of the cloudera data platform (CDP7). But when we are running to the Step7 of the Parcel installation. It is stuck in the status of the distribution. Any suggestion for this step? Thanks.  

I am looking to upgrade the cluster from CR 7.1.7 to CR 7.1.7 SP2. Can it be done directly or do I have to upgrade to CR 7.1.7 SP1 and then to SP2? 

Hi, I'm new to Nifi and currently trying to understand how to improve its performance.  In this use case I am getting large zip files (30 - 700 GB) from a network share using the FetchFile processor and then unpacking with UnpackContent. Files that are in the 500GB+ range are taking several hours to move onto the unpack stage.  Files are eventually make their way through but I'm not sure on the best way to improve performance. The server is beefy with 1TB RAM, plenty of disk space, 40 core CPU.  Utilisation across CPU/Mem/Disk/Network I/O all seem under used whilst the processor is running. I've upped the max timer/event thread counts to 80/5 respectively but that doesn't seem to have made any  difference.  Is tackling the jvm min/max limits a good option to go for next? thanks for any guidance.

My use-case is: 1) Have API credentials2) Use UpdateAttribute to update (1) schema, (2) s3 bucket/location (my list of reports)3) Query API endpoint for report4) API endpoint paginates and gets more records5) Call MergeRecord6) Save to s3 Since 3, 4, 5, 6 are all the same, I'm re-using the processors like below (screenshot).  My problem is (5) MergeRecord will try to merge different schemas together, which is obviously a problem. How can I restructure this?  I'd like to re-use processors as much as possible, but still be able to add more schemas as my needs evolve. 

Hi,we would like install Cloudera CDP 7.1.7 SP1 on linux host with Linux Centos Stream 8.x.The information in Cloudera's support matrix is ​​generically "Centos". Can we consider them valid for Linux Centos Stream 8.x? Thank you.   

Error while processing statement: FAILED: Error in acquiring locks: Error communicating with the metastore (state=42000,code=10) Unable to create Hive Table, while creating a table getting error in acquiring locks: Error communicating with the metastore, ERROR : FAILED: Error in acquiring locks: Error communicating with the metastoreorg.apache.hadoop.hive.ql.lockmgr.LockException: Error communicating with the metastoreat org.apache.hadoop.hive.ql.lockmgr.DbLockManager.lock(DbLockManager.java:178)at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:447)at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocksWithHeartbeatDelay(DbTxnManager.java:463)at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:278)at org.apache.hadoop.hive.ql.lockmgr.HiveTxnManagerImpl.acquireLocks(HiveTxnManagerImpl.java:76)at org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.acquireLocks(DbTxnManager.java:95)at org.apache.hadoop.hive.ql.Dr

Please help design data flow in nifi, I need to insert the incoming JSON data into table which having single column. The condition are I need to concat all the keys and values and insert that data. 

Hi Team, I'm new to CDP and am currently looking at myaccount page "CDP Public Cloud Usage summary" staring to get a feel for the CDP product, but wanted to know if anyone has had experience in Billing and Metering of CDP subscriptions.  I'd like to know if it is at all possible to create rules around subscription spend eg. If estimated cost went above $20, can an alert generated advising me of this.Also is there any other info that might help with managing cost that this team might know about, that might be useful?  Thanks

I have an avro schema that I can get from a registry:  { "name": "bar_tbl", "fields": [ { "name": "foo1", "type": "string" }, { "name": "foo2", "type": "string" } ]}  I know how to use `EvaluateJsonPath` and get `name` and `fields`.I'd like to set an attribute with `UpdateAttribute` to equal:  select foo1, foo2 from bar_tbl  to use it later downstream in another processor.How can I achieve this? Or, more directly, I have a REST api that takes an url-encoded sql statement.  How can I feed the sql statement to `InvokeHttp`?

We are encountering some issues when starting Nifi 1.16.3 after calling encrypt-config.shFor background, Nifi is deployed and managed using an in-house deployment product, and is used as a component within our application stack. In general everything works fine when using Nifi 1.15.1, however, we are encountering issues when using Nifi 1.16.3.  See the following in nifi-bootstrap.log  2022-11-13 06:32:06,488 INFO [main] org.apache.nifi.bootstrap.Command Launched Apache NiFi with Process ID 828732022-11-13 06:32:14,269 INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for Bootstrap requests on port 329082022-11-13 06:34:06,217 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: Decryption Failed with Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]2022-11-13 06:34:06,218 ERROR [NiFi logging handler] org.apache.nifi.StdErr Shutting down...2022-11-13 06:34:06,945 INFO [main] org.apac

So guys,Need your help with something with which I have been struggling in the last couple of days. I just installed a NiFi cluster, composed of 5 machines. I have used NiFI 1.19.1 as it is the latest version.Now, when trying to create a DBCP Connection Pool, I keep on receiving the below error messages. Beside these two, I receive plenty more with other classes. Right now, I am using "openjdk version "11.0.8" 2020-07-14 LTS" so thereafter I have uploaded ojdbc11.jar in my lib folder, on all servers. Afterwards I have restarted each server and tried to create a new DBCP Connection Pool, but kept on receiving the these error messages.Database Connection URL = jdbc:oracle:thin:@HOSTNAME:PORT/SERVICEDatabase Driver Class Name = oracle.jdbc.driver.OracleDriverDatabase Driver Location(s) = /opt/nifi-1.19.1/lib/ojdbc11.jarDo you know what might be the cause of this problem? On a cluster of servers with java 8 I do not encounter such issues.2023-01-27 18:00:08,841 ERROR [Timer-Driven Process

It seems this question gets asked a lot in this forum, but there aren't any good responses.  Hopefully somebody can help me out here.I am attempting to use the InvokeHTTP processor to POST a query to an REST endpoint.  I currently use the InvokeHTTP processor to access data on this data service using the GET method, and that always works fine.  However, there is a specific service endpoint that only accepts POST requests.  I've verified this service is functional using curl.  But when I attempt to use the NiFi InvokeHTTP processor, I get no response.  Hopefully somebody here knows what's going on.  Below are some screen shots of my processor configuration. If I understand the documentation correctly, I need to specify in the "Attributes to Send" field the flowfile attributes I want to send as parameters.  Since this processor is the first processor in the flow, it does't have any attributes yet, so I add a dynamic attribute named "data" (sho

I have a apache/nifi:latest instance spun inside an Amazon Linux 2 EC2. For reference, see this guide: hereI have a QuerySalesforceObject ver. 1.18.0 that makes use of StandardOauth2AccessTokenProvider.  The oauth2 provider url is configured at https://test.salesforce.com/services/oauth2/tokenI can curl this url from the box and from inside the docker container just fine (I don’t get a timeout).[root@ip-10-229-18-107 \~\]# docker exec -it nifi_container_persistent /bin/shprintenv | grep -i proxyHTTPS_PROXY=http://proxy.MY_DOMAIN.com:3128no_proxy=localhost,127.0.0.1,MY_DOMAIN.com,.amazonaws.comNO_PROXY=localhost,127.0.0.1, MY_DOMAIN.com,.amazonaws.comhttps_proxy=http://proxy.MY_DOMAIN.com:3128http_proxy=http://proxy.MY_DOMAIN.com:3128HTTP_PROXY=http://proxy.MY_DOMAIN.com:3128curl https://test.salesforce.com/services/oauth2/token{"error":"unsupported_grant_type","error_description":"grant type not supported"}#But when I run the task, oauth2 fails with an errorjava.io.UncheckedI

Hi,Currently we are using CDP 7.1.7 with phoenix version 5.0.We were told by platform admins that cloudera is dropping support for phoenix from the next cdp version onwards. Is this true? If so, what are the alternative to phoenix are suggested?What is the strategy on embodying phoenix with CDP package? ThanksSagar

Other API calls work such as to /nifi-api/tenants/user-groups. I do not see an entry for /resources in authorizations.xml nor does there seem to be an entry VIA the gui policy editing. NIFI 19.1...

Hello, I have an error I don't understand with ListSFTP processor. When I click on "Verify properties" for this processor it lists well an SFTP folder.Successfully listed contents of Remote Directory [/data/folder_1/] on [hosname:port]. Found 2 objects. Of those, 2 match the filter.But when I execute this processor I have the error I never seen: ListSFTP[id=X] Processing failed: org.apache.nifi.processors.standard.socket.ClientConnectException: SSH Client connection failed [hosname:port] - Caused by: java.net.SocketTimeoutException: connect timed out Why can I list when verify properties but processor doesn't work ? (Nifi version : 1.17.0) I changed nothing on Nifi and don't know what to do to correct the problem or what to do to investigate.

I want to transfer the HR system of our company to BigData. My archive has 500 GB of data on SAP. I want to install Apache Sqoop, Impala and Hadoop HDFS. How much of a resource do I need? What would my Cloudera license cost be? How can I make 2 nodes

Hi, I am using Cloudera Quickstart VM 5.13.0 to write code using pyspark. Trying to import SparkSession using below command:from pyspark.sql import SparkSession It's throwing an error saying Importerrir: cannot import name SparkSession I need help to fix this. Please suggest me if anything missed

hello  I learn sqoop and mysql fist timei use hdp 2.65 and use puttymy user name is root@localhost I want to sqoop movie file when I input order  sqoop import --connect jdbc:mysql://localhost/movielens --driver com.mysql.jdbc.Driver --table movies -m 1 and the following warning message appears. Warning: /usr/hdp/2.6.5.0-292/accumulo does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.23/01/31 19:47:12 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.6.5.0-29223/01/31 19:47:12 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.23/01/31 19:47:12 INFO manager.SqlManager: Using default fetchSize of 100023/01/31 19:47:12 INFO tool.CodeGenToo

 I am having a simple flow where I am fetching the creds for SFTP from DB and then uploading a file using the putSFTP but I am getting UnknownHostException error on PutSFTP processor. I see the creds are being fetched - I see them in logs and also tried connecting to the Server using FileZilla (creds are correct ..no space etc all good with that)..Also I tried hardcoding the creds in PutSFTP and still it did not work.can someone please guide me..

Hi Team, I have an requirement to generate temporary credentials of AWS and fetch files from the S3 bucket  and use this credentials in FetchS3ObjectBut at FetchS3Object processor I am getting below error while fetching files from the bucket: "FetchS3Object[id=xxx] Failed to retrieve S3 Object for FlowFile[filename=xyz.json]; routing to failure: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: To use assume role profiles the aws-java-sdk-sts module must be on the class path., com.amazonaws.auth.profile.ProfileCredentialsProvider@3075e891: profile file cannot be nul

When attempting to insert a timestamp in milliseconds into a flowfile using the UpdateRecord processor, the timestamp comes out expressed as a string (first image) despite the fact that I've explicitly cast it as a number (second image).   How can I get this expressed as a number like it is supposed to?     

I have been working with NiFi to do some basic testing with MQTT ingestion and publishing using protobufs. Everything seems to work fine, but each step takes about 30-60s to process. This seems incredibly long for such a simple process. I tried to see if there were any configuration settings that might need adjusted, but didn't have any success there. I also, setup a pipeline that consumes then publishes what it consumes and that still takes 30+ seconds. Is this a known issue or a setting that I have missed?I have used Streamsets in the past, so I decided to try it there and the same setup executes sub-second. I am using NiFi version 1.19.1Appreciate any thoughts or assistance!