Find answers, ask questions, and share your expertise
Recently active
Hello, I have an error I don't understand with ListSFTP processor. When I click on "Verify properties" for this processor it lists well an SFTP folder.Successfully listed contents of Remote Directory [/data/folder_1/] on [hosname:port]. Found 2 objects. Of those, 2 match the filter.But when I execute this processor I have the error I never seen: ListSFTP[id=X] Processing failed: org.apache.nifi.processors.standard.socket.ClientConnectException: SSH Client connection failed [hosname:port] - Caused by: java.net.SocketTimeoutException: connect timed out Why can I list when verify properties but processor doesn't work ? (Nifi version : 1.17.0) I changed nothing on Nifi and don't know what to do to correct the problem or what to do to investigate.
I want to transfer the HR system of our company to BigData. My archive has 500 GB of data on SAP. I want to install Apache Sqoop, Impala and Hadoop HDFS. How much of a resource do I need? What would my Cloudera license cost be? How can I make 2 nodes
Hi, I am using Cloudera Quickstart VM 5.13.0 to write code using pyspark. Trying to import SparkSession using below command:from pyspark.sql import SparkSession It's throwing an error saying Importerrir: cannot import name SparkSession I need help to fix this. Please suggest me if anything missed
hello I learn sqoop and mysql fist timei use hdp 2.65 and use puttymy user name is root@localhost I want to sqoop movie file when I input order sqoop import --connect jdbc:mysql://localhost/movielens --driver com.mysql.jdbc.Driver --table movies -m 1 and the following warning message appears. Warning: /usr/hdp/2.6.5.0-292/accumulo does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.23/01/31 19:47:12 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.6.5.0-29223/01/31 19:47:12 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.23/01/31 19:47:12 INFO manager.SqlManager: Using default fetchSize of 100023/01/31 19:47:12 INFO tool.CodeGenToo
I am having a simple flow where I am fetching the creds for SFTP from DB and then uploading a file using the putSFTP but I am getting UnknownHostException error on PutSFTP processor. I see the creds are being fetched - I see them in logs and also tried connecting to the Server using FileZilla (creds are correct ..no space etc all good with that)..Also I tried hardcoding the creds in PutSFTP and still it did not work.can someone please guide me..
Hi Team, I have an requirement to generate temporary credentials of AWS and fetch files from the S3 bucket and use this credentials in FetchS3ObjectBut at FetchS3Object processor I am getting below error while fetching files from the bucket: "FetchS3Object[id=xxx] Failed to retrieve S3 Object for FlowFile[filename=xyz.json]; routing to failure: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: To use assume role profiles the aws-java-sdk-sts module must be on the class path., com.amazonaws.auth.profile.ProfileCredentialsProvider@3075e891: profile file cannot be nul
When attempting to insert a timestamp in milliseconds into a flowfile using the UpdateRecord processor, the timestamp comes out expressed as a string (first image) despite the fact that I've explicitly cast it as a number (second image). How can I get this expressed as a number like it is supposed to?
I have been working with NiFi to do some basic testing with MQTT ingestion and publishing using protobufs. Everything seems to work fine, but each step takes about 30-60s to process. This seems incredibly long for such a simple process. I tried to see if there were any configuration settings that might need adjusted, but didn't have any success there. I also, setup a pipeline that consumes then publishes what it consumes and that still takes 30+ seconds. Is this a known issue or a setting that I have missed?I have used Streamsets in the past, so I decided to try it there and the same setup executes sub-second. I am using NiFi version 1.19.1Appreciate any thoughts or assistance!
Hi Team, I am trying to re-build my materialized view (ALTER MATERIALIZED VIEW <db>.<table> REBUILD;). However, I got an error message saying that there is another process re-building the MV.Error message:SQL Error [40000] [42000]: Error while compiling statement: FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: Another process is rebuilding the materialized viewI tried to check if there is any write lock on the MV or the table, however there is none. I also tried to re-build the MV, but then I got the same error again when I tried to rebuild.Any idea why this is happening or how to check the "another process"?
Hello, I am in process of setting up a CDP 7.1.7 cluster.At the moment, CM 7.6.1 is installed and integrated to AD on LDAPS protocol (Had to select authentication type as LDAP for the integration to work even though we have AD being used).Next step is I have added few basic services i.e. HDFS, YARN and Zookeeper and now I am enabling Kerberos.At the step of Generating credentials it fails with attached screenshot.However, in the same window I noticed that CM is trying to connect to AD on LDAP protocol on port 389. Ideally it should be connecting via LDAPS on 636 as we have TLS also configured and enabled. Not sure if this is even relevant. From where does CM gets the LDAP URL? I tried to understand gen_credentials_ad.sh script at /opt/cloudera/cm/bin, however, did not completely interpret. Please help as this is bit urgent. Thankssnm1523
I have a use case that requires malware analysis. The analysis and feature extraction is taking place on a separate designated server/network. A feature file is generated on that server/network. I will need to schedule the analysis to run on the most recent malware and generate feature files. The challenge is how do I send the feature file to Nifi from the server/network that has access to malware repository?
Hi,I have my project deployed in CDSW and am eager to know if there is an option to mask the code being printed/displayed on the output console log? or at least in the email that gets distributed on the job status after the run?
Hi all, I'm very new to NiFi and trying to find ways how to convert json array to be stored in SQL database. I currently have this input json { "QueryHandle" : "#qb9ec959946924620a6f46013d745fe03", "Rows" : [ { "Values" : [ "A1", "Test Data A1" ] }, { "Values" : [ "A2", "Test Data A2" ] }, { "Values" : [ "A3", "Test Data A3" ] }, { "Values" : [ "A4", "Test Data A4" ] } ], "TotalRowCount" : 197, "DataTypes" : { "Values" : [ { "ColumnName" : "Value", "Type" : "String" }, { "ColumnName" : "Description", "Type" : "String" } ] }} Here's the end result I would like to have before I can use PutSQL processor [ { "VALUES" : "A1", "DESCRIPTION" : "Test Data A1"}, { "VALUES" : "A2", "DESCRIPTION" : "Test Data A2"}, { "VALUES" : "A3", "DESCRIPTION" : "Test Data A3",}, { "VALUES" : "A4", "DESCRIPTION" : "Test Data A4",} I have tried to use jolttransform, but couldn't be able to remove the array or t
hiI want to know if its possible to use the LOAD DATA INPATH to load parquet files into an external iceberg table? right now I am getting the following error: Error while compiling statement: FAILED: SemanticException [Error 10101]: A non-native table cannot be used as target for LOAD does hive support LOAD DATA INPATH with Iceberg tables?
I have been trying to send an email via NiFi using PutEmail processor with Gmail/Outlook both and getting same exception in both of the configuration as PutEmail[id=3611dcb8-0183-1000-8a1d-df8f12f92438] Failed to send email for StandardFlowFileRecord[uuid=017c80a8-fad0-493b-a427-fa72b9a530c7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1663081396353-45, container=default, section=45], offset=521106, length=538],offset=0,name=f37f23a1-e328-4eff-bb30-6f125369ebf7,size=538]: Could not convert socket to TLS; routing to failure: javax.mail.MessagingException: Could not convert socket to TLS; nested exception is: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate) The configuration is set as below : Current NiFi version is : 1.13.2Java Version : openjdk version "11.0.16" 2022-07-19OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu120.
We using PutHive3Streaming processor to send data to Hive from NiFi, I have an issue where we are getting LOTS of small delta files on our busier feeds which is causing issues with compaction etc. I have used a series of merges in NiFi to ensure each flowfile contains many thousand records but it still creates many delta files. I wondered if anyone had any advice on tuning 'Records Per Transaction' and 'Transactions per Batch' options on the PutHive3Streaming processor, I believe this could help with my issue but have had mixed/confusing results from testing. There isn't a great deal of information on best practice that I have found. Has anyone else had similar issues/found adjustments helpful?
The server of hive on tez shut down after it restarted. We can find error message of hive's log below:2021-02-25 14:20:06,800 INFO org.apache.hadoop.hive.common.ZooKeeperHiveHelper: [shutdown-hook-0]: Server instance removed from ZooKeeper.2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopping/Disconnecting tez sessions.2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopped tez session pool manager.2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Shutting down HiveServer22021-02-25 14:20:06,800 ERROR org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Error stopping schqjava.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalidat org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-2
Hi, I have a use case, where i have to see the content of each FlowFiles by triggering the API call. API i have explored so far :https://<URL>/nifi-api/flowfile-queues/<Queue-ID>/flowfiles/<FlowFile-UUID>/content I got the response as below : "The id of the node in the cluster is required." Note : My NiFi is setting up in three node cluster. Please suggest.
Dear Team,My api is not retrieving any data using invokehttp. Im able to get expected json data from cli but not from Nifi processor. https://resource-manager-serer:8090/ws/v1/cluster/apps?states=RUNNING -k --negotiate -u : I have https and Kerberos.
Hi I am learning Apache Crunch. But I discovered that https://crunch.apache.org/ is retired.What is the replacement this project?
Can someone tell me if these is even possible. We want to send a file through nifi to another nifi server on a different network, thus nifi is the only tunnel of communication between the two networks. If we can send just the file data I'm ok with that. If we can maintain the attributes and send the actual flowfile then even better.
I have installed Apache nifi 1.19.1 and configured the ssl using below command ./tls-toolkit.sh standalone -n 'nifi1,nifi2,nifi3' -B MyPassword -C 'CN=nifiadmin,OU=NIFI' -O -o /opt/nificert and it was wotking fine after importing the certificate in my browser.Then configured LDAP like below:-loginIdentityProviders <?xml version="1.0" encoding="UTF-8" standalone="yes"?><loginIdentityProviders> <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">cn=Myuser,ou=Myuser,dc=Mydomain,dc=com</property> <property name="Manager Password">Mypassword</property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property
Hello Experts, I am using nifi-api to update the variable registry of specific processor groups.It was working as expected, but in recent time its taking too long (some time infinite) time to complete the variable registry update.However if I restart nifi and try again this var registry update api it works quickly as expected. Is it because of increased number/size of variable registry or any resource crunch ?Could you please suggest a solution on this ?Any suggestion is much appreciated in advance. ThanksMahendra
Hello Experts, We have a nifi instance with 8 core machine, and recently nifi is looking quite slow with its performance (nifi-api response, UI etc).Recently we have added lot of processors but at any point of time the processor which are processing data will be only 10% of total processors. Will nifi perf goes down with increase in number of processors though 90% total processors will be idle? ThanksMahendra
Hi All Good Day! Just want to share our problem after we upgrade our nifi version from 1.9.2 to 1.16.2. all processor running fine but after ingesting a data, We got an error on the nifi bulletin. Upon checking on the logs this the error. Thank you
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.