Solved

 I am having a simple flow where I am fetching the creds for SFTP from DB and then uploading a file using the putSFTP but I am getting UnknownHostException error on PutSFTP processor. I see the creds are being fetched - I see them in logs and also tried connecting to the Server using FileZilla (creds are correct ..no space etc all good with that)..Also I tried hardcoding the creds in PutSFTP and still it did not work.can someone please guide me..

When attempting to insert a timestamp in milliseconds into a flowfile using the UpdateRecord processor, the timestamp comes out expressed as a string (first image) despite the fact that I've explicitly cast it as a number (second image).   How can I get this expressed as a number like it is supposed to?     

I have been working with NiFi to do some basic testing with MQTT ingestion and publishing using protobufs. Everything seems to work fine, but each step takes about 30-60s to process. This seems incredibly long for such a simple process. I tried to see if there were any configuration settings that might need adjusted, but didn't have any success there. I also, setup a pipeline that consumes then publishes what it consumes and that still takes 30+ seconds. Is this a known issue or a setting that I have missed?I have used Streamsets in the past, so I decided to try it there and the same setup executes sub-second. I am using NiFi version 1.19.1Appreciate any thoughts or assistance!

Hi Team, I am trying to re-build my materialized view (ALTER MATERIALIZED VIEW <db>.<table> REBUILD;). However, I got an error message saying that there is another process re-building the MV.Error message:SQL Error [40000] [42000]: Error while compiling statement: FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: Another process is rebuilding the materialized viewI tried to check if there is any write lock on the MV or the table, however there is none. I also tried to re-build the MV, but then I got the same error again when I tried to rebuild.Any idea why this is happening or how to check the "another process"?

Hello, I am in process of setting up a CDP 7.1.7 cluster.At the moment, CM 7.6.1 is installed and integrated to AD on LDAPS protocol (Had to select authentication type as LDAP for the integration to work even though we have AD being used).Next step is I have added few basic services i.e. HDFS, YARN and Zookeeper and now I am enabling Kerberos.At the step of Generating credentials it fails with attached screenshot.However, in the same window I noticed that CM is trying to connect to AD on LDAP protocol on port 389. Ideally it should be connecting via LDAPS on 636 as we have TLS also configured and enabled. Not sure if this is even relevant. From where does CM gets the LDAP URL? I tried to understand gen_credentials_ad.sh script at /opt/cloudera/cm/bin, however, did not completely interpret. Please help as this is bit urgent. Thankssnm1523

Hi,I have my project deployed in CDSW and am eager to know if there is an option to mask the code being printed/displayed on the output console log? or at least in the email that gets distributed on the job status after the run?

Hi all, I'm very new to NiFi and trying to find ways how to convert json array to be stored in SQL database. I currently have this input json  { "QueryHandle" : "#qb9ec959946924620a6f46013d745fe03", "Rows" : [ { "Values" : [ "A1", "Test Data A1" ] }, { "Values" : [ "A2", "Test Data A2" ] }, { "Values" : [ "A3", "Test Data A3" ] }, { "Values" : [ "A4", "Test Data A4" ] } ], "TotalRowCount" : 197, "DataTypes" : { "Values" : [ { "ColumnName" : "Value", "Type" : "String" }, { "ColumnName" : "Description", "Type" : "String" } ] }}  Here's the end result I would like to have before I can use PutSQL processor [ { "VALUES" : "A1", "DESCRIPTION" : "Test Data A1"}, { "VALUES" : "A2", "DESCRIPTION" : "Test Data A2"}, { "VALUES" : "A3", "DESCRIPTION" : "Test Data A3",}, { "VALUES" : "A4", "DESCRIPTION" : "Test Data A4",}  I have tried to use jolttransform, but couldn't be able to remove the array or t

hiI want to know if its possible to use the LOAD DATA INPATH to load parquet files into an external iceberg table? right now I am getting the following error: Error while compiling statement: FAILED: SemanticException [Error 10101]: A non-native table cannot be used as target for LOAD    does hive support LOAD DATA INPATH with Iceberg tables?

I have been trying to send an email via NiFi using PutEmail processor with Gmail/Outlook both and getting same exception in both of the configuration as   PutEmail[id=3611dcb8-0183-1000-8a1d-df8f12f92438] Failed to send email for StandardFlowFileRecord[uuid=017c80a8-fad0-493b-a427-fa72b9a530c7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1663081396353-45, container=default, section=45], offset=521106, length=538],offset=0,name=f37f23a1-e328-4eff-bb30-6f125369ebf7,size=538]: Could not convert socket to TLS; routing to failure: javax.mail.MessagingException: Could not convert socket to TLS; nested exception is: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)   The configuration is set as below :  Current NiFi version is  : 1.13.2Java Version : openjdk version "11.0.16" 2022-07-19OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu120.

We using PutHive3Streaming processor to send data to Hive from NiFi, I have an issue where we are getting LOTS of small delta files on our busier feeds which is causing issues with compaction etc. I have used a series of merges in NiFi to ensure each flowfile contains many thousand records but it still creates many delta files. I wondered if anyone had any advice on tuning 'Records Per Transaction' and 'Transactions per Batch' options on the PutHive3Streaming processor, I believe this could help with my issue but have had mixed/confusing results from testing. There isn't  a great deal of information on best practice that I have found. Has anyone else had similar issues/found adjustments helpful? 

The server of hive on tez shut down after it restarted.  We can find error message of hive's log below:2021-02-25 14:20:06,800 INFO org.apache.hadoop.hive.common.ZooKeeperHiveHelper: [shutdown-hook-0]: Server instance removed from ZooKeeper.2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopping/Disconnecting tez sessions.2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopped tez session pool manager.2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Shutting down HiveServer22021-02-25 14:20:06,800 ERROR org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Error stopping schqjava.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalidat org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-2

Hi, I have a use case, where i have to see the content of each FlowFiles by triggering the API call. API i have explored so far :https://<URL>/nifi-api/flowfile-queues/<Queue-ID>/flowfiles/<FlowFile-UUID>/content I got the response as below : "The id of the node in the cluster is required." Note : My NiFi is setting up in three node cluster. Please suggest.  

Hi I am learning Apache Crunch. But I discovered that https://crunch.apache.org/ is retired.What is the replacement this project?

I have installed Apache nifi 1.19.1 and configured the ssl using below command  ./tls-toolkit.sh standalone -n 'nifi1,nifi2,nifi3' -B MyPassword -C 'CN=nifiadmin,OU=NIFI' -O -o /opt/nificert  and it was wotking fine after importing the certificate in my browser.Then configured LDAP like below:-loginIdentityProviders  <?xml version="1.0" encoding="UTF-8" standalone="yes"?><loginIdentityProviders> <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">cn=Myuser,ou=Myuser,dc=Mydomain,dc=com</property> <property name="Manager Password">Mypassword</property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property

Hello Experts, We have a nifi instance with 8 core machine, and recently nifi is looking quite slow with its performance (nifi-api response, UI etc).Recently we have added lot of processors but at any point of time the  processor which are processing data will be only 10% of total processors. Will nifi perf goes down with increase in number of processors though 90% total processors will be idle? ThanksMahendra

Hi guys,I'm testing an upgrade from CDH 6.3.4 to CDP 7.1.8 , I successfully performed the upgrade but I'm trying to test the rollback process, I'm using  AUTO-TLS, but I'm having a one issue though,  it says the below in the HDFS rollback Cloudera : (Clusters with TLS enabled only) Edit the /etc/hadoop/conf.rollback.namenode/ssl-server.xml file on all NameNode hosts (located in the temporary rollback directory) and update the keystore passwords with the actual cleartext passwords. So the question is : How can I get the cleartext password ? 

I have a JSON dataset and need to create a unique key from it. No single field is unique, but the first two, together, are. What I want to convert from: [{"airport":"KDEN", "runway":"16R", "length":16000}, ...] and I want to concatenate the "airport" and "runway" values to create this: [{"key":"KDEN/16R", "airport":"KDEN", "runway":"16R", "length":16000}, ...] I've got it working with an ugly anti-pattern of SplitRecord, EvaluateJSONPath, UpdateAttribute, and AttributesToJSON  which changes each flow file to several thousand, but am looking for a better way to do this. 

I need help understanding the root cause of a job failure in Cloudera Data Science Workbench.What does it mean when a job is stuck in "Scheduling" phase and terminates as "Failed" after the timeout period? The log file, which usually displays the python code followed by the results from print statements and/or run time errors, states only a single line reading "Engine exited with status 34". Apparently, I was able to re-run the job successfully after a while. It could be a temporary glitch in the system but still trying to understand the root cause of this issue. Any insight on what happens during the "Job Scheduling" is much appreciated.

Hello everyone! Recently login ambari report 500 error box. I've been working on it for ages, but I can't solve it. The following are the error phenomena and related logs, hoping to get your help. Thanks!Error screenshot:500 status code received on GET method for API: /api/v1/clusters/bgybigdata/alerts?format=groupedSummary log:HTTP ERROR 500Problem accessing /api/v1/clusters/bgybigdata/alerts. Reason: Server Error Caused by:java.lang.NullPointerException at org.apache.ambari.server.controller.internal.AlertResourceProvider.toResource(AlertResourceProvider.java:260) at org.apache.ambari.server.controller.internal.AlertResourceProvider.getResources(AlertResourceProvider.java:240) at org.apache.ambari.server.controller.internal.AlertResourceProvider.queryForResources(AlertResourceProvider.java:166) at org.apache.ambari.server.controller.internal.ClusterControllerImpl$ExtendedResourceProviderWrapper.queryForResources(ClusterControllerImpl.java:966) at org.apache.ambari.s

Does anyone have experience with CDP Private Cloud on CentOS 9?Are there any plans to officially support it in the near future?

Hello,I'm brand new to Atlas and trying to see if this capability exists.  I understand that you can use a Hive Hook and whenever a Hive table is modified, Kafka is able to communicate to Atlas and have those changes appear in Atlas.  I'm trying to see if there is any reporting or auditing features that allows you to see what those changes were?For example: If a user modified a Hive table and added two columns and this gets replicated in Atlas, is there a way to see what change happened (columns added) and when it occurred?  I'm not sure if this is something Atlas can do natively or if there might be 3rd party tools I could use to see this?Thanks in advance!

Hi Team, We are currently using Cloudera 6.1.1 and now we are planning to change Log4j version from 2.8.2to 2.16.0 or above.Can you share the link of 2.16.0 or above log4j version so can i download it and apply patch. Thanks  

Hey together, we're running a CDP Private Cloud Base Cluster with actually 6 Nodes. All 6 Nodes running fully on SSD storage. I found out that HDFS thinks the SSDs are normal disk, because it is not configured as SSD storage like this --> "[SSD]/grid/n/dn", actually its "/grid/n/dn". Now we're planning to add four other servers from an older cluster with normal disks. My action plan is as follows: 1. configure existing storage/server to SSD -> "[SSD]/grid/n/dn"2. add the other four server an configure it -> "[DISK]/grid/n/dn"3. restart recommended services I've got the following questions: - Will HDFS distribute the blocks across all nodes (SSD and disk?) without any storage policy configured? So that we have a homogeneous storage. - How will services react to this changes in storage, for example HBase?- If we configure a homogeneous storage, as i described: does reconfiguring the storage paths of the Datanodes to SSDs add any

Hi,All I want to do is replace multiple character in a field's value.The text in the field is:'I have different characters'If I try ${field.value:replace('a', 'i')} it works. The output is 'I hive different chiricters' Bu I want to replace one more character. How can I do this in same processor.I have tried someting like ${field.value:raplace('a', 'i'); field.value:replace('e', 'u')} I couln't find the right syntax. Please help.

we have hadoop cluster based on HDP from hortonworks version HDP 3.1.0.0-78cluster include 2 namenode services when one is the standby namenode and the second is the active namenode , all machines in the cluster are CentOs 7.9 version , and we dont'  see any problem on OS level also cluster include  87 data node machines ( 9 admin nodes various master services working on them ). All physical machines , around 7 PB data volume, 75% is full. the story begin with neither NN1 nor NN2 work in same time , I mean active , standby .  They were working for more that 2-3 years with out issue. Last 2-3 months they dont get running in the same time.  When I look at he NN logs , after one NN get active second one getting running and in some point 3 Journal Nodes get an exception and NN2 get down. /************************************************************have seen NN failover occur after Journal Node got Exception from 3 JN.see below logs. {Truncated unnecessary