Solved

I have a requirement to notify once the files on the destination is processed. The pipeline is as follows: ListSFTP --> Update attribute (adds specific attribute for each state/province) ---> Send to Cloud Storage. Each state has a different count of files. Once the files are sent to cloud storage, notify the end user using email for each state. I have tried merging files based on a common attribute but, I need single files for each state so that I can send one notification for each state.  

Hi Team, Using QueryRecord to filter the incoming array of objects where each object has an array field.The field can be an empty array or it contains some objects in it but it is not null. Tried a lot of SQL queries but didn't succeed. Also, I am trying to consider Record based approach as I want to work on multiple flow files instead of single ones In Actual flow wanted to use ForkRecord on this array but it outputs an empty array if any object contains an empty array in the path so had to use QueryRecord before ForkRecord to filter those empty arrays. Not sure if it's the bug/expected behavior in ForkRecord Flow: GenerateFlowFile:Custom Text: [ { "fields": [ { "field": "field1" }, { "field": "field1" } ] }, { "fields": [] }, { "fields": [ { "field": "field2" }, { "field": "field2" } ] }, { "fields": [ { "field": "field3" }, { "field": "field3" } ] }] Expected output: To filter out the 2nd object whose fie

I am using the importPoliciesFromFile API call but see it deletes and creates the policies. Will this cause any impact on authorization if someone hits the api at the same time or is this API synchronous? By synchronous I mean, does it stop all other code execution and user interaction until the call returns?

Our vulnerability scanning found these two vulnerabilities on our CDP Private Cloud 7.1.7-SP1, CVE-2022-22970 and CVE-2022-22971.  There are several versions of spring-core in the parcel, none of which are the recommended version:./jars/spring-core-4.3.29.RELEASE.jar./jars/spring-core-5.2.18.RELEASE.jar./jars/spring-core-5.3.10.jar./jars/spring-core-5.3.12.jar./jars/spring-core-5.3.13.jar./jars/spring-core-5.3.4.jarIs CDP vulnerable to these vulnerabilities? https://tanzu.vmware.com/security/cve-2022-22970https://tanzu.vmware.com/security/cve-2022-22971 

Hi Team, We have total 46 nodes cluster.9(2 Master & 7 slave )  nodes on Baremetal and 37 nodes on VCP(VM's).  Can we set any limit/quota on CDH side by restrict using maximum bandwidth from N/W switches??

I have 2 process that I need to execute on parallelo and I have a 3rd process what I wanto to start when firsts boths process will been finished, so my question is: There is a some box processor that I can use in order to implement this logic or somebody has a some idea to develop it. Thank a lot and I am waiting for your answersBest regards

I am a beginner. I submitted the HQL(create table bds_hive.test_ft19 (name string)) using spark3-shell and experienced three permission errors. These errors indicate that I do not have permissions for the corresponding paths on HDFS. According to the error message, I configured the corresponding path permission in Ranger, but it did not work. In addition, there is no denied record in the audit log. I hope guys can help me analyze and provide solutions. Hadoop version:3.1.1.7.1.7.1000-141Hive version:3.1.3000.7.1.7.1000-141Spark version:3.2.1.3.2.7171000.1-1 

Hello everyone, I have a NiFi cluster (1.12.1) with 3 secure nodes with SSL and 3 more external zookeeper nodes, I need to update this guy to version (1.17.0). I think about uploading a new environment with 1.17.0 and migrating the flow files? Would work?

we have set up 4 node cluster and all other 3 nodes start without any exceptions. and in NIFI uiit shows our down node is in connecting state.but our down NIFI node is not running.its logs as below.2022-10-17 19:04:51,860 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 114432022-10-17 19:04:51,950 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager Failed to close Leader Selector for Cluster Coordinatorjava.lang.IllegalStateException: Already closed or has not been startedat com.google.common.base.Preconditions.checkState(Preconditions.java:173)at org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:165)at org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1120)at org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:349)at org.apache.nifi.we

HI Team,System Env Details: I am trying setup NIFI on AWS3 node clusterNIFI version:  1.16.0authorisers.xml :       <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Legacy Authorized Users File"></property> <property name="Initial User Identity 1">xxx@xxx.com</property> <property name="Initial User Identity 2">C=US, ST=xxx, O=xxx, OU=xxxx, CN=%%cn_user_identity_1%%</property> <property name="Initial User Identity 3">C=US, ST=xxx, O=xxx, OU=xxxx, CN=%%cn_user_identity_2%%</property> <property name="Initial User Identity 4">C=US, ST=xxxxx, O=xxxx, OU=xxxx, CN=%%cn_user_identity_3%%</property> </userGroupProvider> <accessPolicyProvider>

Hello all...i am very new to big data. I currently have downloaded HDP (sandbox) 2.6.5. I also have downloaded HDF 3.1.1 where I practice. Is there anyway i can download HDF services on HDP 2.6.5? if not is there any way to download only Nifi on hdp2.6.5? It will also be very nice to have any answer to download airflow schduler instead of using bulit-in workflow manager?...i try to look and many documentations that was recommended here but no avail.Thank you

Hi all We have Ambari HDP cluster ( HDP version - 2.6.4 ) , with 420 workers linux machines ( when each worker include data node and node manager service ) Unfortunately Ambari DB is damaged , and we not have Ambari DB dump , so we cant recover Ambari DB  , so actually we not have Ambari and Ambari GUI But HDFS disks on workers machines include HDFS data , and name node is still working with all data  as ( journal/hdfsha/current/ ) and ( namenode/current )So HDFS works without Ambari  So regarding what I said until now - it is possible install new Ambari cluster from scratch , and then add existing working HDFS data to the cluster ?Dose hortonworks / cloudera have procedure for this process ?

Hello all Within NiFi, updateAttribute processor I am trying to change an attribute called 'hive_database' based on the value of another attribute called 'hostname'. For instance if:#1: hostname = Mickey Mouse#2: hostname = James Bond I want to use a 'contains' statement (or similar ) to change the output of new attribute 'hive_database' property:hive_databasevalue:${hostname:contains('Mickey'):<OUTPUT>('cartoon')} <OR>${hostname:contains('Mickey'):<OUTPUT>('movie')} So that #1 output would be: property= hive_database value= cartoon The parts I'm stuck on is the OUTPUT and OR parts above, do you know if this would be possible using the update attribute processor? I am trying to avoid having to use routeOnAttribute to break out multiple different 'hosname' values to send data to the relevant 'hive_database'  Hope this makes sense, unable to share screenshots etc as sensitive vales. Any help greatly appreciated. 

Hello Community,I have a configuration where ListenSysylog is listenig and provides the received data to InvokeHTTP. Also I use a funnel to redirect Http failures to the InvokeHttp once again. The problem occurs when InvokeHttp tries to resend data and it consumes so many threads that ListenSyslog starts droping incoming data. I do not want to lost any incoming data and in the same time I want to be sure that data will be send. Do you have a similar problem? Is there any way to limit proccessors?Thanks

We have newly install CDP-DC 7.1.7 and try to load data using a spark jobs but getting the errors.----Payloadjava -cp /etc/hadoop/conf.cloudera.hdfs/ssl-client.xml:/etc/hbase/conf.cloudera.hbase/hbase-site.xml:/etc/hadoop/conf.cloudera.hdfs/core-site.xml:/etc/hadoop/conf.cloudera.hdfs/hdfs-site.xm:/data/scripts/LeaApp-1.0-SNAPSHOT.jar net.ba.lea.transformation.FileActions "/data/scripts/msc/IN/" "/tmp/nss_processing/" "/data/scripts/msc/reject/" "250" "LEA.DBM_CDR_FILE_HEAD" "NSS" "jdbc:phoenix:gzvlcdpnode01.ba.net:2181:/hbase:phoenix/gzvlcdpnode02@BA.NET:/etc/security/keytab/phoenix.keytab"error :Can't find method newStub in org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService!--------------------------------But the same code is working fine with CDP-DC 7.1.3.We have changed  the version only.Please guide us .

Hello all,I am using Cloudera Datasservices on CDP Private Cloud. I am using using  the embedded container service (ECS).If i check the kubernetes pods i see that the pods of the namespace shared-services are always crashing (Expect of openshift-idling-controller-manager) kubectl get pods -n shared-servicesNAME READY STATUS RESTARTS AGElog-router-fznw2 0/2 Init:CrashLoopBackOff 17 46mlog-router-jnjrb 0/2 Init:CrashLoopBackOff 17 46mlog-router-tptbg 0/2 Init:CrashLoopBackOff 17 46mopenshift-idling-controller-manager-0 1/1 Running 0 45m If i query the logs of the pod I get the following error:kubectl logs -f -n shared-services log-router-jnjrb sdx-init-container I have already tried to delete the pods but without success.But the behaviour is unchanged:Who can help?Thanks in advance.Best regardsUli  {"timestamp":"2022-10-07 07:35:52.120 +0000","level":"WARN","thread":"Thread-8","logger":"c.cloudera.cdp.config.ConfigTemplate","message":"[2022-10-07 07:35:52.12

Hello Guys! I updated HDP from 3.1.0.0-78 to 3.1.5.6189-1 as well as Ambari from 2.7.3 to 2.7.5.I want to remove the old stack of HDP as the files are still on the servers.Is there any official guide for this move? 

I have cluster 1 (the source cluster in this example) which does not have HA enabled, and I have cluster 2 (destination cluster in this example) which does have HA enabled.  Both clusters are running the same version of HDP.  When I attempt to copy data from cluster 1 to cluster 2 via distcp using the active name node instead of the HA nameservice, it works but ultimately causes problems when the active name node switches.  I have tried using the HA nameservice, but without any configurations, it fails (as one would expect).  When modifying the hdfs-site.xml to add cluster 2's nameservice information (following this guide laid out here but only for one HA cluster: https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/scaling-namespaces/topics/hdfs-distcp-between-ha-clusters.html), the HDFS services on cluster 1 fail to restart, as the name nodes appear to be attempting to join cluster 2.  Is this sort of configuration even possible, or do we need HA on both cl

We want to configure listenhttp processor that listen our web service. question is what are the configuration in http listener processor in nifi such that when we will make curl post request this ListenHttp processor will catch the update. basically i didn't understand what i have to put in Base path (is it i have to put nifi address i.e http://localhost:8080/contentListener or our web app service URL???) what is contentListener. And is it i have to put our web app port number in listening port? in the request command what i have to put in url part curl --request POST 'http://(here i have to used the nifi url)' -H 'Authorization: Key demo-key i know this is very basic question but i'm just using this don't know any thing about nifi and curl command .....thanks in advance

Hi all, this may be an easy question, but just wanted to verify.  When adding new brokers to the cluster, after installing the Kafka role/instance on the new hosts, should you start the new broker instances first before you restart the existing brokers?  Or does it matter?  This is CDP Private Cloud 7.1.7Thanks

Is there any other way to migrate the Hortonworks HDP3  cluster to the Apache Hadoop 3.3.2  other than the distcp?is it possible to upgrade by updating the jar and pointing the older metadata to the newer stack of Hadoop? Thanks

Is CDP 7.1.7 SP1 a stable release? Is it recommended upgrade to this version? We still see some log4j 1X findings in this version so we need to now if someone has faced the same issue.

Hi Team, We have 2 Hive Metastore server services configured on cluster in node1 and node2. In node1 Hive Metastore getting alert with below message. "Hive Metastore Canary" The Hive Metastore canary failed to create a database. I am seeing below errors in servicemonitor log file. 2022-10-12 22:40:06,657 WARN com.cloudera.cmf.cdh6client.hive.MetastoreClientImpl: (2 skipped) Could not drop hive database: cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_0137966f79e5f15b3b5d4dec61b7592ecom.cloudera.cdh6client.hive.shaded.org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed outat com.cloudera.cdh6client.hive.shaded.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)at com.cloudera.cdh6client.hive.shaded.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)at com.cloudera.cdh6client.hive.shaded.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)at c

Hello everyone.i need help. I have a JSON file which contains in an object either an array of other objects or a single object. This is the relevant object "Product": [ or "Product": { .How can I configure with the JOLT Transform processor to always expect an array in the object? The product object must always be treated as an array, even if it contains only a single object. { "costc": 9638, "sum_amount": 543, "sum_invitation": 0, "sum_selfconsumption": 0, "sum_loss": 0, "sum_gross": "2764,000", "sum_net": "2392,480", "sum_margin": "2003,780", "sum_costs_total": "388,700", "sum_costs": "388,700", "sum_costs_invitation": 0, "sum_costs_selfconsumption": 0, "sum_costs_loss": 0, "sum_costs_total_percent": "16,25", "sum_costs_percent": "16,25", "Classes": { "ClassSummary": [ { "class": 1102, "sum_amount": 117, "sum_invitation": 0, "sum_selfconsumption": 0, "sum_loss": 0, "sum_gross": "740,500", "sum_net": "692,060

Hi, In BDR HDFS replication I want to exclude all files which end with ".tmp" in the directory "/backups/"   Appreciate it if somebody could give the expression for this to add in BDR "Add Exclusion".    Thanks