Q&A

Hi guys, I have some problem with my NIFI flow-I download records from Kafka, I split it to have single record per split-here i have round robin load balance.Next i add id attribute and then i divide the flow to two paths, on one I do some actions with http iinokes to have some new attributes.Next I clear the context on this path and I would like merge it back with this other path to have this json context and this attributes in one.I use merge content processoe for this, merging using the id attribute that I assigned at the beginning. And now I have a problem because it works but in the case of a larger number of records, eg. 200 it doesn't because I have an error BIN_MANAGER_FULL. Minimum Number of Entries is 2 and Maximum Number of Entries i have 2 too, while Maximum number of Bins is 10 and i would not like to increase it. Is there any way to bypass this and make it work for a larger number of records?For example apply the limit so that it merges only when there is space or wait fo

I am trying to create grafana dashboard for the CM charts . So is it possible to connect grafana directly or if we need to store the data in database . Any samples.

Hello,Is there a way to list all HDFS directories a (human, not Impala, ...) user has access to, using the Ranger API?Many thanks.

I am trying to run an application on using hive and hadoop.And, my YARN cluster is not recognizing any active NodeManagersTerminal output on: yarn node -listanushkakundu@AnushkaHP14:/opt/hadoop/bin$ /opt/hadoop/bin/yarn node -list2024-05-23 17:07:47,933 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032Total Nodes:0 Node-Id Node-State Node-Http-Address Number-of-Running-ContainersOne more thing: My terminal reads: The url to track the job: http://AnushkaHP14:8088/proxy/application_1716465092573_0001/However http://AnushkaHP14:8088/proxy/application_1716465092573_0001/ says This site can’t be reachedI have done the following:Started Hadoop (verified by running jps), started hive (metastore and hiveserver2 are active.)Modified my yarn-site.xml to allocate appropriate resources.Re stared everything.Any suggestions are really welcome. 

We are having some issues while trying to connect with Cloudera Impala through pyobdc in our production enviroment (not in development env, neither in pre-production env).We have configured our Driver ODBC for Impala with v2.6.11.1011-1, Kerberos 5 v1.18.2, pip v23.3 and pyodbc v4.0.38.The Impala odbc driver is installed without problem, the impala ports are open, the kerberos ports are open too, everything was validated with telnet.Our connection string is defined in the following way: connection_string_IaaS = f''' DRIVER=/opt/cloudera/impalaodbc/lib/64/libclouderaimpalaodbc64.so; Host=(name of host); Port=21050; AuthMech=1; Encrypt=no; KrbRealm=(name of data source); KrbFQDN=(name of host); KrbServiceName=impala; TrustedCerts=/opt/cloudera/impalaodbc/lib/64/cacerts.pem; TSaslTransportBufSize=60536; RowsFetchedPerBlock=4000; SSP_BATCH_SIZE=60536'''   Once we ensure the kerberos ticket is right, we try to establish the connection by making pyodbc.connect(connec

Hi there!I am trying to use this processor. I have found a very useful guide: https://docs.cloudera.com/cdf-datahub/7.2.18/mainframe-ebcdic-nifi/topics/cfm-create-mainframe-ebcdic-recordreader.html?I am following the same steps but I am encontering this error:"Copybook Path" validated against #{hdfs.output.raw-input-files}TGSS/copybook.cbl is invalid because The specified resource do not exist or could not be accessed: [#{hdfs.output.raw-input-files}TGSS/copybook.cbl]Is anybody having this issue?How can I solve it?Thanks in advance

Hi Team, I Installed cloudera manager, its running but its not able to take admin admin as username and password. http://35.226.36.214:7180/cmf/login?returnUrl=%2Fcmf%2FpostLogin%3FreturnUrl%3D

Hi Team, I am installing solr on cloudera manager.Getting below error, Zookeeper is running but still getting this error while start error. Solr is not working.Solr Logs:```24/05/09 12:27:16 INFO zookeeper.Login: [Thread-1]: TGT refresh thread started.24/05/09 12:27:16 INFO zookeeper.Login: [Thread-1]: TGT valid starting at: Thu May 09 12:27:16 UTC 202424/05/09 12:27:16 INFO client.ZooKeeperSaslClient: [main-SendThread(cloudera-1.c.cloudera-project-422106.internal:2181)]: Client will use GSSAPI as SASL mechanism.24/05/09 12:27:16 INFO zookeeper.Login: [Thread-1]: TGT expires: Thu May 09 22:27:16 UTC 202424/05/09 12:27:16 INFO zookeeper.Login: [Thread-1]: TGT refresh sleeping until: Thu May 09 20:38:23 UTC 202424/05/09 12:27:16 INFO zookeeper.ClientCnxn: [main-SendThread(cloudera-1.c.cloudera-project-422106.internal:2181)]: Opening socket connection to server cloudera-1.c.cloudera-project-422106.internal/10.128.0.11:2181. Will attempt to SASL-authentic

Hi,I am using InvokeHTTP  processer  in Apache Nifi.  Only 20 requests are getting POST at a time. how can I increase the number of requests post through InvokeHttp.Do we have any parameter we can add or change in the InvokeHTTP processer it self orDo we need to change any config level changes.Thanks In advance. 

Hi!Currently, we're utilizing a 3-node Nifi setup within Kubernetes, incorporating a custom NAR to integrate custom processors (approximately 30 processors developed).When updating Nifi along with our custom processors, we typically follow these steps:- Develop a new docker image containing the updated NAR version.- Modify our Kubernetes deployment to incorporate the new docker image.- Delete the Kubernetes deployment, resulting in a complete Nifi stop.- Recreate the Kubernetes deployment, initiating Nifi with the new docker image.Though effective, this process introduces downtime.We're contemplating transitioning to a rolling update approach, wherein each Nifi node undergoes a stop-start cycle. However, in Nifi 1.7 at least, the node that is stopped and restarted cannot to join the cluster due to flow differences in the version of our custom processors. We're curious whether this issue persists in the latest Nifi version, say 1.25.Another option involves deploying multiple versions of

Hi,we installed a CDP Private Cloud 7.1.9 cluster (CM 7.11.3) on AWS ec2 machines with Postgres RDS as backend database.Hive Metastore Server is on two machines:ap-cdpclusteru1c (role group: Hive Metastore Server Default Group)ap-cdpclusteru2c. (role group: hive-HIVEMETASTORE)Configuration in CM looks identical for both Rolesap-cdpclusteru1c starts successfully but ap-cdpclusteru2c fails to start with following error: + exec /cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/lib/hive/bin/hive --config /var/run/cloudera-scm-agent/process/1546344708-hive-HIVEMETASTORE --service metastore -p 9083SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/jars/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/jars/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.

Hello i just installed APACHE Nifi in Linux CentOS and for testing purposes i have this configuration in nifi.properties <pre>nifi.web.https.host=127.0.0.1 nifi.web.https.port=8443 </pre> I saw a solution from here saying that Jetty 10 doesn't accept IP address but instead hostnames, so what I did was to change etc/hosts to 127.0.0.1 -> nifi.local and used that nifi.web.https.host=nifi.local for the configuration of the nifi.properties file. I run in browser https://nifi.local:8443 and i get the same error:( Can someone assist please?`HTTP ERROR 400 Invalid SNI URI: /nifi STATUS: 400 MESSAGE: Invalid SNI SERVLET: - CAUSED BY: org.eclipse.jetty.http.BadMessageException: 400: Invalid SNI Caused by:org.eclipse.jetty.http.BadMessageException: 400: Invalid SNI at org.eclipse.jetty.server.SecureRequestCustomizer.customize(SecureRequestCustomizer.java:266) at org.eclipse.jetty.server.SecureRequestCustomizer.customize(SecureRequestCustomizer.java:207) at

I have a nifi cluster, which keeps having issue of nodes getting disconnected and cluster is unstable.What should be the approach to debug the issue?

Hello experts,I am facing an issue in one of the Nifi server where we have multiple consume eventhub flows.The flow file repository disc is getting full but content and provenance repos are not. Have attached the screen shot of all repos usage and content of flowfile repo.journals folder is occupying very large amount of data.nifi.properties (related to flofile repo):nifi.flowfile.repository.always.sync=falsenifi.flowfile.repository.checkpoint.interval=2 minsnifi.flowfile.repository.directory=/flowfilenifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepositorynifi.flowfile.repository.partitions=256nifi.flowfile.repository.retain.orphaned.flowfiles=truenifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog  Can anyone help me understand what is the issue? how to resolve this?Thanks,Mahendra 

Hi everyone.How can i change, if possible, the ID of a connection in Apache NiFi?from: 9e16eefa-018c-1000-663d-ec184ba2fc6cto: a0077eefa-023c-1000-423d-ef184ba2fc6c (example)I think it's not possible and certainly it's not recommended, but it's a client request. They have an external software that identify the connection by its ID. They've deleted the connection and recreated it, so the ID has changed, and now the external software cannot read the status of this connection. We also cannot change de external software.

Hi,Anyone Unlucky enough to run into this problem (at least as it happens on version M2.0.0-M2) when you try to run run-nifi.bat file to start nifi to be welcomed with this error:    Could not find or load main class org.apache.nifi.bootstrap.RunNiFi   At this point the nifi-app.log file has not been created so you are out of luck in determining what caused this error.Solution:I'm not saying this is the only reason you could encounter this error. After spending some time researching this error , Idid not find the exact match of this error message but I found something close (Example: https://community.cloudera.com/t5/Support-Questions/Error-Error-Could-not-find-or-load-main-class-org-apache/td-p/141006 )In my case you need to watch for the full error message which is :   'F:\NIFI' is not recognized as an internal or external command,operable program or batch file.The JAVA_HOME environment variable is not defined correctly.Instead the PATH wi

Hi,I had some time to download and play with the latest release 2.0.0.M3 which have some major changes compared to predecessor 2.0 releases specially around introducing new UI and Color theme. Here are my preliminary observations so far:1- It was not clear to me in the release notes the following sentence:  The new Apache NiFi UI is included and available at the /nf context path!  So It took me little while to figure out where are the new changes because everything looked the same using regular url ending  /nifi (Example: https://localhost:8443/nifi  ) before realizing oh its path change in the url which is https://localhost:8443/nf , but maybe that was me because I though initially its some configuration in one of conf files like nifi.properties.2- Before figuring out the new url path (context) for the new UI, I  was trying to figure out how to set the dark theme, initially I thought its the brush icon under the canvas Operate panelIt turns out its a

Hello,We have recently migrated from HDP to CDP 7.1.9 and new Knox topologies created as per Cloudera recommendations (cdp-proxy, cdp-proxy-api and cdp-proxy-token) are in place and configured. However, for some reason the topologies from HDP environment are also inherited and are visible on Knox UI, though are not active / used. Kindly advise on below points:1. Is this a normal behaviour that Knox topologies get inherited OR we missed something?2. How do we delete these unwanted topologies so they don't appear in Knox UI.3. There are some topologies that we would still need, however, few services like Ambari and Zeppelin are not needed. How do I remove these from an existing topology?Thankssnm1523

Hello,We are on CDP 7.1.7 SP1 and as per my understanding when DistCp is executed with below parameters, it does the following: 1. update: this will update the files in destination that have got modified at source2. delete: this will delete the files in destination that exist in destination but not in source. Apart feom above, it will copy over any new files at source that do not exist in destination. While copy and update is happening, it does not delete the files instead duplicates them and hence, the data count at destination is double than the source. Kindly help urgently as we are at the verge of initiating heavy data copy. Thankssnm1523

We have 3 regional intake tables partitioned by date and a client-view table partioned by date and region.is there a way to populate the client-view table with data from the 3 source tables in one atomic transaction instead of three separate insert commands:currently we do multiple insert statement like: insert into client_view_tbl( col, col2, col3...)partition(cobdate='20240915', region='region1')select col2, col2, col3... from region1_table where cobdate='20240915';insert into client_view_tbl( col, col2, col3...)partition(cobdate='20240915', region='region2')select col2, col2, col3... from region2_table where cobdate='20240915';insert into client_view_tbl( col, col2, col3...)partition(cobdate='20240915', region='region3')select col2, col2, col3... from region3_table where cobdate='20240915';   

As you know, the support for RHEL7.x is approaching the end. There is an HDP3.1.5 cluster in my production environment.Is it possible to deploy the HDP 3.1.5 cluster and Ambari 2.7.5And how to solve the dependency issues that previously uses package of el7? 

Hi,I am encountering a network issue with Hive where is trying to connect to a node that does not exist when trying to do a SELECT * query from either Hue or Beeline CLI:Error while compiling statement: FAILED: SemanticException Unable to determine if hdfs://ip-172-19-36-94.ap-southeast-2.compute.internal:8020/user/hive/warehouse/allowed is encrypted: org.apache.hadoop.hive.ql.metadata.HiveException: java.net.NoRouteToHostException: No Route to Host from ip-172-19-36-68/172.19.36.68 to ip-172-19-36-94.ap-southeast-2.compute.internal:8020 failed on socket timeout exception: java.net.NoRouteToHostException: No route to host; For more details see: http://wiki.apache.org/hadoop/NoRouteToHostThe node that is specified by ip-172-19-36-94.ap-southeast-2.compute.internal does not exist in the cluster or anywhere in our environment.I have been looking for the hive-site.xml and core-site.xml but there are no references to this DNS entry.I was just wondering if there was anywhere else H

On the fresh new cluster based on HDP 3.1.0 (Kerberized) I'm still facing to a problem with Spark and Hive reading. Connection via HWC is not working. When a try run hive.table("default.table1").show I'll get an error message:java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: shadehive.org.apache.hive.service.cli.HiveSQLException: java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.NullPointerException at com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.createBatchDataReaderFactories(HiveWarehouseDataSourceReader.java:166) at org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanExec.inputRDD$lzycompute(DataSourceV2ScanExec.scala:64) at org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanExec.inputRDD(DataSourceV2ScanExec.scala:60) at org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanExec.inputRDDs(DataSourceV2ScanExec.scala:79) at org.apache.spark.sql.executi

I am using ValidateCsv 1.21.0 processor and in the documentation a property Include all violations must exist but I can't find it in the configuration. When I add it manually it is not known.What could be the reason ? 

we are planning to deploy rack concept in cdh 6.3.4 clusterthe cluster is currently in the "/default" racki have some questions:1. what would be the best scenario to deploy the rack in the cluster?2. will this rack deployment cause unavailability in the cluster?3. what are the “strengths” and “weaknesses” when deploying the rack in the cluster?