Solved

Hi,We upgrade cloudera from 5.11.2 to 7.1.7 and we have a process of creating Env with that new version.It failed on :"Aborting command 1546335073 (CreateTmp) because timeout value 210 seconds exceeded. "How can I increase the default timeout from 210 to 400 for example?Thanks

Hello guys,Can someone please help me, how I can write a solr query where results to be grouped on one field (text type) and sum on another field. Here is my documents results:  {"responseHeader": {"zkConnected": true,"status": 0,"QTime": 30,"params": {"q": "*:*","doAs": "knoxui","fl": "access, agent, repo, resource, resType, event_count","fq": ["access:read","repo:cm_ozone","resType:key","action:read"],"_forwardedCount": "1","_": "1674633786010"}},"response": {"numFound": 8,"start": 0,"docs": [{"access": "read","agent": "ozone","repo": "cm_ozone","resource": "volume1/fso-bucket/dir1/dir2/dir3","resType": "key","event_count": 1},{"access": "read","agent": "ozone","repo": "cm_ozone","resource": "volume1/fso-bucket/dir1","resType": "key","event_count": 1},{"access": "read","agent": "ozone","repo": "cm_ozone","resource": "volume1/fso-bucket/dir1/test3.txt","resType": "key","event_count": 1},{"access": "read","agent": "ozone","repo": "cm_ozone","resource": "volume1/fso-bucket/dir

While writing data to Hbase, I am seeing the following logs at client end:AsyncRequestFutureImpl [ERROR] No ClientProtos.RegionLoadStats found for server=regionserver-1,16020,1662657306210, region=table1,,1662657579710.841db4e690a77a76e74b322f413b3c96.I am successfully able to write data, but I am seeing these logs for every write request. Looks like this is probably because Region Load Stats are disabled for regions in the region servers. Is there any way to enable them and fix this logging?Hbase Client version : 2.1.7Hbase Server version : 2.1.7

Hi Team, cloudera embedded postgres db taking more memory. How to check and reduce embedded postgres db taking more memory?? Regards,Hanu

Hello Everyone  Can I Create custom processor in apache nifi to convert CSVfile to excel file ?? is that possible and if yes how ? Thank you  

 I am following the Apache nifi instructions:  nifi.apache.org/docs/html/administration-guide.html#basic-cluster-setup. And am receiving the following error dump (see below): One specific error that keeps repeating is: Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: The nifi.web.http.port must be specified if running in a cluster with nifi.cluster.protocol.is.secure set to false.at Can anyone provide any guidance/recommendations for resolving the errors I am encountering? Thank you! VR, Dave org.apache.nifi.processors.hadoop.FetchHDFSorg.apache.nifi:nifi-hadoop-nar:1.19.1 || ./work/nar/extensions/nifi-hadoop-nar-1.19.1.nar-unpackedorg.apache.nifi.processors.cassandra.PutCassandraRecordorg.apache.nifi:nifi-cassandra-nar:1.19.1 || ./work/nar/extensions/nifi-cassandra-nar-1.19.1.na

Hello I'm trying to implement an automation process to instanciate a nifi cluster and I have a problem with Context Parameters.Using the nifi REST API with curl commands (or python scripts), I have found how to create a context parameter, how to populate it with parameters, how to bound it to a process group, but I do not find how to unbind it from a processor group (like if I want to reset the "Process Group Parameter Context" of the processor group configuration window scrolling menu to "No parameter context") I meen it is possible (it is possible using the GUI)Does anyone have any idea how to do this.ThanksBest regards,Absolute Z

I have json file like this and want send it to hive using nifi, how can i change the timestamp type because when i send it to hive, in hive table the column type is string, and I read in hive DDL format for timestamp "yyyy-MM-dd HH:mm:ss.SSS" {"nama": "d", "alamat": "jkt", "_ab_cdc_updated_at": "2023-01-05T05:57:43.499Z", "_ab_cdc_lsn": 1640844128, "_ab_cdc_deleted_at": null}{"nama": "a", "alamat": null, "_ab_cdc_updated_at": "2023-01-05T05:57:43.510Z", "_ab_cdc_lsn": 1640844256, "_ab_cdc_deleted_at": "2023-01-05T05:57:43.510Z"}{"nama": "c", "alamat": "semarang", "_ab_cdc_updated_at": "2023-01-12T03:38:32.136Z", "_ab_cdc_lsn": 1684930552, "_ab_cdc_deleted_at": null}{"nama": "f", "alamat": null, "_ab_cdc_updated_at": "2023-01-12T04:11:53.329Z", "_ab_cdc_lsn": 1685073792, "_ab_cdc_deleted_at": "2023-01-12T04:11:53.329Z"}

Nifi - Is there a way where credentials for sftp NIFI can be read from DB and used

I need to pass headers in order make a WebSocket connection.How can I do this in NiFi? 

Hi Team,We have used the latest Solr version 8..11.1. In this same version, I tried to create a config set but got exceptions like "Can't create a config set with an unauthenticated request from a trusted baseConfigSet". Used below command for creating config set :http://localhost:8984/solr/admin/configs?action=CREATE&name=_client&baseConfigSet=_default&configSetProp.immutable=false&wt=xml&omitHeader=true Response with exception :Please let me know the steps for creating a config-set in New version 8.11.1.

Hi All, Need some guideline for "how to define/design Dynamic SPARK job" any parameter. Please suggest me, it would be more appreciate in advance. Regards,Pankaj

can I get official support if I install the CDH on hygon server

Hi All, Need advice, how we can export all the DDL of one scheme from Hive ( from QA region, a schema with around 2000 tables) to Dev or ACPT region, only DDL's  needed to be in sync , not worried about the data, so do we have any option or script to import / export all the DDL's from one region to another region in one go, please suggest, thank you. PK

Hi All, Do we have any script to export all the DDL's of the table for a schema in Hive ? we have a schema name - staging_DB and its contains around 2000 tables, now we need to export all the DDL's of this tables to a txt file so that we can run this DDL's in another setup. Please suggest if we have any script or any easy way to do it, thank you. - PK

I have a situation, where I have to get the combined data from two tables  from Mysql database, ExecuteSql processor throwing out bellow error,  where as the same query is working fine from Mysql Workbench.   2020-04-22 14:55:06,597 ERROR [Timer-Driven Process Thread-8] o.a.nifi.processors.standard.ExecuteSQL ExecuteSQL[id=9d81c7ae-0171-1000-f638-2f1e60bfe48e] Unable to execute SQL select query selectt1.CODE_CLOCK_ID,t1.COMPANY_CD,t1.GROUP_SEGMENT_L1,t1.GROUP_SEGMENT_CD,t2.CODE_CLOCK_ID,t2.COMPANY_CD,t2.GROUP_SEGMENT_L1,t2.GROUP_SEGMENT_CDfromsheet26 t1INNER JOIN sheet27 t2on t1.CODE_CLOCK_ID = t2.CODE_CLOCK_ID and t1.COMPANY_CD = t2.COMPANY_CD LIMIT 0, 1000; due to org.apache.nifi.processor.exception.ProcessException: org.apache.avro.AvroRuntimeException: Duplicate field CODE_CLOCK_ID in record any.data.sheet26: CODE_CLOCK_ID type:UNION pos:4 and CODE_CLOCK_ID type:UNION pos:0.. No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: org.a

Hi , I have a spark cluster deployed on windows. I'm trying to submit a simple spark job using the rest api. The job is just python code that does simple hello world sentence as follows :from pyspark.sql import SparkSessiondef main(args): print('hello world') return 0if __name__ == '__main__': main(None)The url Im using to submit the job is:http://<Master-IP>:6066/v1/submissions/createWith the following Post Body: { "appResource": "file:../../helloworld.py", "sparkProperties": { "spark.executor.memory": "2g", "spark.master": "spark://<Master IP>:7077", "spark.app.name": "Spark REST API - Hello world", "spark.driver.memory": "2g", "spark.eventLog.enabled": "false", "spark.driver.cores": "2", "spark.submit.deployMode": "cluster", "spark.driver.supervise": "true" }, "clientSparkVersion": "3.3.1", "mainClass": "org.apache.spark.deploy.SparkSubmit", "environmentVariables": { "SPARK_

Hi. I haven't found any documentation for these internals about the InvokeHTTP processor in NiFi. My requirement is to throttle requests to and endpoint (which we also control) that will accept, let's say, 5 simultaneous connections in an address and port, but I need to wait for each of their responses before letting another flowfile to be sent, so: Does this translate to simply using 1 InvokeHTTP processor configured to 5 "Concurrent Tasks" and that's it?Will the processor wait for the remote endpoint's request before sending the next one?How does the "Run Schedule" works together with the previous settings? (if I had, e.g.: 1 sec)I've been proposed with splitting the incoming queue and put 5 InvokeHTTP processors in parallel, each one attending 1/5 of the incoming flowfiles (I'd do the pre-partitioning before with some RouteOnAttribute trick), but I think it's exactly the same outcome as the 1. above. Is it?This all boils down to not knowing exactly how this processor work

Hi,I'm looking at using Apache Nifi as a data integration hub for our case/archive management solution used in the Nordics and our rest-based API. One use case is to first read an XML from one source with a few values and then use these values in a larger JSON payload that creates a new case in our solution with InvokeHTTP.All values in the JSON arrays are inside a parameter{} node and can have objects with child properties. I've done a lot of research and testing with the different processors, like ReplaceText, and AttributesToJSON, but can't find how I easily can do this. Is it needed to use some complex JSON generator like Groovy or Jolt to achieve this with ExecuteScript? With a online tool like make.com I was able to do this really easy, but find it complex to just replace multiple single values in the JSON payload before InvokeHHTP when the JSON payload is nested. Example of a JSON payload for the API can be something like below, but can also contain many

In CDP Public Cloud Machine Learning, we can create a new session with reserved resource, for example 4vCPU and 16 GiB Memory.We can also create spark session inside the machine learning workbench with some memory configuration. For example:spark = (SparkSession.builder.appName(appName).config("spark.driver.memory", "16G").config("spark.executor.instances", "10").config("spark.executor.cores", "4").config("spark.executor.memory", "20G").getOrCreate())My question is, how will the memory be allocated to Spark session now? Is the reserved resource (4vCPU and 16 GiB Memory) in machine learning session the maximal limitation for total spark memory usage? How many work nodes and executors can I configure for the spark session? 

Folks, I have setup the secured nifi cluster on development env and there are few things hitting my mind wrt to production use. - There are few configurations related to database, flow repository, content repository, provenance, components etc and I'm wondering what should be the best practices to manage these files. Should I use the persistence volume/storage on K8 to have these centralized for whole cluster.                    If yes, would it interfere with internal replication? Wouldn't it be SPOF?                    If not, what if my cluster goes down, I will loose all the state and data or shall I use a replica set explicitly? Can someone helps understanding the best practices on production wrt above scenarios and also anything in general Thanks,

Hello, My NiFi flow has two process groups that process data from kafka:main - ~20 processors that processes 100 json messages per secondsupport - ~900 processors that processes 1-2 json messages in a day When the main group is running (started) without message processing the CPU load is ~5%. When the support group is additionally started the CPU load is ~20% (without messages). In addition the main group handles messages with 1-2 sec latency if the support group is not running, but latency increases dramatically (x10-20) if i just start the support group. Is it possible to set additional priorities for the main group or somehow configure the nifi scheduler? Thanks.

hey,when I run a sqoop program. sqoop import "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" --driver org.apache.phoenix.jdbc.PhoenixDriver --connect jdbc:phoenix:172.18.1.16:2181 --query "select sid from test.PV WHERE 1=1 and \$CONDITIONS limit 10" --hive-import --hive-database tmp --hive-table system_stats --target-dir /hivetable/phoenixdb --delete-target-dir --split-by sid --hive-overwrite --null-string '\\N' --null-non-string '\\N' it's out this error  023-01-06 16:53:15,737 INFO [ReadOnlyZKClient-172.18.1.16:2181@0x062577d6] org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/yarn/nm/usercache/root/appcache/application_1672990653606_0007/container_e14_1672990653606_0007_01_0000022023-01-06 16:53:15,738 INFO [ReadOnlyZKClient-172.18.1.16:2181@0x062577d6] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=172.18.1.16:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$16/1413879487

Hi All,   Can someone past procedure to decommission disk from Ambari cluster. It will helpful. As we have 3 data node's and we have 3 disks configured each node and we want to remove one disk from it.    Thanks, Amit Mayekar