Q&A

I have a GenerateFlowFile processor that sends a flowFile to the ExecuteScript processor. In the ExecuteScript processor, I added a new attribute and value to the incoming flowFIle. But this updated flowFIle is not output to the next processorIncoming FlowFile content from GenerateFlowFile  {'name':'myName'}  ExecuteScript script body  var flowFile = session.get();if (flowFile != null) { try { flowFile = session.putAttribute(flowFile, 'gender', 'male'); session.transfer(flowFile, REL_SUCCESS); } catch (e) { session.transfer(flowFile, REL_FAILURE); }}   Output to next processor  {'name':'myName'}  Am I missing something??

I have NIFI single Node and for my learning purpose I'm trying to implement SiteToSiteBulletinReporyingTask (with assuming this will work for single node)I configured StandardRestrictedSSLContextService as belowNote - same keystore and truststore used inside nifi-properties  I configured SiteToSiteBulletinReportingTask as belowNote - the destination URL is the same URL that nifi is currently running.   Now I'm facing below issues -I'm not receiving any data to the bulletin port. I'm getting thebelow warning message in nifi-app.log (couldn't find resource to fix this)The issue is I'm getting the below warning and, to be honest, I don't have a clear idea about this warning, and also2022-10-23 17:57:09,618 WARN [NiFi Site-to-Site Connection Pool Maintenance] o.apache.nifi.remote.client.PeerSelector Unable to refresh remote group peers due to: Certificate for <xxx.xxx.xx.xxx> doesn't match any of the subject a

I have a NiFi flow that reads file from local drive and writes in another folder. But due to some reason it is not reading files on regular basis. I can't find anything on the log. The log file is printed as if nothing was scheduled for that particular time. It's simply skipping the timing and continuing. I'm using Apache NiFi version 1.11.4 Screenshot of the Flow. Screenshot of Scheduling tabScreenshot of the GetFile Processor Scheduling tab. I have scheduled it for 08:05 on daily basis. Screenshot of properties tabScreenshot of GetFile Processor Properties tab. This NiFi is hosted on Windows 2012 server and this problem is only occurring in production environment. Although all the configuration is same in all the environment.  I'm also attaching the sample flow template in case if anyone want to try it out. Sample_flow.xml  <?xml version="1.0" encoding="UTF-8" standalone="yes"?><template encoding-version="1.3"> &lt

The current cluster has 7 tablet servers and 448 CPU cores. It is planned to set Hash partitions to cooperate with Range partitions. The Range partition divides the ten year data into 120 partitions by month. How many Hash partitions based on other fields is appropriate? In addition, the main read/write scenario is a large number of reads/writes based on the current month's data.

when I input:start-all.shin command line, I can open localhost:50070 successfully.However, if I click on the overview tab, nothing is shown on overview subpage.But other subpages work well, like datanode subpage:localhost:8088 also looks well: I don't know whether this case is normal or there's something wrong with my overview page. 

I'm trying to run listenHTTP processor for listening to a webhook notification. I'm relatively new to this processor. When I initialize, I observe java.net.bind exception. Address already in use.  nifi url: http://localhost:8080/nifi/    (it has a public IP but we use NATing)  Existing configuration    Error Observed    

Have been getting the below exception "ReplaceText[id=3fa43c98-5a5a-1c87-8e34-911a06de02aa] Processing failed: org.apache.nifi.processor.exception.ProcessException: IOException thrown from ReplaceText[id=3fa43c98-5a5a-1c87-8e34-911a06de02aa]: java.io.FileNotFoundException: /opt/nifi-1.17.0/content_repository/466/1666653401958-112082 (No space left on device) - Caused by: java.io.FileNotFoundException: /opt/nifi-1.17.0/content_repository/466/1666653401958-112082 (No space left on device)"   It seems like the disk is full which is not allowing the processors to update or modify the data. I notice that the logs directory on all the cluster have taken all the space.  Each node has 64G of storage space and log directory has taken almost 55G of space.    I recently notice my disk is getting full very quickly. I remove the logs and the repository and restart nifi and that seems to work but then again after a few hours it got filled up. I finally notice that the logs direct

So I am  using SplitXml processor to split flowfile. After some transformations I am merging those flowfiles back together using MrgeContent. After mergning I have noticed that not all of the flow files merged together. e.g. I have splitted my flowfile  on 100 parts and after merging I've got 2 files with 50 records of my initial flow file each. When i set  Run schedule option to 10 seconds instead of 1 it merged all files together. But what if i don't know how many flowfiles i will get after split and how much time it will take for merge processor to merge it?  I use flowfile name as Correlation attribute name. Thank you  beforehand!  

 Hello community, well i cannot put data from local to hdfs due to this error.how can i solve it please?!

There are many same errors showing at back-end and nothing return to front-end.error message as below: 2022-10-31T01:42:15.569Z, RpcRetryingCaller{globalStartTime=1667180497142, pause=100, maxAttempts=11}, org.apache.hadoop.hbase.CallQueueTooBigException: Call queue is full on XXXXX,16020,1663311727745, too many items queued ? env: OpenShift container + Hbase

 I am unable to complete hue installation due to the following errorUnexpected error: Unable to verify database connection #Hue 

running the following from hiveSET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;SET hive.support.concurrency=true;INSERT INTO hello_acid partition (load_date='2016-03-01') VALUES (1, 1);getting this error SQL Error [30041] [42000]: Error while processing statement: FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session e73c125d-e40a-49fb-b2e0-28c28ab1ff60_0: java.lang.RuntimeException: spark-submit process failed with exit code 1 and error ? Hive has Spark configured as  dependency  Have this settings in  hive-site.xml <property><name>hive.spark.client.connect.timeout</name><value>30000ms</value></property><property><name>hive.spark.client.server.connect.timeout</name><value>300000ms</value></property> Some Other settings Java Configuration Options for HiveSer

Do table statistics get replicated as part of hive BDR replication ?

Hi We use Cloudera in the environment without internet access These URLs added on Parcel Repository & Network Settings :https://archive.cloudera.com/accumulo-c5/parcels/latest/https://archive.cloudera.com/accumulo6/6.1/parcels/https://archive.cloudera.com/kafka/parcels/latest/http://archive.cloudera.com/kudu/parcels/latest/Someone know how to remove these URLs using the CM API ? Best Regards,  

Hello, As part of a class I am in, we have tasks to edit the configuration files to include or change certain properties.   I have found numerous articles detailing the importance of changing these properties but cannot find any explaining HOW TO change properties, either through the CLI or the web interface.  We are using Hortonworks Hadoop Sandbox HDP through a virtual machine.   For instance, in one of our tasks, we are to configure wire encryption.  I understand that I need to edit the core-site.xml configuration file to include the property  hadoop.rpc.protection=privacy But I do not understand how to do so.  I have tried going into the CLI, and the furthest I can make it is to change the directory to etc/hadoop/conf where the file is located.  I can also view the file properties using the vi command.  But how can I add a property to the file, or replace the value if it an existing property.  I have searche

Hi all, I would like to change the name of objects and add values to the following JSON file.[ { "Ean": 4052269532309, "Bestand": 0 }, { "Ean": 4052269532316, "Bestand": 0 }, { "Ean": 4052269532323, "Bestand": 0 }]The result of my specification is not what I expect.JOLT Script:[ { "operation": "shift", "spec": { "*": { "Ean": "gtin", "ean": "gtin", "EAN": "gtin", "EanCode": "gtin", "EAN-Code": "gtin", "EANCode": "gtin", "Ausdr1": "gtin", "Bestand": "qty", "Menge": "qty", "Stock": "qty", "stock": "qty", "Verfügbare Menge (Kennzahl)": "qty", "Verfügbare Menge": "qty" } } }, { "operation": "modify-default-beta", "spec": { "update_date": "2022-11-03", "supplier_name": "demo", "file_name": "test" } }]I get this result:{ "gtin" : [ 4052269532309, 4052269532316, 4052269532323 ], "qty" : [ 0, 0, 0 ], "update_date" : "2022-11-03",

Hi,I have Spark program written in scala which is continuously consuming data from Kafka topic processing and aggregating data. The program uses structured streaming and is supposed to insert data in Hive (non- acid table) partitioned based on 'startdate' field from the data.I tried below to insert data to Hive table. val query = stream_df_agg_tr1.writeStream.format("orc").partitionBy("startdate").option("checkpointLocation","<>:8020/user/sm/hivecheckpt8").outputMode(OutputMode.Append).start("<>/warehouse/tablespace/external/hive/activecore.db/applicationstats5minute")query.awaitTermination() I tried by specifying the table path using .option("path",<>) as well. I have below properties set (tried in spark program and as spark-shell arguments) ."spark.hadoop.hive.exec.dynamic.partition=true"  "spark.hadoop.hive.exec.dynamic.partition.mode=nonstrict" Still, I do not see the partitions getting created automatically. Once I tried with one date-based pa

Good morning,   I was hoping someone would know if there is a supported method for installing additional libraries onto the NiFi nodes from Cloudera Manager.  For example, I need to include the Oracle JDBC driver JAR on each server so it can be referenced in the controller services.   I know I can load files onto the servers via server deployment tools, but I was hoping there was a way to deploy additional library files to the cluster via the manager tool as part of the configuration.  (Especially as these files may need to be updated over time, and I would rather not run the scripts which impact the server at a lower level.)   Thanks.    

Updating statistics using "COMPUTE INCREMENTAL STATS" produced the following error.  Server version: impalad version 3.2.0-cdh6.3.2 RELEASE (build 1bb9836227301b839a32c6bc230e35439d5984ac)Query: COMPUTE INCREMENTAL STATS hoge_db.huga_tbl PARTITION ( dt >= "20221015" )ERROR: AnalysisException: Incremental stats size estimate exceeds 200.00MB. Please try COMPUTE STATS instead.  In the error message, it says "Please try COMPUTE STATS instead.".I don't understand why. Wouldn't "COMPUTE STATS" give the same result?

Hello all, NiFi Jolt questionDoes anyone know if it's possible to convert timestamp formats using Jolt using 'field.value' the way you can using update Record?  I'm aware that I can use an attribute value in Jolt to convert a field, like:"date" : "${timestamp:toDate('yyyy-MM-dd HH:mm:ss'):toNumber()}" but I was wondering if this function can be done using Jolt on incoming 'field.values', have tried using the default spec as below but it doesn't change the format.  Thanks in advance. Andy

I was testing if mapreduce is working properly on my cloudera 6.3, CDH 6.3.2 cluster.I have installed cloudera using one-click-installer Proof-of-concept one! while i excecuted the below command as it is in this linkhttps://docs.cloudera.com/documentation/other/tutorial/CDH5/topics/ht_usage.html  Command:sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100 i Got below error: WARNING: Use "yarn jar" to launch YARN applications.Number of Maps = 10Samples per Map = 100Wrote input for Map #0Wrote input for Map #1Wrote input for Map #2Wrote input for Map #3Wrote input for Map #4Wrote input for Map #5Wrote input for Map #6Wrote input for Map #7Wrote input for Map #8Wrote input for Map #9Starting Job20/01/02 15:18:01 INFO client.RMProxy: Connecting to ResourceManager at ABC.xyz.net/192.168.xx.xx:8032java.io.FileNotFoundException: File does not exist: hdfs://ABC.xyz.net:8020/user/yarn/mapreduce/mr-framework/3.0.

Hello!I`m newbie in spark and all cloud-data workflow, but I have a problem on my new job where I need to work with PySpark and Hadoop.In my spark-history some applications are "incompleted" for week now. I've tried to kill them, close sparkContext(), kill main .py process, but nothing helped.For example,   yarn application -status <id>  shows:  ...State: FINISHEDFinal-State: SUCCEDED...Log Aggregation Status: TIME_OUT...  But in Spark-History I still see it in incomplete section of my applications. If I open this application there, I can see 1 Active job with 1 Alive executor, but they are doing nothing for all week. This seems like a logging bug, but as I know this problem is only with me, other coworkers doesn't have this problem.This thread doesn't helped me, because I dont have access to start-history-server.sh.I suppose this is because of   Log Aggregation Status: TIME_OUT  because my "completed" app

Hello All, I am trying to deploy a 6-node cluster and one of the nodes I am getting attached error while activating parcels. I had tried to copy the mentioned contents from another host where the parcels are activated but I run out of space on the host where issue persists, surprisingly both the host have the same disk space of 5GB in /opt. Requesting your assistance / guidance in resolving this issue.

I have a Nifi process where I create a Hive table and add a single row to the table in a separate PutHiveQL process.  The 'create table' process works most of the time, but the 'insert' almost always fails with the below error.  This even though I direct the issue to the RetryFlowFile processor and repeat it 5 times.  Note that I can run the same insert statement in Hue without any issues.  What could be the problem?   Is there anything I can tweak to make this work? PutHiveQL[id=27ab3890-0b98-1ba9-8499-5c4985902ebf] Failed to update Hive for StandardFlowFileRecord[uuid=91a0e7ea-5f2d-4530-bcd8-6e650597f6c7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1666794893345-1, container=default, section=1], offset=5660, length=128],offset=0,name=91a0e7ea-5f2d-4530-bcd8-6e650597f6c7,size=128] due to Error while compiling statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask; it is possible that re

also tried on web shell client gettiing  server not yet listening on http port 8080 after 50 seconds Exiting.Tried all passwords plz help?