Solved

Hi All, We are enabling hbase hook in apache atlas 2.2.0 , we followed below steps ,https://atlas.apache.org/2.0.0/Hook-HBase.html After that , when we start to start hbase , failing with below error.ROD.2022-09-05 14:02:23,481 INFO [master/vmazewdranger01:16000:becomeActiveMaster] atlas.ApplicationProperties: Setting solr.wait-searcherproperty 'true'2022-09-05 14:02:23,481 INFO [master/vmazewdranger01:16000:becomeActiveMaster] atlas.ApplicationProperties: Setting index.search.map-name property 'false'2022-09-05 14:02:23,482 INFO [master/vmazewdranger01:16000:becomeActiveMaster] hook.AtlasHook: Failed to load application propertiesorg.apache.atlas.AtlasException: Failed to load application propertiesat org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:150)at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:103)at org.apache.atlas.hook.AtlasHook.<clinit>(AtlasHook.java:80)at org.apache.atlas.hbase.hook.HBaseAtlasCoprocessor.<in

Hi,  I have the requirement where I am converting the base64 Image content to attribute and after that I am preparing the payload including the base64 Image content. But the problem is while converting the base64 Image content (approx. 6-7 MB size ) it is getting truncated in the attribute Value.  Can you please tell me what is the length limit of attribute value in NIFI.Appreciate your help. Thank you.

I've added ranger as service in Cloudera manager, but when I click 'audit' from the menu i get the following error : 

Hello,I am trying to create an Operational DB and getting the error in Subject. Appreciate any inputs.Thanks in advance.  

Hi Cloudera community,   I have been using MapR for sometime and have some regular data pipelines in place. I want to move out from MapR to Couldera.    My clarification with the experts here is, I want to retain the system & data architecture created in MapR(servers, nodes, clusters, data storage) and implement couldera on top of hadoop HDFS. Is this possible?   basically i want to replace MapR with Couldera without disturbing my existing hardware and data storage setup.    Looking forward for some sharing.

Problem: I have set up a listSFTP processor to run between 2am to 2pm everyday.  But the files are not being picked up by the processor.  Existing setup and configuration: The files are generated at the SFTP server at 12 am to 1am every day. ListSFTP ConfigurationSchedule: listing strategy: tracking timestampsother tracking configuration  When I start this processor, it runs as expected for 1 day or 2 and after that, the files are not picked.  Is it the tracking time window of 3 hours that is affecting the file being listed? 

Hello,I have a question about Cloudera Data Platform and SDX :Is data sharing available accros my company only?If I want to share a specific file or a specific table in my datawarehouse with my client outside my company (who has his own account in Cloudera), is it possible? Regards,Mohamed

I have a GenerateFlowFile processor that sends a flowFile to the ExecuteScript processor. In the ExecuteScript processor, I added a new attribute and value to the incoming flowFIle. But this updated flowFIle is not output to the next processorIncoming FlowFile content from GenerateFlowFile  {'name':'myName'}  ExecuteScript script body  var flowFile = session.get();if (flowFile != null) { try { flowFile = session.putAttribute(flowFile, 'gender', 'male'); session.transfer(flowFile, REL_SUCCESS); } catch (e) { session.transfer(flowFile, REL_FAILURE); }}   Output to next processor  {'name':'myName'}  Am I missing something??

I have a NiFi flow that reads file from local drive and writes in another folder. But due to some reason it is not reading files on regular basis. I can't find anything on the log. The log file is printed as if nothing was scheduled for that particular time. It's simply skipping the timing and continuing. I'm using Apache NiFi version 1.11.4 Screenshot of the Flow. Screenshot of Scheduling tabScreenshot of the GetFile Processor Scheduling tab. I have scheduled it for 08:05 on daily basis. Screenshot of properties tabScreenshot of GetFile Processor Properties tab. This NiFi is hosted on Windows 2012 server and this problem is only occurring in production environment. Although all the configuration is same in all the environment.  I'm also attaching the sample flow template in case if anyone want to try it out. Sample_flow.xml  <?xml version="1.0" encoding="UTF-8" standalone="yes"?><template encoding-version="1.3"> &lt

The current cluster has 7 tablet servers and 448 CPU cores. It is planned to set Hash partitions to cooperate with Range partitions. The Range partition divides the ten year data into 120 partitions by month. How many Hash partitions based on other fields is appropriate? In addition, the main read/write scenario is a large number of reads/writes based on the current month's data.

So I am  using SplitXml processor to split flowfile. After some transformations I am merging those flowfiles back together using MrgeContent. After mergning I have noticed that not all of the flow files merged together. e.g. I have splitted my flowfile  on 100 parts and after merging I've got 2 files with 50 records of my initial flow file each. When i set  Run schedule option to 10 seconds instead of 1 it merged all files together. But what if i don't know how many flowfiles i will get after split and how much time it will take for merge processor to merge it?  I use flowfile name as Correlation attribute name. Thank you  beforehand!  

 Hello community, well i cannot put data from local to hdfs due to this error.how can i solve it please?!

There are many same errors showing at back-end and nothing return to front-end.error message as below: 2022-10-31T01:42:15.569Z, RpcRetryingCaller{globalStartTime=1667180497142, pause=100, maxAttempts=11}, org.apache.hadoop.hbase.CallQueueTooBigException: Call queue is full on XXXXX,16020,1663311727745, too many items queued ? env: OpenShift container + Hbase

running the following from hiveSET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;SET hive.support.concurrency=true;INSERT INTO hello_acid partition (load_date='2016-03-01') VALUES (1, 1);getting this error SQL Error [30041] [42000]: Error while processing statement: FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session e73c125d-e40a-49fb-b2e0-28c28ab1ff60_0: java.lang.RuntimeException: spark-submit process failed with exit code 1 and error ? Hive has Spark configured as  dependency  Have this settings in  hive-site.xml <property><name>hive.spark.client.connect.timeout</name><value>30000ms</value></property><property><name>hive.spark.client.server.connect.timeout</name><value>300000ms</value></property> Some Other settings Java Configuration Options for HiveSer

Do table statistics get replicated as part of hive BDR replication ?

Hi We use Cloudera in the environment without internet access These URLs added on Parcel Repository & Network Settings :https://archive.cloudera.com/accumulo-c5/parcels/latest/https://archive.cloudera.com/accumulo6/6.1/parcels/https://archive.cloudera.com/kafka/parcels/latest/http://archive.cloudera.com/kudu/parcels/latest/Someone know how to remove these URLs using the CM API ? Best Regards,  

Hello, As part of a class I am in, we have tasks to edit the configuration files to include or change certain properties.   I have found numerous articles detailing the importance of changing these properties but cannot find any explaining HOW TO change properties, either through the CLI or the web interface.  We are using Hortonworks Hadoop Sandbox HDP through a virtual machine.   For instance, in one of our tasks, we are to configure wire encryption.  I understand that I need to edit the core-site.xml configuration file to include the property  hadoop.rpc.protection=privacy But I do not understand how to do so.  I have tried going into the CLI, and the furthest I can make it is to change the directory to etc/hadoop/conf where the file is located.  I can also view the file properties using the vi command.  But how can I add a property to the file, or replace the value if it an existing property.  I have searche

Hi all, I would like to change the name of objects and add values to the following JSON file.[ { "Ean": 4052269532309, "Bestand": 0 }, { "Ean": 4052269532316, "Bestand": 0 }, { "Ean": 4052269532323, "Bestand": 0 }]The result of my specification is not what I expect.JOLT Script:[ { "operation": "shift", "spec": { "*": { "Ean": "gtin", "ean": "gtin", "EAN": "gtin", "EanCode": "gtin", "EAN-Code": "gtin", "EANCode": "gtin", "Ausdr1": "gtin", "Bestand": "qty", "Menge": "qty", "Stock": "qty", "stock": "qty", "Verfügbare Menge (Kennzahl)": "qty", "Verfügbare Menge": "qty" } } }, { "operation": "modify-default-beta", "spec": { "update_date": "2022-11-03", "supplier_name": "demo", "file_name": "test" } }]I get this result:{ "gtin" : [ 4052269532309, 4052269532316, 4052269532323 ], "qty" : [ 0, 0, 0 ], "update_date" : "2022-11-03",

Hi,I have Spark program written in scala which is continuously consuming data from Kafka topic processing and aggregating data. The program uses structured streaming and is supposed to insert data in Hive (non- acid table) partitioned based on 'startdate' field from the data.I tried below to insert data to Hive table. val query = stream_df_agg_tr1.writeStream.format("orc").partitionBy("startdate").option("checkpointLocation","<>:8020/user/sm/hivecheckpt8").outputMode(OutputMode.Append).start("<>/warehouse/tablespace/external/hive/activecore.db/applicationstats5minute")query.awaitTermination() I tried by specifying the table path using .option("path",<>) as well. I have below properties set (tried in spark program and as spark-shell arguments) ."spark.hadoop.hive.exec.dynamic.partition=true"  "spark.hadoop.hive.exec.dynamic.partition.mode=nonstrict" Still, I do not see the partitions getting created automatically. Once I tried with one date-based pa

Good morning,   I was hoping someone would know if there is a supported method for installing additional libraries onto the NiFi nodes from Cloudera Manager.  For example, I need to include the Oracle JDBC driver JAR on each server so it can be referenced in the controller services.   I know I can load files onto the servers via server deployment tools, but I was hoping there was a way to deploy additional library files to the cluster via the manager tool as part of the configuration.  (Especially as these files may need to be updated over time, and I would rather not run the scripts which impact the server at a lower level.)   Thanks.    

Hello all, NiFi Jolt questionDoes anyone know if it's possible to convert timestamp formats using Jolt using 'field.value' the way you can using update Record?  I'm aware that I can use an attribute value in Jolt to convert a field, like:"date" : "${timestamp:toDate('yyyy-MM-dd HH:mm:ss'):toNumber()}" but I was wondering if this function can be done using Jolt on incoming 'field.values', have tried using the default spec as below but it doesn't change the format.  Thanks in advance. Andy

I was testing if mapreduce is working properly on my cloudera 6.3, CDH 6.3.2 cluster.I have installed cloudera using one-click-installer Proof-of-concept one! while i excecuted the below command as it is in this linkhttps://docs.cloudera.com/documentation/other/tutorial/CDH5/topics/ht_usage.html  Command:sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100 i Got below error: WARNING: Use "yarn jar" to launch YARN applications.Number of Maps = 10Samples per Map = 100Wrote input for Map #0Wrote input for Map #1Wrote input for Map #2Wrote input for Map #3Wrote input for Map #4Wrote input for Map #5Wrote input for Map #6Wrote input for Map #7Wrote input for Map #8Wrote input for Map #9Starting Job20/01/02 15:18:01 INFO client.RMProxy: Connecting to ResourceManager at ABC.xyz.net/192.168.xx.xx:8032java.io.FileNotFoundException: File does not exist: hdfs://ABC.xyz.net:8020/user/yarn/mapreduce/mr-framework/3.0.

Hello All, I am trying to deploy a 6-node cluster and one of the nodes I am getting attached error while activating parcels. I had tried to copy the mentioned contents from another host where the parcels are activated but I run out of space on the host where issue persists, surprisingly both the host have the same disk space of 5GB in /opt. Requesting your assistance / guidance in resolving this issue.

I have to do some custom preprocessing tasks on a huge data file (~200GB). currently, its works as below way.select * from tablepreprocessing line by linereturn a new single flow fileso I decided to convert the above approach to the below way.get the row count from the user (let's assume the user gives 1000)execute select * query as resultSetread the results line by line (rs.next())when the line count reaches 1000 return the flow file and continues to other linesSo my approach is as below onTrigger public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException { logger = getLogger(); FlowFile flowFile = session.get(); if (flowFile == null) { return; } try { final Long rowLimit = context.getProperty(ProcessorUtils.MAX_RECORD).evaluateAttributeExpressions(flowFile).asLong(); Connection conn = DriverManager.getConnection( // db connection properties

Hi guys, Anyone know scripts to remove hdfs file more than 1 years? Thanks,Mazz