Q&A

Hello , can anyone help with a spec to transform this JSON input to the desired output . JSON INPUT=========={"saleStrategy": {"strategy": "strategy1","code": "code1"},"saleStrategyCollection": [{"id": "2","saleStrategy": {"strategy": "strategy2","code": "code2"}},{"id": "3","saleStrategy": {"strategy": "strategy3","code": "code3"}}]}== desired output json =={"saleStrategies" : [{"strategy" : "strategy1","code" : "code1"},{"strategy" : "strategy2","code" : "code2"},{"strategy" : "strategy3","code" : "code3"}]}== current spec === // this currently only transforms the initial single saleStrategy object to an array called salesStrategies. Desired output above shows that If there is a saleStrategy array also present in the input JSON , then the spec needs to add the saleStrategy objects within that array to the salesStrategies output array as well . NOTE the transform must not include "id" in the output.[{"operation": "shift","spec": {"saleStrategy": {"strategy": "saleStrategies[0].st

Hi Community, I'd like to know if it's possible to display a table or column description from Power BI that is connected to Hive using ODBC driver? Issue is following: I set a comment in a Hive table and on its columns and I would expect that these are loaded and visible as "Description" fields from external tools (like Power BI). For example:create table mytable1(mycol1 string comment 'column description') stored as ORC TBLPROPERTIES('comment' = 'table description'); I would expect that when Power BI accesses this table, it would retrieve these comments as descriptions of the table/column, but the "Description" fields are all 'null'. Note that these table/column descriptions are visible from HUE. Thanks in advance for your anwsers.

impala执行sql语句时速度不稳定，同样类型的sql存在十几秒完成，也存在几分钟才完成，从sql执行持续时间上来看，[客户端获取等待时间]占整体时间占比很高，请问是什么因素会影响[客户端获取等待时间]   负载均衡中间件：Haproxy 负载超时时间设置：30m     查询慢的脚本 查询慢的脚本1查询慢的脚本2查询慢的脚本3  

hi  all, hbase version: cdh 6.2.0 balancer config <property><name>hbase.master.loadbalancer.class</name><value>org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer</value><final>false</final><source>hbase-default.xml</source></property> <property><name>hbase.balancer.period</name><value>300000</value><final>false</final><source>hbase-default.xml</source></property> but the balancer not scheduler  interval  5min and no error logs  and use  balancer command in shell  will return  1  hbase(main):007:0> balancertrueTook 0.0746 seconds=> 1 and there any help to fix it ?

Hi,we would like to install Cloudera CDP 7.1.7 SP1 using a Powerscale Storage with OneFS 8.2 or 9.2.Is there any limitations about the Cloudera CDP Runtime features? (Es. Kudu, HDFS ACL, etc.)I can't find any compatibility matrix for this purpose.Thanks in advance. 

Hi, I have inherited some Hadoop 3.1.1 clusters. There is a configured connection between SAP BO and Hadoop using Knox 1.0.0 as the Proxy for authentication. The problem that I have is that I don´t count with user to authenticate to Knox, and I need to edit the gateway. Is there a way to recover the admin password? Thanks for your support.

I am trying update nifi-1.15.3   java version from 1.8 to jdk 11. While trying to start up Nifi getting below error:  org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'niFiWebApiConfiguration': BeanPostProcessor before instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'metaDataSourceAdvisor': Cannot resolve reference to bean 'methodSecurityMetadataSource' while setting constructor argument; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.method.configuration.GlobalMethodSecurityConfiguration': Unsatisfied dependency expressed through method 'setMethodSecurityExpressionHandler' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.

Following the Apache walkthrough titled "NiFi Cluster Using NiFi CA", i created a 3 node cluster using 3 separate aws instances.  However, when I access the GUI at https://nifi1:9337 (or nifi2, or nifi3), for nifi1 and nifi2, the GUI only reflects 2 nodes (nifi1 and nifi2 but not nifi3).  https://nifi3:9337 only reflects 1 node i.e., nifi3. I am using an external zookeeper and these are the properties I changed in nifi.properties:nifi.cluster.is.node = truenifi.cluster.load.balance.host = nifi[1,2,or 3]...depending on which host the nifi.properties file exists innifi.zookeeper.connect.string = zookeeper1 private IP address:2181, zookeeper2 private IP address:2181, etc... Additionally, per the directions, I updated the state-management.xml file and added the list of zookeeper instances in the Connect String property i.e., zoo1private IP address:2181, etc... I also updated the authorizers.xml file, by adding the following:  in userGroupProvide

So I'm working with a batch JSON file with the following value :[ { "eventType": "UPDATE", "eventTime": "2021-12-14T12:34:56.789012Z", "country": "ES", "resourceType": "Party" }, { "eventType": "UPDATE", "eventTime": "2021-12-14T12:34:56.789012Z", "country": "ES", "resourceType": "Party" }, { "eventType": "UPDATE", "eventTime": "2021-12-14T12:34:56.789012Z", "country": "FR", "resourceType": "Party" }, { "eventType": "UPDATE", "eventTime": "2021-12-14T12:34:56.789012Z", "country": "FR", "resourceType": "Party" }]So I'm working with a batch JSON file with the following value :[ { "eventType": "UPDATE", "eventTime": "2021-12-14T12:34:56.789012Z", "country": "ES", "resourceType": "Party" }, { "eventType": "UPDATE", "eventTime": "2021-12-14T12:34:56.789012Z", "country": "ES", "resourceType": "Party" }, { "eventType": "UPDATE", "eventTime": "2021-12-14T12:34:56.789012Z", "country": "FR", "resourc

I've checked the configuration file and other information, and I don't see any problem. 2023-02-08 15:49:16,113 ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch: Unable to recover container container_e05_1612422638528_1073494_02_000023java.io.IOException: Problem signalling container 20735 with NULL; output: and exitCode: 24 at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.signalContainer(LinuxContainerExecutor.java:515) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.isContainerProcessAlive(LinuxContainerExecutor.java:570) at org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor.reacquireContainer(ContainerExecutor.java:179) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.reacquireContainer(LinuxContainerExecutor.java:473) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.RecoveredContainerLaunch.call(RecoveredContainerLaunch.java:83) at org.apa

Hi All, We are facing an issue where no matter what we try Impala queries will randomly throw a "Failed to open HDFS file" error. This seemingly started out of nowhere and we are not sure what else to try. Below are some of the things we have tried. 1. Enforce SYNC_DDL2. We used to have 87 impala daemons (both executor and coordinator). We setup dedicated coordinators for Impala (4 coordinator + 83 executors) and load balanced with haproxy.3. Tried adding invalidate metadata, and then removing it. Below is the sequence of queries. 1. Insert Overwrite a table. (approx every 1 hour)2. Refresh 3. Compute stats.4. Select. The select never fails on the same coordinator as insert, but randomly on other coordinators. And it keeps failing until a refresh. As soon as a refresh is run on the other failing coordinator, query succeeds. This leads me to believe it is a metadata sync issue across coordinators. The problem is that multiple applications/dashboar

Hi, Gor message out of the blue - speaking to agents. Any one know if its part of Cloudera.com or is it a scam? Thanks 

Is there a way to get or see the log message in case of validation failure of a flow-content versus the provided json schema ? 

Please could someone help what can be done here , i'm getting attached error while try to set up the cluster

I am able to create tables and load data in iceberg manually. But how can I do that using nifi? And how can I connect to the remote server using nifi?

I am setting up a Pre prod cluster from Path B. I ran the command "/usr/share/cmf/schema/scm_prepare_database.sh mysql -h mysqldatabase scm temp password" to check the database status. I am getting the error of log4j however the SCM database is configured correctly. Please find the below error and suggest any workaround for this error.[root@clouderamanage cloudera-scm-server]# /usr/share/cmf/schema/scm_prepare_database.sh mysql -h webserver scmmysqldatabase passwordJAVA_HOME=/usr/java/jdk1.7.0_67-clouderaVerifying that we can write to /etc/cloudera-scm-serverCreating SCM configuration file in /etc/cloudera-scm-serverExecuting: /usr/java/jdk1.7.0_67-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.log4j: ERROR Could not find value for key log4j.

I have created a new Nifi registry image using our own base image. I deployed this image in one of my cluster, on creating a processor group in Nifi I can do import from Nifi registry and create a new bucket with flow.But on deploying this same image in the another cluster import from Nifi registry works fine. Creation of new buckets with new flow is taking some time. In some case I have to delete the pod of Nifi registry to get this done.Both nifi and Nifi registry pods are in the same namespace . No DB and PV for Nifi registry.Does anyone here have any idea what might be causing this issue.

Hi team, I'm facing the below-mentioned issue. executorLostFailure (executor 6 exited unrel+details ExecutorLostFailure (executor 6 exited unrelated to the running tasks) Reason: Container marked as failed: container_1677979357691_0001 on host: xyz Exit status: -100. Diagnostics: Container released on a *lost* node.Previously I'm using 18 r5.4xlarge machines, so now I want to move it to 5 r5.16xlarge machines. So I made a few config changesI'm using m5.2xlarge as a master in both the casesI didn't change the EBS volume size. Previously and now also it is 950G spark.executor.cores: "5"spark.driver.cores: "5"spark.executor.memory: "35G"spark.driver.memory: "30G"spark.executor.instances: "60"spark.dynamicAllocation.enabled: "false"spark.executor.memoryOverhead: "6G"spark.excludeOnFailure.enabled: "true"spark.excludeOnFailure.enabled: "true"spark.excludeOnFailure.killExcludedExecutors: "true"spark.excludeOnFailure.application.fetchFailure.enabled: "true"spark.excludeOnFailure.app

hi after running the create_parcel.sh file and copying the .parcel and .parcel.sha file to cloudera i get the following error error Error when distributing to serverxxx.local [Errno 20] Not a directory u'opt/cloudera/parcels/GCSCONNECTOR-3.0.0xxx.sha/meta/parcel.json' and  error Error when distributing to serverxxx.local [Errno 20] Not a directory u'opt/cloudera/parcels/GCSCONNECTOR-3.0.0xxx.parcel/meta/parcel.json'

We have around 6k tables in HDFS. All these tables are stored as sub-dirctories under a parent directory. As of now we're passing the root directory and recursive subdirectories as true in LISTHdfs processor. And later using routeonattribute processor to filter out the relevant tables using the path name. But the problem here is LISTHDFS is taking larger time just to list out as it needs to go through 6K directories. Is there is anyway possible to pass only required directories in listhdfs processor or any workaround available for this?

Hello, Have recently deployed a CDP 7.1.7 SP1 cluster and unable to figure out issues with Solr service. When I open Solr Admin UI, it does opens the UI but with red strips and no information related to system, logs, audits, collections, etc. Ideally we should be able to see some system info on the home page and also various logs when in Logging section, and so on. I checked in logs but do not see any error. Kindly help / guide on how this can be investigated.Thankssnm1523

I want to access the reporting task id of a particular reporting task. For doing the same , I thought about fetching all reporting task id and then searching for a particular one. Is there any nifi-api that can give me all the reporting task ids or any other approach to solve this problem.

How can I use Apache NiFi to transform the current output of my Oracle database to the desired output format, as shown below? Desier out should be as mention below Avro formatCurrent output:{ "type":"record", "name":"nifiRecord", "namespace":"org.apache.nifi", "fields":[ {"name":"AGENT_CODE", "type":["null","string"]}, {"name":"CORR_ACC_NO", "type":["null","string"]}, {"name":"LOCAL_ACC_NO", "type":["null","string"]} ] }Desired output:{ "type": "record", "name": "Bank", "namespace": "com.xyz.model.vo", "fields":[ {"name": "instance", "type": "string"}, {"name": "agent_code", "type": "string"}, {"name": "corr_acc_no", "type": "long"}, {"name": "local_acc_no", "type": "string"} ] }What steps do I need to take in NiFi to achieve this transformation and output the results to the desired format? especially for changing { "type": "record", "name": "Bank", "namespace": "com.xyz.model.vo"

Hi All, I'm new to HBase. I have learnt it by some books and Udemy courses. Can someone help me to understand, how to automate it? For example, if one wants to create tables or wanted to perform some administrative tasks, has to create script and run it periodically. in this case, we can wrte script and schedule it in scheduling tool. Which scripting is being used here? Shell Script or HBase has internal scripting mechanism? Any books, tutorial suggestion are welcome.

We have a large json file which is more than 100GB and we want to split this json file into multiple files. We used Split Text processor to split this json file into mutliple files by specifying Line Split Count. Is there any way we can pass attribute/variable in Line Split Count and then split the records based on the attribute/variable as currently Line Split Count does not support attributes/variables.Kindly suggest if there is another approach to split these json files based on attribute/variables Sample Json File {"name": "John","lastName": "Wick","phoneNumber": "123123123"} {"name": "Paul","lastName": "Jackson","phoneNumber": "123123123"}{"name": "Paul","lastName": "Jackson","phoneNumber": "123123123"}