Q&A

Version 1.15.3 Error: org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable to locate node CN=w-nifi-inf-wf03.dev.Company.com, OU=OCIO, O=Company to seed policies. Authorizers.xml:<userGroupProvider><identifier>file-user-group-provider</identifier><class>org.apache.nifi.authorization.FileUserGroupProvider</class><property name="Users File">./conf/users.xml</property><property name="Legacy Authorized Users File"></property><property name="Initial User Identity 1">user1</property><property name="Initial User Identity 2">CN=w-nifi-inf-wf02.dev.Company.com, OU=OCIO, O=Company</property><property name="Initial User Identity 3">CN=w-nifi-inf-wf03.dev.Company.com, OU=OCIO, O=Company</property></userGroupProvider><accessPolicyProvider><identifier>file-access-policy-provider</identifier><class>org.apache.nifi.authorization.FileAccessPolicyP

Hello,I am trying to move data from hadoop cluster to aws s3 bukcet using  but even a file of size 4MB takes a lot a time and i am unable to understand the bottle neck here.I am running the below command aws s3 cp <source_folder> s3://<path>/ --recursive The file size is 4MB and the upload speed is 50-60 kib/sec. I talked to aws side as well according to them the issue maybe due to networking is the client end ie the cloundera hadoop cluster Can anyone help understand how can i move my data from hadoop hdfs to aws s3 efficiently ?

I'm getting same error again and again. Not able to resolve it. Can anyone help me in it. Nifi running in Linux server 

HiI've zip file that contains several other zips files.I want to extract the main ZIP and all other zips files (Recursive)Can anyone please advise how to do this using NIFI ?

Hi,   I am trying to add a custom runtime from gitlab private docker registry , and I am getting this error "The provided full version does not start with the short version concatenated with maintenance version." it might related to the path... I am trying to follow https://docs.cloudera.com/machine-learning/cloud/runtimes/topics/ml-distribute-the-runtime-image.html are there rules for the 'user' ?gitlab registry has some '/' , but I make sure that the last fields (sep by / ) are user , image-name:tag any idea? thanks

I am Trying running command in impala shell. And i am get error.I am trying Creating a New kudu table from impala CREATE TABLE my_first_table( id BIGINT, name STRING, PRIMARY KEY(id))PARTITION BY HASH PARTITIONS 16STORED AS KUDU; here log output[root@cloudera-master ~]# impala-shellStarting Impala Shell without Kerberos authenticationOpened TCP connection to master.cloudera-testing.io:21000Connected to master.cloudera-testing.io:21000Server version: impalad version 2.12.0-cdh5.16.2 RELEASE (build e73cce22064ef4972312d895bed2cdb8787a4215)***********************************************************************************Welcome to the Impala shell.(Impala Shell v2.12.0-cdh5.16.2 (e73cce2) built on Mon Jun 3 03:32:01 PDT 2019)Every command must be terminated by a ';'.***********************************************************************************[master.cloudera-testing.io:21000] > CREATE TABLE my_first_table > (

I am trying to trigger spilling to disk on impala (Impala Shell v3.2.0-cdh6.3.4 ) The cloudera documentation suggest the steps to do it here: https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/impala_scalability.html Under the section: "Testing performance implications of spilling to disk:" I am extremely confused by the two seemingly contradictory statements there: "Set the MEM_LIMIT query option to a value that is smaller than the peak memory usage reported in the profile output. Do not specify a memory limit lower than reported in the profile output. "So what should be the value of MEM_LIMIT ?When I do set it to be lower than the peak memory usage, i get the following error : "Rejected query from pool root.centos: minimum memory reservation is greater than memory available to the query for buffer reservations. Memory reservation needed given the current...."When I set the MEM_LIMIT to be higher, then I don't see any "SpilledPartit

Hi,  we have a problem with agent installation on hosts which we want to add to the cluster.On the Install Agents page, the process fails with an error "Failed to copy installation files.  Examining cloudera-scm-server.log we got this (The process fails when trying to copy files). 2022-03-21 10:55:01,075 INFO NodeConfiguratorThread-7-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: Transitioning from MAKE_TEMP_DIR (PT0.197S) to COPY_FILES2022-03-21 10:55:01,081 INFO NodeConfiguratorThread-7-0:com.cloudera.server.cmf.node.NodeConfigurator: Using default key bundle URL2022-03-21 10:55:01,111 INFO NodeConfiguratorThread-7-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: Setting COPY_FILES as failed and done state2022-03-21 10:55:01,111 INFO NodeConfiguratorThread-7-0:net.schmizz.sshj.transport.TransportImpl: Disconnected - BY_APPLICATION2022-03-21 10:55:01,111 INFO NodeConfiguratorThread-7-0:com.cloudera.cmf.model.HostInstallArgs: Deprecated option

i have found my one of CDH has so many errors on every datanode, the error logs as below. who have this kind experience on this issue ? and give me some advises  2021-08-03 05:23:43,389 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2123011416-10.37.54.12-1457006347704:blk_3910061604_2849065475 src: /10.37.54.218:36088 dest: /10.37.54.218:10042021-08-03 05:23:43,700 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.37.54.218:36082, dest: /10.37.54.218:1004, bytes: 358, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-859199005_222, offset: 0, srvID: 44713da0-9f69-44ea-b6c0-8f7420a41f83, blockid: BP-2123011416-10.37.54.12-1457006347704:blk_3910061597_2849065468, duration: 597337782021-08-03 05:23:43,700 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2123011416-10.37.54.12-1457006347704:blk_3910061597_2849065468, type=HAS_DOWNSTREAM_IN_PIPELINE terminating2021-08-03 05:23:43,833 INFO org.apache.hadoop.hdfs.se

I am getting following error while running the tpch queryI saw the following link related to error (https://issues.apache.org/jira/browse/HIVE-11427) but not sure how to use this.I am using: Hive 3.1.3000.7.1.6.0-297 Error: Error while compiling statement: FAILED: SemanticException 0:0 Error creating temporary folder on: hdfs://testmach:8020/apps/hive/warehouse/tpch_flat_orc_2.db. Error encountered near token 'TOK_TMP_FILE' (state=42000,code=40000) 0: jdbc:hive2://testmach:10000> use tpch_flat_orc_120;INFO : Compiling command(queryId=hive_20220317154641_5560d6bf-4866-48b0-a990-9c5ea44f6dbd): use tpch_flat_orc_120INFO : Semantic Analysis Completed (retrial = false)INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)INFO : Completed compiling command(queryId=hive_20220317154641_5560d6bf-4866-48b0-a990-9c5ea44f6dbd); Time taken: 0.017 secondsINFO : Executing command(queryId=hive_20220317154641_5560d6bf-4866-48b0-a990-9c5ea44f6dbd): use tpch_flat_or

Solr process keeps stopping each 3 daysI am getting the below error22/02/27 05:15:21 ERROR handler.RequestHandlerBase: [qtp417301556-3371]: org.apache.solr.common.SolrException: Exception writing document id 6ae68838-4669-4f11-9544-53d22025085c-5 to the index; possible analysis error: number of documents in the index cannot exceed 2147483519Caused by: java.lang.IllegalArgumentException: number of documents in the index cannot exceed 2147483519 I have 1 solar server, Cloudera version 7.4.4, solar version 8.4.1what do you suggest to solve the problem?Thanks 

Hi,No incoming connections are permitted with QueryDatabaseTableRecord. How i can populate the database.name attribute for QueryDatabaseTableRecord in order to be able to use DBCPConnectionPoolLookup as database connection as indicated in the documentation ? 

Hi,   I'm having a problem with ClouderaImpalaODBC64.dll version 2.16.14.1016.   SQLExecDirectW function returns an error when executing the following query:   With variable1 as ( SELECT orders.order_id, orders.order_date FROM quest_stage.orders orders WHERE orders.order_date = (CAST(? AS TIMESTAMP)) ) Select * from variable1   Error details: [Cloudera][ImpalaODBC] (110) Error while executing a query in Impala: [HY000]: ParseException: Syntax error in line 5   It seems to be the problem when you combine WITH and CAST keywords.   Both of the following queries below work:   SELECT orders.order_id, orders.order_date FROM quest_stage.orders orders WHERE orders.order_date = (CAST(? AS TIMESTAMP)) ---------------------------------------------------------------------------------------------- With variable1 as(SELECT orders.order_id, orders.order_dateFROM quest_stage.orders orders)Select * from variable1   Could you help me understand what seem

I have created a data pipeline with Apache Ni-Fi (version 1.15.2) to transfer data from a MySQL database in a AWS RDS instance into Timescale (postgreSQL). Ni-Fi runs in Linux Ubuntu 18.04 in Virtual Box on top of Windows 10. The pipeline has been tested successfully by using as a target the Timescale database located in my local Linux instance, without using SSL. Then I have modified the pipeline to use Timescale Cloud as a target (https://portal.timescale.cloud/login). The database name is: periodic-measurements In Timescale Cloud service page I can see these credentials: Host: <host-string>Port: 10250User: tsdbadminPassword: <timescale-cloud-service-password>Service URI: postgres://tsdbadmin:<timescale-cloud-service-password>@<host-string>:10250/defaultdb?sslmode=requireSSL mode: requireAllowed IP addresses: 0.0.0.0/0I use PutDatabaseRecord processor to write data in the postgreSQL database which uses the DBCPConnectionPool controller service.&

We're trying for HDP upgrdation from 3.1.5.0-152 to 3.1.5.6189-1.As we're having four nodes cluster, in two nodes the upgradation part completed, but in other it's failing. Please provide some solutions. Background operations log:Error getting repository data for HDP-3.1-GPL-repo-51, repository not found2022-03-09 11:09:07,436 - Installing package sqoop_3_1_5_6189_1 ('/usr/bin/yum -y install sqoop_3_1_5_6189_1')2022-03-09 11:09:11,891 - Installing package hadoop_3_1_5_6189_1 ('/usr/bin/yum -y install hadoop_3_1_5_6189_1')2022-03-09 11:09:16,296 - Installing package hadoop_3_1_5_6189_1-client ('/usr/bin/yum -y install hadoop_3_1_5_6189_1-client')2022-03-09 11:09:21,000 - Installing package snappy ('/usr/bin/yum -y install snappy')2022-03-09 11:09:25,451 - Installing package snappy-devel ('/usr/bin/yum -y install snappy-devel')2022-03-09 11:09:30,161 - Installing package hadoop_3_1_5_6189_1-libhdfs ('/usr/bin/yum -y install hadoop_3_1_5_6189_1-libhdfs')2022-03-09 11:09:34,700 - Inst

Hi All, Refer to CDSW install doc, create subdomain is part of setup pre-requisite and the subdomain has to such as  "cdsw.<your_domain>.com" and "*.cdsw.<your_domain>.com". My question is, does the CDSW setup must use "cdsw" as a subdomain name? does it support other names?    Thanks in advance RegardsMatt

Hello Team,We are facing some issue related to Hue KT renewal server not starting up.the KT renewal log saying k.urls' from '/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6626826/lib/hue/desktop/libs/notebook/src/notebook/urls.py'> (notebook:notebook) ^notebook/>[16/Mar/2022 01:30:32 +0100] urls DEBUG Dynamic pattern: <RegexURLResolver <module 'dashboard.urls' from '/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6626826/lib/hue/desktop/libs/dashboard/src/dashboard/urls.py'> (dashboard:dashboard) ^dashboard/>[16/Mar/2022 01:30:32 +0100] urls DEBUG Dynamic pattern: <RegexURLResolver <module 'kafka.urls' from '/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6626826/lib/hue/desktop/libs/kafka/src/kafka/urls.py'> (kafka:kafka) ^kafka/>[16/Mar/2022 01:30:32 +0100] urls DEBUG Static pattern: <RegexURLPattern None ^static\/(?P<path>.*)$>[16/Mar/2022 01:30:32 +0100] kt_renewer INFO Keytab renewer not starting, no keytab configured  I have do

Hello,I have a problem with converting json to csv. I use ConvertRecord processor with attached JsonTreeReader and CSVRecordSetWriter.The schema for both might look like this. Attributes in the json are optional.     {"type": "record","name": "some_event","fields": [{ "name": "date_time", "type": ["string","null"] },{ "name": "id", "type": ["string","null"] },{ "name": "is_success", "type": ["string","null"] },{ "name": "username", "type": ["string","null"] },{ "name": "username_type", "type": ["string","null"] }]}   for the input json like   {"date_time": "2017-01-29T13:42:03.965Z","id": "20049015990001584400001""is_success": "TRUE",}   where username and username_type is not set, the output does not correspond to my expectation I get2022-01-29T13:42:03.965Z|20049015990001584400001|trueinstead of2022-01-29T13:42:03.965Z|20049015990001584400001|true||which should be the right format. Note the separator for null values at the end of the second record.   Do

Hi Experts, One of our oozie workflow got failed while sqoop import job. Please find below logs: org.apache.velocity.exception.VelocityException: Error initializing log: Failed to initialize an instance of org.apache.velocity.runtime.log.Log4JLogChute with the current runtime configuration. in other logs as : org.springframework.web.client.HttpServerErrorException: 500 null Please tell what could be the possible cause of this failure, we retried this job but getting the same error.

Hi Experts, One of the Impala daemon suddenly went down and we are unable to bring it up. Some queries were executed on the impala daemon. I have captured the logs when it got shutdown. Could not open log file: /var/log/impalad/lineage/impala_lineage_log_1.0-1647629222243@ 0x95b479 impala::Status::Status()@ 0xd00b95 impala::SimpleLogger::FlushInternal()@ 0xd01fcf impala::SimpleLogger::Init()@ 0xbb9c2a impala::ImpalaServer::InitLineageLogging()@ 0xbcb25e impala::ImpalaServer::ImpalaServer()@ 0xbb676d ImpaladMain()@ 0x8e2593 main@ 0x7f5cf7658495 __libc_start_main@ 0x9299e1 (unknown) ERROR cc:312 Aborting Impala Server startup due to failure initializing lineage logging. Impalad exiting. Can some one please tell what steps should be taken, Just to note Enable Audit Collection and Enable Impala Audit Event Generation is enabled/checked. 

Version 1.15.3Windows Server 2019 Standard After GUI login receive the following error: Invalid StateThe Flow Controller is initializing the Data Flow.

Version 1.15.3 Windows Server 2019 Standard   # zookeeper properties, used for cluster management #nifi.zookeeper.connect.string=nifi-inf-02:2181,nifi-inf-03:2181nifi.zookeeper.connect.timeout=10 secsnifi.zookeeper.session.timeout=10 secsnifi.zookeeper.root.node=/nifinifi.zookeeper.client.secure=falsenifi.zookeeper.security.keystore=nifi.zookeeper.security.keystoreType=nifi.zookeeper.security.keystorePasswd=nifi.zookeeper.security.truststore=nifi.zookeeper.security.truststoreType=nifi.zookeeper.security.truststorePasswd=nifi.zookeeper.jute.maxbuffer=   After configuration of cluster the service spits the following error on startup.   Please check the configuration of the Cluster State Provider with ID [zk-provider] in the file D:\nifi-1.15.3\.\conf\state-management.xmlat org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:176)at org.springframework.beans.factory.support.FactoryBeanRegistrySupport

Hi we have noticed through Ambari that one of the datanode has the status "DEAD"It seems that the process to replicate the block from the datanode to the rest of the datanode has started. In the main time what should be the proper procedure to restart the dead datanode? Thank you

Hello, I have a question regarding hive bucketed tables (bucketed only no partitions) optimization. Now, I know that if we have table A and table B and we want to join them using A.COL1 and B.COL1 ( A.COL1 = B.COL1), we should bucket both table A and B on col1 into same no of buckets or its multiple. But what if we have more than 2 tables ? for example, I have table A and I want to join it on table B and table C. table A is joined with table B using COL1 (A.COL1 = B.COL1) and table A is joined with table C using COL2 (A.COL2 = C.COL2 ) what columns should I cluster by for table A ??? is it bucket by COL1 and COL2 (clustered by col1,col2)?  in summary, how can I optimize one table if it is joined with more than one table using buckets only. Thanks in advance.   

Is it possible to connect to on-prem kafka cluster using Kerberos authentication from cloud deployed service?When we are trying to connect we are getting below error:Caused by: KrbException: Generic error (description in e-text) (60) - Unable to locate KDC for realm "ABC.COM"This is my jaas config:com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/pathtokeytab" principal="principal_name@ABC.COM" ; Please help me if anyone has faced such error.  Full stack trace: Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Cannot locate KDC        at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:162)        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146)        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67)