Q&A

Hi, Below is the logs   Host checks started.Check execute list: db_connection_checkDB connection check started.WARNING: File /var/lib/ambari-agent/cache/DBConnectionVerification.jar already exists, assuming it was downloaded beforeWARNING: File /var/lib/ambari-agent/cache/mysql-connector-java.jar already exists, assuming it was downloaded beforecall['/usr/jdk64/jdk1.8.0_112/bin/java -cp /var/lib/ambari-agent/cache/DBConnectionVerification.jar:/var/lib/ambari-agent/cache/mysql-connector-java.jar -Djava.library.path=/var/lib/ambari-agent/cache org.apache.ambari.server.DBConnectionVerification "jdbc:mysql://host.com/metastore" "hive" [PROTECTED] com.mysql.jdbc.Driver'] {}call returned (1, 'ERROR: Unable to connect to the DB. Please check DB connection properties.\ncom.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.')DB co

Hello,I would like to implement this logic on nifi:I have a json array:[{"att1": "1","att2": "aaa"},{"att1": "2","att2": "bbb"},{"att1": "3","att2": "aaa"},] I would like that, according to the attribute, the json was processed to the processA or to the processB. For example, if att2 is equal "aaa", it was processed to the processA, else it was processed to the processB.I tried to follow the suggestion of the following solution https://community.cloudera.com/t5/Support-Questions/Nifi-processor-route-json-array-into-multiple-json-arrays/td-p/128722 but it didn't work. Can you give me an alternative? Thank you.

Hi, We have  a requirement to retrieve files from SFTP server and place into to different prefixes in a S3 bucket.   Mapping is something like this:   ftp path1 -> file pattern match1 -> s3 prefix1 ftp path1 -> file pattern match2 -> s3 prefix2 ftp path2 -> file pattern match3 -> s3 prefix3 ftp path3 -> match4 -> s3 prefix4   ListSFTP -> FetchSFTP -> RouteOnAttribute  RouteOnAttribute - Match1 (${filename:indexOf('pattern1'):gt(-1)}) - putS3Object1 RouteOnAttribute - Match2 (${filename:indexOf('pattern2'):gt(-1)}) - putS3Object2   We are using the ListSFTP, FetchSFTP and RouteOnAttribute to route it to the a corresponding putS3Object processor with different variables for different patterns.   We initially had only 2 patterns which grew into 10 - 15 patterns (Match1, Match2, .. Match N) to be routed to the corresponding paths. So if we follow the same approach we have to use 10 -15 putS3Object. Is there a way to avoid thi

Hi   I'm new to Nifi and I wanted to orchestrate a data pipeline with micro-services that I have using Nifi. I have two micro-services and I'm using InvokeHTTP and ListenHTTP for them. I have no issue with the invokeHTTP but I'm getting error with the ListenHTTP processor which says "Failed to properly initialize the processor". Can someone help me? Can I use Nifi to orchestrate my data pipeline that has multiple micro-services. Which processor is used to listen to the microservices and how Do i connect them

Hi,I am unable to start node manager on a node and get the attached error, any help is resolving this is much appreciated. Error starting NodeManagerorg.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 1 missing files; e.g.: /var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/000042.sst at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:181) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:245) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:562) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:609)Caused by: org.fus

Hello, Good afternoon.I am new in cluster. I am installing CDH 5.16.2-1.cdh5.16.2.p0.8. I am stuck in hereI am check scm-server log using "tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log"I get this logID '798f1ee7-be79-475c-bc30-28573451693e' that is not recognized2019-10-03 11:48:43,218 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: (27 skipped) Found files KUDU-1.4.0-1.cdh5.12.2.p0.8-el7.parcel,CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel,CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel,CDH-6.2.0-1.cdh6.2.0.p0.967373-bionic.parcel under /opt/cloudera/parcel-repo2019-10-03 11:48:52,417 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Discovered parcel on CM server: CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel2019-10-03 11:48:52,417 WARN ParcelUpdateService:com.cloudera.cmf.persist.ReadWriteDatabaseTaskCallable: Error while executing CmfEntityManager taskjava.lang.IllegalArgumentException: Unknown Distro suffix: bionicat co

I am trying to enable Kerberos on Ambari 2.6.2.2 on CentOS 7.  Below are the errors*********************** 500 status code received on POST method for API: /api/v1/clusters/hdp265/requestsError message: An internal system exception occurred: Failed to execute the command: Broken pipe *********************************** Below is my krb5.conf file nano /etc/krb5.conf # Configuration snippets may be placed in this directory as wellincludedir /etc/krb5.conf.d/ [logging]default = FILE:/var/log/krb5libs.logkdc = FILE:/var/log/krb5kdc.logadmin_server = FILE:/var/log/kadmind.log [libdefaults]dns_lookup_realm = falseticket_lifetime = 24hrenew_lifetime = 7dforwardable = truerdns = falsepkinit_anchors = /etc/pki/tls/certs/ca-bundle.crtdefault_realm = HADOOPSECURITY.COMdefault_ccache_name = KEYRING:persistent:%{uid} [realms]# EXAMPLE.COM = {# kdc = kerberos.example.com# admin_server = kerberos.example.com# }HADOOPSECURITY.COM = {kdc = p1.bigdata.comadmin_server = p1.bigdata.com} [domain_

Hi Team, I want to rename/modify the filename of the flowfile with the timestamp added to it. Let's say the original flowfile name is file1.txt, I want to change the name of file to be file1_<timestamp>.txt. I used updateAttribute processor to do it. The expression which I gave is   ${filename:substringBeforeLast(‘.’)}_${now():format("yyyy-MM-dd HH:mm:ss.SSS’Z’”)}.${filename:substringAfter('.')}   This Giving me error. I'm bad at Expression language. could anyone help me in this?Thanks. 

Hi Team, I have a zipped file which contains so many text files. I used unpackContent processor to unzip the files. I want to store those unzipped files in another location. But the file names should be replaced with file1.txt, file2.txt, file3.txt and so on. How can I give file names dynamically? I thought updateAttribute can be used in this. But I'm not sure at the expression language to get this done. Can anyone help me to write a expression language in updateAttribute processor or any alternative processor/solution for this.   Thanks.

Hello,I am using HDP3.0, Ambari 2.7.1. My MapReduce2 is not running, it reports the following error:   Traceback (most recent call last):File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/historyserver.py", line 126, in <module>HistoryServer().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 351, in executemethod(env)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/historyserver.py", line 94, in startskip=params.sysprep_skip_copy_tarballs_hdfs)File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/copy_tarball.py", line 516, in copy_to_hdfsreplace_existing_files=replace_existing_files,File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__self.env.run()File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in runself.run_action(resource, action)File "/usr/lib/ambari-agent/lib/resource_management

I am using cloudera VB. I am trying to login to beeline. I tried the username and passwod (empty) but it's not working :-!connect jdbc:hive2://localhost:10000/Connecting to jdbc:hive2://localhost:10000/Enter username for jdbc:hive2://localhost:10000/:Enter password for jdbc:hive2://localhost:10000/:Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000/: java.net.ConnectException: Connection refused (state=08S01,code=0)0: jdbc:hive2://localhost:10000/ (closed)>@  [cloudera@quickstart ~]$ beeline2019-10-09 17:06:46,217 WARN [main] mapreduce.TableMapReduceUtil: The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it.Beeline version 1.1.0-cdh5.7.0 by Apache Hivebeeline> !connect jdbc:hive2://localhost:10000/scan complete in 5msConnecting to jdbc:hive2://localhost:10000/Enter username for jdbc:hive2://localhost:10000/: hadoopEnter password for jdbc:hive2://localhost:10000/:Error: Could not open client tra

I am using cloudera virtual box. while creating partitions, it is creating all the partitions whether they are unique or not create table product_order1(id int,user_id int,amount int,product string, city string, txn_date string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';LOAD DATA LOCAL INPATH 'txn' INTO TABLE product_order1;Loading data to table oct19.product_order1Table oct19.product_order1 stats: [numFiles=1, totalSize=303]OK Time taken: 0.426 secondshive> > set hive.exec.dynamic.partition = true;hive> > set hive.exec.dynamic.partition.mode = true;hive> > create table dyn_part(id int,user_id int,amount int,product string,city string) PARTITIONED BY(txn_date string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';OK Time taken: 0.14 secondshive >INSERT OVERWRITE TABLE dyn_part PARTITION(txn_date) select id,user_id,amount,product,city,txn_date from product_order1;Result which i have received :-Loading data to table oct19.dyn_part partition (txn_date=null)

Aloha!Before, I was able to connect to the web UI via host IP and port 7180. After setting up a Static IP for the hosts in my Cluster, I seem to have broken Cloudera Manager. Should I uninstall and reinstall, or is there a simple solution that I can implement to get things working again?

Hi,   I try to replace text with space using ReplaceTextWithMapping processor. I have file with multiple lines   "action":"account sing up""action":"account sign in""action":"account sign out"   And I have a mapping file with \t separating the search value and replace value     account sing up asuaccount sign in asiaccount sign out aso     Regular expressions "action":"(.*)" Matching group: 1   Result   "action":"account sing up""action":"account sign in""action":"account sign out"   Expected:   "action":"asu""action":"asi""action":"aso"     Does anyone know what I did wrong?   Thanks David

HI Guys,   I upgrade to the latest CDSW 1.6 over the weekend. The upgrade was successful, however i am having trouble lunching a CLI terminal once i have created a project. Every time i do, i get the below error.      This page isn’t working tty-8c51gbduvszddduehl.domain.co.za is currently unable to handle this request. HTTP ERROR 500   My cluster is fully TLS'ed and only having this problem since the upgrade. 

Hello. I've been following this tutorial (https://www.batchiq.com/nifi-configuring-ssl-auth.html) to the best of my ability, yet the web UI still does not want to start. I get the error message ERR_CONNECTION_REFUSED" when trying to access the nifi server through Google Chrome. It has also been this way when trying to access through Firefox, so I suppose this is more of a server error than anything else. The following is a relevant section of my conf file...   # web properties #nifi.web.war.directory=./lib#nifi.web.http.host=#nifi.web.http.port=8080nifi.web.http.network.interface.default=nifi.web.https.host=nifi.web.https.port=8443nifi.web.https.network.interface.default=nifi.web.jetty.working.directory=./work/jettynifi.web.jetty.threads=200nifi.web.max.header.size=16 KBnifi.web.proxy.context.path=nifi.web.proxy.host=     # security properties #nifi.sensitive.props.key=SyoerSecretFlyingPasswordnifi.sensitive.props.key.protected=SuperSecretFlyingPasswordnifi.sensitive.pro

Why isn't the root user showing up on ranger? I intend to give permission to the root user on Hbase but I'm getting the following error:     hbase(main):001:0> grant 'root','RWXCA'ERROR: org.apache.hadoop.hbase.coprocessor.CoprocessorException: HTTP 400 Error: root is Not Found at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.grant(RangerAuthorizationCoprocessor.java:1261) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.grant(RangerAuthorizationCoprocessor.java:1072) at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService$1.grant(AccessControlProtos.java:10023) at org.apache.hadoop.hbase.protobuf.generated.AccessControlProtos$AccessControlService.callMethod(AccessControlProtos.java:10187) at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8065) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.ja

I'm just curious and also unsure... When I'm creating a cluster (in this case, 3 kafka hosts and zookeeper), should I be adding the SCM host to this cluster as well, or does that operate separately?I've read that each cluster should have a Master host, Utility host (with cloudera manager), and 3 worker hosts. Is it a bad idea to add the host that is running SCM to my new cluster, or should I leave it separate? I've noticed that when I add the SCM host to a cluster, it is using a LOT of memory by default, which leads me to think that I should not add the SCM host to the new cluster.

Hi there. I find CDH does not support Ubuntu 18.04 yet, and plan to support ? I hope Ubuntu 18.04 can be supported, and Canonical extends Ubuntu 18.04 support to 10 years, really LTS. So I hope CDH could support Ubuntu 18.04. Thanks

Hello All,We are facing issue with one of our clients cluster, we see some files over 2GB, and would like to know the purpose of these files, and if these could be deleted to make way for some space.1. Eventserver[root@TestBed ~]# du -sh /var/lib/cloudera-scm-eventserver/v3/* |grep G5.4G /var/lib/cloudera-scm-eventserver/v3/_1bx6l.fdt1.2G /var/lib/cloudera-scm-eventserver/v3/_1hgcy.fdt1.4G /var/lib/cloudera-scm-eventserver/v3/_1mz17.fdt 2. Cloudera-Scm-Headlamp[root@TestBed ~]# du -sh /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/* |grep G2.3G /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/_2t.fdt2.2G /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/_5m.fdt2.1G /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/_8f.fdt Also is there a way to limit TS data in SMON & HMON, I understand that 10GB is the minimum requirement for these roles but what could be done if a clients has limited resources. Any help / guidance is appreciated Thanks

I have JSON input of the following format:{  "Id": 1000000,  "ReportName": TestReport,  "Results": [{    "Id": 1,    "ResultId": "1000000-0",    "Query": {      "Id": 001,      "Name": "TestQuery0",    }  }, {    "Id": 2,    "ResultId": "1000000-1",    "Query": {      "Id": 002,      "Name": "TestQuery1",    }  }]}These file can become quite large depending on the number of Results in the Report and I was hoping to convert the single Flowfile to multiple records for processing. However, due to the format of the JSON a SplitRecord will result in one record per split. There is one report per FlowFile and therefore only 1 root level element.I am looking for a method or strategy to split the Flowfile into smaller Records while still maintaining the cohesiveness of the report in the end when it put in HDFS. Current Strategy:Use

Hey, folks, I have a Kerberized server, and for the police I'm using Ranger, via console I can publish and consume the messages of Kafka, But now I must do through Nifi and that I'm very lost someone could help me. Thank you

Hi Cloudera Community!   I`m trying to start going through the tutorial but cannot overcome the following problem:   [cloudera@quickstart java]$ sqoop import-all-tables -m 1 --connect jdbc:mysql://quickstart:3306/retail_db --username=retail_dba --password=cloudera --compression-codec=snappy --as-parquetfile --warehouse-dir=/user/hive/warehouse --hive-import19/10/04 01:46:39 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.16.219/10/04 01:46:39 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.19/10/04 01:46:39 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override19/10/04 01:46:39 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.19/10/04 01:46:39 WARN tool.BaseSqoopTool: It seems that you're doing hive import directly into default19/10/04 01:46:39 WARN tool.BaseSqoopTool: hive warehouse directory which is not supported. Sqoop is19/10/04 01:46:39 WARN tool.BaseSqoopTool

Hi Cloudera Team, I am getting issue while sending message to Kafka topic from outside of sandbox, but same application in sandbox is able to send message to Kafka topic.Below is the screen print of these casesIn First case I am trying to send test message to my Kafka topic; that application runs without giving any error. But at cluster location topic was not showing any messages In Second case when I ran the same application in sandbox messages were send and displayed in Kafka topic.  Could you please assist me what wrong I am doing (or what extra I have to do) to make my First case successful.