Q&A

We have a requirement wherein we get different type of input files i.e avro, json and csv (example: client1_product.avro,client2_product.json,client3_product.csv, etc) and we have to convert these files to csv and load to db2.Currently I have designed a job with GetFile to fetch the avro files and ConvertRecordProcessor to convert records (avro to csv) and load it to db2 and this works fine but we need to design a generic job to read any file format (i.e avro, json and csv) and load it to db2 table.Can we design Nifi job to read different format files and convert it to a csv format in same job?Could you please advise. Thanks,Avi

Hello Experts,   I can no longer find the CDP Data Center upgrade docs on the website. I only see CDP Private Cloud. For example OS reqs link https://docs.cloudera.com/cdp/latest/release-guide/topics/cdpdc-os-requirements.html the url has cdpdc-os in it but the content is private cloud. I thought I saw Data Center few days back but cannot find now. Any ideas?   Thanks!

Hello all,I have HDF 3.4 cluster A with nifi and nifi registry integrated, cluster B with NIFI. both are tls/ssl secured. now I'm trying to use the cluster A NIFI registry for NIFI running on cluster B. noticing below error when trying to version a flow from cluster B NIFI  integrated with cluster A registry  I have added cluster B nifi node cert to registry users list but still same errorCN=its-nifi-node-dev-nifipoc1-01, OU=NIFI @alim @MattWho @sunile_manjee please advice 

I'm trying to overwrite a parquet file in hdfs for a Hive table, but I noticed a bunch of individual parquet files with unique names like this, table_directory /part-00199-edf111ef-0f7a-43ab-9393-8fc620195d4c-c000.snappy.parquet How do I add a parquet file to a Hive table directory that's composed of several parquet file?  

Hi,Due to space constraints, I would like to reduce the following settings in CM:1. Maximum Number of Events in the Event Server Store-5000000 To 40000002. Event Server Max Log Size - 200MB To 100MB Would like to know if this is can be done or would hamper CM services. Appreciate any assistance. ThanksWert 

ZooKeeper server creates snapshot and log files, but never deletes them. So we need to care about the retention policy. How to decide on the right amount of remaining Zookeeper snapshot files? Need to say that ZooKeeper server itself only needs the latest complete fuzzy snapshot and the log files from the start of that snapshot. But since ZooKeeper creates a backup of snapshot file, how many ZooKeeper snapshot backups do we need to retain? Sometimes snapshots can be corrupted, so the backup of snapshot files should take this into consideration. In our ZooKeeper server we saw that snapshot backup is created each day. Example of snapshot file from my ZooKeeper server:  -rw-r--r-- 1 ZooKeeper hadoop 458138861 Aug 10 07:12 snapshot.19000329d1-rw-r--r-- 1 ZooKeeper hadoop 458138266 Aug 10 07:13 snapshot.19000329de-rw-r--r-- 1 ZooKeeper hadoop 458143590 Aug 10 09:24 snapshot.1900032d7a-rw-r--r-- 1 ZooKeeper hadoop 458142983 Aug 10 09:25 snapshot.1900032d84-rw-r--r-- 1 Zoo

Hi, I am trying to import a hive table that has an MYSQL DB as metastore into Atlas by using import-hive.sh command but even when I explicitly pass the database name it fails to detect the same. @Shelton,Can you please suggest on this?  The above snippet shows AMBDB existing in the MySQL DB.

I'm trying to set up a Load Balancing Demo.   I have a GetFile processor that watcches a directory for incoming files and then a flow that processes them.   To try to understand LoadBalancing, I have created a second NiFi instance on the same host (my dev machine), but on a different port 8090 to the first 8080. I have then checked out the flow from the Registry so that it is identical on both instances.   Next, I have set the Load Balancing Strategy on the output queue from both nodes first processsor, (the GetFile proc) in order to see if I can get data going through both instances. I can then drop files into the watch dir and see them hit my flow. However, all the files are going into the same node, even if I wait 10 minutes between dropping them into the dir. But, when I run again it switches to the other node and all the files go through there. Is there a timeout somewhere that I can set to speed up the switch from one node to the other? Or better to switch files to

I want to setup cluster of 3 nodes. Suppose below are the 3 nodes, Node 1 = 192.168.0.10 Node 2 = 192.168.0.20 Node 3 = 192.168.0.30 I am able to setup the cluster by using below zookeeper string in the respective nodes, Node 1 = 192.168.0.10:2181 Node 2 = 192.168.0.10:2181,192.168.0.20:2181 Node 3 = 192.168.0.10:2181,192.168.0.20:2181,192.168.0.30:2181 If Node 1 is up and running and I am turning of any other node than Node 1, then cluster is working without any issue.   But problem with above configuration is, if I am turning off the Node 1 (192.168.0.10), then cluster is failing with the connection loss status. As Node 2 and Node 3 waiting for Node 1 which is down which turns into cluster failure.   So I am trying to setup same zookeeper string for all the node at once and trying to start server one by one. Node 1 = 192.168.0.10:2181,192.168.0.20:2181,192.168.0.30:2181 Node 2 = 192.168.0.10:2181,192.168.0.20:2181,192.168.0.30:2181 Node 3 = 192.168.0.10:2181,192.168.0.

Hi Team ,I need help in converting the given csv to json format .This is the input csv.EMP,EVSEID,SessionID,Starttime,Endtime,Totalchargeamount,Totalduration,Totalcosts3023513,AT*HTB*E1002760,b8ed2c41-4a0a-4528-9f14-4c4f040de81e,15:06:56 26.10.2019,15:42:59 26.10.2019,5890,2163,2.5922899202,AT*HTB*E1001927,ee3a1d64-2a69-4d43-87a4-4e23cb992660,16:24:11 08.10.2019,19:00:24 08.10.2019,6430,9373,3.782974330,DE*BOX*EAC0016,25543dea-6d58-4c27-a924-dacc01a83162,12:33:09 19.10.2019,15:29:58 19.10.2019,13700,10609,6.028.I am able to convert the given csv to json output .Need to add additionalheader and footer tag, Footer tag will be holding the aggregated value of the totalcosts given in the csv file.I am stuck in the process of performing aggregation based on the totalcosts. Attached the generated json and expected json format Below table is having generated json.[ { "record" : { "EMP" : "3023513", "EVSEID" : "AT*HTB*E1002760", "SessionID" : "b8ed2c41-4a0a-4528-9f14-4c4f040

Hello all, I want to see the size of each table in Hive residing in multiple databases. There are around 3000 tables, so it is difficult to do it one by one for each table. How can I do it at one go? Regards,Manu

My processor is querying a DB2 table and loads data to S3 and then ingest it into Snowflake. I have unique account numbers in my table and I need to add a column with unique identifier for each unique account number.

I am inserting records in maria db table from a file using python. Population column in the file is empty. I want it to go as empty value in table as well. Population column in table is set as integer and can accept null value. I am trying the below code -Table Definition -CREATE TABLE `local_db`.`table_x` (`Unique_code` varchar(50) NOT NULL,`city` varchar(200) DEFAULT NULL,`state` varchar(50) DEFAULT NULL,`population` bigint(20) DEFAULT NULL,`Govt` varchar(50) DEFAULT NULL) ENGINE=InnoDB DEFAULT CHARSET=utf8;```input_file = "input_file"csv_data = csv.reader(open(input_file))try:connection = mysql.connector.connect(host='localhost',database='local_db',user='root',password='root',port = '3306')cursor = connection.cursor()for row in csv_data:cursor.execute("""INSERT INTO table_x(Unique_code,city,state,population,Govt)VALUES("%s", "%s", "%s","%s", "%s")ON DUPLICATE KEY UPDATE city = VALUES(city),state = VALUES(state), \population = VALUES(population),Govt = VALUES(Govt)""")connection.comm

hello, i want to know if the processor of nifi listSFTP does remove files from path directory after listing them?

We are planning to install KMS and KTS but do we need Cloudera Navigator to Install KMS and KTS? if not required, then how to install it without Cloudera Navigator?

I can't open /run/cloudera-scm-agent/process/256-yarn-NODEMANAGER/container-executor.cfg: Permission denied.+ perl -pi -e 's#{{CGROUP_GROUP_CPU}}##g' /run/cloudera-scm-agent/process/256-yarn-NODEMANAGER/yarn-site.xmlI am getting this error after enabling Kerberos in CDH cluster, HDFS  and yarn are not able to start. After checking the yarn Node Manager logs I see the below error. yarn nodemanager logs:: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.security.authorize.AuthorizationException: : User: cloudera@CLUSTERIE.LOCAL is not allowed to impersonate yarn/ip-10-0-xxxxx@xyz.comAny suggestion why am I getting this error, when I disable the Kerberos everything works well. Please assist as the severity of this is very high.

Hi,  I am facing the below error when running the import-hive.sh to import hive metadata to Apache Atlas.  2020-08-08T07:41:25,369 ERROR [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Import failedcom.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed outat com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155) ~[jersey-client-1.19.jar:1.19]at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:105) ~[jersey-client-1.19.jar:1.19]at com.sun.jersey.api.client.Client.handle(Client.java:652) ~[jersey-client-1.19.jar:1.19]at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) ~[jersey-client-1.19.jar:1.19]at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.19.jar:1.19]at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:634) ~[jersey-client-1.19.jar:1.19]at org.apac

Installed Ambari 2.7 HDP 3.1 version, after added Ranger service I couldn't able to login. From Ranger UI "Unable to connect db" and in ambari UI it shows "User:admin credentials on Ambari UI are not in sync with Ranger" I checked Ranger access logs, Checked DB connection, grant permissions everythings is fine. Also from Ranger Config, passwords for "Ranger Admin username for Ambari" and "Ranger Admin user's password" are same and I thought this was a bug. Can anyone help me on this? Thanks in advance.

@Shelton@deepesh1, I need your help in figuring out why this Hive Metastore and Hive Server 2 are not working. Please find the ambari-error.log below: Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 995, in restart self.status(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_metastore.py", line 87, in status check_process_status(status_params.hive_metastore_pid) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/check_process_status.py", line 43, in check_process_status raise ComponentIsNotRunning()ComponentIsNotRunningThe above exception was the cause of the following exception:Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_metastore.py", line 201, in <module> HiveMetastore().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/scr

Hi everyone.   Before anything I'm going to describe the environment where I'm trying to install the Cluster on Amabri:   Static hostname: sandbox-hdp.hortonworks.comIcon name: computer-vmChassis: vmMachine ID: d8323e61d06a489f97b6175ebe8ff9c7Boot ID: a0b2c8c30d5345f796a51dc754e807e9Virtualization: vmwareOperating System: CentOS Linux 7 (Core)CPE OS Name: cpe:/o:centos:centos:7Kernel: Linux 3.10.0-1127.13.1.el7.x86_64Architecture: x86-64   Amabri Agent Version: ambari-agent.x86_64 2.7.4.0-118 @Ambari-2.7.4.0   Ambari Server Version: ambari-server.x86_64 2.7.4.0-118 @Ambari-2.7.4.0   Ambari Agent ini file: [server]hostname=sandbox-hdp.hortonworks.comurl_port=8440secured_url_port=8441connect_retry_delay=10max_reconnect_retry_delay=30   [agent] .... public_hostname_script=/var/lib/ambari-agent/hostname.shhostname_script=/var/lib/ambari-agent/hostname.sh   [security] ....force_https_protocol=PROTOCOL_TLSv1_2    /var/lib/ambari-agent/hostname.sh

Hi,Im new to Nifi and Groovy. I have been trying to run a groovy script using the ExecuteScript Processor but not matter what my groovy code is I keep getting the exception "Java.Lang.NoClassDefFoundError: Couldnot Initialize class groovy.lang.groovySystem" not sure why? I have installed the groovy sdk and set the environment variable PATH accordingly. Not sure what else I have to do? Thanks for your help.

Failed to Start Infra Solr Instance in Ambari,  Please find the logs, -------------------------------------- Traceback (most recent call last):File "/var/lib/ambari-agent/cache/common-services/AMBARI_INFRA_SOLR/0.1.0/package/scripts/infra_solr.py", line 171, in <module>InfraSolr().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in executemethod(env)File "/var/lib/ambari-agent/cache/common-services/AMBARI_INFRA_SOLR/0.1.0/package/scripts/infra_solr.py", line 58, in startsetup_solr_znode_env()File "/var/lib/ambari-agent/cache/common-services/AMBARI_INFRA_SOLR/0.1.0/package/scripts/setup_infra_solr.py", line 142, in setup_solr_znode_envjava_opts=java_optsFile "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/solr_cloud_util.py", line 162, in setup_kerberos_pluginExecute(setup_kerberos_plugin_cmd)File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__self.env.run()File "/usr/lib

Hi All, We're trying to install new cluster with Cloudera Manager 6.2.0 ( CDH 6.2.1 ), while adding Hue service, I am getting an error as " Unexpected error. Unable to verify database connection." in Cloudera Manager. I checked cloudera manager server logs in /var/log/cloudera-scm-server/cloudera-scm-server.log Here is the log I found at the time Hue test connection failure  2020-08-05 11:49:55,499 INFO scm-web-436:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult2020-08-05 11:49:55,506 INFO scm-web-436:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:2002020-08-05 11:49:57,541 INFO scm-web-370:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult2020-08-05 11:49:57,548 INFO scm-web-370:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operatio

Consumer IMAP - Reset State?Can anyone tell me how to control state on Consume IMAP (v1.11.4)I have successfully implemented a flow where I receive emails using the ConsumeIMAP ExtractEmailHeaders and ExtractEmailAttachements, all is working as expected.I have set the properties:Delete Messages = falseMark Message as Read = false so my emails remain in the INBOX.I would like a way to "reset" -  so that the processor would receive all the emails again, however I can't seem to find where state is controlled.Stopping and starting the processor does not reset;Stopping/Starting Nifi does not reset. Does this processor hold some other state or does it query my provenance to workout where it is up to? Would I need to delete the provenance to reset it? I was expecting some kinds of state similar to ListSFTP? Any help appreciated John

Hi, I need to add new hosts to an existing cluster using Ambari but the new hosts have more disks than the old nodes that I want to add to the HDFS (old nodes have /data/disk1& /data/disk2 while new nodes have/data/disk1, /data/disk2, /data/disk3 & /data/disk4). How can I add those disks after adding the nodes? can I just update  dfs.datanode.data.dir?