Solved

NIFI flow.xml.gz is getting cleared out during NIFI restartHDF 3.4NIFI 1.9 no configs have been changes but when restarted the existing flwo.xml.gz of size 70MB is getting overwritten. though restored from a backup and placed back in, again after restarting it's getting cleared.   @MattWho  please advice  

Hi, I created a external table with 'A' as string but stores a decimal value. Loaded data into the table then during report creation I realized my mistake and altered the column 'A' to double. Now I want to access that data but get the below error. I tried  set hive.msck.path.validation=ignore;msck repair table tablename;   But still get the below error.  Bad status for request TFetchResultsReq(fetchType=0, operationHandle=TOperationHandle(hasResultSet=True, modifiedRowCount=None, operationType=0, operationId=THandleIdentifier(secret='\nv\xe01\x14\x7fD\xd2\xb0\xadk\xd1\xbe\x9a\xd2\xd6', guid='\x19\xc2\x13b\x82\x96O\xd1\x93\x04\xf7\xdd\xc5\xcfz\xeb')), orientation=4, maxRows=100): TFetchResultsResp(status=TStatus(errorCode=0, errorMessage='java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.DoubleWritable', sqlState=None, infoMessages=['*or

I have a custom variable in Updatedate processor( state = success), now i want to access those variable in my groovy script which is in invokescriptor processor.def status= flowFile.getAttribute('status')i tried below mentioned both syntax as wellAttemp1:if( status.equals("success")){//}Attemp 2:if(status == "success"){}the above 2 attempts flowFile.getAttribute('status') returnning null value.Please correct my mistake!

We have a user interface based on Java. This application connects Hive as data source. Please find the product version of Apache Hive below. Application is working fine when single user has login with the application to access data from Hive. It gives error while concurrent users login to same application and access the Hive data.. apache-hive-2.3.7db-derby-10.14.2.0hadoop-2.10.0 Further investigation –The authentication mechanism of Hive is NOSASL. We did a small mutli threaded java application for debugging. Please find Java code and log for your analysis.Thanks in advance for your help!! import java.sql.Connection;import java.sql.DriverManager;import java.sql.PreparedStatement;import java.sql.ResultSet;import java.sql.SQLException;public class MapReduceThreadClass implements Runnable {String name;Thread t;String query;MapReduceThreadClass (String thread,String query){name = thread;this.query=query;t = new Thread(this, name);System.out.println("New thread: " + t);t.sta

I can't find location of boostrap.conf of nifi to inrease heap memort, and in the UI it listed as below  

To Schedule the exam, I have to create an account at www.examslocal.com. The step by step guide said that I have to use the same email I used to register on university.cloudera.com. I did it.After that, I have discovered that I'm unable to login because my PSI account is managed by AWS.The PSI support asked me to create a new PSI account, with a different email. Since the email used on niversity.cloudera.com and PSI must be the same, How can I migrate my purchased exam to a newly created Cloudera account?

I have a incoming csv file which I need to route to different nifi flows based on the values in one of the column. Each csv has a column called "Country" which has same country name in all of its rows. What I want is File 1 - 50 rows (All rows will only one value in country column which will be any of the below)England, england, Britain, UK, United kingdom, USA, United States So the file which contain values like England, england, Britain, UK, United kingdom in country column should be routed to one flow and if file contain USA or United States then it should be routed to another flow.Note - One file will only contain one value in all rows. So routing based on first row value of country column for entire file is fine. How to achieve this?

Hi,I am trying to install a node manager in a host. However, I got timeout status via ambari web ui upon the installation of the node manager. All other components successfully installed. Did you guys encounter this error?

We have a requirement wherein we get different type of input files i.e avro, json and csv (example: client1_product.avro,client2_product.json,client3_product.csv, etc) and we have to convert these files to csv and load to db2.Currently I have designed a job with GetFile to fetch the avro files and ConvertRecordProcessor to convert records (avro to csv) and load it to db2 and this works fine but we need to design a generic job to read any file format (i.e avro, json and csv) and load it to db2 table.Can we design Nifi job to read different format files and convert it to a csv format in same job?Could you please advise. Thanks,Avi

Hello Experts,   I can no longer find the CDP Data Center upgrade docs on the website. I only see CDP Private Cloud. For example OS reqs link https://docs.cloudera.com/cdp/latest/release-guide/topics/cdpdc-os-requirements.html the url has cdpdc-os in it but the content is private cloud. I thought I saw Data Center few days back but cannot find now. Any ideas?   Thanks!

Hello all,I have HDF 3.4 cluster A with nifi and nifi registry integrated, cluster B with NIFI. both are tls/ssl secured. now I'm trying to use the cluster A NIFI registry for NIFI running on cluster B. noticing below error when trying to version a flow from cluster B NIFI  integrated with cluster A registry  I have added cluster B nifi node cert to registry users list but still same errorCN=its-nifi-node-dev-nifipoc1-01, OU=NIFI @alim @MattWho @sunile_manjee please advice 

I'm trying to overwrite a parquet file in hdfs for a Hive table, but I noticed a bunch of individual parquet files with unique names like this, table_directory /part-00199-edf111ef-0f7a-43ab-9393-8fc620195d4c-c000.snappy.parquet How do I add a parquet file to a Hive table directory that's composed of several parquet file?  

Hi,Due to space constraints, I would like to reduce the following settings in CM:1. Maximum Number of Events in the Event Server Store-5000000 To 40000002. Event Server Max Log Size - 200MB To 100MB Would like to know if this is can be done or would hamper CM services. Appreciate any assistance. ThanksWert 

ZooKeeper server creates snapshot and log files, but never deletes them. So we need to care about the retention policy. How to decide on the right amount of remaining Zookeeper snapshot files? Need to say that ZooKeeper server itself only needs the latest complete fuzzy snapshot and the log files from the start of that snapshot. But since ZooKeeper creates a backup of snapshot file, how many ZooKeeper snapshot backups do we need to retain? Sometimes snapshots can be corrupted, so the backup of snapshot files should take this into consideration. In our ZooKeeper server we saw that snapshot backup is created each day. Example of snapshot file from my ZooKeeper server:  -rw-r--r-- 1 ZooKeeper hadoop 458138861 Aug 10 07:12 snapshot.19000329d1-rw-r--r-- 1 ZooKeeper hadoop 458138266 Aug 10 07:13 snapshot.19000329de-rw-r--r-- 1 ZooKeeper hadoop 458143590 Aug 10 09:24 snapshot.1900032d7a-rw-r--r-- 1 ZooKeeper hadoop 458142983 Aug 10 09:25 snapshot.1900032d84-rw-r--r-- 1 Zoo

Hi, I am trying to import a hive table that has an MYSQL DB as metastore into Atlas by using import-hive.sh command but even when I explicitly pass the database name it fails to detect the same. @Shelton,Can you please suggest on this?  The above snippet shows AMBDB existing in the MySQL DB.

I'm trying to set up a Load Balancing Demo.   I have a GetFile processor that watcches a directory for incoming files and then a flow that processes them.   To try to understand LoadBalancing, I have created a second NiFi instance on the same host (my dev machine), but on a different port 8090 to the first 8080. I have then checked out the flow from the Registry so that it is identical on both instances.   Next, I have set the Load Balancing Strategy on the output queue from both nodes first processsor, (the GetFile proc) in order to see if I can get data going through both instances. I can then drop files into the watch dir and see them hit my flow. However, all the files are going into the same node, even if I wait 10 minutes between dropping them into the dir. But, when I run again it switches to the other node and all the files go through there. Is there a timeout somewhere that I can set to speed up the switch from one node to the other? Or better to switch files to

I want to setup cluster of 3 nodes. Suppose below are the 3 nodes, Node 1 = 192.168.0.10 Node 2 = 192.168.0.20 Node 3 = 192.168.0.30 I am able to setup the cluster by using below zookeeper string in the respective nodes, Node 1 = 192.168.0.10:2181 Node 2 = 192.168.0.10:2181,192.168.0.20:2181 Node 3 = 192.168.0.10:2181,192.168.0.20:2181,192.168.0.30:2181 If Node 1 is up and running and I am turning of any other node than Node 1, then cluster is working without any issue.   But problem with above configuration is, if I am turning off the Node 1 (192.168.0.10), then cluster is failing with the connection loss status. As Node 2 and Node 3 waiting for Node 1 which is down which turns into cluster failure.   So I am trying to setup same zookeeper string for all the node at once and trying to start server one by one. Node 1 = 192.168.0.10:2181,192.168.0.20:2181,192.168.0.30:2181 Node 2 = 192.168.0.10:2181,192.168.0.20:2181,192.168.0.30:2181 Node 3 = 192.168.0.10:2181,192.168.0.

Hi Team ,I need help in converting the given csv to json format .This is the input csv.EMP,EVSEID,SessionID,Starttime,Endtime,Totalchargeamount,Totalduration,Totalcosts3023513,AT*HTB*E1002760,b8ed2c41-4a0a-4528-9f14-4c4f040de81e,15:06:56 26.10.2019,15:42:59 26.10.2019,5890,2163,2.5922899202,AT*HTB*E1001927,ee3a1d64-2a69-4d43-87a4-4e23cb992660,16:24:11 08.10.2019,19:00:24 08.10.2019,6430,9373,3.782974330,DE*BOX*EAC0016,25543dea-6d58-4c27-a924-dacc01a83162,12:33:09 19.10.2019,15:29:58 19.10.2019,13700,10609,6.028.I am able to convert the given csv to json output .Need to add additionalheader and footer tag, Footer tag will be holding the aggregated value of the totalcosts given in the csv file.I am stuck in the process of performing aggregation based on the totalcosts. Attached the generated json and expected json format Below table is having generated json.[ { "record" : { "EMP" : "3023513", "EVSEID" : "AT*HTB*E1002760", "SessionID" : "b8ed2c41-4a0a-4528-9f14-4c4f040

Hello all, I want to see the size of each table in Hive residing in multiple databases. There are around 3000 tables, so it is difficult to do it one by one for each table. How can I do it at one go? Regards,Manu

My processor is querying a DB2 table and loads data to S3 and then ingest it into Snowflake. I have unique account numbers in my table and I need to add a column with unique identifier for each unique account number.

I am inserting records in maria db table from a file using python. Population column in the file is empty. I want it to go as empty value in table as well. Population column in table is set as integer and can accept null value. I am trying the below code -Table Definition -CREATE TABLE `local_db`.`table_x` (`Unique_code` varchar(50) NOT NULL,`city` varchar(200) DEFAULT NULL,`state` varchar(50) DEFAULT NULL,`population` bigint(20) DEFAULT NULL,`Govt` varchar(50) DEFAULT NULL) ENGINE=InnoDB DEFAULT CHARSET=utf8;```input_file = "input_file"csv_data = csv.reader(open(input_file))try:connection = mysql.connector.connect(host='localhost',database='local_db',user='root',password='root',port = '3306')cursor = connection.cursor()for row in csv_data:cursor.execute("""INSERT INTO table_x(Unique_code,city,state,population,Govt)VALUES("%s", "%s", "%s","%s", "%s")ON DUPLICATE KEY UPDATE city = VALUES(city),state = VALUES(state), \population = VALUES(population),Govt = VALUES(Govt)""")connection.comm

hello, i want to know if the processor of nifi listSFTP does remove files from path directory after listing them?

We are planning to install KMS and KTS but do we need Cloudera Navigator to Install KMS and KTS? if not required, then how to install it without Cloudera Navigator?

I can't open /run/cloudera-scm-agent/process/256-yarn-NODEMANAGER/container-executor.cfg: Permission denied.+ perl -pi -e 's#{{CGROUP_GROUP_CPU}}##g' /run/cloudera-scm-agent/process/256-yarn-NODEMANAGER/yarn-site.xmlI am getting this error after enabling Kerberos in CDH cluster, HDFS  and yarn are not able to start. After checking the yarn Node Manager logs I see the below error. yarn nodemanager logs:: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.security.authorize.AuthorizationException: : User: cloudera@CLUSTERIE.LOCAL is not allowed to impersonate yarn/ip-10-0-xxxxx@xyz.comAny suggestion why am I getting this error, when I disable the Kerberos everything works well. Please assist as the severity of this is very high.

Hi,  I am facing the below error when running the import-hive.sh to import hive metadata to Apache Atlas.  2020-08-08T07:41:25,369 ERROR [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Import failedcom.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed outat com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155) ~[jersey-client-1.19.jar:1.19]at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:105) ~[jersey-client-1.19.jar:1.19]at com.sun.jersey.api.client.Client.handle(Client.java:652) ~[jersey-client-1.19.jar:1.19]at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) ~[jersey-client-1.19.jar:1.19]at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.19.jar:1.19]at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:634) ~[jersey-client-1.19.jar:1.19]at org.apac