Solved

I am trying to copy CSV files from my local directory into a SQL Server database running in my local machine by using Apache NiFi. I am new to the tool and I have been spending few days googling and building my flow. I managed to connect to source and destination but still I am not able to populate the database since I get the following error: "None of the fields in the record map to the columns defined by the tablename table." I have been struggling with this for a while and I have not been able to find a solution in the Web.Any hint would be highly appreciated. Here are further details. I have built a simple flow using GetFile and PutDatabaseRecord processors My input is a simple table with 8 columns My configurations for GetCSV process are here (I have added the input directory and left the rest as default) The configuration for PutDatabaseRecord process is here (I have referred to the CSVReader and DBCPConnectionPool controller services, used the

How to enable Authentication for Hadoop HTTP web-consoles? running a cluster with older software cloudera-manager-server-5.15.0-1.cm5150.p0.62.el6.x86_64hadoop-httpfs-2.6.0+cdh5.15.0+2783-1.cdh5.15.0.p0.52.el6.x86_64cloudera-manager-daemons-5.15.0-1.cm5150.p0.62.el6.x86_64cloudera-manager-agent-5.15.0-1.cm5150.p0.62.el6.x86_64 

Hi, as we all know CDH 6.3.3 and subsequent versions are not available anymore under "Express licensing" model. Yet, I was in the process of setting up a 6.3.1 installation and apparently this is not possible, because a valid authentication (enterprise) is required. What am I doing wrong? Below, a screenshot from the Installation Guide:  But when I try to import the 6.3.1 Repository (WITHOUT Username and Password as stated on the documentation itself for version < 6.3.3 I get the following error:     # rpm --import https://archive.cloudera.com/p/cm6/6.3.1/redhat7/yum/RPM-GPG-KEY-clouderacurl: (22) The requested URL returned error: 401 Authentication requirederrore: https://archive.cloudera.com/p/cm6/6.3.1/redhat7/yum/RPM-GPG-KEY-cloudera: lettura importazione fallita(2).     What am I doing wrong? @Cloudera1 Is this a mistake I'm committing or is it a problem with Cloudera implementing their policies in a wro

I installed Cloudera Quickstart VM 5.13 on virtualbox and I'm trying to start hadoop server with the command sudo service hadoop-hdfs-namenode startBut i'm getting these two errors: log4j:ERROR Could not find value for key log4j.appender.RFAlog4j:ERROR Could not instantiate appender named "RFA" This is the full output of the command: [root@quickstart etc]# sudo service hadoop-hdfs-namenode startstarting namenode, logging to /var/log/hadoop-hdfs/hadoop-hdfs-namenode-quickstart.cloudera.outlog4j:ERROR Could not find value for key log4j.appender.RFAlog4j:ERROR Could not instantiate appender named "RFA".log4j:WARN No appenders could be found for logger.(org.apache.hadoop.hdfs.server.namenode.NameNode).log4j:WARN Please initialize the log4j system properly.log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.Failed to start Hadoop namenode. Return value: 1 [FAILED] Can anyone can help me with this issue?

Server returned HTTP response code: 401 for URL: https://archive.cloudera.com/p/cm6/6.3.1/allkeys.asc

Hi Cloudera Community, greetings from Florida I am very new to the new Cloudera Technologies. What I need is working with Impala and get used to that technology. Before, I used to download the Cloudera VM and that is it. All the technologies came installed already.  Would you please help me by saying step by step what I need in order to have a working environment in which I can use Impala?How that is work?Is there any other free IDE different than web based HUE? If not, that is OK. How much will it cost to have a small dataset and run some queries? But mainly I need to have it everything set up.  Thanks for your help and time VV

When running Hive queries from NiFi 1.10 that contain backticks, I get the following error: 2019-12-23 15:17:00,191 WARN [Timer-Driven Process Thread-2] o.a.nifi.processors.hive.SelectHiveQL SelectHiveQL[id=075b3a1e-7632-1647-68d9-338231b5921b] Failed to parse query: select 1 as `asdf` due to java.lang.NullPointerException: I thought Hive queries allowed backticks to escape column names, so I'm not sure why NiFi can't parse this.  The actual query runs fine on the Hive server, and I get a valid flow file with the query results, but it still raises a red NiFi bulletin (which we would prefer not to have if there is not a real problem). The Hive server running the query is the one that comes with HDP 2.5.

Hi Team,Please help me on the below issueI am ingesting the Data From Oracle to PostgresSql Using NiFi.We have huge data and continuously generating from sources, I want to update attributes and delete attributes using NiFi My Flow is: QueryDatabaseRecord-->UpdateAttribute-->PutDatabaseRecordwhen matched record found on the final table then define which action you need to take either Update (or) Deleteif the record not matched in the final dataset then insert the record.Appreciate your help.Thank you,

Hi All, I am building Nifi flow which fetch the filename.zip from shared server and unpack the .zip file then distributes the files across nifi cluster and finally put the files in HDFS. By doing this process , i am collecting success or failure status of each processors using custom message inUpdateattribute processor . Refer the sample attached screen-print for updateattribute processor . Next i am planning to update overall status(success or error) of .zip file along with each file processed status ,like combining all information into one file and push the file to shared server to know about the status. I tried to use Mergecontent processor but i am unable to update the .zip file status and also noticed that if there is a delay in processing  How to build the flowfile like below  Filename.zip:staus:error_messageFilename1.status:sucess_messageFilename2:status:success_messageFilename3:status:error_message Going forward i may also update the status me

Hi, I am trying to download the ambari repo for RHEL 6 from the cloudera site. but its asking for username and password. Not sure why ? has any policy been changed ? I was able to download this till last month. Below is URL for reference: http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.3.0/ambari.repo Below error shows on web page:<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>6AB2932EB30E54AC</RequestId> </Error> Thanks

Hello ,We have one use case where we need to connect to HDF schema registry using Python Library .We have installed python-schema-registry-client  library. But Schema registry URL is kerberos authenticated URL so couldn't connect with below code from schema_registry.client import SchemaRegistryClient, schemaclient = SchemaRegistryClient(url="http://127.0.0.1:8081") provided here https://marcosschroh.github.io/python-schema-registry-client/Getting Authentication error 401.Is there any extra arguments or methods to get connect to Schema registry using python library? Thanks and Regards

Hi, I was trying to use nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository. To do so i have to comment java.arg.13=-XX:+UseG1GC to get rid of any performance issues as per https://community.cloudera.com/t5/Community-Articles/HDF-CFM-NIFI-Best-practices-for-setting-up-a-high/ta-p/244999However i find it difficult on how to do it via Cloudera Manager. Please help.

Hue audits are not logging consistently /var/log/hue/audit/ on CDH 5.16.2 any ideas, is this a bug?

Hello,We have upgraded CM from 5.16.2 to 6.3.4 has finished successfuly. However, after restarting the CM agents and checking the "Review Upgrade Status" we are getting the following warnings: "View Details 2 hosts are running CDH 5 and 0 hosts are running CDH 6. There are mismatched versions across the system, which will cause failures. See below for details on which hosts are running what versions of components.  Component : Supervisord Version / Host:3.4.0 / Server1 3.4.0-cm6.3.4 / Server2 " We have also tried a hard_restart of the CM agents and we are still getting the error message.Does anyone know why this can happen?Thank you in advance.

Hello,I try to work out some automated deployment situation in NiFi (no UI usage, just cli toolkit or alternatives). Let us asume, you have a nifi flow under a load which need to be updated.There are some data in in-between connection queues.What happens when you try to update the flow?Will some data be lost or will the update be refused? Are there any solution for procedure like this.1) stop the data receiving processor2) check if the success connection queue is empty3) apply steps 1 and 2 to the next processor till the last processor is stopped.4) update the flow ignoring any data in failure or error connection queues Do you think it is reasonable approach? Is there something ready to use?Best regardsJaro 

http://archive-primary.cloudera.com/cdh5/cdh/5/ for download client tarballs is not available. Has it been moved to somewhere else ? Thanks in advance.

Hello, I am working through this tutorial: https://www.cloudera.com/tutorials/tour-of-cloudera-data-science-workbench.html. I'm able to install the libraries just fine. When I go to run the python script (Sharing Results section), it appears to start processing the code, then it refreshes a few times, and finally it crashes. Is this a known issue?

Hello,We are uploading the version of our clusters to 6.3. During this process, we have found the need to also upgrade the Oracle database to version 19.The Oracle upgrade was done but the process slowed down due to many Cloudera Manager objects in the Oracle database recycle bin.A simply and maybe a stupid question but... Do you know if Cloudera Manager can be affected if we disable the Oracle recycle bin?Thank you

Hi All,  Currently we are building a Java application to connect hive database to perform data transfer. We are encountering issue while issuing the below query through java application"select count(*) from tablename where partition_key='xxxxx'".The above table is partitioned by partition_key column. below is the complete error log from Java ### Error querying database. Cause: java.sql.SQLException: [Cloudera][HiveJDBCDriver](500051) ERROR processing query/statement. Error Code: [Cloudera][JSQLEngine](12010) The table "dbname"."tablename" could not be found., SQL state: HY000, Query: select count(*) from dbname.tablename where partition_key='201912310002' .### The error may exist in com/hds/datatransfer/mapper/hive/HiveRepository.java (best guess)### The error may involve com.hds.datatransfer.mapper.hive.HiveRepository.test1### The error occurred while executing a query### SQL: select count(*) from dbname.tablename where partition_key='201912310002' ### Cause: java.

Hi,  these repos now ask for authentication. https://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5/https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/ are they still available?

I'm trying to convert local json file with getfile processor into parquet file on HDFS.  I'm following this guide: https://medium.com/@abdelkrim.hadjidj/using-apache-nifi-for-json-to-parquet-conversion-945219d5caba I choose Record Reader of JsonTreeReader, but I don't see any of the schema properties in the putparquet processor? (Schema Access Strategy, Schema Text).  How do I get those to show up, or how do I enter them manually?

The GetSFTP processor is throwing the FlowFileAccessException when trying to read a file from a remote directory.If I look at the error stack trace I can see that at StreamUtils.copy is where IOException has happened.Is it something like file was moved when actually reading it ?Can you guys please help me understand why this issue occurring and how I can avoid it ?o.a.nifi.processors.standard.GetSFTP GetSFTP[id=7d5fe4b5-7b67-3779-8de7-00e903c20c34] Unable to retrievefile FILENAME due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.IOException: error: org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from com.jcraft.jsch.ChannelSftp$2@6e1e8c5e for StandardFlowFileRecord[uuid=a50179f7-b631-4aa8-9e74-a2dca44d89bd,claim=,offset=0,name=a50179f7-b631-4aa8-9e74-a2dca44d89bd,size=0] due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.

Hi All, I need to migrate existing MySql DBs used for metastore, cmserver, amon, rman, oozie, hue and sentry to another MySql DB instance (MySql version 5.5). We are using CDH 5.16 and it contains 300TB data. Migration need to be done smoothly with less interruption to production workload. Appreciate if you can share the instructions/ related documents to do this. Thank you

Hello,I have tried to use nifi cli toolkit (nifi 1.11.4) to set parameter like this. However, I can't see any effect.What have I doing wrong? #> nifi get-param-context -u http://localhost:8080 -pcid 828f0882-0177-1000-aabd-d680a070c5c5# Name Value Sensitive Description- ------------- ------------- --------- -----------1 sample.Passwd ******** true2 sample.Schema SAMPLE001 false3 sample.URL jdbc://sample false4 sample.User dbsample false #> nifi set-param -u http://localhost:8080 -pcid 828f0882-0177-1000-aabd-d680a070c5c5 -pn sample.URL newvalueWaiting for update request to complete... #> nifi get-param-context -u http://localhost:8080 -pcid 828f0882-0177-1000-aabd-d680a070c5c5# Name Value Sensitive Description- ------------- ------------- --------- -----------1 sample.Passwd ******** true2 sample.Schema SAMPLE001 false3 sample.URL jdbc://sample false4 sample.User dbsample falseBest regardsJaroJaro

we have kafka cluster with 3 brokers machines and 3 zookeeper servers machines all servers are installed on redhat 7.2 version but when we run the following cli ( to know that all brokers ids are exist in zookeeper , we get:   zookeeper-shell.sh zoo_server:2181 <<< "ls /brokers/ids"WATCHER::WatchedEvent state:SyncConnected type:None path:null[3, 2]instead to get that:[3, 2 , 1] we checked the first broker ( kafka01 ) by searching errors in server.log And we not see any related error in the log! port 2181 from kafka broker to zookeeper machine is working we also restart kafka01 , but this not help to get the broker id in zookeeper cli we try also to restart all zookeeper servers ( there are 3 ) , and then again to restart kafka01 , but still without results so any suggestion to this behavior? can we add the missing broker to zookeeper cli ? , if yes then how?     note - I see another thread - https://community.cloudera.com/t5/Support-Questions/Specified-config-do