Q&A

I'm using PublishKafkaRecord processor. Everything works fine, but if I stop the processor and multiple flowfiles stuck in queue when I run the processor again if any of files is invalid all files get into failed queue. Is there any batch setting or any other way to solve this?

I am trying to read a csv file and put those records into a postgres table. To do it, I have used some tips given Loading a simple CSV file using nifi.   My template is : GetFile -> UpdateAttribute -> PutDatabaseRecord   In UpdateAttribute I have used `schema.name` -> `postgres database`   In PutDatabaseRecord ->   CSVReader   AvroSchemaRegistry   DBCPConnectionPool   My CSV file ``` userid,username 1,'sachith' ```   When I put this to my source folder, it goes to PutDatabaseRecord and gives following error   > None of the fields in the record map to columns defined by the nifi.nifiTest table.   First I used MySQL just with GetFile -> PutDatabaseRecord, it was working properly without error.   What am I missing here? I am pretty new for Nifi.

Hi Support Team,   We are planning to setup cloudera on one of our cloud environment.We have RHEL 7.7 installed on our servers. Does currently Cloudera support RHEL 7.7 version ??? or any possibility in near future to support latest RHEL version.   i also want to know what happened if we deploy current cloudera versions on RHEL 7.7.   Thanks,      

Hi, I tried to insert some data in postgresql db using PUTSQL processor. Below example of sql statement in putSql properties:   INSERT INTO myTable(date_response, success, code, payload_response) values ('${now()}', '${status:toNumber()}', '${code:toNumber()}', ${payload});    The problem is when this insert is executed, i have this error: org.postgresql.util.PSQLException: ERROR: column "zxlbavuzvmlvm2x6vedsemrdstzjrnnnzxlbavuzvmlvm2x6swpvz2v5qwlhv1f" does not exist. What happen ? .... the payload_response column name is replace by the value of my flow file attribute ${payload}. Why ? any hints ?

Getting error when inserting a csv-file into a MSSQL database using the PutDatabaseRecord (1.9.2) processor in Nifi.ListFile → FetchFile → PutDatabaseRecord     The schema used:     { "type" : "record", "name" : "mibas_t005_talkgroup", "fields" : [ { "name" : "talkgroup_ref", "type" : ["null","long"] },{ "name" : "fleet_ref", "type" : ["null","long"] },{ "name" : "talkgroup_id", "type" : ["null","long"] },{ "name" : "talkgroup_name", "type" : ["null","string"] },{ "name" : "group_type", "type" : ["null","string"] },{ "name" : "dispatcher_ref", "type" : ["null","string"] },{ "name" : "multigroup_ref", "type" : ["null","string"] },{ "name" : "owner_", "type" : ["null","string"] },{ "name" : "fne_state", "type" : ["null","string"] },{ "name" : "addl_attrvar", "type" : ["null","string"] },{ "name" : "last_user_modified", "type" : ["null","string"] },{ "name" : "time_stamp", "type" : ["null", { "type":"long", "logicaltype":"timestamp-millis"} ]},{ "name" : "fne_oi

I have installed the HDP sandbox HDP-3.0.1.0 (3.0.1.0-187) using Docker and Kerberized the cluster.    I can list the principles from the keytab file like this:   klist -kte /etc/security/keytabs/atlas.service.keytab  Which gives the following output:   Keytab name: FILE:/etc/security/keytabs/atlas.service.keytabKVNO Timestamp Principal---- ------------------- ------------------------------------------------------ 7 12/04/2019 10:27:18 atlas/sandbox-hdp.hortonworks.com@MYREALM.TLD (des-cbc-md5) 7 12/04/2019 10:27:18 atlas/sandbox-hdp.hortonworks.com@MYREALM.TLD (des3-cbc-sha1) 7 12/04/2019 10:27:18 atlas/sandbox-hdp.hortonworks.com@MYREALM.TLD (arcfour-hmac) 7 12/04/2019 10:27:18 atlas/sandbox-hdp.hortonworks.com@MYREALM.TLD (aes256-cts-hmac-sha1-96) 7 12/04/2019 10:27:18 atlas/sandbox-hdp.hortonworks.com@MYREALM.TLD (aes128-cts-hmac-sha1-96)     Please forgive my stupidity, but how can I now login to Atlas UI?   I've tr

hi all   we want to identify kafka alert from ambari by rest api   Example when we get alert     what is the AMBARI REST API in order to verify alert on kafka broker01 or kafka broker02 , etc 

Hey, folks,I know how to remove and add a server what I don't know how I can see the data replicating in the new node. someone could help me. Greetings

Hi team,Iam creating materialized view for a table ,for that getting below error.can any one help on this? create materialized view car1_mv as select * from cars;Error: Error while compiling statement: FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: Automatic rewriting for materialized view cannot be enabled if the materialized view uses non-transactional tables (state=42000,code=40000)

we are runs spark application on hadoop cluster ( HDP version - 2.6.5 from hortonworks )   from the logs we can see the following Diagnostics   User: airflowApplication Type: SPARKUser class threw exception: org.apache.hadoop.security.AccessControlException: Permission denied. user=airflow is not the owner of inode=alapati not clearly what we need to search in `HDFS` in order to find why we get Permission denied

Hi folks,   We have been struggling for the past 3 weeks trying to secure a cluster. We have 3 nodes. They were working ok in a cluster (but not secured). When trying to secure those (and following the guide suggested here), we came across the following message when trying to access the UI:   javax.net.ssl.SSLPeerUnverifiedException: Hostname <my_fqdn_is_here> not verified: certificate: sha256/716mOuXyoAKqzNrXrNnG2ozHXwN8WWJsVxzWzfQzpNV= DN: CN=xxx-xxxx-xxxx.xxx.xxx.net, OU=XXXXXXXXX XXXXXX, O=XXXXX, L=XXXXXX, ST=XXXXXX XXXXX, C=CA subjectAltNames: []    We've been going over and over the configs and nothing seems to point to the right direction.Would anyone point us to at least the right direction? 

Hello I'm new in Big data and I'm trying to create a cluster, but I have an issue with the registration of hosts, I saw a lot of posts about this problem but the problem is persist (You find below the Registration Log for the ambari server):   ========================== Creating target directory... ========================== Command start time 2019-11-18 10:23:23 Connection to ambari.server closed. SSH command execution finished host=ambari.server, exitcode=0 Command end time 2019-11-18 10:23:23 ========================== Copying ambari sudo script... ========================== Command start time 2019-11-18 10:23:23 scp /var/lib/ambari-server/ambari-sudo.sh host=ambari.server, exitcode=0 Command end time 2019-11-18 10:23:24 ========================== Copying common functions script... ========================== Command start time 2019-11-18 10:23:24 scp /usr/lib/python2.6/site-packages/ambari_commons host=ambari.server, exitcode=0 Command end time 2019-11-

#/usr/jdk64/jdk1.8.0_112/bin/java -cp/usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/share/java/mysql-connector-java-5.1.37.jarorg.apache.ambari.server.DBConnectionVerification "jdbc:mysql://xxxxxxx:3306/ambari" "ambari" "xxxxx" com.mysql.jdbc.DriverERROR: Unable to connect to the DB. Please check DB connection properties.com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communicationslink failureThe last packet sent successfully to the server was 0 milliseconds ago. Thedriver has not received any packets from the server.#ambari-server setup --jdbc-db=mysql--jdbc-driver=/usr/share/java/mysql-connector-java-5.1.37.jarUsing python /usr/bin/pythonSetup ambari-serverCopying /usr/share/java/mysql-connector-java-5.1.37.jar to/var/lib/ambari-server/resources/mysql-connector-java-5.1.37.jarCreating symlink/var/lib/ambari-server/resources/mysql-connector-java-5.1.37.jar to/var/lib/ambari-server/resources/mysql-connector-java.jarIf you are updating existing jdbc driver jar for mys

I'm attempting to enable authentication on my NiFi server that is in a cluster with several other hosts. If I enabled HTTPS for webUI authentication, without changing any of the other hosts on the cluster, will this cause any complications with their communication with one another?

Hello everyone,, When we assign Operate the Component to a process group, it allows users to start/stop/disable process group, but it doesn't allow the user to start/stop the Processors inside the Process group. How can we enable Operate the Component so that the users can start/stop the Processors as opposed to the all the processors inside the Process Group? Steps Taken:1. Go inside the process Group2. Click on Access Policy icon3. Click "Operate the component" in the drop down4. If needed, click either add or override link at the top of the page5. Assign a "User Group" to this policy Testing1. Login as a user in the User Group2. Navigate to Process Group3. You will be able to start or stop4. Click on Processor5. User unable to start or stop (Problem} 

Hi,   My Ambari server is offline (ambari.properties file is empty/virgin - don't ask 😞 ) but my cluster is functional, that is, HDFS is fine and so on. My Question is this, can I rebuild my Ambari server and "import" the running HDP cluster or do I have to start from scratch?   Thanks,   john

Hello All, We have applications in our data organization like Oracle Data Integrator and Jupyter Hub. Those set of applications require up to date jars, core-site.xml, hdfs-site xml etc to integrate properly with the cluster. Our Application Architects suggests we should install those applications' clients and/or agents on edge nodes of our cluster. I don't want to do this because: *I might need to deal with conflicting environment requirements. *I will have limited control over server resource consumption *Application level operations such as upgrades might have adverse affects on the cluster I am considering to offer mounting jars/conf directories of edge nodes on read only mode to application servers. Any suggestions and evaluations about both architectures or alternatives are more than welcome. Best regards

sudo apt-get install libmysql-javaReading package lists... DoneBuilding dependency treeReading state information... Donelibmysql-java is already the newest version (5.1.45-1).0 upgraded, 0 newly installed, 0 to remove and 92 not upgraded.     /usr/jdk64/jdk1.8.0_112/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/share/java/mysql-connector-java.jar org.apache.ambari.server.DBConnectionVerification "jdbc:mysql://XXXXXX:3306/ambari" "ambari" "password" com.mysql.jdbc.DriverERROR: Unable to connect to the DB. Please check DB connection properties.java.lang.ClassNotFoundException: com.mysql.jdbc.Driver  

Hello,I am facing an issue with Impala queries, though the query status says 'finished' on Impala status page it still shows 'executing'. I was following Cloudera Article : https://my.cloudera.com/knowledge/Finished-Queries-show-as-Executing-in-the-Cloudera-Manager?id=71576the difference  in my case is queries were run through impala shell via jdbc connection.would the resolution outlined in the article be applicable, for queries submitted through impala jdbc connection? and secondly what would be the fix for such queries which are in hung state. RegardsAmn

Hello Team,     We were planning to implement Role-Based-Access-Control(RBAC) for our impala tables we tried Sentry but there is a prerequisite of installing Kerberos which we don't want to use. So Apache Ranger is the other option we have but the latest CDH version 6.3 has Impala 3.2 which does not support Ranger. So just wanted to if there is going to be release of new CDH bundle which will have Impala 3.3.0 and support for ranger? What will be approximate release date of the new CDH bundle?       Regards Parth

  Hello  i need to connect Nifi to Oracle DB but i faced a problem in the connection. i copy Ojdbc.jar to lib directory     i choose QueryDatabaseTable processor ( i am not sure if it's correct )  -  Database Connection URL :  jdbc:oracle:thin:@//HostName:XXXX/XXXXXXX -  the Database Driver Class Name : oracle.jdbc.driver.OracleDriver -  Database Driver Location : C:\Users\d1156\Desktop\nifi-1.10.0\lib\ojdbc8.jar   after i start the processor this error appears : unable to execute SQL select query - cannot create JDBC driver of class ' Oracle.jdbc.driver.oracleDriver'    Thank you .  

I am trying to use Ranger for Authorization with Hive and HDFS. But somehow Ranger doesn't recognize Hive or HDFS. I don't see the default policies in Ranger for Hive and HDFS. I activated the Ranger Plugins for Hive and HDFS and checked the Advanced ranger-hive-plugin-properties and Advanced ranger-hdfs-plugin-properties. For me everything seems fine, but the services don't appear in Ranger. I also activated Ranger as authorization method for Hive. But that also didn't helpThis is how my configuration looks:Ranger Services:Ranger Plugins:Advanced ranger-hdfs-plugin-properties.  Advanced ranger-hive-plugin-properties What am I missing here? I would be happy about hints!Thanks

I have a problem with the HandleHTTPRequest Processor, I'm using it to receive Data from Sensors via a path and a port, the problem is that even If sensors send Data and when using wireshark I see Data coming from sensors to my Integration server in the port assigned , but the processor doesn't show anything until I restart NIFI , Data reach the processor for some minutes or sometime for 2 days and after it stop again , don't know what is the problem , because restarting Nifi every time is not a good solution for me , someone can help me on that ?

Hi,  I am wondering if it is possible to get Service-Wide configurations via read_config method of the RoleConfigGroupsResourceApi class. https://archive.cloudera.com/cm6/6.3.0/generic/jar/cm_api/swagger-html-sdk-docs/python/docs/RoleConfigGroupsResourceApi.html#read_config   The read_roles method of RolesResourceAPI returns theses roles:   CD-HDFS-eHtEMKVf-DATANODE-BASECD-HDFS-eHtEMKVf-SECONDARYNAMENODE-BASECD-HDFS-eHtEMKVf-HTTPFS-BASECD-HDFS-eHtEMKVf-DATANODE-BASECD-HDFS-eHtEMKVf-DATANODE-BASECD-HDFS-eHtEMKVf-NAMENODE-BASE    But when I query all these roles, I cannot find the Service-Wide property of Advanced configuration for core-site.xml   Reading configuration for CD-HDFS-eHtEMKVf-DATANODE-BASE{'items': [{'default': None, 'description': 'For advanced use only, key-value pairs (one on ' "each line) to be inserted into a role's " 'environment. Applies to configurations of this '

CDH6.2.1imapla 3.2.0-CDH6.2.1 the below errors happened every day,  i also find some similar issue on issue.apache.org.  it seems a bug .  here is some issue list :https://issues.apache.org/jira/browse/IMPALA-8561https://github.com/cloudera/Impala/commit/9e4bf4c99f48a29e1c1d46d49c5f8452b777e2db it seems this issue also happened on Impala3.3, not juse impala 3.2, but it's fixed in 3.3. so, Cloudera support, how to fix this issue on imapla-3.2( CDH6.2.1), this issue is so critical cause many users encounter this issue and ask me what's happening, and i just can tell them this is a bug and i can't do anything. may i ask another question ? if Impala3.2 can't resolve this issue ,may i update imapla 3,2 to impala 3.3 , i mean i don't update Cloudera Manager  and CDH other components , just update impala . i hope Cloudera Supporter can give me advises , thank you very much .