Q&A

Hello,   We are currently trying to fetch changes from SQL Server CDC tables and push them into a Kafka topic with Nifi. Basically the flow is working but we would like to have feedback on how to make it more reliable and optimized.   Here is the SQL used to fetch changes from MY_TABLE from offset 37979520     SELECT CT.SYS_CHANGE_VERSION, CASE CT.SYS_CHANGE_OPERATION WHEN 'D' THEN CT.T_PK ELSE NULL END as DeletedPK, CHANGE_TRACKING_CURRENT_VERSION() as LastTransactionId, T.*FROM CHANGETABLE(CHANGES MY_TABLE, 37979520) CT LEFT JOIN MY_TABLE T on CT.T_PK = T.T_PK     Here is the Nifi implementation: (bigger picture in attachment)   To avoid threading issues, all processors are targeting the primary node with a single thread.   Here are the steps   1 - GenerateFlowFile To start the process, I generate a FlowFile with a custom property "increment.value" = cache_key.   2 - FetchDistributedMapCache I fetch the distributed cache w

Hi! I have an array with dates and I want to insert it into my flowFile through a groovy script but I can't find a way to do it, could you help me?that's my script with which I get dates.and this is my groovy script but it always goes to the queue of failure.

recently ambari server logs are showing some warnings /var/log/ambari-server/ambari-server.log 02 Jul 2021 18:43:52,514 INFO [ambari-client-thread-792188] AmbariAuthToLocalUserDetailsService:109 - Translated knox/<knox_gateway>@<REALM> to knox using auth-to-local rules during Kerberos authentication.02 Jul 2021 18:43:52,515 WARN [ambari-client-thread-792188] AmbariAuthToLocalUserDetailsService:143 - Failed find user account for user with username of knox during Kerberos authentication.02 Jul 2021 18:43:52,516 WARN [ambari-client-thread-792188] AmbariKerberosAuthenticationFilter:149 - Negotiate Header was invalid: Negotiate YIIDlAYGKwYBBQUCoIIDiDCCA4SgDTALBgkqhkiG9xIBAgKhBAMCAXaiggNrBIIDZ2CCA2MGCSqGSIb3EgECAgEAboIDUjCCA06gAwIBBaEDAgEOogcDBQAgAAAAo4IB/WGCAfkwggH1oAMCAQWhMxsxMjZGNURFMDEtNUU0MC00RDhBLTk4QkQtQTQzNTNCN0JGNUUzLkRBVEFMQUtFLk9WSKJPME2gAwIBAKFGMEQbBEhUVFAbPG92aC1lbm9kZTYuMjZmNWRlMDEtNWU0MC00ZDhhLTk4YmQtYTQzNTNiN2JmNWUzLmRhdGFsYWtlLm92aKOCAWYwggFioAMCARKh

I run hadoop with ldap   hadoop.security.group.mapping=org.apache.hadoop.security.LdapGroupsMapping hadoop.security.group.mapping.ldap.bind.user=uid=ldapadmin,ou=people,dc=join,dc=com hadoop.security.group.mapping.ldap.bind.password=00000 hadoop.security.group.mapping.ldap.base=dc=join,dc=com hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=posixAccount)(uid={0})) hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=posixGroup) hadoop.security.group.mapping.ldap.search.attr.member=memberUid hadoop.security.group.mapping.ldap.search.attr.group.name=cn   but I run spark,I get a WARN   (LdapGroupsMapping:290)2021-07-13 13:02:45,523 WARN  - [pool-2-thread-4:] ~ Failed to get groups for user jztwk (retry=0) by javax.naming.OperationNotSupportedException: [LDAP: error code 53 - unauthenticated bind (DN with no password) disallowed]   and I ldapsearch -x -D "uid=ldapadmin,ou=people,dc=join,dc=com" -W -b "dc=jo

Trying to troubleshoot an issue in which we want to look at Cloudera Impala ODBC Driver logs. I open the ODBC Admin set the logging level to LOG_DEBUG, give it a path etc.,click OK and close the admin. Then when I run a query, nothing happens, no logs were created, I open the ODBC config again and notice that the Logging Level is back to Log_off. I've tried turning on logging multiple times with different logging levels and paths, every time I close it to run something, it reverts back to Log_Off.  Looking for any suggestions on what might be causing this to happen, what can I do to fix this?  Thanks All! Cloudera ODBC Driver for Impala V2.6.11.1011 

Hi All, We have Hue (version 4.2.0) in our prod environment and it is coupled with our LDAP. We manually onboard new users by going to "Manage Users" -> "Add/Sync Manage Users". Here, we add the LDAP ID and later when the user gets added. Once the user is added, we again search the newly added LDAP user, click on that and go to "Profiles and Group". Here, we select "temporary-access" so that User can able to access the UI. Without Temporary-access selection, user will not be able to see the UI. We want to get this automated by writing a Py program and we are looking for the right REST API for adding users to Hue and give him/her "temp-access" in profiles and group. The following link wasn't helpful - http://cloudera.github.io/hue/latest/developer/api/ Please assist us. I would highly appreciate any form of assistance or help. Regards,Shesh Kumar

Flow: QueryDatabaseTable >> PutDatabaseRecord (using the same MS SQL table) Original Date: '2020-01-01 01:00:00.000'Saved Date: '2019-12-31 22:00:00.000' Another:Original Date: '1753-01-01 00:00:00.000' (SQL Server minimum date)Saved Date: Routing to failure.: The conversion of a datetime2 data type to a datetime data type resulted in an out-of-range value. My timezone: (GMT-3) 

Hello, I am trying to load bmp image data to hive and getting below error. I went through many past discussions but didn't find an exact solution to implement. Appreciate your help in solving this.  Steps I followed. 1: Copied the images to HDFS2: Created a table with a column Binary3: Load using data in path. Error:  Error while compiling statement: FAILED: SemanticException Unable to load data to destination table. Error: The file that you are trying to load does not match the file format of the destination table. My scripts:create table wrk_image (id int, filename STRING, data BINARY);load data inpath '/warehouse/tablespace/external/hive/cadt.db/scriptpro/' into table wrk_image;

Hi We are periodically getting errors in our resource manager logs, which then cause no Spark jobs to be accepted until we restart the YARN ResourceManager.We are on HDP 3.1.4 and we have Kerberos enabled.  The stack trace is below. Does anyone have ideas, please?    2021-05-08 04:15:59,407 ERROR yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(68)) - Thread Thread[Thread-20,5,main] threw an Exception.java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.ResourceCommitRequest.<init>(ResourceCommitRequest.java:101) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.createResourceCommitRequest(CapacityScheduler.java:2848) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.submitResourceCommitRequest(CapacityScheduler.java:2685) at org.apache.hadoop.yarn.server.resourceman

I have a json like this coming from an input port: {"url": "blablabla","keyword": "foo"} then I have to generate a new json to pass to a post call. The new json is something like this: { "requests": [ { "source": "blablabla", "params": { "keywords": [ "something" ], "sub-field1": "1" } } ], "field1": "1", "field2": "2", "field3": false} where the array keywords should be replaced with a new array with the value of the previous Json ("foo"). The resulting is:   { "requests": [ { "source": "blablabla", "params": { "keywords": [ "foo" ], "sub-field1": "1" } } ], "field1": "1", "field2": "2", "field3": false} then i pass this json to a invokehttp to call the post.  Could you please explain me ho

I used to have some tables in HIVE but after an incident HIVE went down and some new tables erased and old tables went back to a previous schema state. What I want, is to ask if it is there any option to backup tables' schemas into MySQL which stores HIVE metastore in order to retrieve them if there occurred any incident again.

"The conversion from UNKNOWN to CHAR is unsupported."I'm using only 2 processors:QueryDatabaseTable (reading UniqueIdentifier Column) >> PutDatabaseRecord (writing in the same table and column)Table:CREATE TABLE TableTest(OID [uniqueidentifier] NULL,Bit1 [bit] NULL)QueryDatabaseTable:PutDatabaseRecord:Error:The conversion from UNKNOWN to CHAR is unsupported.Supposedly, Nifi's internal array is inverting the columns, putting the bit column in place of the UniqueIdentifier column, producing this "Unknown to char" error. Bug.I tried to install Nifi 1.14 but, it keeps asking for username and password and there is no ".bat" to generate username and password, only the shell ".sh". That is, it only works on Linux.

Hi Everyone, The goal is to be able to extend to several servers, in order to collect the files in a parallel way to count the number of lines and the number of files to store in a database each day. I collect files I count the number of files and the number of lines in the files. I want to store these values in a database. I installed apache cassandra and created a database and a table. when I insert the number of lines and documents, I check my table and I don't see any data. GetFile-->MergeContent-->CountText-->ReplaceText-->PutCassandraRecordI want to check the data entered in the database table but I don't see any data.here is the configuration of the PutcassandraRecord processor.PutCassandraRecord

I'm quite new in Apache Nifi but I'm encountering some issues trying to connect a kafka microservice (with a producer) with Apache nifi consumer.Basically, I have a docker-compose like this: zookeeper: container_name: zookeeper_test image: wurstmeister/zookeeper #zookeeper:3.5.7 ports: - 2181:2181kafka: container_name: kafka_test image: wurstmeister/kafka #:2.13-2.6.0 ports: - 9092:9092 environment: KAFKA_ADVERTISED_HOST_NAME: kafka KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true' KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 depends_on: - zookeeper - kafkauikafkaui: container_name: kafka-ui_test image: provectuslabs/kafka-ui:latest environment: - KAFKA_CLUSTERS_0_NAME=kafka - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092 - KAFKA_CLUSTERS_0_ZOOKEEPER= zookeeper:2181 ports: - 6789:8080test: container_name: test build: context: ./test dockerfile: Dockerfile depends_on: - kafka command: python test.pytest is my producer:

hello,hue is installed using tarball and now trying to synchronize users with ldap group. i am able to synchronize group and able to add all users into hue. however when user is trying to login, they are getting Error 500 ({"auth": false})in error log, i can see hue is trying to add this user again and db and getting duplicate error TypeError: 'AnonymousUser' object is not iterable[14/Jul/2021 23:05:04 -0700] forms ERROR LDAP auth error: (1062, "Duplicate entry 'chalkeg' for key 'username'") in hue.ini  [[ldap]]base_dn="DC=<MY_COMPANY>,DC=COM"ldap_url=ldaps://<company_ldap_host>:636bind_dn="<valid_bind_dn>"bind_password=<password>ldap_username_pattern="sAMAccountName=<username>,DC=<my_company>,DC=COM"search_bind_authentication=truesync_groups_on_login=truecreate_users_on_login=false * ignore <> values as they are correct and username is getting resolved.

Hello, Is there a way to find which kudu table is stored in which tablet server ? Thanks Amn

It's hard to describe the situation  The custom service was started in cm console. There were no errors in the log, but it was retried four times The truth is that the host has been started successfully.Therefore, we want to know how the CM console perceives whether the service startup is successful, and how to solve the problem.

Please help me out in this scenario:1. I have to create a new project. The details are accessed using GetFileProcessor.2. I have passed two input to Invokehttp Processor(one Authorization token using CustomField and other as data to be used for POST action). However, this process is not working as expected. Please find the attached screenshot for reference. Data To be used for POST:  Early Response much appreciated!Thanks!

Hello Team, can you please advise how to replace nulls by a default value? select concat(cast(glb.budget_version_id as string)," ", cast(glc.segment1 as string)) as balidfrom oracle_financial.gl_balances glb,oracle_financial.gl_code_combinations glc,oracle_financial.gl_ledgers glg ; If I add a null to a value, null will be returned. I tried NVL it is not working. Thanks, Roshan 

Hello everyone,My NIFI pipeline gets data from DB2 using QuaryDatabaseTable processor. When I launch it get exception (org.apache.nifi.processor.exception.ProcessException: Error during database query or conversation of record. I found out that the problem with some char in a table's field. I see this in the nifi log (nifi-app.log) and I see there that it falls with an exception java.nio.charset.UnmappableCharsetException: Input length = 1.What I tried. I used another JDBC connection driver. I tried to set UTF8 in bootstrap.conf and nifi-env.sh (I found this tip here). It has not helped. Could you help me, please? 

I have a requirement to check the count of files in a directory (say 10) everyday after it is transferred and to send Just one email alert if it is not satisfied. I have to check this condition every day whether these 10 files have been transferred without any fail . Also my previous files Would not be deleted from the directory as well. I am really new to nifi hence has a doubt the n how to proceed .@MattWho Could you please help with this somehow

Hi, We are processing ZIP file contains multiple timestamp files (.hiveql,.csv) in distributed manner .We check the file extension whether it is .hql or .csv then we route the file to execute it PutHiveQL and PutHDFS processor respectively.The files(timestamp order starts with for example t1 or system timestamp)  below contains in ZIP file to be extracted and processed in order.table_info.zip  table_info_t1.hqltable_info_t1_1.csvtable_info_t1_2.csvtable_info_t2.hqltable_info_t2_1.csvtable_info_t2_2.csvtable_info_latest.hqltable_info_latest.csv   Please find the below NiFi flow and RouteonAttribute property  Is there any way to make us to wait first puthivesql executes first and give indication to putHDFS execution next for each timestamp file one by one order. Can we group each timestamp files into group and process the .hql file and the put .csv file into HDFS?  @Nifi

I want to download http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo.But got ERROR 403: Forbidden.Any help is appreciated!

Hi Team, I am trying to connect the hive table from CDSW data science workbench using python. i am able to connect the hive from python using impala, when i try to execute the query i am getting below error,  SemanticException [Error 10265]: This command is not allowed on an ACID table default.atlastest with a non-ACID transaction manager. Failed command: SELECT * FROM `default`.`table_name`'), operationHandle=None) PFB code ref: import osimport pandasfrom impala.dbapi import connectfrom impala.util import as_pandas# Specify HIVE_HS2_HOST host name as an environment variable in your project settingsHIVE_HS2_HOST='1.1.1.1'# This connection string depends on your cluster setup and authentication mechanismconn = connect(host=HIVE_HS2_HOST,port=10000,user = 'username',password = 'password',auth_mechanism='PLAIN',kerberos_service_name='hive')cursor = conn.cursor()res = cursor.execute('SELECT * FROM `default`.`table_name`')  Kindly help me to close this iss

Hello! Can you kindly help me in an NiFi upgrade error? I followed this tutorial: https://cwiki.apache.org/confluence/display/NIFI/1.x.0+to+1.x.0+Upgrade Following it, i stopped my processors, waited until my flows had no queued flowfiles, stopped old nifi services, changed property names to my new nifi version, moved the flow.xml.gz to my new installation and started the services in the new version. After this, i get the error shown bellow:  Is this a regular mistake? Do you know any alternatives in this case? Thank you,Lucas