Q&A

Hi Everyone. I already have a Data Lake and Data Hub running in my CDP environment but I’m having issues with registering the ‘Classic Clusters’.After clicking ‘Get Started’ the next screen is basically a blank page? Attached are screenshots for reference. Please help to provide your inputs. Appreciate your help 

Hi, I am using the ExecuteSQL processor in NiFi to write records to PostgreSQL.The problem I have is when running on multi-thread (for example 10), the processor gets stuck and runs too slow. This doesn't happen in a single thread. However, single thread is not efficient so doesn't work for me.After getting slow, I stop the processor, terminate it, and start it again. For almost 5-10 seconds everything is ok and then the same problem.I also have a similar problem with the InvokeHTTP processor. Thank you in advance.

Hello . I am trying to read the mysql bin log data from mysql database and i am getting the error as"Could not connect to Distributed Map Cache server to get table information"when i added the Distributed Map Cache Client value in CaptureChangeMysql Aapache nifi processor .screenshots are attached for the reference  OS :Ubuntu (AWS AC2)NIFI : 1.8.0Mysql : 5.7.*i   

does CDP still needs rack topology/awareness for 10 + nodes cluster? if yes may I know the steps on doing this please. thanks

Hi eveyone, Ambari :- 2.7.4.0HDP :- 3.1.4.0 I want to know whether postgresql in addition to MYSQL ( I will be using MYSQL in advanced database configuration while setting up Ambari server as my choice ) is needed for Ambari ( 2.7.4 ) installation or not. If it is needed should I install it on both Ambari Server host and also Ambari Agent hosts or only Ambari Server host. On which hosts is it required? Please help!@KuldeepK @jsensharma @MattWho @Shelton     

The command show databases list all databases in a Hive instance. The command list databases that I have and don't have access to. When I try to list the tables in a DB that I don't have access to, using the command show tables from forbidden_db, it returns an empty list. Which command would list all databases that I have access to at least one table?

Hello! I'm trying to install Ambari Server as instructed at https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/download_the_ambari_repo_amznlnx2.html I tried the command (replacing username with my username and respective password), but I get the "Authentication Failed" for both commands below.  sudo wget -nv https://username:password@archive.cloudera.com/p/ambari/amazonlinux2/2.x/updates/2.7.5.0/ambari.repo -O /etc/yum.repos.d/ambari.repo   sudo wget -nv https://archive.cloudera.com/p/ambari/amazonlinux2/2.x/updates/2.7.5.0/ambari.repo -O /etc/yum.repos.d/ambari.repo  I'm using Amazon Linux 2 AMI  I also tried on my local machine (Mac OS) and got the same error message. How should I proceed? Thank you!

Following latest Cloudera announcement about moving to a subscription model.https://www.cloudera.com/downloads/paywall-expansion.html Does that mean that if you have a cluster running 5.x without a subscription, you can no longer access the cluster or Cloudera manager ? or might this still work if you have already downloaded the parcels using Cloudera Manager?

Following latest Cloudera announcement about moving to a subscription model.https://www.cloudera.com/downloads/paywall-expansion.html Does that mean that if you have a cluster running 5.x without a subscription, you can no longer access the cluster or Cloudera manager ? or might this still work if you have already downloaded the parcels using Cloudera Manager?

 After making changes  asssh-copy-id -i /home/oozie/.ssh/id_rsa.pub cloudera@example.local.com getting permission issue in oozie workflow oozi-W@ssh-34f1] Error starting action [ssh-34f1]. ErrorType [NON_TRANSIENT], ErrorCode [AUTH_FAILED], Message [AUTH_FAILED: Not able to perform operation [ssh -o PasswordAuthentication=no -o KbdInteractiveDevices=no -o StrictHostKeyChecking=no -o ConnectTimeout=20 abc.com  mkdir -p oozie-oozi/0000000-201209133111712-oozie-oozi-W/ssh-34f1--ssh/ ] | ErrorStream: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive).

I'm leveraging CDF and Nifi to orchestrate copying data from a relational database to Hive v3 on a daily basis.  This is using CDF and CDP (on-prem).  I have logic working for creating the tables via an AvroToORC conversion and then using the hive.ddl attribute to create the table.  I have two primary issues: 1) There are no unique columns or primary keys in the source system.  I have a Nifi processor that runs "delete from database.tablename" on Hive.  This clears out the data and leaves the table structure.  On large tables this can take some time.  The reason I have to do this is the PutHive3Streaming processor is not able to recognize duplicates and thus will continually append to the database and over-inflate it with duplicate records.  Are there other options for not needing to drop all the entries but still inserting the data? 2) From a performance standpoint, PutHive3Streaming is working but it's quite slow.  I've compared

Hi ,I have to log Nifi processor success or errors status in shell script to monitor the flow . Please help with solution . I have tried using Updateattribute and log the message in a filesystem and could not complete it . Also i can not access nfi-app.log file into Edge node. Would like to do through REST API calls or some other solution .  Thanks

I have a scenario where list of files are coming from previous processor, where for each file, I have to create json file with attributes of the flowfile. In AttributesToJSON processor configuration there is option to extract pipeline attributes and can create json files/object, if we set Include Core Attributes to true, it will read some of the file properties and forms the json file. the out for the above case in my scenario is …{"fragment.size":"125"file.group:"root",file.lastModifiedTime:"2020-12-22T15:09:13+0000",fragment.identifier:"ee5770ea-8406-400a-a2fd-2362bd706fe0",fragment.index:"1",file.creationTime:"2020-12-22T15:09:13+0000",file.lastAccessTime:"2020-12-22T17:34:22+0000",segment.original.filename:"Sample-Spreadsheet-10000-rows.csv",file.owner:"root",fragment.count:"2",file.permissions:"rw-r--r--",text.line.count:"1"}} But the files has other properties, like absolute.path, filename, uuid are missing in the above json file. My requirement is, get the absolute.path

Hi, I have sensitive parameter defined in process group's parameter context. The documentation says that this parameter can be assigned to a sensitive property only. So I want to know how to create custom sensitive property specific to my nifi processor. I did see few posts where they are asking to use InvokeScriptedProcessor, but I did not find any example. Any suggestions here would be of great help. Thanks, 

After Setup for the Tez Shuffle Handler following instructions on : https://tez.apache.org/shuffle-handler.html I'm getting below error in query execution:Vertex failed, vertexName=Map 2, vertexId=vertex_1608273679503_0002_2_01, diagnostics=[Task failed, taskId=task_1608273679503_0002_2_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1608273679503_0002_2_01_000000_0:java.lang.RuntimeException: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) at java.security.Acces

Hello Team, We are using Impala 3.2.0 there are some queries which were working 3-4 weeks back and now they have suddenly started throwing "IllegalStateException: null" exception. In last 3-4 nothing has changed on our Impala cluster. Error logs also does not suggest the cause of the error.  Can anybody help us as to why this issue arising.Query which failing looks something like this WITH banner_test AS (WITH layout_id AS ( SELECT <some columns> FROM table_v1 WHERE ( layout_component_id = <some_id>) ), layouts AS (SELECT <some columns>, coalesce() AS col1, <some columns> FROM table_v2 lch WHERE lch.some_id IN (SELECT some_id FROM table_v3 WHERE some_field_v1 IN (SELECT some_field from layout_id))

Hello All, I have an HDP 3.1 cluster that is neither kerberized nor it has ranger installed. Now, I have a requirement to Kerberize it and also enable Role Based Access Control using Ranger. 1. Should I first go with Kerberization and then installation of Ranger or vice versa?2. Any cautions to be taken with both of these approaches? 3. Is there a requirement/recommended for implementing TLS/SSL for all the services while Kerberizing the cluster? Thank You,Megh Vidani

Hi,     Im new to Apache Nifi and i'm looking on how to filter the CSV data using specific column.      I'm able to convert to JOSN without filtering, but SplitText and RouteonAttribute processors are not helping to filter the data. Below is my input CSV. Input:number,name,resourceState,location,manufacturer111897,lok,INSTALLED,HYD,ABC115677,redd,RETIRED,BLR,ABC1108448,eswar,PROP_INITIAL,CLT,ABC1116740,wqwq,INITIAL,AA,ABC Filtering should be based on resourceState column to consider INSTALLED and RETIRED data. So, converted JSON should have only 2 rows like below. Expected JSON Output:[{"number": "111897","name": "lok","resourceState": "INSTALLED","location": "HYD","manufacturer": "ABC"},{"number": "115677","name": "redd","resourceState": "RETIRED","location": "BLR","manufacturer": "ABC"}] Please help me on the CSV filtering part. Thanks in Advance.

i'm planning to simulate install Cloudera Manager in 1 server, but the MYSQL where cloudera metadatas resides is installed in a different server? how can I run the scm_database.sh if the mysql is in another different/separate server?

I have a set of tables (20 tables) which fetches the last loaded date and count of the tables. INSERT OVERWRITE TABLE dbo.table PARTITION (last_load_date )select 'table' as tablename,current_date,count(*) as count,last_load_date from table1 where last_load_date in (select max(last_load_date) from table1)group by last_load_dateunionselect 'table' as tablename,current_date,count(*) as count,last_load_date from table2 where last_load_date in (select max(last_load_date) from table2)group by last_load_date...Union of 20 TABLES and configured the script  like below and ran the script with command sh table.sh  Table.shhive --hiveconf tez.queue.name=Last_date --hiveconf hive.session.id=data_xxx -f /mypath/union_query.sql I could see the error:  Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Arrays.java:3664) at java.lang.String.<init>(String.java:207) at java.lang.StringB

I'm totally new to NiFi and learning as I go, but am against a hard deadline and need urgent assistance. I was tasked to create a data interface between two learning systems to send course completion data.I'm creating a flow to send course completion data from one system to another and have a JSON formatted file (to send as HTTP body to send as a CURL post?)This is the sample JSON data that I've created so far:{"user_id":10042,"username":"de","course_id":"C00230","completed_on":"12-02-2020","SCORE":100}The suggested HTTPS Curl request format is provided below.curl -v -X POST -d "j_username=admin&j_password=admin&userID=MORRW001&crsID=C00565&dateCompleted=2020-11-16&testScore=100&courseType=a" https://lms.dev.test.net/jw/web/json/data/How do I configure the InvokeHTTP processor to send it correctly? Do I need to reformat the JSON data so it's in the correct format for the curl format shown above? Remember I'm just a NiFi novice, so details, including screens

Hi, Generate the following error in the postgres database installed with the ambari. Can you please help? Postgres cannot be started. Ambari-server cannot be started.   /var/log/postgresql/postgresql-10-main.log   2020-12-16 16:39:01.601 +03 [1688] LOG: worker process: logical replication launcher (PID 1821) exited with exit code 12020-12-16 16:39:01.602 +03 [1814] LOG: shutting down2020-12-16 16:39:05.512 +03 [1688] LOG: database system is shut down2020-12-16 16:43:03.421 +03 [1772] LOG: listening on IPv4 address "0.0.0.0", port 54322020-12-16 16:43:03.454 +03 [1772] LOG: listening on IPv6 address "::", port 54322020-12-16 16:43:03.764 +03 [1772] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"2020-12-16 16:43:05.405 +03 [1809] LOG: database system was shut down at 2020-12-16 16:39:05 +032020-12-16 16:43:06.942 +03 [1772] LOG: database system is ready to accept connections2020-12-16 16:43:07.547 +03 [1845] [unknown]@[unknown] LOG: incomplete startup

Added 6 new DN nodes for scaling but HBASE is not assigning any regions on new DN'sHave errors like this on HBASE master -  AppDispatcher,\xEB\xFDE\x08\xEB\xDD\x11\xE9\xA4I\x0AX\x0A\xF4\x8Ar,1574112884653.8808c0e1917bf0b4acea2d83d9548463. state=FAILED_CLOSE, ts=Wed Dec 16 16:53:03 UTC 2020 (3008s ago), server=cdh-dn-28.prod.mcs.az-eastus2.mob.nuance.com,60020,1608136647199

I have inherited a problem. 48 regions in hbase:meta in transition. Table has had data removed - most likely manually in an attempt to fix RIT issues. These RITs are probably a result of a network outage mid-operation. Table is currently ENABLED and cannot be DISABLED (this has already been attempted by previous techie, which resulted in LOCKS/Procedures for DISABLE and DELETE as well as RITs). Table is no longer required so can be deleted.  HDFS reported it as being only 6k so I removed the table directories and zapped the znodes via ZK shell.  This fixed the locks/procedures messages but CManager still reports 48 regions in transition and, as a result of this, balancing is not working. What I need is a way to remove the rows from 'hbase:meta' as this is the only place where this table is still referenced. Sample output: alfa:rfilenameext column=table:state, timestamp=1604493139455, value=\x08\x00alfa:rfilenameext,,1557760164826.35925292c25

Hello Team, We have 5 node Impala cluster (1 co-ordinator and 4 executors) we are running Impala 3.2.0. Each Impala node is of size 32 GB and 4 cores. Now we are facing an issue sometimes 2-3 Impala executors out of 5 are over utilised (80 - 90 %) memory usage  and other are not, for example if executor 1 and 3 have memory usage of more than 80% and some new query issued it fails saying could allocate space(512MB) on executor 1 even tough there is more than enough memory on the other executors (2,4 and 5) whose memory utilisation is under 20%. Following is the error which I receive Memory limit exceeded: Failed to allocate row batch EXCHANGE_NODE (id=148) could not allocate 16.00 KB without exceeding limit. Error occurred on backend impala-node-executor-1:22000 Memory left in process limit: 19.72 GB Query(2c4bc52a309929f9:2fa5f79d00000000): Reservation=998.62 MB ReservationLimit=21.60 GB OtherMemory=104.12 MB Total=1.08 GB Peak=1.08 GB Unclaimed reservations: Reservation