Solved

Hi, I would like to create the access privileges in database and table level.   So, what I had performed is:   1. Create user in hue (Manage User) a) testuser1 (non-sentry admin group), assigned under "testgroup" b) admin (Sentry-admin-group), assigned under "admingroup"   2. Grant role to the user "testuser1": create role role_test; grant all on server server1 to role role_test; grant role role_test to group testgroup;   However, when login in as "testuser1" in hue, I can't see any database. And check via query "show current role" and no record is returned.    Is there any steps I have missed?  Thank you!    

Hi Team, following the resolution thread Re: How to report NiFi lineage to Atlas I am now able to use the Atlas lib of Nifi called "ReportLineageToAtlas". I configured the settings as the following (Atlas URLs is changing day by day, after the restart of the servers) : In Kerberos Credential Service I inserted Atlas Kerberos Keytab and Atlas Kerberos Principal.The error I receive while the "ReportLineageToAtlas" is running is about Authentication to Atlas to reach the API : Indeed if I open the Atlas web URL : http://34.134.205.221:31000/api/atlas/v2/types/typedefs?name=nifi_output_port  I have no data.Therefore we have two issues:1) it seems there's an authentication issue from Atlas towards Nifi API.2) it seems in the API there are no data named "nifi_output_port" (maybe because it's not yet able to authenticate?). Maybe the two issues are linked.Do you have any hints about how to fix this problem? Were am I wrong in the settings? I can pr

Hi All,I have Nifi 1.14.0 currently installed and I would like to connect this to Apache Atlas available in Cloudera Cluster.I read here Configure NiFi for Atlas Integration (cloudera.com) there should be the possibility to do this by configuring ReportLineageToAtlas task in NiFi. Unfortunately this option is missing in my NiFi installation. Do you have any hints about how to add this feature in my NiFi Centos installation? Thanks a lot for your support.Best Regards,Daniele.

Dears, one of my kafka node's one of the disk is got full. Could you please suggest steps to make sure the 4 disks of my kafka node get sync in terms of disk occupancy. Ref Snapshot is below   Swift response are highly appreciated  

I have a cluster that uses a Postgres db with version 9.4.17 installed on a host in the cluster and I want to migrate to an external Oracle db. I have researched about the methods to do this in Cloudera documentations and I found that I have to have a Cloudera version higher than 5.15. Would it make any difference if we do the migration using Cloudera 5.13? what could be the differences between 5.13 and 5.15 in terms of db? """Configuration uses Cloudera Manager 5.15.0 or later on supported platforms."""https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ag_migrate_postgres_db_to_oracle_mysql.html  

Hi, I am using CFM 1.0.1.7, when I am using the remote process group to transfer file between two cluster. There is an error message "Failed to persist link of Peer due to java.io.FileNotFoundException:./conf/state/xxxxxxxxxxx_HTTP.peer (No such file or directory); If restarted and peer's NCM is down, may be unable to transfer data unitil communuication with NCM is restored. How shall I proceed with this error. The NIFI version is 1.9.0 and according to the online documentation, NCM is only avaliable for NIFI 0.X, what am I seeing this error?

I am unable to get sandbox to work without problems on my mac. I have limited memory 8 GB therefore, I installed 2.6.5 which I think may require less than the 10 GB required by 3.0.1. After I install and run the vbox .ova file, I get the url which I use to connect to Ambari dashboard.  No matter what I do, I always end up with a dashboard full of red flags, and most of the services won't work. For some reason, I keep getting 502 Bad Gateway for 5-10 minutes when I try to connect to the url above, followed by the dashboard which is clearly full of issues as shown below: Let's say I want to access hive view:This is what I actually get:So, I tried appending the following to /private/etc/hosts127.0.0.1 sandbox-hdp.hortonworks.combut the problem persists and I am unable to determine what causes this weird behavior since there are too many problems.

Attempting to add a client node to cluster via Ambari (v2.7.3.0) (HDP 3.1.0.0-78) and seeing odd error stderr: Traceback (most recent call last😞 File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 38, in <module> BeforeAnyHook().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 31, in hook setup_users() File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py", line 51, in setup_users fetch_nonlocal_groups = params.fetch_nonlocal_groups, File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124

Hi After installing HDP_3.0.1_virtualbox_181205.ova on Oracle Virtual Box, I get the window with the Ambari URL and the SSH details. However, the user/password admin/admin is not working and the SSH (with Putty) is giving me a connection error as well. I appreciate any help, Regards,Tamir 

Hello experts, I opened a support case and got a run around with many links to all kinds of generic documentation . We have 7.1.6 with Cloudera Manager 7.3.1 and we need to upgrade to 7.1.7 and Cloudera manager 7.4.xto latest and we are all beginners in Cloudera and Hadoop . So looking for some better documentation than https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-cdh/topics/cdpdc-pre-upgrade.html I am thinking from a high level stepsa) Get RPM/bits from archive.cloudera.comb)Shutdown agents and CMc)Take backup per https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-cdh/topics/ug_cm_upgrade_backup.htmld) Upgrade Agentsc) Upgrade Cloudera ManagerThen how do you exactly upgrade CDP from 7.1.6 to 7.1.7 runtime ? I couldnt really find step by step document with screenshot etc . I found this https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-cdh/topics/ug_cdh_upgrading_top.html but doesnt really tell you how

Hi All,   I have to display the records of csv file over mail body in tabular format using Putmail.   I am extracting records by executing sql which is scheduled at some duration and than forwarding the same using putmail to some specific users.   Can someone please tell me how to display the data of generated csv file in tabular format.    

Hello,I am facing an issue with MarkLogic Connector, some times the connection hangs for ever while writing records to Marklogic and we had to restart nifi server to resolve it. Is there a way to set timeout for the connection so that it should continue processing next records with new connection? It should retry or give up, we should not be expected to restart the server. are there any connection properties to handle this, i do not find any in the documentation ?We are using NiFi Version 1.12.1 of Docker image and below Marklogic connectors.DefaultMarkLogicDatabaseClientService 1.9.1.7PutMarkLogic 1.9.1.7

I need to load the CSV data into hive table but i am facing issues with embedded double quotes in few column values as well embedded commas in other columns .Because of this, wherever embedded double quotes and embedded commas are occured , the data from there not loading properly and filled with nulls.I have tried using below openCSV serde options and all. But no luck! ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'WITH SERDEPROPERTIES ("separatorChar" = "\t","quoteChar" = "'","escapeChar" = "\\")theCSV data looks like thisid,name,description,role"1","peter","He is Data enginer"", "Senior Engineer""2","Anee","Hadoop Engineer"","Lead""3","James","Data, Architect"","Sr Architect" hive data should look like this after loading  IdNameDescription    Role1peterHe is Data EngineerSenior Engineer2aneeHadoop EngineerLead3jamesData, ArchitectSr Architect 

Hi all, I'm trying to connect NiFi (installed in my windows host) to HDP sandbox 3.0.1 (vmware) , but I have some trouble. For example if i try to use ListHDFS processor, it works, but if I try to use FetchHDFS it doesn't work, I got the error: ERROR [Timer-Driven Process Thread-7] o.a.nifi.processors.hadoop.FetchHDFS FetchHDFS[id=32589ddd-017c-1000-5c99-2e601313c742] Failed to process session due to org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from org.apache.hadoop.hdfs.client.HdfsDataInputStream@2dec7fd9: org.apache.hadoop.hdfs.DFSInputStream@7eb12001 for StandardFlowFileRecord[uuid=8a5cd0d3-8a89-43e7-a3af-7f369a8e8412,claim=,offset=0,name=00000_0,size=0] due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1419118625-172.17.0.2-1543512323726:blk_1073742033_1212 file=<path>; Processor Administratively Yield

I'm trying to use NiFi expression language with a custom property on the ExtractText processor. It appears that the ExtractText processor interprets the value literally and does not interpolate the value. Is there a way to achieve this?    java.util.regex.PatternSyntaxException: Illegal repetition near index 2${start}^at java.base/java.util.regex.Pattern.error(Pattern.java:2028)at java.base/java.util.regex.Pattern.closure(Pattern.java:3374)at java.base/java.util.regex.Pattern.sequence(Pattern.java:2229)at java.base/java.util.regex.Pattern.expr(Pattern.java:2069)at java.base/java.util.regex.Pattern.compile(Pattern.java:1783)at java.base/java.util.regex.Pattern.<init>(Pattern.java:1430)at java.base/java.util.regex.Pattern.compile(Pattern.java:1095)at org.apache.nifi.processors.standard.ExtractText.onScheduled(ExtractText.java:366)at jdk.internal.reflect.GeneratedMethodAccessor483.invoke(Unknown Source)at java.base/jdk.internal.reflect.DelegatingMethodAccessorImp

I'm trying to read tables and columns and columns' type from Metastore DB. I checked "COLUMN_V2" table and it doesn't have TBL_ID, but it has CD_ID ! Then, I released that there is a table named CDS having one column which is "CD_ID" and the values are as the same as TBL_ID in TBLS table. I want to know if below query using CD_ID is a reliable query or not.Select tbls.TBL_NAME, cv2.COLUMN_NAME,cv2.TYPE_NAME from TBLS tblsJOIN COLUMNS_V2 cv2 on tbls.TBL_ID = cv2.CD_ID And I would be grateful if I can know more about below concerns:1. What is the purpose of CDS table?2. why TBL_ID was not enough and CD_ID was needed?3. What does "CD" stands for?4. Does CD_ID has the same value as TBL_ID all the time? or there are scenarios that they can be different? 

Dear all,  I'm installing CDP private cloud base and i have installed Hive, Hive on Tez, Impala and Kudu services on the cluster But for my use case i don't want impala to see hive database because i will use impala to work with kudu and i will use hive on tez to work with hive. So, how can i separate impala from hive ?  

Dear Team, can you please advise why varchar is not acceptable when replicating from Oracle to Kudu though I can create table with varchar on Kudu? Error from our CDC tool.Error while writing batch on table impala::cbs.cb_ad_x Reason account_type_v isn't [Type: string], it's varchar. Cause: account_type_v isn't [Type: string], it's varchar CREATE TABLE cbs.cb_address_xm (addressid varchar(65535) ,account_link_code_n BIGINT ,account_type_v varchar(65535) ,address_type_n INT ,account_type varchar(65535) , address_format varchar(65535) ,address_type INT , building varchar(65535) , city varchar(65535) ,city_desc varchar(65535) , country varchar(65535) , country_desc varchar(65535) ,district varchar(65535) , district_desc varchar(65535) , floor varchar(65535) , landmark varchar(65535) , postal_code varchar(65535) , po_code varchar(65535) , street varchar(65535) , street_desc varchar(65535) , sub_locality_code varchar(65535) , sub_locality_desc varchar(65535) , op_insert_date

Hello,   I started a test CDP 7.1.7 cluster, added some files in HDFS and created tables in Hive. However Atlas is empty. I expected to find at least some hive_table and hdfs_path items.   The result is the same if I use a custom user (with Atlas ADMIN profile) or with the predefined admin user.   No error is displayed.   Is there something I was supposed to do I missed ? What is my next action to populate Atlas or to find what is missing ?   Thank you, Jérôme

Dears,How send data from nifi cluster to 3 minifi ( same config ) "Data" -> NiFi Cluster -> RPG ->  MiNiFi 1, MiNiFi 2, MiNiFi 3 (receive same "Data" in 3 minifi) Thank you in advance.Best Regards,

Hi Team,How do we identify hive metastore performance and which tables/dbs causing more pressure on Hivemetastore and number of connections on HMS? Regards,naresh.B

从MQTT接收到的数据，要读入json数组Spoiler (Highlight to read){"device_name":"Device37","device_id":"GanghuaLianXian","date":"2021-09-24 17:45:44","sensor":[{"sensor_id":"Data05_37_alarm","value ":"0"},{"sensor_id":"Data06_37_alarm","value":"0"},{"sensor_id":"Data07_37_alarm","value":"0"}, {"sensor_id":"Data08_37_alarm ","value":"0"},{"sensor_id":"Data09_37_alarm","value":"0"},{"sensor_id":"Data10_37_alarm","value":"0"},{"sensor_id ":"Data11_37_alarm","value":"0"}, {"sensor_id":"Data12_37_alarm","value":"0"},{"sensor_id":"Data13_37_alarm","value":"0"},{"sensor_id":"Data14_37_alarm","value":"0"},{"sensor_id":"Data15_37_alarm", "value":"0"}, {"sensor_id":"Data16_37_alarm","value":"0"},{"sensor_id":"Data17_37_alarm","value":"0"},{"sensor_id": " Data18_37_alarm","value":"0"},{"sensor_id":"Data19_37_alarm","value":"0"},"0"},Data15_37_alarm","value":"0"}, {"sensor_id":"Data16_37_alarm","value":"0"},{"sensor_id":"Data17_37_alarm","value":"0"},{" sensor_id":"Data18_37_alarm","value":"0"},{"sensor_id":"Data1

Hi,   I am using CM 6.3.1 with CFM. As I am exploring NIFI -> ControllerStatusReportingTask and I wanted to log the output to a log file. I realised the logback.xml is auto generated by Cloudera Manager, how am I going to add customize logback configuration for the ControllerStatusReportingTask into the logback.xml?   From the CM, there is a NIFI Node XML Override (logback_safety_valve). Should I use this? It seems that, the input require a complete logback XML file. Just wanted to be sure of the approach as I will be upgrading on my production system.  

Hello Team,I need help in geting the below issue fixed.I have ftp server and in the ftp server i have a path where we will keep our daily files in a folder Path: /user/Mahesh/testFolder under the path : 202100923,20210924 Now i need to transfer the files and the folders and place them in s3 bucket How do i get the folder name. Since i have so many folders do the below command work in update attribute processor ${absolute.path:getDelimitedField('4','/')} please advise any other way to extract folder name. Remember this folder name changes daily. and  can have  multiple folders also at a time.   

Hi, I have downloaded the latest version of Nifi 1.13.2. In one of my flows Im using an Execute script processor that utilizes python engine  which I know it was working without problems in an older (version 1.11.4). I noticed when the processor executes for the first time it works then if you stop it and re run again it hangs and the flow files get stuck in the upstream queue. Initially I thought something in my script  is causing the issue  so I change it to something as simple as this flowFile = session.get() if flowFile != None:     # All processing code starts at this indent     session.transfer(flowFile, REL_SUCCESS)Even with this simple script the processor hangs and the flow files remain in the queue. This is a major issue and its causing me not to upgrade. Not sure if you are aware of it and if if there is anything that can be done to mitigate. please advise.