Q&A

Hi all I am absolute beginner in apache nifi as well as in DWH field, I want to create replica of my database table, using nifi. I am taking Mysql table data using QueryDatabaseTableRecord processor. Now I want this data to be inserted into another mysql database table which I have created. I was trying using the putSQL processor but that didn't worked I got no record inside of my table.I tried things which I could think of myself, but that was not enough. Please help me !!!Here is the screenshot of my flow,

Hello, I'm trying to use a ODBC connexion with SAP BI 4.2 app, on linux environment. Here are my settings :  --------- odbc.ini file -----------------[ODBC Data Sources]Impala=Simba Impala ODBC Driver (64-bit)[Impala_TEST]Description=Simba Impala ODBC Driver (64-bit)Driver=/data/BO/sap_bobj/enterprise_xi40/linux_x64/odbc/simbaimpala/2.0/lib/libimpalaodbc_sb64.soHiveServerType=2ServiceDiscoveryMode=0Host=portalgps.idf.fr.gin.intraorangePort=443schema=defaultAuthMech=4KrbFQDN=_HOSTUID=hiveThriftTransport=2HttpPath=/knox/cdp-proxy-api/hiveServiceDiscoveryMode=0AllowSelfSignedServerCert=1UseNativeQuery=1SSL=1 AllowSelfSignedCerts=1CAIssuedCertNamesMismatch=1TrustedCerts=/data/BO/sap_bobj/enterprise_xi40/linux_x64/odbc/simbaimpala/2.0/lib/cacerts.pemTwoWaySSL=0   --------- odbcinst.ini -----------------[ODBC Drivers]Simba Impala ODBC Driver 64-bit=InstalledImpala_TEST=Installed[Simba Impala ODBC Driver 64-bit]Description=Simba Impala ODBC Driver (64-b

Hi, I am using nifi 1.14. My entire workflow has been crashed and suddenly Nifi is showing blank canvas. Is there any safe way to recover all the jobs? Please provide step by step solution. Any help in this issue will be greatly appreciated.Thank You!

Hello, everyone,I'm just starting to get to grips with NiFi and I'm developing a flow that involves taking csv files from an SFTP server and transferring them locally.The task itself seems very simple to me, I found that a viable solution would be to use ListSFTP and FetchSFTP for the purpose. My problem lies in the connection to the SFTP server and in particular in passing the 'Private Key Path' parameter. My private key is stored in a .ppk file on the virtual machine I am using, which uses UNC paths. The moment I pass the UNC path to the processor with the pointing to the .ppk file, the processor returns an alert, telling me that the file does not exist. I can't figure out if the problem is in the way I passed the path to the processor or if it's something else.The path I passed to the processor looks like this:\\computer_name\View_UserData$\user_name\Desktop\private_key.ppk In addition, I have set the Port parameter to 22, and the connection also requires a Password.Can anyone

Hi experts: We are on the version Cloudera CDH-6.3.4. When users exit from login nodes, we see the error appearing in /var/log/messages from time to time: Sep 23 15:06:37 hlog-2 cm: Process Process-10786:Sep 23 15:06:37 hlog-2 cm: Traceback (most recent call last):Sep 23 15:06:37 hlog-2 cm: File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrapSep 23 15:06:37 hlog-2 cm: self.run()Sep 23 15:06:37 hlog-2 cm: File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in runSep 23 15:06:37 hlog-2 cm: self._target(*self._args, **self._kwargs)Sep 23 15:06:37 hlog-2 cm: File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/host/filesystem_map.py", line 27, in disk_usage_wrapperSep 23 15:06:37 hlog-2 cm: usage = psutil.disk_usage(p)Sep 23 15:06:37 hlog-2 cm: File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/psutil/__init__.py", line 1947, in disk_usageSep 23 15:06:37 hlog-2 cm: return _psplatform.disk_usage(path)Sep 23 15:06:

I am working off of a 3-node NiFi cluster and that is kicked off by a GenerateFlowfileProcessor run on the primary node, performs some NiFi processing, and then writes the files to the server that I will then run an ExecuteStreamCommand Python script on. The problem I’m running into is I can’t figure out a way to ensure that the processors picking up the first output are run on the same node as the processors that produced the first output. What is the best way to handle producing files that can be accessed by all nodes?Is there a way to specify the node for a process will be run on? (using “run on primary” is not working as the primary node cycles over the process)

Hi, I would like to create the access privileges in database and table level.   So, what I had performed is:   1. Create user in hue (Manage User) a) testuser1 (non-sentry admin group), assigned under "testgroup" b) admin (Sentry-admin-group), assigned under "admingroup"   2. Grant role to the user "testuser1": create role role_test; grant all on server server1 to role role_test; grant role role_test to group testgroup;   However, when login in as "testuser1" in hue, I can't see any database. And check via query "show current role" and no record is returned.    Is there any steps I have missed?  Thank you!    

Hi Team, following the resolution thread Re: How to report NiFi lineage to Atlas I am now able to use the Atlas lib of Nifi called "ReportLineageToAtlas". I configured the settings as the following (Atlas URLs is changing day by day, after the restart of the servers) : In Kerberos Credential Service I inserted Atlas Kerberos Keytab and Atlas Kerberos Principal.The error I receive while the "ReportLineageToAtlas" is running is about Authentication to Atlas to reach the API : Indeed if I open the Atlas web URL : http://34.134.205.221:31000/api/atlas/v2/types/typedefs?name=nifi_output_port  I have no data.Therefore we have two issues:1) it seems there's an authentication issue from Atlas towards Nifi API.2) it seems in the API there are no data named "nifi_output_port" (maybe because it's not yet able to authenticate?). Maybe the two issues are linked.Do you have any hints about how to fix this problem? Were am I wrong in the settings? I can pr

We have a development cluster consists of CDP 7.1.6 with Nifi 1.13.2. The enviromment is Kerberised and use Ranger for the security, and connected to the AD server through sssd service.We already setup the Flows policies on the ranger, by defining user and group that exisiting in the ranger.When we try to access the Nifi UI, for user which define on Users of Flows policies, it works properly. But when login using user that define on the Groups of Flows policies, it gives error message: “INFO [NiFi Web Server-225822] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[18330301], groups[] does not have permission to access the requested resource. Unable to view the user interface. Returning Forbidden response.” Nifi could not identified this user group.Could you please give advice how to resolve this issues, so Nifi could give access to this user, which define in the Groups of Flow policies. Thank you in advance for you help.

I have installed CDSW in my cluster and created a project. But while launching the session I see the following error.  I'm adding the details of 'cdsw status' here, [root@edge ~]# cdsw statusSending detailed logs to [/tmp/cdsw_status_mrI87T.log] ...CDSW Version: [1.9.1.10118148:81531c5]Installed into namespace 'default'OK: Application running as root checkOK: NFS service checkOK: System process check for CSD installOK: Sysctl params checkOK: Kernel memory slabs check-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| NAME | STATUS | CREATED-AT | VERSION | EXTERNAL-IP | OS-IMAGE | KERNEL-VERSION | GPU | STATEFUL |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

I have installed CDH on RHEL 7.9 . The cloudera installation files are located in  - /opt/cloudera/ java files are in /usr/java/jdk1.8.0_181-cloudera   I am trying to use apis in libhdfs.so from my application -  when I launch my application through command line I get the below error -    could not find method getRootCauseMessage from class (null) with signature (Ljava/lang/Throwable;)Ljava/lang/String;could not find method getStackTrace from class (null) with signature (Ljava/lang/Throwable;)Ljava/lang/String;FileSystem: loadFileSystems failed error:(unable to get root cause for java.lang.NoClassDefFoundError)(unable to get stack trace for java.lang.NoClassDefFoundError)# A fatal error has been detected by the Java Runtime Environment:# SIGSEGV (0xb) at pc=0x00007f21231904f1, pid=28247, tid=0x00007f1f5e7fc700# JRE version: Java(TM) SE Runtime Environment (8.0_181-b13) (build 1.8.0_181-b13)# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode linu

Hi All,I have Nifi 1.14.0 currently installed and I would like to connect this to Apache Atlas available in Cloudera Cluster.I read here Configure NiFi for Atlas Integration (cloudera.com) there should be the possibility to do this by configuring ReportLineageToAtlas task in NiFi. Unfortunately this option is missing in my NiFi installation. Do you have any hints about how to add this feature in my NiFi Centos installation? Thanks a lot for your support.Best Regards,Daniele.

We have an hdp cluster 2.6.5, hive service is installed with HA3 Hive server3 Metastores1 HiveServer2 Interactive1 WebHCat Server We are receiving "memory high usage" alerts from our monitoring tool, when i check the memory consumption on that nodes i can see that hive is consuming more than 80% of memory node When memory usage reach 98%, the hive server crash with the following error message[root@mnode4 hive]# head -n 20 hs_err_pid27508.log## There is insufficient memory for the Java Runtime Environment to continue.# Native memory allocation (mmap) failed to map 1732247552 bytes for committing reserved memory.# Possible reasons:# The system is out of physical RAM or swap space# In 32 bit mode, the process size limit was hit# Possible solutions:# Reduce memory load on the system# Increase physical memory or swap space# Check if swap backing store is full# Use 64 bit Java on a 64 bit OS# Decrease Java heap size (-Xmx/-Xms)# Decrease number of Java threads# De

What is the suggested method to extract data from SQL server to hdfs? Is there any performance advantage in using spark jdbc over sqoop import ?Also as per apache.org, apache sqoop project is retired as of June 2021, Is it better to go with the Spark approach in new ETL pipelines developments?

Dears, one of my kafka node's one of the disk is got full. Could you please suggest steps to make sure the 4 disks of my kafka node get sync in terms of disk occupancy. Ref Snapshot is below   Swift response are highly appreciated  

I have a cluster that uses a Postgres db with version 9.4.17 installed on a host in the cluster and I want to migrate to an external Oracle db. I have researched about the methods to do this in Cloudera documentations and I found that I have to have a Cloudera version higher than 5.15. Would it make any difference if we do the migration using Cloudera 5.13? what could be the differences between 5.13 and 5.15 in terms of db? """Configuration uses Cloudera Manager 5.15.0 or later on supported platforms."""https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ag_migrate_postgres_db_to_oracle_mysql.html  

Hi, I am using CFM 1.0.1.7, when I am using the remote process group to transfer file between two cluster. There is an error message "Failed to persist link of Peer due to java.io.FileNotFoundException:./conf/state/xxxxxxxxxxx_HTTP.peer (No such file or directory); If restarted and peer's NCM is down, may be unable to transfer data unitil communuication with NCM is restored. How shall I proceed with this error. The NIFI version is 1.9.0 and according to the online documentation, NCM is only avaliable for NIFI 0.X, what am I seeing this error?

I am unable to get sandbox to work without problems on my mac. I have limited memory 8 GB therefore, I installed 2.6.5 which I think may require less than the 10 GB required by 3.0.1. After I install and run the vbox .ova file, I get the url which I use to connect to Ambari dashboard.  No matter what I do, I always end up with a dashboard full of red flags, and most of the services won't work. For some reason, I keep getting 502 Bad Gateway for 5-10 minutes when I try to connect to the url above, followed by the dashboard which is clearly full of issues as shown below: Let's say I want to access hive view:This is what I actually get:So, I tried appending the following to /private/etc/hosts127.0.0.1 sandbox-hdp.hortonworks.combut the problem persists and I am unable to determine what causes this weird behavior since there are too many problems.

Attempting to add a client node to cluster via Ambari (v2.7.3.0) (HDP 3.1.0.0-78) and seeing odd error stderr: Traceback (most recent call last😞 File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 38, in <module> BeforeAnyHook().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 31, in hook setup_users() File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py", line 51, in setup_users fetch_nonlocal_groups = params.fetch_nonlocal_groups, File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124

Hi After installing HDP_3.0.1_virtualbox_181205.ova on Oracle Virtual Box, I get the window with the Ambari URL and the SSH details. However, the user/password admin/admin is not working and the SSH (with Putty) is giving me a connection error as well. I appreciate any help, Regards,Tamir 

Hello experts, I opened a support case and got a run around with many links to all kinds of generic documentation . We have 7.1.6 with Cloudera Manager 7.3.1 and we need to upgrade to 7.1.7 and Cloudera manager 7.4.xto latest and we are all beginners in Cloudera and Hadoop . So looking for some better documentation than https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-cdh/topics/cdpdc-pre-upgrade.html I am thinking from a high level stepsa) Get RPM/bits from archive.cloudera.comb)Shutdown agents and CMc)Take backup per https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-cdh/topics/ug_cm_upgrade_backup.htmld) Upgrade Agentsc) Upgrade Cloudera ManagerThen how do you exactly upgrade CDP from 7.1.6 to 7.1.7 runtime ? I couldnt really find step by step document with screenshot etc . I found this https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/upgrade-cdh/topics/ug_cdh_upgrading_top.html but doesnt really tell you how

Hi All,   I have to display the records of csv file over mail body in tabular format using Putmail.   I am extracting records by executing sql which is scheduled at some duration and than forwarding the same using putmail to some specific users.   Can someone please tell me how to display the data of generated csv file in tabular format.    

Hello,I am facing an issue with MarkLogic Connector, some times the connection hangs for ever while writing records to Marklogic and we had to restart nifi server to resolve it. Is there a way to set timeout for the connection so that it should continue processing next records with new connection? It should retry or give up, we should not be expected to restart the server. are there any connection properties to handle this, i do not find any in the documentation ?We are using NiFi Version 1.12.1 of Docker image and below Marklogic connectors.DefaultMarkLogicDatabaseClientService 1.9.1.7PutMarkLogic 1.9.1.7

I need to load the CSV data into hive table but i am facing issues with embedded double quotes in few column values as well embedded commas in other columns .Because of this, wherever embedded double quotes and embedded commas are occured , the data from there not loading properly and filled with nulls.I have tried using below openCSV serde options and all. But no luck! ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'WITH SERDEPROPERTIES ("separatorChar" = "\t","quoteChar" = "'","escapeChar" = "\\")theCSV data looks like thisid,name,description,role"1","peter","He is Data enginer"", "Senior Engineer""2","Anee","Hadoop Engineer"","Lead""3","James","Data, Architect"","Sr Architect" hive data should look like this after loading  IdNameDescription    Role1peterHe is Data EngineerSenior Engineer2aneeHadoop EngineerLead3jamesData, ArchitectSr Architect 

Hi all, I'm trying to connect NiFi (installed in my windows host) to HDP sandbox 3.0.1 (vmware) , but I have some trouble. For example if i try to use ListHDFS processor, it works, but if I try to use FetchHDFS it doesn't work, I got the error: ERROR [Timer-Driven Process Thread-7] o.a.nifi.processors.hadoop.FetchHDFS FetchHDFS[id=32589ddd-017c-1000-5c99-2e601313c742] Failed to process session due to org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from org.apache.hadoop.hdfs.client.HdfsDataInputStream@2dec7fd9: org.apache.hadoop.hdfs.DFSInputStream@7eb12001 for StandardFlowFileRecord[uuid=8a5cd0d3-8a89-43e7-a3af-7f369a8e8412,claim=,offset=0,name=00000_0,size=0] due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1419118625-172.17.0.2-1543512323726:blk_1073742033_1212 file=<path>; Processor Administratively Yield