Q&A

Hello, my team is considering upgrading our source oracle to either version 12c or 19c, so we want to see if anyone knows if these versions are fully compatible with Nifi, or if there're any known issues? Any info is greatly appreciated. Thanks!!!

Hi ,   We are facing issue while connecting phoenix-sqlline command line. Getting error: Caused by: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region SYSTEM.CATALOG,,1584547942548.9dca4950cba77ee68c34f17fdb7867b9. is not online on xxxxxxxxxxxxxxxxxxx,16020,1596706210559 From the HBase UI, I could see two regions server are present at xxx servers.     Please give a solution ASAP.   Thanks,  

Hi Experts,We are currently in middle of upgrade from HDP 2.6.5 to HDP 3.1.5. Currently we are just before finalizing the upgrade state.As instructed we wanted to make sure everything is fine before we finalize the upgrade.However when we checked we found that the HDFS namenode is not starting. We have HA configuration for our HDFS and both the ZKFAILOVERCONTROLLER is running but HDFS namenode or standby namenode is not starting and it keep on trying without any success.I am attaching the logs and screenshots for your analysis.Please help.Thanks and Regards,Ananya

Hello I have some cloudera installation experience including production clusters of CDH 5 and 6.I once installed the CDP 7.0.3 trial version using the automatic installation script and it was good. Now I wanted to see how the latestCDP 7.1.1 looks like. So I followed the instructions and downloaded cloudera-manager-installer.bin script.Then I prepared several virtual CentOs 7.6 hosts with OpenJDK 8 and SELinux turned off and ran the script on one of them.It was running fine for a while, then after installing the embedded database it halted for a long time.The installation logs looked like this:-rw-r--r-- 1 root root 0 Jun 12 16:16 0.check-selinux.log-rw-r--r-- 1 root root 0 Jun 12 16:16 1.install-repo-pkg.log-rw-r--r-- 1 root root 1843 Jun 12 16:20 2.install-openjdk8.log-rw-r--r-- 1 root root 2205 Jun 12 17:03 3.install-cloudera-manager-server.log-rw-r--r-- 1 root root 22355 Jun 12 17:03 4.check-for-systemd.log-rw-r--r-- 1 root root 3824 Jun 12 17:03 5.install-cloudera-manager-ser

Hello, We are facing a problem in our production environment. The Hbase shell command "remove_peer '<peer>'" is not working.  The peer was previsously disabled ( took a very very long time to get back to the prompt ).In our others environment, removing a peer was ok. Any ideas of the problem and how to fix it?Thanks for any help. RegardsDaniel 

Hi Team, I want the list of all running yarn/spark applications along with the actual user name for the queries running with user as hive. 1) I was aware that we can use yarn application -list to get the list of running applications and 2) hive.access.subject.name in the configuration page of yarn history server/resource manager. But for point#2, I need to manually check it for each application which is not a viable solution.So, I was trying to check if there is any API available for YARN & SPARK history server to get the actual user of the query/application.(OR)It would great if we have any command to get the actual user name for a running application. Kindly let me know if any additional information required.

Jupiter is working fine in cluster but not able to login with Default usename and passord.admin/admin.  Please advise.  

I'm trying to test Sqoop action in the following environment:CDP 7.1.3  (new instalation)Sqoop 1.4.7Via the command line, the following sqoop command works:sudo -su hdfs sqoop import "-Dorg.apache.sqoop.splitter.allow_text_splitter = true" --connect jdbc:oracle:thin:@XXXXXXXXXXXX:1521/orcl --username XXXXX --password XXXX --table "HR.JOBS" --delete-target-dir --as-parquetfile --null-string '\\N' --null-non-string '\\N' --hive-import --create-hive-table --hive-table default.jobs Sqoop connects in Oracle Database and write parquet files into HDFS.  But when try load data into Hive. getting this error: Import failed: java.io.IOException: Hive exited with status 4: 20/09/30 23:32:03 INFO hive.HiveImport: Loading uploaded data into Hive20/09/30 23:32:03 INFO hive.HiveImport: Cannot find hadoop installation: $HADOOP_HOME or $HADOOP_PREFIX must be set or hadoop must be in the path20/09/30 23:32:03 ERROR tool.ImportTool: Import failed: java.io.IOException: Hive ex

Hi there, What warnings there are around upgrading hdp 3.1.4 to hdp 3.1.5we have already installed:hdfs 3.1.1yarn 3.1.1mapreduce 3.1.1tez 0.9.1hive 3.1.0sqoop 1.4.7oozie 4.3.1zookeeper 3.4.6spark2 2.3.2 Our principal goal is to solve hive - spark metadata issue. Regards! 

Hello everyone,first at all, i am kind of newbie with nifi so feel free to change everywhting you see.I am ussing JoltTransformJSON. My input JSON is the next one:{"ID": "123","Text1": "aaa","Text2": "aaa","Text3": "aaa"}And i need the next output:{"data": {"ID": "123","Text1": "aaa","Text2": "aaa","Text3": "aaa"},"date": "","dataset": "","ID": "123"}The var date and dataset are attributes from the flow, so there is no problem here, the problem is i need to extract the value of ID, in this case 123 but this is just an example, without deleted the ID inside de field data. My Jolt Specification is the next one:[{"operation": "shift","spec": {"*": "data.&"}}, {"operation": "default","spec": {"dataset": "${dataset:toLower()}","date": "${date}"}}]and with that i have all EXCEPT the ID field, that i dont know how to do it :(.Thankss

In the picture I attached to this post you can see my current log level value. This does not work. In /var/log/hadoop/hdfs/hadoop-hdfs-namenode-sandbox-hdp.hortonworks.com.log I can only see INFO and WARN messages.

Hi, I'm planning to upgrade my CM and CDH 5.8 to CM, CDH 6.3 but my Rhel OS Version is just 7.2, planning also to upgrade it to 7.4. just want to know if, what steps need to made? do I need to upgrade the OS first then upgrade my cloudera? or I'll upgrade the OS then uninstall cloudera, then after the OS Upgrade, i'll reinstall cloudera?

I have some JSON that contains two arrays that I would like to convert to key/value pairs. I've already create a spec to convert my input data into the format below. Input   { "rows" : [ { "row" : [ "row1", "row2", "row3" ], "header" : [ "header1", "header2", "header3" ] }, { "row" : [ "row4", "row5", "row6" ], "header" : [ "header1", "header2", "header3" ] } ]}    Is it possible to convert it again with Jolt to something like:    {"header1" : "row1","header2" : "row2","header3" : "row3","header1" : "row4","header2" : "row5","header3" : "row6"}    Any guidance would be highly appreciated 🙂

I want to populate my target JSON with a field based on value of another field. e.g. Input JSON:[ {"examUrl": "https://exam.test.com/page/1473161074","urlTypeName": "Spring URL"}, {"examUrl": "https://exam.test.com/page/12423222","urlTypeName": "Java URL"}, ]   Expected Output: { "ExamDashBoardURL" : "https://exam.test.com/page/1473161074" } In the above example, I will populate ExamDashBoardURL only if urlTypeName is "Spring URL". Can anyone please help me on this. Thanks and Regards, Biswa    

Json Input:[{"BRANCH_CODE": "7Z3","CUST_AC_NO": "Z256VM3K6872CY1F8","AC_DESC": "437T7F97CET","CUST_NO": "RX1"},{"BRANCH_CODE": "7Z3","CUST_AC_NO": "Z256VM3K6872CY1F8","AC_DESC": "437T7F97CET","CUST_NO": "RX1"}] Expected Output:[ {"ExternalSystemIdentifier" : "Z256VM3K6872CY1F8","Fields" : {"FLD0001" : "7Z3","FLD0002" : "RX1","FLD0003" : "437T7F97CET"},"InstitutionId" : "1"}, {"ExternalSystemIdentifier" : "Z256VM3K6872CY1F8","Fields" : {"FLD0001" : "7Z3","FLD0002" : "RX1","FLD0003" : "437T7F97CET"},"InstitutionId" : "1"} ] Jolt Spec:[{"operation": "shift","spec": {"*": {"BRANCH_CODE": "[&1].FLD0001","CUST_NO": "[&1].FLD0002","AC_DESC": "[&1].FLD0003"}}},{"operation": "shift","spec": {"*": {"$": "[#2].ExternalSystemIdentifier","@": "[#2].Fields"}}},{"operation": "default","spec": {"*": {"InstitutionId": "1"}}}]

Using:HDP 3.0.1HDFS 3.1.0 NAMENODE HEAP: 84.8% 3.3 GB / 4.0 GBDISK USAGE (DFS USED): 71.20% 63.9 TB / 89.7 TBDISK USAGE (NON DFS USED): 0.74% 676.2 GB / 89.7 TBDISK REMAINING: 28.06% 25.2 TB / 89.7 TBBlock Size : 128 MB Any idea how to solve to reduce reduce the name node heap size usage?Thank you

I'm usingGetDateandServer processor to fetch file names ,decompress the files ,remove header using executestreamcommand Process  Command arguments- 1dCommand path- sed IgnoreSTDIN_ False​  MergeContent Processor to merge files   Merge strategy: Bin-Packing algorithm Merge format: Bin concatenation Merge data strategy: Do not merge uncommon metadata Min no of entries: 180 Max no of entries: 1000 Minimum Group Size: 60GB Max Bin age: 5 min Max no of bins: 1  UpdateAttribute to Create a files name Compress filesputHDFS-To put data into HDFSMy problem is after ExecuteStreamcommand Processor is triggered in the queue I could see two positions with same valueFor every half hour I should get 1 output file but I could see 2 files as output. NIFi is running on 3 nodes, and list queue on 2 nodes. Files running on node1 is merged as 1 file and files running on node 2 is fetched as 2nd file so I'm getting 2 files. Could you please let me

We are trying to use hbase with adls as a storage layer. Is this still possible in cdh 6.1? This is described in the docs for cdh 5.12 : https://docs.cloudera.com/documentation/enterprise/5-12-x/topics/admin_using_adls_storage_with_hbase.html#hbase_adls_configuration_steps but we are using cdh 6.1 and there I don't see a similar page anymore.    Off course I ask since I have an issue following the 5.12 docs.When I want to change in Cloudera Manager in the hbase configuration,    hbase.root.dir = adl://ourproject.azuredatalakestore.net/hbase   I can not save this method as I get an error saying the variable is not in the allowed format:   HDFS Root Directory: Path adls://ourproject.azuredatalakestore.net/hbase does not conform to the pattern "(/[-+=_.a-zA-Z0-9]+)+(/)*".    

Hello All, We would like to install a cluster CDP 7.1 on our DEV environment composed of 8 nodes :3 Masters3 Slaves1 Edge1 Cloudera-manager What are the best practices for distributing the services (Spark,Hbase,Zookeeper...) on the different nodes of the cluster? Thanks for your feedback.

Hi All, When we Load data into Hive table from HDFS, it deletes the file from source directory(HDFS) is there a way we can keep the file in the source directory and load the data into hive table as well. I used the below query; LOAD DATA INPATH 'source_file_path' OVERWRITE INTO TABLE TABLENAME;

Hello, I'm using Ubuntu 16 and Cloudera 6.2.1. I need to startup a couple other agents only after CDH HDFS has completely started upon OS start, which can take several minutes. I am unsure how to do this. Anyone have any pointers?   Thank you.

Hello All, I could see error messages being displayed every minute on top most corner of Processor groups.The errors being displayed is regarding Processor FETCHSFTP trying to fetch an old file which doesn't exist in source, I don't want that error to be displayed. Could you please let me know what settings to be changed. Thank you in advance

Hello All and @Shu_ashu,@MattWhoI have a CSV file with a time column "Apr 15, 2020 02:02:16.388267000 Paris, Madrid (heure d’été)".... "Apr 15, 2020 02:02:16.986879000 Paris, Madrid (heure d’été)"...etc.Then I to convert the field time using UpdateRecord processor by the following code:${field.value:substring(0,28):toDate("MMM dd,yyyy HH:mm:ss.SSSSSS"):format("yyyy-MM-dd HH:mm:ss.SSSSSS")}But after conversion, the output I get is:nbtimedelta12020-04-15 02:08:44.0002670.022020-04-15 02:18:42.0008790.598612 But the output should be:nbtimedelta1Apr 15, 2020 02:02:16.3882670.02Apr 15, 2020 02:02:16.9868790.598612 So, I always get 000 before the milliseconds. As per the example above, I got 000267 instead of 388267 and 000879 instead of 986879.Why I am getting the 000 instead of the real value? How can I modify me code to resolve this problem? Any help on this issue will be highly appreciated.

How should I install impala-udf-devel package? Is it installed along with another services?

Hi,i read that usualy the result should be ready in few hours, but i am still waiting for.My exam was yestarday on 3 May 2020 at 10:00 a.m CEST, more then 1 day passed. Just to be informed, it is normal?  If not, what i should do?  Thanks in advance.