Solved

Hi everyone,I am new to Ambari and have just installed Ambar 2.7.4 with HDP stack 3.1.4 with a cluster having 1 node running the Ambari Server, one being the master node of all service components and there is one more. The installation completed with few warnings, all the services on the web UI are running with green light on. The problem is an alert that says.. SmartSense Bundle Capture FailureLast SamrtSense bundle was not successful. Bundle key.unspecified_xxxxxx_0_2020-10-04_02-00-01, status:failedAlso, the Activity Analyzer stops again and again after restarting it. All other services are running.PS: I have just installed this and haven't loaded any data or anything or tested anything.Please provide help!                                        Thankyou!@jsensharma  @MattWho @Shelton @nsabharwal 

I was curious if it was possible to move flows from one bucket to another and keep the version history in Registry. The way we have our environment set up today is we just have one bucket that we use in our DEV environment, as well as in production. It gets messy and we would like to clean it up. Here is my idea: We have a bucket for each, DEV, Prod, & Archive. We keep versions controlled for the development process, if any, and when its ready to be deployed to another environment we can the move the flow versioning info over to the production bucket. And then if a flow is old but we don't want to delete it, we can then move it over to the archive bucket.  The DEV to Prod bucket is not as important because we frequently are updating things that are in prod through DEV so it wouldn't be necessary to always move it back into the dev bucket since prod still tracks the version, however, the archive portion is important to us. If moving flows from bucket to bucket is

I have installed a an HDF v3.3 cluster. everything seems to be working fine except Schema Registry which starts fine but then shuts down immediately giving the following errors:The mysql drivers have been loaded fine in ambari and work for all other modules.java.lang.RuntimeException: java.lang.ClassNotFoundException: com.mysql.jdbc.jdbc2.optional.MysqlDataSource at com.zaxxer.hikari.util.PoolUtilities.createInstance(PoolUtilities.java:141) at com.zaxxer.hikari.util.PoolUtilities.initializeDataSource(PoolUtilities.java:153) at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:146) at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:113) at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:78) at com.hortonworks.registries.storage.impl.jdbc.connection.HikariCPConnectionBuilder.prepare(HikariCPConnectionBuilder.java:49) at com.hortonworks.registries.storage.impl.jdbc.connection.HikariCPConnect

From this document it seems the query results can be redacted however I don't see the same. Is there a known bug?https://www.hadoopandcloud.com/hadoop/enableconfigure-log-and-query-redaction/ 0: jdbc:hive2://calingita.pheonix.co> select * from pheonix_global.redaction;INFO : Compiling command(queryId=hive_20201005072828_53b14d1f-5f0a-4ce6-894d-debfeade3103): select * from pheonix_global.redaction;INFO : Semantic Analysis CompletedINFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:redaction.id, type:int, comment:null), FieldSchema(name:redaction.email, type:string, comment:null)], properties:null)INFO : Completed compiling command(queryId=hive_20201005072828_53b14d1f-5f0a-4ce6-894d-debfeade3103); Time taken: 0.322 secondsINFO : Executing command(queryId=hive_20201005072828_53b14d1f-5f0a-4ce6-894d-debfeade3103): select * from pheonix_global.redactionINFO : Completed executing command(queryId=hive_20201005072828_53b14d1f-5f0a-4ce6-894d-debfeade3103); Time taken: 0

Hello all, I tried to search Hive related topics but did not find the issue I face so I wanted to ask here. This is CDH 6.3.2 (Parcels) on Linux (CentOS) I have Kafka and Hive and I am able to create external Kafka tables after putting hive-exec-3.1.3000.7.1.3.0-100.jar and kafka-handler-3.1.3000.7.1.3.0-100.jar in Hive under Hive Auxiliary JARs Directory However, when I try to select data from the external table I get this error: 0: jdbc:hive2://localhost:10000/testdb> select * from kafka_table;Error: java.io.IOException: java.io.IOException: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;JJ)V (state=,code=0) After looking for this error I've found that it is most likely related to issues with different versions of guava in Hadoop and Hive: https://issues.apache.org/jira/browse/HIVE-22915 I see that Hadoop is using  /opt/

I'm trying to identify why I am seeing the error given in the subject of this question. The error appears nightly on each host running cloudera-scm-agent (6.2).   I suspect the following: When cron.daily is run, the host monitor appears to start monitoring the disk_usage of /run/user/0. Due to the temporary nature of /run/user/*, when the directory is removed the cm-agent throws 'No such file or directory' errors into /var/log/messages.   Has anyone else encountered these errors? What's the rationale of monitoring /run/user/*? Could this be handled better?   Thanks, Deaks

I'm testing the CDH upgrade from v5.16.1 to v6.3.3. Cloudera manager server v6.3.3 upgrade was done successfully. But I have a trouble in Upgrading the Cloudera Manager Agents section. https://docs.cloudera.com/cdp-private-cloud/latest/upgrade-cdh/topics/ug_cm_upgrade_agent.html I can see The Upgrade Cloudera Manager Agent Packages page. But the Install agents fails due to Failed to copy installation files. I tried both repository location of Cloudera repository and custom repository but the both was failed.   I think inter hosts ssh communications are no problem because I tested ssh login b/w them and I confirmed the /tmp/scm_prepare_node.XXXXXXXX was made on target agent host. I suspects key handling is the cause but I'm not sure how to confirm it. Here's the cloudera-scm-server.log. (modified URL because it's raw username and password)   2020-10-06 14:54:10,500 INFO NodeConfiguratorThread-12-0:com.cloudera.server.cmf.node.NodeConfigurator: Executing mktemp -d /tmp/scm_prep

Hello, my team is considering upgrading our source oracle to either version 12c or 19c, so we want to see if anyone knows if these versions are fully compatible with Nifi, or if there're any known issues? Any info is greatly appreciated. Thanks!!!

Hi ,   We are facing issue while connecting phoenix-sqlline command line. Getting error: Caused by: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region SYSTEM.CATALOG,,1584547942548.9dca4950cba77ee68c34f17fdb7867b9. is not online on xxxxxxxxxxxxxxxxxxx,16020,1596706210559 From the HBase UI, I could see two regions server are present at xxx servers.     Please give a solution ASAP.   Thanks,  

Hi Experts,We are currently in middle of upgrade from HDP 2.6.5 to HDP 3.1.5. Currently we are just before finalizing the upgrade state.As instructed we wanted to make sure everything is fine before we finalize the upgrade.However when we checked we found that the HDFS namenode is not starting. We have HA configuration for our HDFS and both the ZKFAILOVERCONTROLLER is running but HDFS namenode or standby namenode is not starting and it keep on trying without any success.I am attaching the logs and screenshots for your analysis.Please help.Thanks and Regards,Ananya

Hello I have some cloudera installation experience including production clusters of CDH 5 and 6.I once installed the CDP 7.0.3 trial version using the automatic installation script and it was good. Now I wanted to see how the latestCDP 7.1.1 looks like. So I followed the instructions and downloaded cloudera-manager-installer.bin script.Then I prepared several virtual CentOs 7.6 hosts with OpenJDK 8 and SELinux turned off and ran the script on one of them.It was running fine for a while, then after installing the embedded database it halted for a long time.The installation logs looked like this:-rw-r--r-- 1 root root 0 Jun 12 16:16 0.check-selinux.log-rw-r--r-- 1 root root 0 Jun 12 16:16 1.install-repo-pkg.log-rw-r--r-- 1 root root 1843 Jun 12 16:20 2.install-openjdk8.log-rw-r--r-- 1 root root 2205 Jun 12 17:03 3.install-cloudera-manager-server.log-rw-r--r-- 1 root root 22355 Jun 12 17:03 4.check-for-systemd.log-rw-r--r-- 1 root root 3824 Jun 12 17:03 5.install-cloudera-manager-ser

Hello, We are facing a problem in our production environment. The Hbase shell command "remove_peer '<peer>'" is not working.  The peer was previsously disabled ( took a very very long time to get back to the prompt ).In our others environment, removing a peer was ok. Any ideas of the problem and how to fix it?Thanks for any help. RegardsDaniel 

Hi Team, I want the list of all running yarn/spark applications along with the actual user name for the queries running with user as hive. 1) I was aware that we can use yarn application -list to get the list of running applications and 2) hive.access.subject.name in the configuration page of yarn history server/resource manager. But for point#2, I need to manually check it for each application which is not a viable solution.So, I was trying to check if there is any API available for YARN & SPARK history server to get the actual user of the query/application.(OR)It would great if we have any command to get the actual user name for a running application. Kindly let me know if any additional information required.

Jupiter is working fine in cluster but not able to login with Default usename and passord.admin/admin.  Please advise.  

I'm trying to test Sqoop action in the following environment:CDP 7.1.3  (new instalation)Sqoop 1.4.7Via the command line, the following sqoop command works:sudo -su hdfs sqoop import "-Dorg.apache.sqoop.splitter.allow_text_splitter = true" --connect jdbc:oracle:thin:@XXXXXXXXXXXX:1521/orcl --username XXXXX --password XXXX --table "HR.JOBS" --delete-target-dir --as-parquetfile --null-string '\\N' --null-non-string '\\N' --hive-import --create-hive-table --hive-table default.jobs Sqoop connects in Oracle Database and write parquet files into HDFS.  But when try load data into Hive. getting this error: Import failed: java.io.IOException: Hive exited with status 4: 20/09/30 23:32:03 INFO hive.HiveImport: Loading uploaded data into Hive20/09/30 23:32:03 INFO hive.HiveImport: Cannot find hadoop installation: $HADOOP_HOME or $HADOOP_PREFIX must be set or hadoop must be in the path20/09/30 23:32:03 ERROR tool.ImportTool: Import failed: java.io.IOException: Hive ex

Hi there, What warnings there are around upgrading hdp 3.1.4 to hdp 3.1.5we have already installed:hdfs 3.1.1yarn 3.1.1mapreduce 3.1.1tez 0.9.1hive 3.1.0sqoop 1.4.7oozie 4.3.1zookeeper 3.4.6spark2 2.3.2 Our principal goal is to solve hive - spark metadata issue. Regards! 

Hello everyone,first at all, i am kind of newbie with nifi so feel free to change everywhting you see.I am ussing JoltTransformJSON. My input JSON is the next one:{"ID": "123","Text1": "aaa","Text2": "aaa","Text3": "aaa"}And i need the next output:{"data": {"ID": "123","Text1": "aaa","Text2": "aaa","Text3": "aaa"},"date": "","dataset": "","ID": "123"}The var date and dataset are attributes from the flow, so there is no problem here, the problem is i need to extract the value of ID, in this case 123 but this is just an example, without deleted the ID inside de field data. My Jolt Specification is the next one:[{"operation": "shift","spec": {"*": "data.&"}}, {"operation": "default","spec": {"dataset": "${dataset:toLower()}","date": "${date}"}}]and with that i have all EXCEPT the ID field, that i dont know how to do it :(.Thankss

In the picture I attached to this post you can see my current log level value. This does not work. In /var/log/hadoop/hdfs/hadoop-hdfs-namenode-sandbox-hdp.hortonworks.com.log I can only see INFO and WARN messages.

Hi, I'm planning to upgrade my CM and CDH 5.8 to CM, CDH 6.3 but my Rhel OS Version is just 7.2, planning also to upgrade it to 7.4. just want to know if, what steps need to made? do I need to upgrade the OS first then upgrade my cloudera? or I'll upgrade the OS then uninstall cloudera, then after the OS Upgrade, i'll reinstall cloudera?

I have some JSON that contains two arrays that I would like to convert to key/value pairs. I've already create a spec to convert my input data into the format below. Input   { "rows" : [ { "row" : [ "row1", "row2", "row3" ], "header" : [ "header1", "header2", "header3" ] }, { "row" : [ "row4", "row5", "row6" ], "header" : [ "header1", "header2", "header3" ] } ]}    Is it possible to convert it again with Jolt to something like:    {"header1" : "row1","header2" : "row2","header3" : "row3","header1" : "row4","header2" : "row5","header3" : "row6"}    Any guidance would be highly appreciated 🙂

I want to populate my target JSON with a field based on value of another field. e.g. Input JSON:[ {"examUrl": "https://exam.test.com/page/1473161074","urlTypeName": "Spring URL"}, {"examUrl": "https://exam.test.com/page/12423222","urlTypeName": "Java URL"}, ]   Expected Output: { "ExamDashBoardURL" : "https://exam.test.com/page/1473161074" } In the above example, I will populate ExamDashBoardURL only if urlTypeName is "Spring URL". Can anyone please help me on this. Thanks and Regards, Biswa    

Json Input:[{"BRANCH_CODE": "7Z3","CUST_AC_NO": "Z256VM3K6872CY1F8","AC_DESC": "437T7F97CET","CUST_NO": "RX1"},{"BRANCH_CODE": "7Z3","CUST_AC_NO": "Z256VM3K6872CY1F8","AC_DESC": "437T7F97CET","CUST_NO": "RX1"}] Expected Output:[ {"ExternalSystemIdentifier" : "Z256VM3K6872CY1F8","Fields" : {"FLD0001" : "7Z3","FLD0002" : "RX1","FLD0003" : "437T7F97CET"},"InstitutionId" : "1"}, {"ExternalSystemIdentifier" : "Z256VM3K6872CY1F8","Fields" : {"FLD0001" : "7Z3","FLD0002" : "RX1","FLD0003" : "437T7F97CET"},"InstitutionId" : "1"} ] Jolt Spec:[{"operation": "shift","spec": {"*": {"BRANCH_CODE": "[&1].FLD0001","CUST_NO": "[&1].FLD0002","AC_DESC": "[&1].FLD0003"}}},{"operation": "shift","spec": {"*": {"$": "[#2].ExternalSystemIdentifier","@": "[#2].Fields"}}},{"operation": "default","spec": {"*": {"InstitutionId": "1"}}}]

Using:HDP 3.0.1HDFS 3.1.0 NAMENODE HEAP: 84.8% 3.3 GB / 4.0 GBDISK USAGE (DFS USED): 71.20% 63.9 TB / 89.7 TBDISK USAGE (NON DFS USED): 0.74% 676.2 GB / 89.7 TBDISK REMAINING: 28.06% 25.2 TB / 89.7 TBBlock Size : 128 MB Any idea how to solve to reduce reduce the name node heap size usage?Thank you

I'm usingGetDateandServer processor to fetch file names ,decompress the files ,remove header using executestreamcommand Process  Command arguments- 1dCommand path- sed IgnoreSTDIN_ False​  MergeContent Processor to merge files   Merge strategy: Bin-Packing algorithm Merge format: Bin concatenation Merge data strategy: Do not merge uncommon metadata Min no of entries: 180 Max no of entries: 1000 Minimum Group Size: 60GB Max Bin age: 5 min Max no of bins: 1  UpdateAttribute to Create a files name Compress filesputHDFS-To put data into HDFSMy problem is after ExecuteStreamcommand Processor is triggered in the queue I could see two positions with same valueFor every half hour I should get 1 output file but I could see 2 files as output. NIFi is running on 3 nodes, and list queue on 2 nodes. Files running on node1 is merged as 1 file and files running on node 2 is fetched as 2nd file so I'm getting 2 files. Could you please let me

We are trying to use hbase with adls as a storage layer. Is this still possible in cdh 6.1? This is described in the docs for cdh 5.12 : https://docs.cloudera.com/documentation/enterprise/5-12-x/topics/admin_using_adls_storage_with_hbase.html#hbase_adls_configuration_steps but we are using cdh 6.1 and there I don't see a similar page anymore.    Off course I ask since I have an issue following the 5.12 docs.When I want to change in Cloudera Manager in the hbase configuration,    hbase.root.dir = adl://ourproject.azuredatalakestore.net/hbase   I can not save this method as I get an error saying the variable is not in the allowed format:   HDFS Root Directory: Path adls://ourproject.azuredatalakestore.net/hbase does not conform to the pattern "(/[-+=_.a-zA-Z0-9]+)+(/)*".