Solved

Hi All,  Downloaded ambari 2.7.0.0 from http://public-repo-1.hortonworks.com/ambari/sles12/2.x/updates/2.7.0.0/ambari.repo    Downloaded the tarballs from https://docs.cloudera.com/HDPDocuments/Ambari-2.7.1.0/bk_ambari-installation/content/hdp_30_repositories.html   Using Local Repository. Using postgresql for ambari, druid, hive, oozie, ranger and rangerkms.  SLES 12 SP5   In the Install phase, i get to Hive Client Install, it fails with this error: Traceback (most recent call last):File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_client.py", line 60, in <module>HiveClient().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in executemethod(env)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_client.py", line 39, in installself.install_packages(env)File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/scrip

I am encountering this error when I'm executing this command : /opt/cloudera/cm/schema/scm_prepare_database.sh mysql cloudera_manager_db cloudera_db Enter SCM password:JAVA_HOME=/usr/lib/jvm/java-openjdkVerifying that we can write to /etc/cloudera-scm-serverCreating SCM configuration file in /etc/cloudera-scm-serverExecuting: /usr/lib/jvm/java-openjdk/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.[ main] DbCommandExecutor ERROR Error when connecting to database.java.sql.SQLException: The server time zone value 'CDT' is unrecog

Hi guys   I just install a fresh cluster with CDP 7.0.3 (7.0.3-1.cdh7.0.3.p0.1635019) in "Trial period" to practice and probe the environment...   The cluster has 3 masters and 5 workers and only has HDFS (no HA mode), Hive, Impala, Zookeeper, Spark and Yarn services installed with integrated postgre database, no ldap, no tls...   All works fine except HIVESERVER2 that doesn't start... i try to debug the error but i can't   The messages from /var/log/hive/hadoop-cmf-hive-HIVESERVER2-hostname.log     2020-03-13 13:15:54,948 INFO org.apache.hadoop.hive.conf.HiveConf: [main]: Found configuration file file:/run/cloudera-scm-agent/process/1546333137-hive-HIVESERVER2/hive-site.xml2020-03-13 13:15:55,681 WARN org.apache.hadoop.hive.conf.HiveConf: [main]: HiveConf of name hive.query.redaction.rules does not exist2020-03-13 13:16:02,898 INFO org.apache.hadoop.hive.conf.HiveConf: [main]: Found configuration file file:/run/cloudera-scm-agent/process/154633313

Greetings,   I am looking for a way to enforce password complexity requirements, mandatory rotation of passwords and mitigation of brute force password cracking attacks on Cloudera Manager (CM) accounts.   I was researching user authentication options in the free version of CM and came to the conclusion that the only available option is Kerberos and SPNEGO, paired with Kerberos password policies.   In CM I enabled "Enable SPNEGO/Kerberos Authentication for the Admin Console and API", but I didn't notice anything different after restarting the service.   Are there other options (for the free version of CM 6.3) that I am missing? What are the options in the paid version?   Thank you, Kind regards, Julius

Hi, We are running Hue as part of CDH 6.2 (and currently upgrading to 6.3.3). We have an AD group - hue-users - which is the base_dn in the CDH Hue config. This group controls basic access to Hue. I would like to add another AD group. How do I do this? For context, this is so that we can split permissions between groups. One group (hue-users) is for user access (all modules). The other is for unattended service accounts, so we only want them be able to access/schedule/run Oozie jobs. Edit: they're all in the same AD realm, although not necessarily the same OU path. My current config is (the numbers are for reading nested AD groups)(memberOf:1.2.840.113556.1.4.1941:=CN=hue-users,OU=x,DC=x,DC=x,DC=x) I've tried (with various brackets etc):((memberOf:1.2.840.113556.1.4.1941:=CN=hue-users,OU=x,DC=x,DC=x,DC=x) || (memberOf:1.2.840.113556.1.4.1941:=CN=service-users,OU=x,DC=x,DC=x,DC=x)) (|(memberOf:1.2.840.113556.1.4.1941:=CN=hue-users,OU=x,DC=x,DC=x,D

Hi,Am getting below error from Ambari UI i.e., while starting or stopping or restarting the components present in Hadoop cluster. Could any one help please ______________________________________________________________________________Caused by: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.4.0.v20120608-r11652): org.eclipse.persistence.exceptions.DatabaseExceptionInternal Exception: java.sql.BatchUpdateException: Batch entry 0 INSERT INTO request (request_id, cluster_id, command_name, create_time, end_time, exclusive_execution, inputs, request_context, request_type, start_time, status, request_schedule_id) VALUES (820, 2, NULL, 1587733705827, -1, 0, NULL, 'Start History Server', 'INTERNAL_REQUEST', -1, NULL, NULL) was aborted. Call getNextException to see the cause.Error Code: 0Call: INSERT INTO request (request_id, cluster_id, command_name, create_time, end_time, exclusive_execution, inputs, request_context, request_type, start_time, status, request_schedule_id)

After a few months, my cluster is working properly. However, yesterday I've noticed that one of the services went down. I've tried to restart the service but it won't start. The action stuck in the background UI window seen in the image. I've tried to restart some services. However, the same issue encountered with the other service. I am thinking that there is something with Ambari UI sending the command to the cluster. Any suggestions would be helpful for me. Thank youI've already restarted the ambari-server and ambari-agent. H

I'm trying to discern what services will be available to us when we upgrade from CDH 5.16.2 to CDP-DC with CDF. From what I can tell, we should have access to any components part of CDF, as well as anything in the Cloudera Runtime. The thing is, I'm seeing some discrepancies.   In the maven artifacts for Cloudera Runtime 7.0.3, it mentions a lot of services, including Druid and Zeppelin: https://docs.cloudera.com/cdpdc/7.0/release-guide/topics/cdpdc-runtime-maven-703.html   But in the component versions list of Cloudera Runtime 7.0.3, Druid and Zeppelin are both not included: https://docs.cloudera.com/runtime/7.0.3/release-notes/topics/rt-runtime-component-versions.html   Additionally, in the documentation for CDF, I don't see Druid listed, even though in the past HDF has had Druid included: https://www.cloudera.com/content/dam/www/marketing/images/diagrams/cdf-diagram.png    Can someone verify for me whether Druid and/or Zeppelin is available for use by users

Hi Experts.  We recently upgraded our Nifi Version from 1.9.2 to 1.11.4 however many of our flows arent working as some processors arent available in the new version. One of them being the "Infer Avro Schema" . I am not sure why this should be the case as every new release should be backward compatible.  Is there a way to restore this processor onto the latest version as our code is impacted.  We can also see that this has been an issue and raised separately as well. Please find the link here http://apache-nifi-users-list.2361937.n4.nabble.com/GRPC-processors-missing-after-upgrade-to-1-11-3-td9161.html#a9166 cheers Abhinav

Hello, everyone,I'm trying to set up user authentication by username/password in Apache NiFi, but without using Kerberos or LDAP.her used nifi toolkit to generate ssl certificates, i have started nifi by https. but i want you to ask me to log in as minimally as possible without installing extras.is this possible? or is it required to have an external user management system? * i am using Ubuntu localhost, one node in my laptop for instance

In Data Science Workbench, I would like to create one file with a number of types/functions defined. Then, in a separate file, I'd like to be able to import those types/functions (the first file is essentially acting as a library that can be shared among many other scripts.) How do I go about this? In the docs, I see a mention of it being possible, but no example and the few obvious things I've tried haven't worked, so I'm clearly overlooking something simple.   Thanks in advance for any help!   - J.

Team , HUE UI caches username password .i have to restart HUE to prompt username and password . I made below change but it did not change . # UI customizations# -------------------[[custom]]# Top banner HTML code# e.g. (H4)Test Lab A2 Hue Services(/H4)## banner_top_html=# Login splash HTML code# e.g. WARNING: You are required to have authorization before you proceed## login_splash_html=## logo_svg=cacheable_ttl=0  RegardsBharad

There events error is: "Operation timed out. Template install timed out with this command id: 1546332222"Here's the complete sequence of events:   Operation timed out. Template install timed out with this command id: 15463322224/28/2020, 2:10:09 PMBuilding cluster; Cluster manager ip:xxx.xxx.xxx.xxx4/28/2020, 1:21:37 PMRetrying Create cluster operation from the failed step.4/28/2020, 1:21:37 PMCloudera Manager reported that node(s) data-lake-eastern-us-master0.dev-test.udum-f5f2.cloudera.site,data-lake-eastern-us-idbroker1.dev-test.udum-f5f2.cloudera.site status became HEALTHY.4/28/2020, 1:12:44 PMOperation timed out. Template install timed out with this command id: 15463322224/28/2020, 1:12:19 PMBuilding cluster; Cluster manager ip:xxx.xxx.xxx.xxx4/28/2020, 12:23:36 PMStarting cluster services4/28/2020, 12:21:14 PMStarting cluster services4/28/2020, 12:19:14 PMSetting up infrastructure metadata4/28/2020, 12:17:34 PMBootstrapping infrastructure cluster4/28/2020, 12:17:09

how to create new keytab if my previous keytab is expired?

I'm trying to run a sed command through a ExecuteCommandStream but I get the error pasted below. The ExecuteCommandStream has the input from a EvaluateXPath processor. How do I give the flowfile argument? I'm getting this error: ExecuteStreamCommand[id=9c7cfd48-0171-1000-1a59-064ce951124a] Transferring flow file StandardFlowFileRecord[uuid=e448ca5a-8dc8-4754-a0d4-549db92d1f79,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1587501858734-9104, container=default, section=912], offset=375435, length=-1],offset=0,name=DL_prelchim1971_20190302_142639_20190302_145642.xml,size=0] to nonzero status. Executable command sed ended in an error: Usage: sed [OPTION]... {script-only-if-no-other-script} [input-file]. 

I have scenario where I am getting the file(stream:flowfile of NIFI) as of type csv file, then creating the dataframe and dumping it thats it. But after creating the dataframe the structure of the file got disturbed, if I open the same flowfile on my disk i could see clear structure with columns separated with tab, but with python dataframe I am not getting the same structure, if I get same structure i can perform row manipulation. Here What I am doing:1: using ExecuteSQL processor, I am getting database record,2: then passing this record to ConvertRecord processor to convert this avro record type csv file separated by tab.convertRecordSetWriter settings...The output of the flowfile is ...3: Then the reading flowfile(step 2 folowfile) as python data using ExecuteStreamCommand, coz I am want to perform some action on the database record, to do this my record structure has been changed in data frame. 

I have Kerberos and sentry enabled for the cluster. Solr is also configured through sentry. The hue admin user is having all the access.   While testing out the collection creation and indexing using the hue indexes. I am getting the error 403. But I can see that collection being created, somehow the user which created the collection does not have access over the collection. Any other permission I need to add for this.  

Hi, After recreating hadoop cluster on HDInsight and connecting to the external metastore I get an error "Custom Ambari Metastore DB must be empty". Why I get this error? The goal was to keep all the metadata in external DB in case to preserve these data every time when I delete and recreate the cluster.

 i have got the below details other than normal host url,POST /j_security_checkContent-type: application/x-www-form-urlencodedj_username=<userid>&j_password=<password> 

hi all     we have two master machines    master01 master02     on both machines yarn resource manager is running as active/standby )   how to verify which machine is with active resource manager by ambari rest api ? 

 hi all the following ambari API will stop only the data-nodes on workers machines # curl -s -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Stop DataNodes","operation_level":{"level":"SERVICE","cluster_name":"$CLUSTER_NAME"},"query":"HostRoles/component_name=DATANODE&HostRoles/host_name.in(dn1.example.com,dn2.example.com,dn3.example.com)&HostRoles/maintenance_state=OFF"},"Body":{"HostRoles":{"state":"INSTALLED"}}}' "h ttp://$AMBARI_FQDN:8080/api/v1/clusters/$CLUSTER_NAME/host_components" now we want to replace the list - n1.example.com,dn2.example.com,dn3.example.com with variable - DATANODES=n1.example.com,dn2.example.com,dn3.example.com so $DATANODES will replce the list in the API as the following  # curl -s -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Stop DataNodes","operation_level":{"level":"SERVICE","cluster_name":"$CLUSTER_NAME"},"query":"HostRoles/compone

Hi I appeared for exam yesterday its been 24 hours but no news from cloudera yet on the Result of my exam..How much time does it take to notify Result... Tried to search this on website but no luck neither getting a reply on email Please advise.Thanks,

hi all  in ambari we have the following hive components   how to stop Hive Metastore and HiveServer2 by cli  ?

This is a partition move from one volume to another in a single broker.  I have started a kafka-reaasign-partitions of a partition from one volume to another but it never completes. Has anyone faced this ? Kafka cluster is in a healthy state Controller is fine source and destination is also fine The replica is not a leader. So far havent seen any errors in logs  The directories look like the below. Here i am moving a partition from sdc to sdf volume. inodes also look fine. desind@xxx:~#> sudo du -sh /kafka/data/sdc/prod-events-4876G /kafka/data/sdc/prod-events-48 desind@xxx:~#> sudo du -sh /kafka/data/sdf/prod-events-48.b82f63b489554ef4b2e2f4c514bc1bc0-future76G /kafka/data/sdf/prod-events-48.b82f63b489554ef4b2e2f4c514bc1bc0-futuredesind@xxx:~#> find /kafka/data -type d -name 'prod-events-48*'/kafka/data/sdf/prod-events-48.b82f63b489554ef4b2e2f4c514bc1bc0-future/kafka/data/sdc/prod-events-48   

I am trying to create hive table to read data from kafka topics.I am using CDH 6.2.0.I am adding the below jar before creating the table : kafka-handler-3.1.0.3.1.0.0-78.jar;hive-serde-0.10.0.jar;hive-metastore-0.9.0.jar; below is the create table statement: CREATE EXTERNAL TABLE kafka_table  (`timestamp` timestamp , `page` string,  `newPage` boolean,  added int, deleted bigint, delta double)  STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'  TBLPROPERTIES  ("kafka.topic" = "topic-name", "kafka.bootstrap.servers"="hostname:9092");  giving me below error:  Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.DefaultHiveMetaHookorg/apache/hadoop/hive/metastore/DefaultHiveMetaHookError: Error running query: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook (state=,code=0)