Q&A

Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 225, in <module> RangerAdmin().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 97, in start self.configure(env, upgrade_type=upgrade_type, setup_db=params.stack_supports_ranger_setup_db_on_start) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 119, in locking_configure original_configure(obj, *args, **kw) File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 139, in configure setup_ranger_db() File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py", line 258, in setup_ranger_db user=params.unix_user, File

Hi community I am trying to debug a simple query in spark SQL that is returning incorrect data.In this instance the query is a simple join between two hive tables .. The issue seems tied to the fact that a the physical plan that spark has generated (with catalyst  optimization) looks to be corrupted where some of the steps in the physical plan have not been assigned an evaluation order id and thus all evaluation on the right side of the join is not completed in the spark queryThis error is on a HDP3.1.4 cluster running   >>> sc.versionu'2.3.2.3.1.4.0-315'   here is the example query ..   from pyspark_llap import HiveWarehouseSessionhive = HiveWarehouseSession.session(spark).build()filter_1 = hive.executeQuery('select * from 03_score where scores = 5 or scores = 6')filter_2 = hive.executeQuery('select * from 03_score where scores = 8')joined_df = filter_1.alias('o').join(filter_2.alias('po'), filter_1.encntr_id == filter_2.

  2020-02-10 16:28:56 Starting to launch local task to process map join; maximum memory = 477626368 SELECT n.Id,t.job_id,t.job_title,n.Name,n.Email,n.Mobile_Number,n.Education,n.Total_Experiance,n.project_id,((count(n.new_skills)*100)/s.skill_count) AverageFROMrmt_demo.resume_convert nJOINrmt_demo.job_description_convert t ON n.new_skills = t.skills and n.job_position = t.job_titleJOINrmt_demo.skill_count s ON n.job_position = s.job_titleJOINrmt_demo.education e ON n.education = e.educationGROUP BYn.Id,t.job_id,t.job_title, n.Name, n.Email, n.Mobile_Number, n.Education, n.Total_Experiance,n.project_id,s.skill_count;      

Hi AllIf in connection(queue between two processors)  I switch on RoundRobin strategy flow files start to stuck there for minutes. Some investigation showed, that flow files stuck only on the primary node.No any Warns or Errors in logs.I am ready to provide any informationPlease, any help.Thanks

We have an HDP 3.1.0 cluster (single node) without Hbase.   The ambari-metrics-collector is constantly stopping with the following error     2020-01-13 22:09:55,247 INFO org.apache.hadoop.hbase.client.RpcRetryingCallerImpl: Call exception, tries=6, retries=16, started=4195 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.NotServingRegionException: METRIC_AGGREGATE_UUID,<^\xC6\xEA\x9A\x7F^\x02\x8C\x98\xE5\xCD\x83\xAC\xEC\xDB\x00\x00\x00\x00\x00\x00\x00\x00,1556526940170.07aab84f4d80a29d771d895a77185269. is not online on xxxxxxx,61320,1578982133933 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3273) at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3250) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2964) at org.apache.

Hello,We have been getting alerts for Open file descriptors on Kudu T-servers “Concerning: Open file descriptors: 16,954. File descriptor limit: 32,768. Percentage in use: 51.74%. Warning threshold: 50.00%.” request assistance in resolving this. Also does restarting services on  affected T-Servers have any performance impact on Kudu as whole.[root@myserver ~]# ulimit -acore file size          (blocks, -c) 0data seg size           (kbytes, -d) unlimitedscheduling priority             (-e) 0file size               (blocks, -f) unlimitedpending signals                 (-i) 1546606max locked memory       (kbytes, -l) 64max memory size   &nbs

Hello I have got strange permission problem with Ambari.I have 2 stages, both have same configs.Only difference in config are information about Postgres DB server. I'm trying to stop services with curl.Curl command: curl -i -u admin:password -H "X-Requested-By: ambari" -X PUT -d "{"HostRoles": {"state": "INSTALLED"}}" https://ambari_server:8443/api/v1/clusters/DEV/hosts/slave01/host_components/NODEMANAGER In Dev stage it all working.  But in test stage Ambari throws following message whet ansible triggers service stop: INFO [ambari-client-thread-1456334] HostComponentResourceProvider:668 - Ignoring ServiceComponentHost as operation is not allowed, What option in ambari is causing this "not allowed"?   

we are running spark streaming app that consumes data from kafka topics we want to know how to tune the parameter - `yarn.scheduler.capacity.maximum-am-resource-percent`   according to documentation:   yarn.scheduler.capacity.maximum-am-resource-percent: Maximum percent of resources in the cluster which can be used to run application masters i.e. controls number of concurrent running,   on some document we even see that recomneded to utilise it to `90 percent` for best results, but the default is `10%`   so my question is do we need to tune this parameter according to cluster size?   or what is the best practice in order to get good results?

I'm able to run cli.sh registry commands, but not cli.sh nifi commands for some reason. They're both secured and use the same props files (except for different baseurls) as defined in ~/.nifi-cli.config: $ cat ~/.nifi-cli.config nifi.reg.props=/appl/nifi-registry/nifi-registry-current/conf/cli-propsnifi.props=/appl/nifi/nifi-current/conf/cli-props $ ./bin/cli.sh registry list-buckets# Name Id Description - -------------- ------------------------------------ ----------- 1 abc 87959437-dfbf-4fa2-9a4d-c0509afaff17 (empty) 2 def 4864d2de-0af5-4a69-a584-d88c108b774f (empty) 3 ghi 20f8069e-60bb-437b-ac3a-f949ffa4582b (empty) But nifi commands fail:   $ ./bin/cli.sh nifi current-userERROR: Error executing command 'current-user' : Error retrieving current $ ./bin/cli.sh nifi current-user -verboseERROR: Error executing command 'current-user' : Error retrieving current

how can i drop column from csv  i only want to keep one selected feild  

Hi,  I have a database and i want to find the best way to import it into impala. Please help me.

sudo su - ${user} -c "beeline -u 'jdbc:hive2://server:port,connectionstring' -e 'insert statement'; "   above line works fine if I take out sudo, with sudo it doesnt work, any help or advice is much appreciated.   Thanks!

Recently I moved from the Oracle JDK 8 to Open JDK 8 (version: openjdk-1.8.0.242.b08). Cloudera version is 5.16.2. I have Kerberos, TLS enabled. When I am using Oracle JDK the below issue doesn't occur but when I switch to Open JDK then I am getting the below error. Please note that the proxy user is already enabled From the CM -> HDFS -> configuration.    JobHistoryServer Error starting JobHistoryServerorg.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [hdfs://XXXX.YYY.COM:8020/user/history/done]at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:680)at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.createHistoryDirs(HistoryFileManager.java:616)at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.serviceInit(HistoryFileManager.java:577)at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)at org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceI

while i executing the program i getting these error can anyone help me with that [cloudera@quickstart ~]$ spark-shell --master yarn-clientSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel).SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/lib/flume-ng/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/lib/parquet/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/usr/lib/avro/avro-tools-1.7.6-cdh5.13.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]Welcome to____ __/ __/__ ___ _____/ /___\ \/ _ \/ _ `/ __/ '_//___/ .__/\_,_/_/ /_/\_\ vers

Hi all!I have a docker-compose created with an image of apache/nifi 1.10.0 into a linux machine.Something like this:version: '2'services: nifi-1: image: 'apache/nifi:1.10.0' hostname: nifi ports: - 8080:8080 - 8082:8082 environment: - NIFI_WEB_HTTP_PORT=8080 - NIFI_ELECTION_MAX_WAIT=1 min - TZ=America/Argentina/Buenos_Aires - LANG=es_AR.UTF-8 Into my development i have a QueryDatabaseTable that read a entire oracle view and then insert this record in other oracle database table.My problem is that this in docker not run.If i run this in a local nifi server this run with success. QueryDatabaseTable configurations: The view: The problem (Im trying with a string field and a date field); The log2020-01-30 12:26:14,004 ERROR [Timer-Driven Process Thread-10] o.a.n.p.standard.QueryDatabaseTable QueryDatabaseTable[id=72b4c9f0-e289-33d7-f82d-50bb8f23d4f1] Unable to execute SQL select query SELECT TRUNC(FECHA_EVALUACION) FECHA_E

I am trying to run a sqoop job using oozie but it log shows it stucks on connecting to hive. The Hive action is working fine and connecting to hive but sqoop action and shell action both are unable to connect to hive ... please suggest   

Dears,   I have to two different databases and the data inside them are identical: MemSql DB Postgre DB Frequently data changed In future, I would like to sync the data between those DBs to capture the data when its changed in the main DB which is the PostgreDB and reflect that changes in the memSql DB. Is there a way using Nifi to capture data changed in the Postgre DB and move the data changed to MemSql DB?   I look forward to hearing from you soon. Regards, Ahmed

Hi All,   I'm trying to enable TLS for Cloudera Manager admin console , below are the commands I followed.   1. Received signed host certificate from IT team ( <hostname>.pem ) , also rootca (root.pem) 2. Copy the JDK cacerts file to jssecacerts as follows:     sudo cp $JAVA_HOME/jre/lib/security/cacerts $JAVA_HOME/jre/lib/security/jssecacerts     3. import rootca cert into JKS store     sudo $JAVA_HOME/bin/keytool -importcert -alias rootca -keystore $JAVA_HOME/jre/lib/security/jssecacerts -file /opt/cloudera/security/pki/root.pem     4. Created JKS and imported host certificate in keystore.     $JAVA_HOME/bin/keytool -genkeypair -alias $(hostname -f) -keyalg RSA -keystore /opt/cloudera/security/pki/$(hostname -f).jks -keysize 2048 -dname "CN=$(hostname -f),OU=Engineering,O=Cloudera,L=Singapore,ST=Singapore,C=Singapore" -ext san=dns:$(hostname -f)sudo $JAVA_HOME/bin/keytool -importcert -alias $(hostname -f) -file /opt/cl

It seems hiveserver2 can't connect to hive metastore,here is my log. 2020-01-31T15:47:46,804 INFO [main]: metastore.HiveMetaStoreClient (:()) - Closed a connection to metastore, current connections: 02020-01-31T15:47:46,804 INFO [main]: metastore.HiveMetaStoreClient (:()) - Trying to connect to metastore with URI thrift://novalinq-server:90832020-01-31T15:47:46,805 INFO [main]: metastore.HiveMetaStoreClient (:()) - Opened a connection to metastore, current connections: 12020-01-31T15:47:46,806 INFO [main]: metastore.HiveMetaStoreClient (:()) - Connected to metastore.2020-01-31T15:47:46,814 WARN [main]: metastore.RetryingMetaStoreClient (:()) - MetaStoreClient lost connection. Attempting to reconnect (8 of 24) after 5s. My jbdc is:jdbc:hive2://novalinq-server:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 And the last error hiveserver2 gives me is: Connection failed on host novalinq-server:10000 (Traceback (most recent call last): File "/

Hi All, I have directory structure like below on Unix file system,/DIR1/DIR2/DIR3/DIR4/DIR5   I have files under the directories DIR3 & DIR5. However I want to read files only from DIR5.I tried with various regex but could not get the right one . Could you please help me with the Regex to be used in ListFile processor to read files from DIR5 only.   Thanks in Advance!  

Hello all,   Problem statement: We have 3 layers in our project,for each layer we have  nifi flow. Currently, the layers or nifi flow for each layer is getting executed one after another (sequentially).   Execution flow: layer1->layer2->layer3->  so on.   The trigger point or input parameters to the flow is nothing but number of source files. Which will be processed further in every layer one by one.   I am looking for such solution ,where my next nifi flow for next layer should not get triggered until completion of previous one for previous layer and for all files.   And if current layer got succeeded for all the files then only next one will kick off.   Please suggest me your thought process on the same. I will be very appreciated on your response.   Thank you in advance.    

Hello Team,   Consider i have a client which is negotiating a timeout from the ZK server.   I have observed that 0x36e18413b196cfc  is the session id and i could see that ZK is terminating the connection after 23,419 seconds ??? Could anyone suggest why ??      2020-01-30 01:03:23,762 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x36e18413b196cfc at /xx.xx.xx.xx:379522020-01-30 01:03:23,762 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x36e18413b196cfc2020-01-30 01:03:23,762 - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:ZooKeeperServer@617] - Established session 0x36e18413b196cfc with negotiated timeout 10000 for client /xx.xx.xx.xx:37952EndOfStreamException: Unable to read additional data from client sessionid 0x36e18413b196cfc, likely client has closed socket2020-01-30 07:33:42,962 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@103

Hi,I have a CDH 6.1 cluster which has < 100 nodes(free Cloudera Express) . I'm thinking about extending the cluster to > 100 nodes.I've found this link:https://www.cloudera.com/content/dam/www/marketing/resources/datasheets/cloudera-enterprise-datasheet.pdf.landing.html Unfortunately it seems like the enterprise license is not only for CDH but for all Cloudera products(CDP, Data Hub, HDP Enterprise). What's the cost per node of a CDH Cloudera Express cluster? How is it related to this pricing?:https://www.cloudera.com/products/pricing.htmlRegards

Hi all, Anyone can help me to Access HDFS directory from  windows machine ? I want to backup folder and file from hdfs to my Windows 10 Machine. I am now using HDP 2.7.4.0 Thank You.

Hello everyone, I'm running HUE version 4. I can create workflow and schedule it as well just fine, but only if the account is in "SUPERUSER" mode. i.e. under user profile, if the user has all the role permissions but no superuser is checked under user profile, no Scheduler (Workflow, Scheduling, Bundle) is shown at all in the menu. I want to grant a standard user the workflow/scheduling creating option but I don't want it to be a superuser, can I accomplish this somehow? Please guide me, thanks.