Q&A

sqoop import --connect jdbc:mysql:///gaian-lap386.com/ambari1 --username ambari1 --table emp --m 1/usr/hdp/3.1.0.0-78/hadoop/libexec/hadoop-functions.sh: line 2363: HADOOP_ORG.APACHE.SQOOP.SQOOP_USER: bad substitution/usr/hdp/3.1.0.0-78/hadoop/libexec/hadoop-functions.sh: line 2458: HADOOP_ORG.APACHE.SQOOP.SQOOP_OPTS: bad substitution19/08/28 17:20:25 INFO sqoop.Sqoop: Running Sqoop version: 1.4.719/08/28 17:20:25 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.19/08/28 17:20:25 INFO tool.CodeGenTool: Beginning code generationLoading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.19/08/28 17:20:26 ERROR manager.SqlManager: Error executing statement: java.sql.SQLException: Access denied for user 'ambari1'@'localhost' (using password: NO)java.sql.SQLException: Access denied for user 'ambari1'@'lo

Caused by: java.net.ConnectException: Connection refused (Connection refused)                 at java.net.PlainSocketImpl.socketConnect(Native Method)                 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)                 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)                 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)                 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)                 at java.net.

Ambari-version：2.7.3 HDP-version：3.1.0   In the yarn log file,I didn't find any error about "ats-hbase service". Where should I find the log file about alter information?

Using HDP 3.1 and unable to run spark2 despite the clients being installed on all nodes (via Ambari), eg.    (venv) ➜ ~ sparkzsh: spark: command not found...zsh: command not found: spark(venv) ➜ ~ spark2zsh: spark2: command not found...zsh: command not found: spark2   Checking the filesystem, nothing seems to be related directly to any spark binaries:   (venv) ➜ ~ find / -name spark 2>&1 | grep -v "Permission denied"/home/spark/var/lib/smartsense/hst-agent/resources/collection-scripts/spark/var/log/spark/var/spool/mail/spark/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/parse/spark/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/optimizer/spark/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/exec/spark/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/common/jsonexplain/spark/tmp/hadoop-unjar3014181574139383154/org/apache/hive/spark/tmp/hadoop-unjar3014181574139383154/biz/k11i/xgboost/spark/usr/hdp/3.1.0.

Hello, I would like your help to secure my Ranger, currently configured by HTTP. But I would like it to be by HTTPS, I have configured 5 Servers. *Edged*Master*worker[1-3]. As an element I have active Kerberos. You could help me with HTTPS for Ranger. Greetings

Greetings,   We are using HDP 2.6.5 mainly for Storm, Kafka, Zookeeper, Spark etc services. We have developed our application in Java and we are deploying our application using JBoss EAP.    Now If I change HDP version 2.6.5 to CDH 6.3 or upgrade it to HDP 3.1.0, do I need to change anything in JBoss configuration or JBoss development perspective? Will there be any development changes from JBoss and Java side if all my code for storm and kafka is in Java.    How JBoss application server will use HDP/CDH services?

I am using GetSFTP processor to get files from Filezilla and PutFile processor to move the file to a directory in my local machine. Is there a way to trigger GetSFTP processor every time a new file is uploaded.? I am trying to create a flow where - every time a new file is uploaded to the server I want it automatically downloaded to my local machine. Can this be done using nifi.? Thanks.  

Looking at the docs for installing NiFi on HDP 3.1 via management pack and looking at list of repository locations here for HDF 3.4. Was wondering if the HDF management pack version needs to be the same as the HDP version for correct installation (else how to tell if the version are compatible with each other)?

I am using HDP-3.1 and YARN 3.1.1. The YARN application type is TEZ.The hive mapreduce job used to work fine. But no mapduce job can run after a complete restart of the HDP. The MR job just hanged and I had to kill them. The job diagnosis show that Application is added to the scheduler and is not yet activated. Skipping AM assignment as cluster resource is empty. Details : AM Partition = <DEFAULT_PARTITION>; AM Resource Request = <memory:4096, vCores:1>; Queue Resource Limit for AM = <memory:0, vCores:0>; User AM Resource Limit of the queue = <memory:0, vCores:0>; Queue AM Resource Usage = <memory:0, vCores:0>; I changed the configuration of yarn.scheduler.capacity.maximum-am-resource-percent from 0.2 to 0.5 but still got the same result. Another issue is that I do not know how to restart YARN. From Ambari UI, I used the menu button YARN - Actions - Restart All, and then clicked the "CONFIRM RESTART ALL" button to confirm. But nothing happens.

Using CM with CDF, using Parcels, trying to Add NiFi CA Service to cluster. Got the following error while trying to activate roles: at com.cloudera.cfg.tools.XmlTools.parseXml(XmlTools.java:122) at com.cloudera.cfg.tools.XmlTools.parseHadoopXmlConf(XmlTools.java:51) at com.cloudera.cfg.configuration.Configuration.getGlobals(Configuration.java:73) at com.cloudera.cfg.configuration.Configuration.substitute(Configuration.java:89) at com.cloudera.cfg.configuration.Configuration.load(Configuration.java:57) at com.cloudera.cfg.JobProcessor.<init>(JobProcessor.java:45) at com.cloudera.cfg.Main.main(Main.java:30)Caused by: java.io.FileNotFoundException: /var/run/cloudera-scm-agent/process/59-nifitoolkitca-NIFI_TOOLKIT_SERVER/staging/globals.xml (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileInputStream.<init>(Fi

Hello,   We can find this ansible playbook on github but are they certified by cloudera ? I'm asking about security. We want to install an hadoop cluster in production with this but we are not sure about the contributors on this github link :  https://github.com/hortonworks/ansible-hortonworks   Thanks.  

Hi, As there are what seems to be 1000s of settings for HDFS, Hive, CM, Hue, Spark, etc. I was wondering if there is a way to see the actual settings that are being used for all applications in the CM/Hadoop suite? I know you can see some of them in config files, some of them using CM's interface.  But not all of them are in both.Defaults, for example, don't show up anywhere as sometimes code sets them.  CM does not cover all the parameters as it has a special location it stores Safety Valve settings, as I've been told??? I would like to be able to see what is resident in memory and actually being used as you may add a setting and it not take as you could have put it in the wrong area or is being overridden by the same setting somewhere else. Is there a way to actually see what is being used for any/all parameters in the suite? Thanks!

its showing this error   stderr: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ACCUMULO/package/scripts/accumulo_master.py", line 24, in AccumuloScript('master').execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ACCUMULO/package/scripts/accumulo_script.py", line 60, in start accumulo_service( self.component, action = 'start') File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ACCUMULO/package/scripts/accumulo_service.py", line 46, in accumulo_service user=params.accumulo_user File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py",

I am creating ansible scripts to bootstrap our ambari servers. I need to check if ambari-server setup has already been ran so that re-running the ansible scripts do not error out when it gets to that step. The easiest way to make these idempotent in ansible is to check for the existence of a file, but I haven't been able to find any obvious file created by the setup process that I can rely on.

Hello, CDH version 5.8, Kafka 2.1.1-1.2.1.p0.18 We have 5 Kafka brokers on production cluster. One of the broker is unavailable due to host issue.  Is there any impact on running streaming applications, that Kafka was using in real time mode to get message from service. 

Brand new NiFi user here. I created a process group and have the QueryDatabaseTable → PutFile processors linked. I funnel the output for failures and successes from the PutFile processor. I'm getting data passed through without issue, however, it is not writing the output file to my C:\users\<username>\nifi_output on my local machine. I have changed the configuration of the PutFile to try \\<machine-name>\C$\users\<username>\nifi_output for the output file and that hasn't worked either. I haven't received any errors. Like I said, I'm a newbie, so I don't know where to look any further. Thanks!

Hello, I am regularly seeing this error in the state-change.log on my production cluster and was wondering if this is something to worry about.   prod.cmla.metrics-15 in response to UpdateMetadata request sent by controller 1005 epoch 84 with correlation id 1083 (state.change.logger)[2019-08-21 04:18:18,862] TRACE Controller 1005 epoch 84 received response {error_code=0} for a request sent to broker  (id: 1005 rack: null) (state.change.logger)[2019-08-21 04:18:18,863] TRACE Controller 1005 epoch 84 received response {error_code=0} for a request sent to broker  (id: 1002 rack: null) (state.change.logger)

Can we have a shell script to delete the nodes in Ambari which were already deleted in Azure

Hi   I have been searching for sometime for a command reference/API manual for PySpark-Kudu and I have been unsuccessful so far. Does Cloudera have something that can be of help?   Thanks.

I am using HDP-2.6.5.0 with kafka 1.0.0; I have to process large (16M) messages so i setmessage.max.bytes=18874368replica.fetch.max.bytes = 18874368socket.request.max.bytes =18874368From Ambary/Kafka configs screen and restarted Kafka servicesWhen I try to send 16M messages:/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list <broker-ip>:6667 --topic test < ./big.txtI still have the same error:ERROR Error when sending message to topic test with key: null, value: 16777239 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)org.apache.kafka.common.errors.RecordTooLargeException: The message is 16777327 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.I tried to set max.request.size in producer.properties file but still have the same error. What am I missing?Thank you,

Hello,   Cloudera version 5.8(Parcels), Impala version is 2.6 and we are facing lot of issues. So we decided to upgrade the Impala.   What all Impala versions are compatible with CDH 5.8. Please provide me the documentations to follow and Impala parcels location.   Please suggest me the upgrade possibilities.

I am trying to install HDP 3.1 with Ambari 2.7 following this document https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/set_up_the_ambari_server.htmlAll software installation completed and mysql database had been created. I then setup ambari with "ambari setup" followed by "ambari start". But I got follwing errors. I had listed the screen output for ambari setup and part of ambari-server.log to show the error message. Please help. Thanks,HarrySetup and start ambari-server [root@msl-dpe-perf80-100g resources]# ambari-server setupUsing python /usr/bin/pythonSetup ambari-serverChecking SELinux...SELinux status is 'disabled'Customize user account for ambari-server daemon [y/n] (n)? Adjusting ambari-server permissions and ownership...Checking firewall status...Checking JDK...Do you want to change Oracle JDK [y/n] (n)? Check JDK version for Ambari Server...JDK version found: 8Minimum JDK version is 8 for Ambari. Skipping to setup differe

Hi Hortonworks Team,   We are getting error "DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020" during pip install. I can see that HDP3.0/HDF3.2 are having python 2.7. My question is post January 1st, 2020. will HDP3.0/HDF3.2 stop working or we will not be able to install any python library through pip or there will be new release of HDP/HDF sandboxes with python version > 2.7.

Hello Community, I have a CDH 5.14 cluster hosted on EC2 machines which is kerberized with Active Directory and have Sentry for authorization of databases.  I want to use SSSD to secure my Linux hosts (RHEL 7.x) with Active Directory.  I have been going through this post but there are few queries which are bothering me to proceed forwards: 1. There are service-users (Hive, YARN, etc.) in AD that are already created during Kerberization of my cluster. So, if I go ahead and implement SSSD, then will these pre-existing service-users be able to communicate? 2. If something goes wrong will I be able to rollback? If yes, how?  

after confirming hosts these two are the warnings   Process Issues (2) The following processes should not be runningProcess /usr/jdk64/jdk1.8.0_112/bin/java -c. Running on 1 host /usr/jdk64/jdk1.8.0_112/bin/java c...  Running on 1 host