Solved

Hey, folks, I'm running a NIFI test on Hortonwork behind a firewall and I want you to help me, I have two nifi installed on my servers. ev01.pepe.local:9091mv01.pepe.local:9091 Generate my Internal CA and at the same time generate certificate for the external name that would be proxev01.pepe.com and also generate with SAN for the internal HOST names, now I'm doing a NAT for the firewall to the ports, it works for me and I have it with permissions for the ranger and ldap. it works for me but when I try to log in with the user I get an error Untrusted proxy CN=proxev01.pepe.com, OU=IT, O=PEPE, L=Madrid, ST=Madrid, C=ES. Could you help me? Thank you Translated with www.DeepL.com/Translator

I want to add 2 host in running Nifi server, which steps i have to follow.. please suggest you input

The reason why I am asking this is a little complicated. So I will not mention too much about it (unless somebody is interested). In fact, I have more than one question.   1. When Kerberos is enabled, does CM add service principals based on hostname shown/stored in CM or every host's actual hostname ? Otherwise, are they always the same ? 2. If hostnames in CM can be different from actual hostnames, how do I achieve that ? (I have tried changing database records and the host cannot connect to HostMonitor afterwards)   Thanks.

I use this command to start:sudo systemctl start cloudera-scm-server. I 'm using CDH-6.2.0-1.cdh6.2.0. and I use mysql as metastore.Then it tells me as below: ```2019-07-16 15:06:38,947 ERROR main:com.cloudera.server.cmf.Main: Server failed.org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Unable to obtain CM release version.at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java

Hi,Pretty new to NiFi and trying to understand the difference between Fetch,Get and List processors.List - As I understand, creates flow files with only metadata and not the data. This information can be further passed to downstream to read the file contents.I am pretty confused about Get/Fetch and which one to be used under what situation.

I'm fairly new to NiFi and trying to execute a Python script stored on Local FS using NiFi. There are couple of XLSB files stored in HDFS. I would want to build a NiFi flow that reads files from HDFS and passes the filename to Python script so that it can convert those to CSV and store it back to HDFS.What should be the flow I need to use to get the above working. I tried using ListHDFS -> ExecuteStream but dont know if that's correct. Also, how do I just test the output of ListHDFS to see the output.

is it possible to degrade hdp version from 3.1 to 2.7?

Deal all.   CM has below query to draw "Total slow get operations across RegionServers"   SELECT integral(total_slow_get_rate_across_regionservers) WHERE entityName = "hbase" AND category = SERVICE   What are the criteria for slow? 1sec ? or  hbase.ipc.warn.response.time?   Thanks in advance  

Hello Team, Is it mandatory to enable ssl for Nifi and Nifi registry in order to configure Ldap Authentication? I followed all steps mentioned in article https://docs.hortonworks.com/HDPDocuments/CFM/CFM-1.0.1/securing-cfm/content/cfm-configure-ldap.html When I open Nifi Ui no login window pops up. Regards, Hemanth 

where i need to check error and solve to launch the browser foe above ink

sqoop import --connect jdbc:mysql:///gaian-lap386.com/ambari1 --username ambari1 --table emp --m 1/usr/hdp/3.1.0.0-78/hadoop/libexec/hadoop-functions.sh: line 2363: HADOOP_ORG.APACHE.SQOOP.SQOOP_USER: bad substitution/usr/hdp/3.1.0.0-78/hadoop/libexec/hadoop-functions.sh: line 2458: HADOOP_ORG.APACHE.SQOOP.SQOOP_OPTS: bad substitution19/08/28 17:20:25 INFO sqoop.Sqoop: Running Sqoop version: 1.4.719/08/28 17:20:25 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.19/08/28 17:20:25 INFO tool.CodeGenTool: Beginning code generationLoading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.19/08/28 17:20:26 ERROR manager.SqlManager: Error executing statement: java.sql.SQLException: Access denied for user 'ambari1'@'localhost' (using password: NO)java.sql.SQLException: Access denied for user 'ambari1'@'lo

Caused by: java.net.ConnectException: Connection refused (Connection refused)                 at java.net.PlainSocketImpl.socketConnect(Native Method)                 at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)                 at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)                 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)                 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)                 at java.net.

Ambari-version：2.7.3 HDP-version：3.1.0   In the yarn log file,I didn't find any error about "ats-hbase service". Where should I find the log file about alter information?

Using HDP 3.1 and unable to run spark2 despite the clients being installed on all nodes (via Ambari), eg.    (venv) ➜ ~ sparkzsh: spark: command not found...zsh: command not found: spark(venv) ➜ ~ spark2zsh: spark2: command not found...zsh: command not found: spark2   Checking the filesystem, nothing seems to be related directly to any spark binaries:   (venv) ➜ ~ find / -name spark 2>&1 | grep -v "Permission denied"/home/spark/var/lib/smartsense/hst-agent/resources/collection-scripts/spark/var/log/spark/var/spool/mail/spark/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/parse/spark/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/optimizer/spark/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/exec/spark/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/common/jsonexplain/spark/tmp/hadoop-unjar3014181574139383154/org/apache/hive/spark/tmp/hadoop-unjar3014181574139383154/biz/k11i/xgboost/spark/usr/hdp/3.1.0.

Hello, I would like your help to secure my Ranger, currently configured by HTTP. But I would like it to be by HTTPS, I have configured 5 Servers. *Edged*Master*worker[1-3]. As an element I have active Kerberos. You could help me with HTTPS for Ranger. Greetings

Greetings,   We are using HDP 2.6.5 mainly for Storm, Kafka, Zookeeper, Spark etc services. We have developed our application in Java and we are deploying our application using JBoss EAP.    Now If I change HDP version 2.6.5 to CDH 6.3 or upgrade it to HDP 3.1.0, do I need to change anything in JBoss configuration or JBoss development perspective? Will there be any development changes from JBoss and Java side if all my code for storm and kafka is in Java.    How JBoss application server will use HDP/CDH services?

I am using GetSFTP processor to get files from Filezilla and PutFile processor to move the file to a directory in my local machine. Is there a way to trigger GetSFTP processor every time a new file is uploaded.? I am trying to create a flow where - every time a new file is uploaded to the server I want it automatically downloaded to my local machine. Can this be done using nifi.? Thanks.  

Looking at the docs for installing NiFi on HDP 3.1 via management pack and looking at list of repository locations here for HDF 3.4. Was wondering if the HDF management pack version needs to be the same as the HDP version for correct installation (else how to tell if the version are compatible with each other)?

I am using HDP-3.1 and YARN 3.1.1. The YARN application type is TEZ.The hive mapreduce job used to work fine. But no mapduce job can run after a complete restart of the HDP. The MR job just hanged and I had to kill them. The job diagnosis show that Application is added to the scheduler and is not yet activated. Skipping AM assignment as cluster resource is empty. Details : AM Partition = <DEFAULT_PARTITION>; AM Resource Request = <memory:4096, vCores:1>; Queue Resource Limit for AM = <memory:0, vCores:0>; User AM Resource Limit of the queue = <memory:0, vCores:0>; Queue AM Resource Usage = <memory:0, vCores:0>; I changed the configuration of yarn.scheduler.capacity.maximum-am-resource-percent from 0.2 to 0.5 but still got the same result. Another issue is that I do not know how to restart YARN. From Ambari UI, I used the menu button YARN - Actions - Restart All, and then clicked the "CONFIRM RESTART ALL" button to confirm. But nothing happens.

Using CM with CDF, using Parcels, trying to Add NiFi CA Service to cluster. Got the following error while trying to activate roles: at com.cloudera.cfg.tools.XmlTools.parseXml(XmlTools.java:122) at com.cloudera.cfg.tools.XmlTools.parseHadoopXmlConf(XmlTools.java:51) at com.cloudera.cfg.configuration.Configuration.getGlobals(Configuration.java:73) at com.cloudera.cfg.configuration.Configuration.substitute(Configuration.java:89) at com.cloudera.cfg.configuration.Configuration.load(Configuration.java:57) at com.cloudera.cfg.JobProcessor.<init>(JobProcessor.java:45) at com.cloudera.cfg.Main.main(Main.java:30)Caused by: java.io.FileNotFoundException: /var/run/cloudera-scm-agent/process/59-nifitoolkitca-NIFI_TOOLKIT_SERVER/staging/globals.xml (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileInputStream.<init>(Fi

Hello,   We can find this ansible playbook on github but are they certified by cloudera ? I'm asking about security. We want to install an hadoop cluster in production with this but we are not sure about the contributors on this github link :  https://github.com/hortonworks/ansible-hortonworks   Thanks.  

Hi, As there are what seems to be 1000s of settings for HDFS, Hive, CM, Hue, Spark, etc. I was wondering if there is a way to see the actual settings that are being used for all applications in the CM/Hadoop suite? I know you can see some of them in config files, some of them using CM's interface.  But not all of them are in both.Defaults, for example, don't show up anywhere as sometimes code sets them.  CM does not cover all the parameters as it has a special location it stores Safety Valve settings, as I've been told??? I would like to be able to see what is resident in memory and actually being used as you may add a setting and it not take as you could have put it in the wrong area or is being overridden by the same setting somewhere else. Is there a way to actually see what is being used for any/all parameters in the suite? Thanks!

its showing this error   stderr: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ACCUMULO/package/scripts/accumulo_master.py", line 24, in AccumuloScript('master').execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ACCUMULO/package/scripts/accumulo_script.py", line 60, in start accumulo_service( self.component, action = 'start') File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/ACCUMULO/package/scripts/accumulo_service.py", line 46, in accumulo_service user=params.accumulo_user File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py",

I am creating ansible scripts to bootstrap our ambari servers. I need to check if ambari-server setup has already been ran so that re-running the ansible scripts do not error out when it gets to that step. The easiest way to make these idempotent in ansible is to check for the existence of a file, but I haven't been able to find any obvious file created by the setup process that I can rely on.

Hello, CDH version 5.8, Kafka 2.1.1-1.2.1.p0.18 We have 5 Kafka brokers on production cluster. One of the broker is unavailable due to host issue.  Is there any impact on running streaming applications, that Kafka was using in real time mode to get message from service.