Solved

i'm currently using cloudera community editition 5.15.0 from past 90 days I had External authentication configured over the Cloudera community edition , after few days of utilization i'm noticing that external authentication is not avaliable under the Administrattion-> Security Options  Does External Authentication comes only will limited period of time ?  please advice 

I have Hadoop classpath defined but when I use it as per documentation I am getting error [root@hadoop1 conf]# hadoop classpath/usr/hdp/2.6.5.0-292/hadoop/conf:/usr/hdp/2.6.5.0-292/hadoop/lib/*:/usr/hdp/2.6.5.0-292/hadoop/.//*:/usr/hdp/2.6.5.0-292/hadoop-hdfs/./:/usr/hdp/2.6.5.0-292/hadoop-hdfs/lib/*:/usr/hdp/2.6.5.0-292/hadoop-hdfs/.//*:/usr/hdp/2.6.5.0-292/hadoop-yarn/lib/*:/usr/hdp/2.6.5.0-292/hadoop-yarn/.//*:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/lib/*:/usr/hdp/2.6.5.0-292/hadoop-mapreduce/.//*:/usr/jdk64/jdk1.8.0_112/lib/tools.jar:mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:ojdbc6.jar:/usr/hdp/2.6.5.0-292/tez/*:/usr/hdp/2.6.5.0-292/tez/lib/*:/usr/hdp/2.6.5.0-292/tez/conf[root@hadoop1 conf]#[root@hadoop1 conf]#[root@hadoop1 conf]#  echo $JAVA_HOME/usr/jdk64/jdk1.8.0_112[root@hadoop1 conf]#[root@hadoop1 conf]# javac -cp  `hadoop classpath`  TestHbaseTable.javaTestHbaseTable.java:3: error: package org.apache.hadoop.hbase does not existimport org.apache.had

Good evening. This message I receive after add two new hosts to the hadoop cluser.Tehnical info:Nodes - Debian GNU/Linux 8.9 (jessie)cloudera-manager-agent 5.13.0-1.cm5130.p0.55~jessie-cm5 amd64cloudera-manager-daemons 5.13.0-1.cm5130.p0.55~jessie-cm5 allEarlier we add nodes using web interface. But this 2 nodes was added by hand:1. Add repo cloudera2. Install packages from repo3. Configure /etc/cloudera-scm-agent/config.ini:server_host=hadoop-server.test.comAfter start clouder agents, I saw tho strangeness In cloudera web interface:1. All Hosts - my new host - CDH Version is None2. Error from topic: "The hostname and canonical name for this host are not consistent when checked from a Java process."I dont know is connected to each other this issues or not. I try resolve second issue:python -c 'import socket; print socket.getfqdn(), socket.gethostbyname(socket.getfqdn())'regensburg.test.com 1.1.1.1This command use from /etc/cloudera-scm-agent/config.ini file. Re

Hi, I need to get the health check report of server, which is found in cloud era manager. Using APIs I need to get those details from cloud era. I need the information of list of Web services that is to be enabled to proceed on using APIs for retrieval of information from cloud era. Thanks.

We are running hive with udf vs spark comparison.The aim is to choose a faster solution for encrypting/decrypting data.Tech stack we are using is as follows:HDP 2.6.5Hive 1.2.1000Spark2 2.xYARN + MapReduce2 2.7.3Data are stored on HDF as csv files: Data set 1 (big data set): 1M rows, 44 columns, 19000 unique customers;Data set 2 (small data set): 25k rows, 44 columns, 494 unique customers;22 columns are being encrypted using unique key for unique customer.UDF on Hive is written in Java. Function on Spark is written in Scala. Encrypting decrypting functions are basically the same.To compare those two frameworks we run count (to decrypt the whole table and get one value as output).Hive is run from beeline command line. Spark is run from Zeppelin.Keys are stored in a file on HDFS.Spark (results in Zeppelin):The average time for querying encrypted small data set with decryption: 11.8 sThe average time for querying encrypted big data set with decryption: 54.8 s---The average time for queryi

Hi, Currently we are using NIFI 2.1, and one workflow is to running SQL from DBCPConnectionPool every 10 minutes to extract data, then do the following logic.Due to the source is the MySQL database in another data center, sometimes the network connection is broken due to network or firewall issues. We would like to have email notification when this kind of thing happens in NIFI.Right now, our first Executor SQL processor, when there is a network connection issue to the source MySQL DB, the only error coming out is in the bulletin or log, which hardly monitored by a human.I wonder if we can design something in the NIFI processor, or it is natively available in NIFI, will trigger Email notification when the DBCP connection pool is broken.Thanks

I can select a particular version from a table , but how can I select a range of versions ? e.g say I want to select all the rows from timestamp 1551108073557 to 1551108018724 ?hbase(main):005:0> scan 'PERSON',{NAME => 'student',VERSIONS => 5}ROW COLUMN+CELL r1 column=student:student_name, timestamp=1551108086986, value=John r1 column=student:student_name, timestamp=1551108073557, value=Mary r1 column=student:student_name, timestamp=1551108037609, value=Nancy r1 column=student:student_name, timestamp=1551108018724, value=Ram r1 column=student:student_name, timestamp=1551108002231, value=Sam1 row(s) in 0.0190 secondshbase(main):006:0>hbase(main):007:0*hbase(main):008:0* get 'PERSON'

Hello All, Need some suggestions on the exact reason of below heath tests getting failed multiple times, almost 3 times every day and generates atleast 3-4 alerts each time. 1. HIVEMETASTORE_CANARY_HEALTH2. REGION_SERVER_AUDIT_HEALTH3. IMPALAD_QUERY_MONITORING_STATUS4. HIVESERVER2_SCM_HEALTH5. NAME_NODE_AUDIT_HEALTH6. NODE_MANAGER_WEB_METRIC_COLLECTION Any help or suggestion to permanently fix these alerts would of great help. Also, if anyone could also guide to reach root cause of this would also be helpful Thankssnm1523

Hi All, We have just productionized our Nifi flows which is a real time streaming application . We have a cluster of 2 Nodes and each node has 6 processors and a high memory of 28 GB. On the Nifi side the memory has been set to 24 GB. Moreover , we tend to see a queue size reaching to around 90,000 or more. For this I have set the property nifi.queue.swap.threshold to 90000 . We also have a disk space of 200 GB. Despite having such a powerful configuration , we still get flow file repository failed to update. I understand that initially the JVM memory is uitilized for the flow file processing and if that gets overused , it spills over to disk space . Our disk space is also pretty high at 200 GB however we still keep on seeing this error.Any pointers or suggestion in this space will be of great help.

HI All , I have created 2 nifi instances on Google Cloud Platform. I now plan to bring them onto a cluster. However I am getting this errorserver.quorum.QuorumPeerConfig does not have the form host:port or host:port:port or host:port:port:typeI have checked my zoo keeper properties file again and again but not sure whats the issue. My zoo keeper properties file is as follows . I am not sure as to why this issue is cropping up. My nifi servers are by the name nifi-server2 and nifi-server1 and ports are 2888 and 3888 . I have followed the documentation by Pierre Villard.server.1=nifi-server2:2888:3888server.2=nifi-server1:2888:3888# server.3=nifi-node3-hostname:2888:3888Can someone please help me here.

I am getting error while installing namenode where it says "-safemode get | grep 'Safe mode is OFF''"Ambari:2.7.0.0HDP:3.0I got the below error and I ran :>hdfs dfsadmin -safemode leave>Output:safemode: Call From devaz01/10.161.137.4 to devaz01:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefusedthen I ran>chmod -R 644 /var/log/hadoop/hdfsSince then I am getting: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/3.0.1.0-187/hadoop/bin/hdfs --config /usr/hdp/3.0.1.0-187/hadoop/conf --daemon start namenode'' returned 1. ERROR: Unable to write in /var/log/hadoop/hdfs. Aborting.019-02-08 03:31:21,394 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://machine:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. safemode: Call From machine/xx.xxx.xxx.x to machine:8020 fail

Basic error on the created notes is that the resolve.conf fails to parse. in looking at a failed node, the resolve.conf file was poorly created - see below:    # Generated by NetworkManagernameserver [an ip address]search     Note that there's nothing after above "search". I fixed this issue by running the setup script on the Altus host.  Problem is that the generated nodes fail to create well constucted "resolve.conf" files - which then leads to the parse error. 

I want to store data in hdfs as parquet file and then create an external table over it as parquet.  i used sqoop command to pull data from Oracle into hdfs.Column4 is date field in Oracle table so i want to store it as timestamp in hive.When i ran select * from table in hue i got the error below. Please help , looks like data type issue because of column 4 .If i import only first 3 fields (string) from oracle to hdfs  and create hive table using first 3 columns then it is working fine. Issue came once i added column4. Step1 : CREATE external TABLE IF NOT EXISTS `abc`(column1 STRING,column2 STRING,column3 STRINGcolumn4 TIMESTAMP)STORED AS PARQUETLOCATION '/data/encrypt/abc';Step 2:sqoop import --connect *** --username *** --password ****** --query 'select column1,column2,column3,column4 from abc where $CONDITIONS' --split-by column1 --delete-target-dir --target-dir /data/encrypt/abc --compression-codec org.apache.hadoop.io.compress.SnappyCodec --a

I'd like to turn on SSL mode for Nifi UI.I'm having a single Nifi node which is running:- HDF 3.1.2- Nifi 1.5This is what I have done:- I used TLS Toolkit (standalone mode) to generate keystore/truststore and P12 file as instructed in here: https://community.hortonworks.com/articles/58233/using-the-tls-toolkit-to-simplify-security.html- I imported above P12 to my browser in my laptop (note that my Nifi is running in a cloud server - not my laptop).- I'm able to access Nifi UI in secured port but it shows "Insufficient Permissions" error from the UI:- I have tried deleting "authorizations.xml" and "users.xml" under "/var/lib/nifi/conf" before restating Nifi but that doesn't help to resolve it.Can anyone please help me figure out what I would miss?- I have attached content of my "users.xml" and "authorizations.xml".- This is content of my authorizers.xml:<authorizers><authorizer><identifier>file-provider</identifier><class>org.apache.nifi.authorization.FileA

Hi Team , We are currently in Operations space with Minifi installed on remote server and successfully transmitting data to the Nifi server overlooked by a C2 server. One question that bothers us is that if Nifi went down or Minifi isnt able to connect to Nifi due to some reason or the other , would the Minifi logs grow huge as it is unable to transmit the data ???Any information / best practices on this will be greatly appreciated @Matt Burgess @Matt Clarke

I am very new to the nifi and have problem casting stream data are generated through sensors over the Internet of Things environment as follows.I want to convert that data to the Jason format using nifi. Sensor data in edge side saved into the sensordata.csv file and using [Tailfile processer] in Minifi send data to the Nifi using S2S(siteTosite). Data reached to the nifi side and s2s works fine. I tried to do this using below procesers in nifi, but not going well. convertCSVtoAvro > convertAvroToJason Anyone has good idea to do this? Thanks in advance!Edge data as follows."{'magnetometer': (75.26935286935287, 0.14993894993894993, -13.194627594627594), 'barometer': (27.7, 1010.44), 'temperature': 0.0, 'accelerometer': (-0.01123046875, 0.0419921875, 2.02001953125), 'battery': 79, 'timestamp': '2019-02-20 17:52:39.099323+09:00', 'humidity': (27.42401123046875, 46.136474609375), 'lightmeter': (-0.00341796875, 0.03564453125, 2.0185546875), 'address': u'98:07:2d:26:d6:82', 'gyroscop

Hi. I am Mauro Beltrame from Argentina, and i have a problem with cluster NiFi. When node 1 ceases to be the primary node and begins to be node 2, some processes configured as primary nodes stop working. I attach the xml flow. flow.xml Thank you!

Hello All, We have a HP OVO monitoring tool monitoring all the alerts from Cloudera Manager and raising a BMC Remedy Incident accordingly. However, we are at times flooded with these monitoring alert tickets and when it is immediately checked in the cluster, everything looks green. When we dig in for detailed analysis and check logs of the respective alert, we do not have anything major and also the service is green. It looks like it heals by itself. However, this flood of tickets, raises concerns with Management and questions for a root cause, which we do not have in reality and also unable to find anything as it was auto fixed with no special traces behind. I am looking out for recommendations / best practices on how exactly this should be setup, so we get only the actual / required alerts. Is there any configuration we need to do in CM or there is something that can be configured in HP OVO. Any suggestions would be of a great help. Also, if anyone has a suggestion tha

Hi Team,I have a table in Oracle DB with one column holding BLOB data. Does NiFi support Blob file from oracle to postgressql? if yes, what are the processors do I need to use for convert and store the data? I want to see the original content of the blob data in postgressql.Thank you.

Nifi has HBase 1.1.2 client version. Can I use HBase 2.x version this HBase controller service?

I am trying to adjust the yarn capacity settings..for a newly setup hive/druid stack.The system was 'working' but then this week it started complaining about Failed to submit application_1550602511762_0005 to YARN : org.apache.hadoop.security.AccessControlException: Queue root.default already has 0 applications, cannot accept submission of application: application_1550602511762_0005it was suggested I needed to reduce the capacity of the llap queue.my 'scheduling' config I am trying..yarn.scheduler.capacity.default.minimum-user-limit-percent=100yarn.scheduler.capacity.maximum-am-resource-percent=0.2yarn.scheduler.capacity.maximum-applications=10000yarn.scheduler.capacity.node-locality-delay=40yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DefaultResourceCalculatoryarn.scheduler.capacity.root.accessible-node-labels=*yarn.scheduler.capacity.root.acl_administer_queue=*yarn.scheduler.capacity.root.acl_submit_applications=*yarn.scheduler.capacity.root.capaci

when submitting a job that failed, it try to run with another attemped, how i can disable the second run ?what configuration param i must set?

i have a large cluster installed currently it is used only for spark on yarn jobswhat services on the master i can remove\uninstall to free up some ram?one server to accept spark jobs currently have the following services:Timeline Service V1.5 / YARN MasterHistory Server / MapReduce2 MasterHive Metastore / Hive MasterHiveServer2 / Hive MasterMetrics Collector / Ambari Metrics MasterGrafana / Ambari Metrics MasterNameNode / HDFS MasterResourceManager / YARN MasterSNameNode / HDFS MasterSpark2 History Server / Spark2 MasterTimeline Service V2.0 Reader / YARN MasterYARN Registry DNS / YARN MasterZooKeeper Server / ZooKeeper MasterDataNode / HDFS SlaveMetrics Monitor / Ambari Metrics SlaveHDFS Client / HDFS ClientHive Client / Hive ClientMapReduce2 Client / MapReduce2 ClientSpark2 Client / Spark2 ClientTez Client / Tez ClientYARN Client / YARN ClientZooKeeper Client / ZooKeeper Clientand multiple clients with:Metrics Monitor / Ambari Metrics SlaveNodeManager / YARN Slave

Hi,Am trying to replace all characters after ";" including ; in the flowfile record.I was using replacetext processor with ;.*$ as the search value and set empty value for Replacement value properties to achieve this. However it is not working out.Any inputs in this regard would be really helpful.Example:InputReplace;text;processorExpected Output:ReplaceThanks,Bala.

Hi,I was trying to deploy the latest hdp sandbox on docker (hosted on a CentOS 7 machine).With issuing the commandsudo sh docker-deploy-hdp30.shThe whole output:+ registry=hortonworks+ name=sandbox-hdp+ version=3.0.1+ proxyName=sandbox-proxy+ proxyVersion=1.0+ flavor=hdp+ echo hdp+ mkdir -p sandbox/proxy/conf.d+ mkdir -p sandbox/proxy/conf.stream.d+ docker pull hortonworks/sandbox-hdp:3.0.13.0.1: Pulling from hortonworks/sandbox-hdp70799bbf2226: Pull complete 40963917cdad: Pull complete 3fe9adbb8d7e: Pull complete ee3ec4e8cb3d: Pull complete 7ea5917732c0: Pull complete 2d951411620c: Pull complete f4c5e354e7fd: Pull complete 22ffa6ef360f: Pull complete 2060aa0f3751: Pull complete ca01ba34744d: Pull complete 83326dded077: Pull complete eb3d71b90b73: Pull complete bdd1cab41c81: Pull complete 500cc770c4bd: Pull complete 0cb1decd5474: Pull complete b9591f4b6855: Pull complete f28e56086127: Pull complete e7de4e7d0bca: Pull complete ec77967d2166: Pull complete 4fdcae170114: Pull complete 6347