Q&A

I want to store data in hdfs as parquet file and then create an external table over it as parquet.  i used sqoop command to pull data from Oracle into hdfs.Column4 is date field in Oracle table so i want to store it as timestamp in hive.When i ran select * from table in hue i got the error below. Please help , looks like data type issue because of column 4 .If i import only first 3 fields (string) from oracle to hdfs  and create hive table using first 3 columns then it is working fine. Issue came once i added column4. Step1 : CREATE external TABLE IF NOT EXISTS `abc`(column1 STRING,column2 STRING,column3 STRINGcolumn4 TIMESTAMP)STORED AS PARQUETLOCATION '/data/encrypt/abc';Step 2:sqoop import --connect *** --username *** --password ****** --query 'select column1,column2,column3,column4 from abc where $CONDITIONS' --split-by column1 --delete-target-dir --target-dir /data/encrypt/abc --compression-codec org.apache.hadoop.io.compress.SnappyCodec --a

I'd like to turn on SSL mode for Nifi UI.I'm having a single Nifi node which is running:- HDF 3.1.2- Nifi 1.5This is what I have done:- I used TLS Toolkit (standalone mode) to generate keystore/truststore and P12 file as instructed in here: https://community.hortonworks.com/articles/58233/using-the-tls-toolkit-to-simplify-security.html- I imported above P12 to my browser in my laptop (note that my Nifi is running in a cloud server - not my laptop).- I'm able to access Nifi UI in secured port but it shows "Insufficient Permissions" error from the UI:- I have tried deleting "authorizations.xml" and "users.xml" under "/var/lib/nifi/conf" before restating Nifi but that doesn't help to resolve it.Can anyone please help me figure out what I would miss?- I have attached content of my "users.xml" and "authorizations.xml".- This is content of my authorizers.xml:<authorizers><authorizer><identifier>file-provider</identifier><class>org.apache.nifi.authorization.FileA

Hi Team , We are currently in Operations space with Minifi installed on remote server and successfully transmitting data to the Nifi server overlooked by a C2 server. One question that bothers us is that if Nifi went down or Minifi isnt able to connect to Nifi due to some reason or the other , would the Minifi logs grow huge as it is unable to transmit the data ???Any information / best practices on this will be greatly appreciated @Matt Burgess @Matt Clarke

I am very new to the nifi and have problem casting stream data are generated through sensors over the Internet of Things environment as follows.I want to convert that data to the Jason format using nifi. Sensor data in edge side saved into the sensordata.csv file and using [Tailfile processer] in Minifi send data to the Nifi using S2S(siteTosite). Data reached to the nifi side and s2s works fine. I tried to do this using below procesers in nifi, but not going well. convertCSVtoAvro > convertAvroToJason Anyone has good idea to do this? Thanks in advance!Edge data as follows."{'magnetometer': (75.26935286935287, 0.14993894993894993, -13.194627594627594), 'barometer': (27.7, 1010.44), 'temperature': 0.0, 'accelerometer': (-0.01123046875, 0.0419921875, 2.02001953125), 'battery': 79, 'timestamp': '2019-02-20 17:52:39.099323+09:00', 'humidity': (27.42401123046875, 46.136474609375), 'lightmeter': (-0.00341796875, 0.03564453125, 2.0185546875), 'address': u'98:07:2d:26:d6:82', 'gyroscop

Hi. I am Mauro Beltrame from Argentina, and i have a problem with cluster NiFi. When node 1 ceases to be the primary node and begins to be node 2, some processes configured as primary nodes stop working. I attach the xml flow. flow.xml Thank you!

Hello All, We have a HP OVO monitoring tool monitoring all the alerts from Cloudera Manager and raising a BMC Remedy Incident accordingly. However, we are at times flooded with these monitoring alert tickets and when it is immediately checked in the cluster, everything looks green. When we dig in for detailed analysis and check logs of the respective alert, we do not have anything major and also the service is green. It looks like it heals by itself. However, this flood of tickets, raises concerns with Management and questions for a root cause, which we do not have in reality and also unable to find anything as it was auto fixed with no special traces behind. I am looking out for recommendations / best practices on how exactly this should be setup, so we get only the actual / required alerts. Is there any configuration we need to do in CM or there is something that can be configured in HP OVO. Any suggestions would be of a great help. Also, if anyone has a suggestion tha

Hi Team,I have a table in Oracle DB with one column holding BLOB data. Does NiFi support Blob file from oracle to postgressql? if yes, what are the processors do I need to use for convert and store the data? I want to see the original content of the blob data in postgressql.Thank you.

Nifi has HBase 1.1.2 client version. Can I use HBase 2.x version this HBase controller service?

I am trying to adjust the yarn capacity settings..for a newly setup hive/druid stack.The system was 'working' but then this week it started complaining about Failed to submit application_1550602511762_0005 to YARN : org.apache.hadoop.security.AccessControlException: Queue root.default already has 0 applications, cannot accept submission of application: application_1550602511762_0005it was suggested I needed to reduce the capacity of the llap queue.my 'scheduling' config I am trying..yarn.scheduler.capacity.default.minimum-user-limit-percent=100yarn.scheduler.capacity.maximum-am-resource-percent=0.2yarn.scheduler.capacity.maximum-applications=10000yarn.scheduler.capacity.node-locality-delay=40yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DefaultResourceCalculatoryarn.scheduler.capacity.root.accessible-node-labels=*yarn.scheduler.capacity.root.acl_administer_queue=*yarn.scheduler.capacity.root.acl_submit_applications=*yarn.scheduler.capacity.root.capaci

when submitting a job that failed, it try to run with another attemped, how i can disable the second run ?what configuration param i must set?

i have a large cluster installed currently it is used only for spark on yarn jobswhat services on the master i can remove\uninstall to free up some ram?one server to accept spark jobs currently have the following services:Timeline Service V1.5 / YARN MasterHistory Server / MapReduce2 MasterHive Metastore / Hive MasterHiveServer2 / Hive MasterMetrics Collector / Ambari Metrics MasterGrafana / Ambari Metrics MasterNameNode / HDFS MasterResourceManager / YARN MasterSNameNode / HDFS MasterSpark2 History Server / Spark2 MasterTimeline Service V2.0 Reader / YARN MasterYARN Registry DNS / YARN MasterZooKeeper Server / ZooKeeper MasterDataNode / HDFS SlaveMetrics Monitor / Ambari Metrics SlaveHDFS Client / HDFS ClientHive Client / Hive ClientMapReduce2 Client / MapReduce2 ClientSpark2 Client / Spark2 ClientTez Client / Tez ClientYARN Client / YARN ClientZooKeeper Client / ZooKeeper Clientand multiple clients with:Metrics Monitor / Ambari Metrics SlaveNodeManager / YARN Slave

Hi,Am trying to replace all characters after ";" including ; in the flowfile record.I was using replacetext processor with ;.*$ as the search value and set empty value for Replacement value properties to achieve this. However it is not working out.Any inputs in this regard would be really helpful.Example:InputReplace;text;processorExpected Output:ReplaceThanks,Bala.

Hi,I was trying to deploy the latest hdp sandbox on docker (hosted on a CentOS 7 machine).With issuing the commandsudo sh docker-deploy-hdp30.shThe whole output:+ registry=hortonworks+ name=sandbox-hdp+ version=3.0.1+ proxyName=sandbox-proxy+ proxyVersion=1.0+ flavor=hdp+ echo hdp+ mkdir -p sandbox/proxy/conf.d+ mkdir -p sandbox/proxy/conf.stream.d+ docker pull hortonworks/sandbox-hdp:3.0.13.0.1: Pulling from hortonworks/sandbox-hdp70799bbf2226: Pull complete 40963917cdad: Pull complete 3fe9adbb8d7e: Pull complete ee3ec4e8cb3d: Pull complete 7ea5917732c0: Pull complete 2d951411620c: Pull complete f4c5e354e7fd: Pull complete 22ffa6ef360f: Pull complete 2060aa0f3751: Pull complete ca01ba34744d: Pull complete 83326dded077: Pull complete eb3d71b90b73: Pull complete bdd1cab41c81: Pull complete 500cc770c4bd: Pull complete 0cb1decd5474: Pull complete b9591f4b6855: Pull complete f28e56086127: Pull complete e7de4e7d0bca: Pull complete ec77967d2166: Pull complete 4fdcae170114: Pull complete 6347

Hi. I need to import data from non-trivial dbms: SAS (it isn't db at all)...So, i have jdbc-driver (class "com.sas.rio.MVADriver"), and i have socket (host and port). And i have no connection-manager at all. What should i do? How to write working import-string?1. What about connection-manager2. "driver" looks like jdbc:mysql://<ip>:<port>/ So "mysql" is strange part of it. How to replace it?3. Where must i put drives (jar-files) to use it from sqoop?

Hello Everyone, The scenario I'm trying to do is as follows:1- Flume TAILDIR Source reading from a log file and appending a static interceptor to the beginning of the message. The interceptor consists of the host name and the host IP cause it's required with every log message I receive.2- Flume Kafka Producer Sink that take those messages from the file and put them in a Kafka topic. The Flume configuration is as follows: tier1.sources=source1tier1.channels=channel1tier1.sinks =sink1tier1.sources.source1.interceptors=i1tier1.sources.source1.interceptors.i1.type=statictier1.sources.source1.interceptors.i1.key=HostDatatier1.sources.source1.interceptors.i1.value=###HostName###000.00.0.000###tier1.sources.source1.type=TAILDIRtier1.sources.source1.positionFile=/usr/software/flumData/flumeStressAndKafkaFailureTestPos.jsontier1.sources.source1.filegroups=f1tier1.sources.source1.filegroups.f1=/usr/software/flumData/flumeStressAndKafkaFailureTest.txttier1.sources.so

the server and the agents are located on container(LXC) basd on proxmox virtual env 5.3 with the latest versioni have some issues with ambari metricstail /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out File "/usr/lib/python2.6/site-packages/resource_monitoring/core/metric_collector.py", line 45, in process_event self.process_host_collection_event(event) File "/usr/lib/python2.6/site-packages/resource_monitoring/core/metric_collector.py", line 80, in process_host_collection_event metrics.update(self.host_info.get_combined_disk_io_counters()) File "/usr/lib/python2.6/site-packages/resource_monitoring/core/host_info.py", line 276, in get_combined_disk_io_counters rawdict = psutil.disk_io_counters(True) File "/usr/lib/python2.6/site-packages/resource_monitoring/psutil/build/lib.linux-x86_64-2.7/psutil/__init__.py", line 1726, in disk_io_counters raise RuntimeError("couldn't find any physical disk")RuntimeError: couldn't find any physical disk

Hi,  I got an error during an attempt to access a remote cluster secured by Kerberos and I dont know why the client is trying to find out the hdfs principal in the local KDC.  The setup is as follows (intentionally ommit full domain names and host names to keep it tidy):   - each cluster (CLUSTERDEV and CLUSTERPROD) has its own KDC (DEVREALM and PRODREALM) - the KDC trusts each other (verified by kvno hdfs/<namenodehost>@REMOTE_REALM from boths sides) - both clusters are running NameNode in HA mode   I have configured the Trusted realm in ClouderaManager for CLUSTERDEV set to CLUSTERPROD. (This triggered the RULE change in auth_to_local for core-site.xml). I have done the same for CLUSTERPROD and set trusted CLUSTERDEV.   - each krb5.conf in CLUSTERDEV has also a PRODREALM in [realms] (I can kinit with "remote" account) - each krb5.conf has [capaths]  DEVREALM  = { PRODREALM = . } - and vice versa, in CLUSTERPROD each krb5.conf the DEVRELAM is a

Hello, I would like to know the various activities performed by respective users on Cloudera Manager. For example, which user restarts a service or moves a cluster / service to maintenance mode. Would be great if any one could share some information on this. Thankssnm1523

Hi,Is it possible to perform Namenode and Resource Manager failover through rest api in Ambari?Any help and suggestions will be greatly appreciated.Thank you.

HDP-2.6.5, RHEL 7, Kerberized, Ambari-2.6 upgraded to Ambari-2.7.3Oozie Service Check fails:STDERRTraceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 139, in <module> OozieServiceCheck().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 52, in service_check OozieServiceCheckDefault.oozie_smoke_shell_file(smoke_test_file_name, prepare_hdfs_file_name) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 70, in oozie_smoke_shell_file raise Fail(format(NO_DOCS_FOLDER_MESSAGE)) NameError: global name 'Fail' is not definedSTDOUT2019-02-15 19:02:10,245 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.5.74-2 -> 2.

Hey guys, I have already asked this on multiple forums but never got a reply, so I thought that I might get one here. I have an about 1 gig dataset, and it's got a "cityid" column of which there are 324 unique values, so after partitioning I should get 324 folders in hdfs. But whenever I partition, it fails, you can look at the exception messages here https://community.hortonworks.com/questions/238893/notenoughreplicasexception-when-writing-into-a-par.html It's definitely an HDFS issue, because everything worked out on MapR. What could possible be the problem? Btw, I tried this on a fresh install of hortonworks and cloudera and with default settings, so nothing was compromised. If you need any more details please ask. Could this be a setup issue or something? Like maybe I need to increase memory somewhere in the HDFS or something?

Hi Team,I have two questions, can anyone help me 1) How will I get the sysdate -1 updated records using NiFi(Ex: I have a timestamp 09:00 today but when I get incremental the processor need to check since 09:01 yesterday or else less than 2 hours of timestamp)?2) When delete happens in the source how will affect in target? and I need to store delete records in another table in the target how will I achieve using NiFi?Thank you.

does sharelib jars like hive-exechive-commonhive-jdbchive-serivcelibthriftneeds to be included explicitly in java action <file> tag or those are auto included?From this document i see they should be auto included if i am not wrong? how-to-use-the-sharelib-in-apache-oozie-cdh-5 

Context: Hive3, HDP 3.1. Tests done with Python/odbc (official HDP driver) under Windows and Linux.I ran the following queries: "select ? as lic, ? as cpg""select * from (select ? as lic, ? as cpg) as t""with init as (select ? as lic, ? as cpg) select * from init",1) and 2) work fine, and give me the expected result. 3 gives me a ParseException : Error while compiling statement: FAILED: ParseException line 1:21 cannot recognize input near '?' 'as' 'lic' in select clause (80) (SQLPrepare)")The exact same statements ran with java/jdbc work fine. Note that 2) looks like is a workaround for 3) but it works for this tiny example, not for bigger queries.Is there something I can do to have ODBC working as expected? Alternatively, where can I find the limits of the ODBC driver?For full context, the full test code is as follow:cnxnstr = 'DSN=HiveProd'cnxn = pyodbc.connect(cnxnstr, autocommit=True)cursor = cnxn.cursor()queries = [ "with init as (select ? as lic, ? as cpg) select * from init"

I am trying to toggle another remote port through the Remote Proc Group however it only lets me toggle the first 'input port' and the others are greyed out. Does anyone know how to make the remote proc group have the ability to select the other input ports on the remote instance?thanks in advance.