Solved

Hi,   I did nifi and mysql setup in stand alone machine, where I have scenario the input will be in excel form, i need to create table in mysql db. I have seen processors where we can do UPDATE, INSERT, or DELETE SQL, does create will be possible?   Also Do I need to write custom processor to extract the column names from the excel?    

I have a problem when init Hive view. It inits ds_jobimpl_xx table but miss add ds_jobimpl_xx_id_seq to ambari_sequences. I go through all Hive view code but can not find any info about those. Any expert can help me to clarify it?  Thank you.

org.apache.ambari.view.PersistenceException: Caught exception trying to store view entity JobImpl{id='null, owner='admin, hiveQueryId='null, dagId='null, queryId='null} org.apache.ambari.view.PersistenceException: Caught exception trying to store view entity JobImpl{id='null, owner='admin, hiveQueryId='null, dagId='null, queryId='null}at org.apache.ambari.server.view.persistence.DataStoreImpl.throwPersistenceException(DataStoreImpl.java:666)at org.apache.ambari.server.view.persistence.DataStoreImpl.store(DataStoreImpl.java:166)at org.apache.ambari.view.hive20.persistence.DataStoreStorage.store(DataStoreStorage.java:61)at org.apache.ambari.view.hive20.resources.CRUDResourceManager.save(CRUDResourceManager.java:117)at org.apache.ambari.view.hive20.resources.PersonalCRUDResourceManager.save(PersonalCRUDResourceManager.java:68)at org.apache.ambari.view.hive20.resources.PersonalCRUDResourceManager.save(PersonalCRUDResourceManager.java:34)at org.apache.ambari.view.hive20.resources.CRUDResour

Hi friends,   I have CDH 6.2.1 setted up in my local server. I can login to cloudera manager to monitor all the components and its running status. Meanwhile im trying to found how to access Cloudera navigator and stuck with it. Can you guide me how to access the same.   Even i followed the below steps, it dint help as i couldnt locate the navigator How can I access the Cloudera Navigator console? The Cloudera Navigator console can be accessed from the Cloudera Manager Admin Console or directly on the Navigator Metadata Server instance. Using the Cloudera Manager Admin Console as a starting point requires the Cloudera Manager roles of either Navigator Administrator or Full Administrator. From the Cloudera Manager Admin Console: Select Clusters > Cluster-n. Select Cloudera Navigator from the menu. To access the Cloudera Navigator console directly: Open the browser to the host name of the node in the cluster that runs the Cloudera Navigator Metad

I need help with config of  wait module. I am trying to execute lookup and if not matched , retry loop is entered with wait 1sec. second loop the message is dropped. THe mechanism works, but I worry of numbers provided by wait module. it shows 8mil files IN, and 134k files out.  Is this a correct setup ?Why and where is the 8mil IN comming from? How to avoid this?I worry it will crash when more traffic comes in. Thank you  

@MattWho  unable to log in seems authorization issue whereas cert authentication is successful. for initial admin cert user.     HDF 3.4 nifi new cluster self-signed ssl enabled - unable to login nifi with cert "Insufficient Permissions"   Insufficient Permissions home No applicable policies could be found. Contact the system administrator.   nifi-user.log:   2020-03-26 04:47:13,898 INFO [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (CN=nifiadmin, OU=NIFI) GET https://lpappnifixx:9091/nifi-api/flow/current-user (source ip: 72.191.44.165)2020-03-26 04:47:13,899 INFO [NiFi Web Server-17] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for CN=nifiadmin, OU=NIFI2020-03-26 04:47:13,901 INFO [NiFi Web Server-17] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[CN=nifiadmin, OU=NIFI], groups[] does not have permission to access the requested resource. No applicable policies could be found. Returning Forbidden respons

HI Can someone let me know if CDH 5.15 and CDH 5.16 is supported on Ubuntu 18? The below doc does not mention anything about Ubuntu 18 so just confirming https://docs.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#c516_supported_os Thanks,

I have a pipeline where input source is GetFile , reading csv files ,updating date format using UpdateRecord and then saving in my sql database. Configuration to update time format is :property=>${field.value:toDate("mm/dd/yyyy hh:mm:ss"):format("YYYY-MM-DD HH:mm:ss")} Problem is I am getting date parsing exception at Update Record and date value read by CSV reader is 4/5/2019 0:00. Getting : Cannot parse attribute value as a date; date format: mm/dd/yyyy hh:mm:ss; attribute value: 4/5/2019 0:00However when i open csv in excel , i get the date as 4/5/2019 12:00:00 AM , but in Notepad++ it is 4/5/2019 0:00. I am confused now where exactly the problem is with csv reader or updateRecord configuration. 

Hello All, We have 3 noded nifi cluster env set up for dev, uat and prod in our organisation. We don't have registry installed yet. From the disaster recovery perspective can anyone guide us what are the things need to be backed up ? We have flowfile xml in mind , however any pointer would really help.   Regards , Jasmine

Trying to export a hive table from an encrypted HDFS.The distcp portion of the export fails. Every file segment throws one of these...2020-03-26 14:36:17,124 ERROR [IPC Server handler 25 on 37768] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1585157114387_3676_m_000000_0 - exited : java.io.IOException: File copy failed: hdfs://myserver/warehouse/tablespace/tmp/hive/e18e8f44-9d0e-4f10-9490-2703bbdc84e9/_tmp_space.db/7579e917-2de7-4923-887a-df739f3a98e1/000006_0 --> hdfs://myserver/warehouse/tablespace/hive-export/username/tablename/data/000006_0 at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:263) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:220) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at org.apache.hadoop

Hi All,   using CDH 5.15.1 security : kerberos enabled whenever i clicked on oozie workflow under my documents getting error 403 forbidden i want to edit oozie workflow

Hi Friends,   Im trying to follow the cloudera get started tutorial and in that im following the below link Exercise 3: Explore log events interactively https://www.cloudera.com/developers/get-started-with-hadoop-tutorial/exercise-3.html   In that when im trying to create the collection using solr with the below command   solrctl --zk cloudera.myhost.com:2181/solr collection --create live_logs -s 1 Im getting the below error like   [root@cloudera flume]# solrctl --zk cloudera.myhost.local:2181/solr collection --create live_logs -s1{ "responseHeader":{ "status":0, "QTime":32369}, "failure":{ "cloudera.myhost.local:8983_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at http://cloudera.myhost.local:8983/solr: Error CREATEing SolrCore 'live_logs_shard1_replica_n1': Unable to create core [live_logs_shard1_replica_n1] Caused by: solr.IntField"}}[root@cloudera flume]# org.apache.solr.common.SolrException: Error CREATEing SolrC

I have done the Getting started with HDP sandbox tutorial so I have created hive tables with DAS and done some Spark using Zeppelin. However, as I try to use Atlas, it is suggested on the tutorial that you turn off both Spark and Zeppelin so that the Sandbox works properly (I tried to enable Atlas with Spark and Zeppelin running and it crashed and I had to restart the VM). My question is, do I need Spark running to see Data Lineage in Atlas? I am able to see hive tables, hive processes... and everything I did on DAS but there is nothing showing in relation to Spark.   Thank you.

My file: "ws:Operation": "ADD","ws:Supervisor_Name": {"ws:PriorValue": "","content": "Dr. Leah Thaler"   In above example i need to send the content:Dr. Leah Thaler to insert  My file: "ws:Operation": "MODIFY","ws:Supervisor_Name": {"ws:PriorValue": "","content": "Dr. Leah Thal" In above example i need to send the content:Dr. Leah Thaler to update. Database is Oracle - the flow: Getfile-> EvaluateJSONPath -> send the data to two tables using AttributesToJSON for flatten JSON -> ConvertJSONSQL to Putsql.    Note: I dont want to write AVRO schema. So we build this flow 

Hello together,   I splitted a flow file for each x value. With value x I fetch value y and set it to the Attribute. Is it possible if I merge the FlowFiles together, that the Attribute values can be merged comma-seperated too?

I recently found NiFi 1.10.0 was released: https://nifi.apache.org/download.html   Does anyone have an ETA on a public repo and management pack?   I started on a management pack to get 1.10.0, but after I get into the HDP/HDF stack it's still using public repos reporting only 1.9.x.   I am going to start working on my own repos, but wanted to see if anyone was already working on the next public repo and mpack?    

i'm not able to process hive queries using Hive view can you help me to find any alternative client tool for HDP-3 cluster

INFO tools.DistCp: Input Options: DistCpOptions ooxxINFO client.AHSProxy: Connecting to Application History server at ooxxINFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 256 for oo at xxINFO security.TokenCache: Got dt for hdfs://clusterA:8020; Kind: HDFS_DELEGATION_TOKEN, Service: ...INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1; dirCnt = 0INFO tools.SimpleCopyListing: Build file listing

I am trying to rebalance hdfs with Cloudera Manager 6.3 with HDFS Balancer Document It says add the same property dfs.datanode.balance.max.concurrent.moves into different section  DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml  Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml But before adding the property I searched and saw dfs.datanode.balance.max.concurrent.moves was already there.  Nevertheless, I did what the document says. After adding properties Cloudera Manager asked me to restart/redeploy stale configurations. Before restart, I saw totally different properties added.  I don't understand although we seem to add the same property why different properties are added to hdfs-site.xml?  

Hello All - Wondering, if there is a feature in CDSW to see "History and Rollback" information by any chance?   Thanks, Dil

from pyspark sql import SparkSessionspark = SparkSession.builder.appName("aaa").master("local") \ .getOrCreate()df = spark.read.csv("hdfs://ec2-18-234-71-106.compute-1.amazonaws.com:8020/dataset/Tech.csv", header=True, inferSchema=True)df.show() Spoiler (Highlight to read) 20/03/22 11:08:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableUsing Spark's default log4j profile: org/apache/spark/log4j-defaults.propertiesSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).[Stage 0:> (0 + 1) / 1]20/03/22 11:08:57 WARN BlockReaderFactory: I/O error constructing remote block reader.java.net.ConnectException: Connection timed out: no further informationat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)at org.apache.hadoop.net.SocketIOWithTimeout.connec

Hello,   I would like to know if there a way to rebalance data in Kudu evenly across all kudu t-servers. Our Kudu deployment is as follows: 3 Kudu Masters 9 Tablet Servers kudu 1.7.0-cdh5.16.2/ CM 5.16.2   Data across these 9 T-Servers is not evenly distributed, out these 9 t-servers, I see data is more stored on 3 t-servers and not distributed evenly. I was going through some articles and found that currently there is no rebalance tool like HDFS (https://community.cloudera.com/t5/Support-Questions/Kudu-Tablet-Server-Data-Directories-rebalancing/td-p/79649),   However if we Go to Clusters > Kudu > Click Actions I see Run Kudu Rebalancer Tool, would like to know, what is the purpose of this. Will this distribute data for overall Kudu or just Kudu Master or Kudu T-Servers too. Request some advice  / assistance on the same.   Thanks Amn

  May I know please how to enable this feature if we got this error/issue message?

Hi Guys   I need to know the following, thanks in advance:   i) Do List processors w/ timestamp tracking store state locally?    ii) Does this state survive NiFi restarts?   iii) If running on primary node only, would this mean when another primary node is chosen, the List processor would list any files it hasn't tracked (and re-ingress a large backlog of files if still there)?   iv) What out-of-box solutions can help to get around the issue of non-persisted non-distributed listing, or do we need custom auditing triggering individual listings?

Hi, currently  I have 2 process groups on NiFi 1.11.3, the native one is showing data provenance on every processor as it should. The imported template (from 1.9.2) inside another group shows a blank provenance window. Any idea of what can be causing this? Thank you!