Q&A

I have a pipeline where input source is GetFile , reading csv files ,updating date format using UpdateRecord and then saving in my sql database. Configuration to update time format is :property=>${field.value:toDate("mm/dd/yyyy hh:mm:ss"):format("YYYY-MM-DD HH:mm:ss")} Problem is I am getting date parsing exception at Update Record and date value read by CSV reader is 4/5/2019 0:00. Getting : Cannot parse attribute value as a date; date format: mm/dd/yyyy hh:mm:ss; attribute value: 4/5/2019 0:00However when i open csv in excel , i get the date as 4/5/2019 12:00:00 AM , but in Notepad++ it is 4/5/2019 0:00. I am confused now where exactly the problem is with csv reader or updateRecord configuration. 

Hello All, We have 3 noded nifi cluster env set up for dev, uat and prod in our organisation. We don't have registry installed yet. From the disaster recovery perspective can anyone guide us what are the things need to be backed up ? We have flowfile xml in mind , however any pointer would really help.   Regards , Jasmine

Trying to export a hive table from an encrypted HDFS.The distcp portion of the export fails. Every file segment throws one of these...2020-03-26 14:36:17,124 ERROR [IPC Server handler 25 on 37768] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1585157114387_3676_m_000000_0 - exited : java.io.IOException: File copy failed: hdfs://myserver/warehouse/tablespace/tmp/hive/e18e8f44-9d0e-4f10-9490-2703bbdc84e9/_tmp_space.db/7579e917-2de7-4923-887a-df739f3a98e1/000006_0 --> hdfs://myserver/warehouse/tablespace/hive-export/username/tablename/data/000006_0 at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:263) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:220) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at org.apache.hadoop

Hi All,   using CDH 5.15.1 security : kerberos enabled whenever i clicked on oozie workflow under my documents getting error 403 forbidden i want to edit oozie workflow

Hi Friends,   Im trying to follow the cloudera get started tutorial and in that im following the below link Exercise 3: Explore log events interactively https://www.cloudera.com/developers/get-started-with-hadoop-tutorial/exercise-3.html   In that when im trying to create the collection using solr with the below command   solrctl --zk cloudera.myhost.com:2181/solr collection --create live_logs -s 1 Im getting the below error like   [root@cloudera flume]# solrctl --zk cloudera.myhost.local:2181/solr collection --create live_logs -s1{ "responseHeader":{ "status":0, "QTime":32369}, "failure":{ "cloudera.myhost.local:8983_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at http://cloudera.myhost.local:8983/solr: Error CREATEing SolrCore 'live_logs_shard1_replica_n1': Unable to create core [live_logs_shard1_replica_n1] Caused by: solr.IntField"}}[root@cloudera flume]# org.apache.solr.common.SolrException: Error CREATEing SolrC

I have done the Getting started with HDP sandbox tutorial so I have created hive tables with DAS and done some Spark using Zeppelin. However, as I try to use Atlas, it is suggested on the tutorial that you turn off both Spark and Zeppelin so that the Sandbox works properly (I tried to enable Atlas with Spark and Zeppelin running and it crashed and I had to restart the VM). My question is, do I need Spark running to see Data Lineage in Atlas? I am able to see hive tables, hive processes... and everything I did on DAS but there is nothing showing in relation to Spark.   Thank you.

My file: "ws:Operation": "ADD","ws:Supervisor_Name": {"ws:PriorValue": "","content": "Dr. Leah Thaler"   In above example i need to send the content:Dr. Leah Thaler to insert  My file: "ws:Operation": "MODIFY","ws:Supervisor_Name": {"ws:PriorValue": "","content": "Dr. Leah Thal" In above example i need to send the content:Dr. Leah Thaler to update. Database is Oracle - the flow: Getfile-> EvaluateJSONPath -> send the data to two tables using AttributesToJSON for flatten JSON -> ConvertJSONSQL to Putsql.    Note: I dont want to write AVRO schema. So we build this flow 

Hello together,   I splitted a flow file for each x value. With value x I fetch value y and set it to the Attribute. Is it possible if I merge the FlowFiles together, that the Attribute values can be merged comma-seperated too?

I recently found NiFi 1.10.0 was released: https://nifi.apache.org/download.html   Does anyone have an ETA on a public repo and management pack?   I started on a management pack to get 1.10.0, but after I get into the HDP/HDF stack it's still using public repos reporting only 1.9.x.   I am going to start working on my own repos, but wanted to see if anyone was already working on the next public repo and mpack?    

i'm not able to process hive queries using Hive view can you help me to find any alternative client tool for HDP-3 cluster

INFO tools.DistCp: Input Options: DistCpOptions ooxxINFO client.AHSProxy: Connecting to Application History server at ooxxINFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 256 for oo at xxINFO security.TokenCache: Got dt for hdfs://clusterA:8020; Kind: HDFS_DELEGATION_TOKEN, Service: ...INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1; dirCnt = 0INFO tools.SimpleCopyListing: Build file listing

I am trying to rebalance hdfs with Cloudera Manager 6.3 with HDFS Balancer Document It says add the same property dfs.datanode.balance.max.concurrent.moves into different section  DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml  Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml But before adding the property I searched and saw dfs.datanode.balance.max.concurrent.moves was already there.  Nevertheless, I did what the document says. After adding properties Cloudera Manager asked me to restart/redeploy stale configurations. Before restart, I saw totally different properties added.  I don't understand although we seem to add the same property why different properties are added to hdfs-site.xml?  

Hello All - Wondering, if there is a feature in CDSW to see "History and Rollback" information by any chance?   Thanks, Dil

from pyspark sql import SparkSessionspark = SparkSession.builder.appName("aaa").master("local") \ .getOrCreate()df = spark.read.csv("hdfs://ec2-18-234-71-106.compute-1.amazonaws.com:8020/dataset/Tech.csv", header=True, inferSchema=True)df.show() Spoiler (Highlight to read) 20/03/22 11:08:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableUsing Spark's default log4j profile: org/apache/spark/log4j-defaults.propertiesSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).[Stage 0:> (0 + 1) / 1]20/03/22 11:08:57 WARN BlockReaderFactory: I/O error constructing remote block reader.java.net.ConnectException: Connection timed out: no further informationat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)at org.apache.hadoop.net.SocketIOWithTimeout.connec

Hello,   I would like to know if there a way to rebalance data in Kudu evenly across all kudu t-servers. Our Kudu deployment is as follows: 3 Kudu Masters 9 Tablet Servers kudu 1.7.0-cdh5.16.2/ CM 5.16.2   Data across these 9 T-Servers is not evenly distributed, out these 9 t-servers, I see data is more stored on 3 t-servers and not distributed evenly. I was going through some articles and found that currently there is no rebalance tool like HDFS (https://community.cloudera.com/t5/Support-Questions/Kudu-Tablet-Server-Data-Directories-rebalancing/td-p/79649),   However if we Go to Clusters > Kudu > Click Actions I see Run Kudu Rebalancer Tool, would like to know, what is the purpose of this. Will this distribute data for overall Kudu or just Kudu Master or Kudu T-Servers too. Request some advice  / assistance on the same.   Thanks Amn

  May I know please how to enable this feature if we got this error/issue message?

Hi Guys   I need to know the following, thanks in advance:   i) Do List processors w/ timestamp tracking store state locally?    ii) Does this state survive NiFi restarts?   iii) If running on primary node only, would this mean when another primary node is chosen, the List processor would list any files it hasn't tracked (and re-ingress a large backlog of files if still there)?   iv) What out-of-box solutions can help to get around the issue of non-persisted non-distributed listing, or do we need custom auditing triggering individual listings?

Hi, currently  I have 2 process groups on NiFi 1.11.3, the native one is showing data provenance on every processor as it should. The imported template (from 1.9.2) inside another group shows a blank provenance window. Any idea of what can be causing this? Thank you!

Hi, I have a hadoop environment that has 2 nifi nodes running. I am not able to connect to the nifi UI since yesterday and i try do restart one of the node without any success....   In in nifi-app.log, file seeing this as only Error level error: 2020-03-17 13:42:37,974 ERROR [main] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for ***-nf02.****.ca:9091 -- Node disconnected from cluster due to org.apache.nifi.controller.UninheritableFlowException: Failed to connect node to cluster because local flow is different than cluster flow.   Could please help?

Hi all,I'm trying to create a function to use in imapla.my function is simply re-using  hive's sha2() function.the creation of the function goes smoothly :create function to_sha2(string,int) returns string location 'user/hive/hive.jar' symbol='org.apache.hadoop.hive.ql.udf.generic.GenericUDFSha2' ;  but when I try to use it doesn't work launching this warning  : select to_sha2('test',256);   Query State: EXCEPTION Query Status: ClassCastException: class org.apache.hadoop.hive.ql.udf.generic.GenericUDFSha2     I have tried to search for UDFSha2 that doesn't contain the Generic word in the hive's jar but I couldn't find it.the original built-in function in hive :sha2(string/binary, len) - Calculates the SHA-2 family of hash functions (SHA-224, SHA-256, SHA-384, and SHA-512). |the other functions are working normally in impala (for example  I have tried to create UDF MD5 function from hive's jar and it was working ).so my question

Restart Spark2 ClientTask Logstderr: /var/lib/ambari-agent/data/errors-8648.txt 2020-03-20 16:05:33,015 - The 'spark2-client' component did not advertise a version. This may indicate a problem with the component packaging.Traceback (most recent call last):File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/spark_client.py", line 55, in <module>SparkClient().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in executemethod(env)File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 966, in restartself.install(env)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/spark_client.py", line 35, in installself.configure(env)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/spark_client.py", line 41, in configuresetup_spark(env, 'client', upgrade_type=upgrade_type, action = 'config')File "/var/lib/ambari-agent/cache/sta

I would like to know what is the difference between the HDP Sandbox VM and CDH.   I am looking to setup CDH on GCP,can anyone let me know would i still be able to use the services after the 60 days trial phase if not what is the cost like?This is for personal use.   Is it also possible with HDP Sandbox on GCP?Again for own use and not for any business account.  

Hi, Can i install cloudera manger offline?  Thanks

i want to install Ambari 2.7.5, refer to page  https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/access_ambari_paywall.html .   after log in to cloudera and navigate to https://my.cloudera.com/ . I'm can't file a non-technical case within the support portal (https://my.cloudera.com) to obtain credentials. because i haven't Permission.   what should i do.   Thank everyone.

Hi all,    We've recently upgraded our cluster from Cloudera Enterprise 5.13.1 to 5.16.1, and it seems the value of the Impala Query Monitoring Failures Threshold cannot be changed anymore.    By default, the Warning and Critical fields are set to "Never" and "Any" respectively, but when we try to switch them to "Specify" and set a value, we get an error that says "<value> is less than the minimum allowed value 0". We've tried typing in all sorts of values, including zero, and negatives...none are accepted. We know for sure this error didn't exist in the previous version.                 To us, it appears to be some sort of logical error in the value check, but maybe there's some other reason for this error we haven't thought of...   Does anyone have the same issue, or can think of a root cause or fix ?   cbfr