Find answers, ask questions, and share your expertise
Recently active
Hi there,I am trying to load simple CSV to mySQL but i have some difficulties with NULL and empty string values.This is my table for example:create table test1 (id int not null, name varchar(255) not null,age int null);and my data looks like this in the mySQL:*the blank values are not NULL they were inserted like:insert into test(id,name) values (2,'');Now when I export this into CSV and try to load this CSV back to mySQL it fails to understand that those empty values are just blank and not NULLs.This is my NiFi configuration: I followed the template from another question I found here and removed some parts that I find like I don't need (or maybe I need??) = removed the AvroSchemaRegistry and instead used Use String Fields From Header.I tried many things from exporting values with double quotes,single quotes but still didn't manage to figure out how to load empty/blank/'' values instead of NULLs.Please help 🙂
HI Team,System Env Details: I am trying setup NIFI on AWS3 node clusterNIFI version: 1.16.0authorisers.xml : <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Legacy Authorized Users File"></property> <property name="Initial User Identity 1">xxx@xxx.com</property> <property name="Initial User Identity 2">C=US, ST=xxx, O=xxx, OU=xxxx, CN=%%cn_user_identity_1%%</property> <property name="Initial User Identity 3">C=US, ST=xxx, O=xxx, OU=xxxx, CN=%%cn_user_identity_2%%</property> <property name="Initial User Identity 4">C=US, ST=xxxxx, O=xxxx, OU=xxxx, CN=%%cn_user_identity_3%%</property> </userGroupProvider> <accessPolicyProvider>
Hello all...i am very new to big data. I currently have downloaded HDP (sandbox) 2.6.5. I also have downloaded HDF 3.1.1 where I practice. Is there anyway i can download HDF services on HDP 2.6.5? if not is there any way to download only Nifi on hdp2.6.5? It will also be very nice to have any answer to download airflow schduler instead of using bulit-in workflow manager?...i try to look and many documentations that was recommended here but no avail.Thank you
Hi all We have Ambari HDP cluster ( HDP version - 2.6.4 ) , with 420 workers linux machines ( when each worker include data node and node manager service ) Unfortunately Ambari DB is damaged , and we not have Ambari DB dump , so we cant recover Ambari DB , so actually we not have Ambari and Ambari GUI But HDFS disks on workers machines include HDFS data , and name node is still working with all data as ( journal/hdfsha/current/ ) and ( namenode/current )So HDFS works without Ambari So regarding what I said until now - it is possible install new Ambari cluster from scratch , and then add existing working HDFS data to the cluster ?Dose hortonworks / cloudera have procedure for this process ?
Hi,I want to execute a flow in two NiFi nodes and I'm fetching data from Cassandra running in the primary node only, but then I have a DistributeLoad with "round robin" strategy to distribute the data for the two nodes.The rest of the processors are with Execution on "All Nodes", is the DistributeLoad redundant? Meaning, if I don't use the DistributeLoad but still use the Execution on "All Nodes" for the rest of the processors, I can still have parallel processing jobs across the nodes? Or are the FlowFiles only processed in the current node?Thanks
Hi Team,I am creating year date time folder thru groovy ExecuteScript processor and using command execute using creating sh file and writing some content in that.. and passing that sh file to downstream ExecuteStreamCommand processor. Its failing with permission error on sh file.How to give full permission to the file which we create. def commmnd1 = "mkdir -p /shared_path/data/poc/test/1/script"def commmnd3 = "mkdir -p /shared_path/data/poc/test/1/output"def process1 = commmnd1.execute();def process3 = commmnd3.execute();def filePath = "${script_dir}/abcd.sh"def file2 = new File(filePath)file2.write "#!/bin/bash\n";and one more, ExecuteStreamCommand executing sh file and it generating some files in dynamically created by previous ExecuteScript processor and these are files does not have full permission. How to give full permission which we create folder and files dynamically. Please suggest. Thanks in Advance.
Hello all Within NiFi, updateAttribute processor I am trying to change an attribute called 'hive_database' based on the value of another attribute called 'hostname'. For instance if:#1: hostname = Mickey Mouse#2: hostname = James Bond I want to use a 'contains' statement (or similar ) to change the output of new attribute 'hive_database' property:hive_databasevalue:${hostname:contains('Mickey'):<OUTPUT>('cartoon')} <OR>${hostname:contains('Mickey'):<OUTPUT>('movie')} So that #1 output would be: property= hive_database value= cartoon The parts I'm stuck on is the OUTPUT and OR parts above, do you know if this would be possible using the update attribute processor? I am trying to avoid having to use routeOnAttribute to break out multiple different 'hosname' values to send data to the relevant 'hive_database' Hope this makes sense, unable to share screenshots etc as sensitive vales. Any help greatly appreciated.
Hello Community,I have a configuration where ListenSysylog is listenig and provides the received data to InvokeHTTP. Also I use a funnel to redirect Http failures to the InvokeHttp once again. The problem occurs when InvokeHttp tries to resend data and it consumes so many threads that ListenSyslog starts droping incoming data. I do not want to lost any incoming data and in the same time I want to be sure that data will be send. Do you have a similar problem? Is there any way to limit proccessors?Thanks
We have newly install CDP-DC 7.1.7 and try to load data using a spark jobs but getting the errors.----Payloadjava -cp /etc/hadoop/conf.cloudera.hdfs/ssl-client.xml:/etc/hbase/conf.cloudera.hbase/hbase-site.xml:/etc/hadoop/conf.cloudera.hdfs/core-site.xml:/etc/hadoop/conf.cloudera.hdfs/hdfs-site.xm:/data/scripts/LeaApp-1.0-SNAPSHOT.jar net.ba.lea.transformation.FileActions "/data/scripts/msc/IN/" "/tmp/nss_processing/" "/data/scripts/msc/reject/" "250" "LEA.DBM_CDR_FILE_HEAD" "NSS" "jdbc:phoenix:gzvlcdpnode01.ba.net:2181:/hbase:phoenix/gzvlcdpnode02@BA.NET:/etc/security/keytab/phoenix.keytab"error :Can't find method newStub in org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService!--------------------------------But the same code is working fine with CDP-DC 7.1.3.We have changed the version only.Please guide us .
Hello all,I am using Cloudera Datasservices on CDP Private Cloud. I am using using the embedded container service (ECS).If i check the kubernetes pods i see that the pods of the namespace shared-services are always crashing (Expect of openshift-idling-controller-manager) kubectl get pods -n shared-servicesNAME READY STATUS RESTARTS AGElog-router-fznw2 0/2 Init:CrashLoopBackOff 17 46mlog-router-jnjrb 0/2 Init:CrashLoopBackOff 17 46mlog-router-tptbg 0/2 Init:CrashLoopBackOff 17 46mopenshift-idling-controller-manager-0 1/1 Running 0 45m If i query the logs of the pod I get the following error:kubectl logs -f -n shared-services log-router-jnjrb sdx-init-container I have already tried to delete the pods but without success.But the behaviour is unchanged:Who can help?Thanks in advance.Best regardsUli {"timestamp":"2022-10-07 07:35:52.120 +0000","level":"WARN","thread":"Thread-8","logger":"c.cloudera.cdp.config.ConfigTemplate","message":"[2022-10-07 07:35:52.12
Hello Guys! I updated HDP from 3.1.0.0-78 to 3.1.5.6189-1 as well as Ambari from 2.7.3 to 2.7.5.I want to remove the old stack of HDP as the files are still on the servers.Is there any official guide for this move?
I have cluster 1 (the source cluster in this example) which does not have HA enabled, and I have cluster 2 (destination cluster in this example) which does have HA enabled. Both clusters are running the same version of HDP. When I attempt to copy data from cluster 1 to cluster 2 via distcp using the active name node instead of the HA nameservice, it works but ultimately causes problems when the active name node switches. I have tried using the HA nameservice, but without any configurations, it fails (as one would expect). When modifying the hdfs-site.xml to add cluster 2's nameservice information (following this guide laid out here but only for one HA cluster: https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/scaling-namespaces/topics/hdfs-distcp-between-ha-clusters.html), the HDFS services on cluster 1 fail to restart, as the name nodes appear to be attempting to join cluster 2. Is this sort of configuration even possible, or do we need HA on both cl
We want to configure listenhttp processor that listen our web service. question is what are the configuration in http listener processor in nifi such that when we will make curl post request this ListenHttp processor will catch the update. basically i didn't understand what i have to put in Base path (is it i have to put nifi address i.e http://localhost:8080/contentListener or our web app service URL???) what is contentListener. And is it i have to put our web app port number in listening port? in the request command what i have to put in url part curl --request POST 'http://(here i have to used the nifi url)' -H 'Authorization: Key demo-key i know this is very basic question but i'm just using this don't know any thing about nifi and curl command .....thanks in advance
Hi all, this may be an easy question, but just wanted to verify. When adding new brokers to the cluster, after installing the Kafka role/instance on the new hosts, should you start the new broker instances first before you restart the existing brokers? Or does it matter? This is CDP Private Cloud 7.1.7Thanks
Hello everyone, I am trying to run a python script in a dockerized enviornment using spark and yarn, the cluster is kerberised and I provided the needed realm and keytab for the spark-submit command. I still face this issue where java only sees user null for some reason. What I tried : spark-submit \--master yarn \--deploy-mode cluster \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=MyImage \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro,/opt/cloudera/parcels/:/opt/cloudera/parcels/:ro,/etc/krb5.conf:/etc/krb5.conf:ro" \--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=MyImage \--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro,/opt/cloudera/parcels/:/o
I have a hive table with three columns, delimited by tabshive (database)> describe formatted my_table;# col_name data_type commentfield1 string field2 string field3 string... ...Storage Desc Params: input.regex (\\S+)\\s+(\\S+)\\s+(.*)hive (database)> select * from my_table limit 2;DataX DataX DataXDataX DataX DataXI want to add two new columns to the hive table schema, so I did:// add two columnshive (database)> alter table my_table add columns(field4 string,field5 string);// add two more (\\S+)\\s+ for the new columnshive (database)> alter table my_table SET SERDEPROPERTIES ("input.regex" = "(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+(.*)");But seems it doesn't work with the old dataset (the old data file with three columns):hive (database)> select * from my_table limit 2;NULL NULL NULL NULL NULLNULL NULL NULL NULL NULLI expected something like this:DataX DataX DataX NULL NULLDataX DataX DataX NULL NULL 
Is there any other way to migrate the Hortonworks HDP3 cluster to the Apache Hadoop 3.3.2 other than the distcp?is it possible to upgrade by updating the jar and pointing the older metadata to the newer stack of Hadoop? Thanks
Is CDP 7.1.7 SP1 a stable release? Is it recommended upgrade to this version? We still see some log4j 1X findings in this version so we need to now if someone has faced the same issue.
Hi Team, We have 2 Hive Metastore server services configured on cluster in node1 and node2. In node1 Hive Metastore getting alert with below message. "Hive Metastore Canary" The Hive Metastore canary failed to create a database. I am seeing below errors in servicemonitor log file. 2022-10-12 22:40:06,657 WARN com.cloudera.cmf.cdh6client.hive.MetastoreClientImpl: (2 skipped) Could not drop hive database: cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_0137966f79e5f15b3b5d4dec61b7592ecom.cloudera.cdh6client.hive.shaded.org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed outat com.cloudera.cdh6client.hive.shaded.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)at com.cloudera.cdh6client.hive.shaded.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)at com.cloudera.cdh6client.hive.shaded.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)at c
Hello everyone.i need help. I have a JSON file which contains in an object either an array of other objects or a single object. This is the relevant object "Product": [ or "Product": { .How can I configure with the JOLT Transform processor to always expect an array in the object? The product object must always be treated as an array, even if it contains only a single object. { "costc": 9638, "sum_amount": 543, "sum_invitation": 0, "sum_selfconsumption": 0, "sum_loss": 0, "sum_gross": "2764,000", "sum_net": "2392,480", "sum_margin": "2003,780", "sum_costs_total": "388,700", "sum_costs": "388,700", "sum_costs_invitation": 0, "sum_costs_selfconsumption": 0, "sum_costs_loss": 0, "sum_costs_total_percent": "16,25", "sum_costs_percent": "16,25", "Classes": { "ClassSummary": [ { "class": 1102, "sum_amount": 117, "sum_invitation": 0, "sum_selfconsumption": 0, "sum_loss": 0, "sum_gross": "740,500", "sum_net": "692,060
Running shell actions(Only sleep command) in Oozie workflows for more than 7 days throws JA017 error on exit.We saw this problem in a shell script that submits a spark job.Spark job succeeded but submit shell action failed.We tested by creating a shell action that does a simple sleep and we see the same failure.https://blog.cloudera.com/hadoop-delegation-tokens-explained/I saw this blog and tried to increase max-lifetimedfs.namenode.delegation.token.max-lifetime (hdfs-site.xml)restart NameNode, Oozie, YARN ResourceManagerbut the same fails. If I run the Oozie workflow and look at the logs, I see that I get 3 delegation tokens.RM_DELEGATION_TOKEN, MR_DELEGATION_TOKEN, HDFS_DELEGATION_TOKENBecause the dfs.namenode.delegation.token.max-lifetime setting value is increased, the maxDate of HDFS_DELEGATION_TOKEN is increased.The maxDate of RM_DELEGATION_TOKEN and MR_DELEGATION_TOKEN tokens did not increase.Why does authentication problem occur due to expiration of delegation token when O
Hi, In BDR HDFS replication I want to exclude all files which end with ".tmp" in the directory "/backups/" Appreciate it if somebody could give the expression for this to add in BDR "Add Exclusion". Thanks
Hi , We upgraded OS of one of the server in cluster from RHEL 7.5 to 7.9. After that we are unable to start cloudera services from webui, getting below errors in agent log file. can anyone help me to get rid of this issue. AttributeError: 'NoneType' object has no attribute 'get'[30/Sep/2022 19:37:38 +0000] 1048 MainThread agent WARNING Long HB processing time: 7.00362992287[30/Sep/2022 19:38:01 +0000] 1048 DnsResolutionMonitor throttling_logger INFO DnsTest not running. Java not located.[30/Sep/2022 19:38:37 +0000] 1048 MonitorDaemon-Reporter firehoses INFO Creating a connection to the ACTIVITYMONITOR.[30/Sep/2022 19:38:37 +0000] 1048 MonitorDaemon-Reporter firehoses INFO Creating a connection to the SERVICEMONITOR.[30/Sep/2022 19:38:37 +0000] 1048 MonitorDaemon-Reporter firehoses INFO Creating a connection to the HOSTMONITOR.[30/Sep/2022 19:38:37 +0000] 1048 MonitorDaemon-Reporter throttling_logger ERROR Error sending messages to firehose: mgmt-HOSTMONITOR-d85e01b86fca15e352
hello cloudera community we are having a problem with zookeeper on cdp after enabling kerberos on the cluster zookeeper instances are healthy (status green), but the general status of zookeeper shows the message: Canary test of client connection to ZooKeeper and execution of basic operations succeeded though a session could not be established with one or more servers how can we solve this problem?
Hi All,I am trying to run a Spark job using a python script in CDE cluster. The script is from a CDE tutorial.The job fails due to the error message in log files (in Subject). Appreciate any inputs.Thanks in advance .... Sekhar
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.