Q&A

Hi Team,I already created 3 node Nifi secured cluster with self signed certificated , where I used nifi-toolkit to do the task. But now I have signed certificate (shared by IT team), the pem and key file. Can you please help (step by step guide) to create 3 node nifi cluster using that certificate?

Hello,We have several Data Hub Clusters on CDP - PC, currently conducting health checks manually, which is time consuming and repetitive. Would like to know if there are some automated scripts / ways, we could get this task completed which would give us the health status of nodes, Cloudera services running on them.Any assistance / guidance is appreciated.

I am trying to run a merge query in PUTSQL after using ExecuteSQL to pull from source database. I then pass the results through ConvertAvrotoJSON and then to convertJSONToSQL (set to INSERT since there is no UPSERT for SQL server database). I want to now run the merge query to update if the primary key is matched or insert if not, but I have been getting all sorts of errorsInput flowfile has the following attributessql.args.1.type12sql.args.1.valueA-001sql.args.2.type12sql.args.2.valueDDDsql.args.3.type12sql.args.3.value2Bsql.args.4.type12sql.args.4.valueAAAsql.args.5.type12sql.args.5.value2Bsql.args.6.type12sql.args.6.valueBBBsql.args.7.type93sql.args.7.value1954-08-01 22:46:50.73sql.catalogCUSDBsql.tableCUSTOMERSuuid50a3daf9-e5d8-422b-b6cb-9c92da723c55I created the following attributes in UpdateAtributesI directed this flowfile to UpdateAttribute where I created above properties like this  sql.args.1.type = 12;  sql.args.1.value = ${vkey}sql.args.2.type = 12;

Hello,trying to setup zeppelin + livy-on-spark3.Livy-on-spark2 is up and rspark3.sql runs fine, any spark3.pyspark code returns error "Interpreter died:".What could be wrong?

Hello Experts,I have a requirement where I need to re-load my custom processors nar without restarting Nifi.I tried with nifi.nar.library.autoload.directory option, If I freshly place new nar first time in autoload directory it loads automatically.But if I replace the nar in autoload directory again with some enhancements, it does not get reloaded in nifi.I tried with changing nar version, with that nifi loaded that new version nar but that resulted in 2 different versions processors in Nifi (it does not unload the pre version nar)Is there any any "auto reload" feature for custom nar or something like unload prev version and load new version automatically without restart?Any help/suggestion is much appreciated.ThanksMahendra

Hi Support,Would like to ask is that possible to set the Global_parameter in processor configuration via nifi rest api instead of using UI ? 

I developed a custom hook on CDH (6.2. X), implement the HiveDriverRunHook interface and ExecuteWithHookContext interface. It runs smoothly, but when I migrate the hook to CDP (7.1. X), I report an error and cannot execute it. I suspect it was caused by the Tez engine, but I cannot solve it and need help，thanks very much

 

Hello Team,I am trying to use the underscore "_" in the column name example "_name" but the create statement is failing. i also wanted to use the reserved keywords example "transform" as column name but that is also not working.I tried setting the values for keyword and for identifiers while running command still it is giving me error when running via beeline  Following are the commands and the error that i am getting  :Command : sudo -u hive beeline -u "jdbc:hive2://cdp1:2181,cdp2:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;;principal=hive/_HOST@hdpcluster.test" -n hive --showHeader=false --outputformat=tsv2 -e "SET hive.support.quoted.identifiers=column;use jayesh; create external table testkeyword1( `transform` string);";error : Error: Error while compiling statement: FAILED: ParseException line 1:45 cannot recognize input near ')' '<EOF>' '<EOF>' in column type (state=42000,code=40000)------------------------------------------

Hi,I am trying to convert json to csv in nifi. My json is something like below {"tokens":["12343455577666","2134421241241","124142141411445","21425654768686"]}and my csv should be like belowtokens12343455577666213442124124112414214141144521425654768686Please help me. Thanks!

Hi all, Have hit a snag when trying to setup a Ozzie workflow with combination of CTE and Insert Statement, not sure why, please help me look at the code, thanks.  set hive.exec.dynamic.partition=true;set hive.exec.dynamic.partition.mode=nonstrict;insert into table A partition(date)with test as(select col1 as name1 , col2 as name2, col3 as name3 from table B),select * from test   

Hello everyone,I am new for spark processing. I need you help my problem when transform data from plat file to hive orc table. Below is my process flow using pyspark:1 - Use pyspark to load flat file to dataframe2 - Transform data in dataframe and insert to hive table(parquet)3 - Insert data from hive(parquet) to orc format Step 1 and 2 is fast, but step 3 is too slow because it use much memory. Sometime it stuck cannot continue. Please help to advice and recommend the better flow. Thanks Here is sample code:-- loading.pyimport pysparkfrom pyspark import SparkContext, SparkConffrom pyspark.conf import SparkConffrom pyspark.sql import SparkSessionfrom pyspark.sql import HiveContext,SQLContextfrom pyspark.sql.types import StructType,StructField, StringType, IntegerTypefrom pyspark.sql.types import ArrayType, DoubleType, BooleanTypefrom pyspark.sql.functions import input_file_name,col,array_containsspark = SparkSession.builder.appName("testing..").enableHiveSupport().g

Hi, i need to know how to create a code in both PIG and Hive for this data: I need to obtain the sum of the newcases for the continents, as well as which country has had the highest number of cases in each continent and their number. somethig like: For pig i was trying somethig like: Any one knows how to do it? Thanks.

after trying a bunch of things I would like to know if there's a way to connect local spark to Hive without getting this error -  This happens right after I successfully login through kerberos.  I have copied my hive-site.xml from server to local, as spark on server is working perfectly fine.  Am I missing any files? Hi, after trying a bunch of things I would like to know if there's a way to connect local spark to Hive without getting this error -  ```23/07/12 13:26:58 ERROR metastore: Couldn't create client transportjava.io.IOException: Could not instantiate SASL transportat org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Client.createClientTransport(HadoopThriftAuthBridge.java:222)at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:432)at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:245)at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<i

Hi everyone, So I've inherited a kerberized Cloudera cluster and I'm learning as I go. Right now I'm trying to get Airflow to work with our Spark jobs but without success. As I understand Airflow was installed by our OS team only after the cluster was configured by Cloudera. It runs on our edge node from where we run our jobs. Basically I'm using bash operators for my test DAG with the following tasks: Task 1:Kinit the user that is running the script:"echo 'password' | kinit user@domain" Task 2:Download some files from some location. Task 3:spark-submit /path/to/script.py  Task 1 and 2 work fine, but task 3 fails with the following: py4j.protocol.Py4JJavaError: An error occurred while calling o32.csv.: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS] I am a bit confused by this as I am am authenticating the user as a f

Hello Experts,After changing MIT Kerberos to AD Kerberos and Regenerating all the Kerberos credentials in CM the zookeeper, YARN etc. is not starting. There is an error about the Active Directory samaccount not able to login as the zookeeper principal. I checked that the principals are created in the AD OrgUnit for Cloudera. And the $ kinit -kt zookeeper.keytab zookeeper/redacted@ADREALM on the linux servers works fine.  Any thoughts how to fix? SERVICE_TYPEZOOKEEPER SEVERITYCRITICAL STACKTRACEjavax.security.sasl.SaslException: Problem with callback handler [Caused by javax.security.sasl.SaslException: redacted@ADREALM.COM is not authorized to connect as zookeeper/redacted@ADREALM.COM] at com.sun.security.sasl.gsskerb.GssKrb5Server.doHandshake2(GssKrb5Server.java:333) at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:161) at org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer.authenticate(SaslQuorumAuthServer.java:

Hi Experts, We are on 7.1.6 CDP version and exposed to CVE-2022-25168.  Versions affected: 2.0.0 to 2.10.1, 3.0.0-alpha to 3.2.3, 3.3.0 to 3.3.2 To address this vulnerability, we are planning to upgrade to CDP 7.1.7 as I got this information from one of the posts in the cloudera community. My question is does anyone did this upgrade before from 7.1.6 to 7.1.7 to eliminate this CVE. If yes, then how easy is it do we need to make changes in the code level also or we can do the upgrade from the GUI by just downloading, distributing and activating the new version.   Please help with your suggestion and if possible, please share the documentation for this upgrade. Thanks 

Hello,I am trying to get an key from GetMongo Processor and then I am trying to replace the value  in the format key=valueusing  ReplaceText where value here is the result which I am getting and then passing  to different POST API which supports `application/x-www-form-urlencoded`From GetMongo Processor I am getting value {match_id: 789456}and In replaceText Processor I am using in Search Value : (?s)(^.*$) and Replacement Value in match_id=${match_id:urlEncode()}But in API data is getting passed as { match_id: '' } though it should pass as  { match_id: 789456 } What I am doing wrong in ReplaceText Processor Can you please help me out with this ?  

Hello, - I've setup a Sandbox HDP 3.0.1.- Determine Network Adapter of Your VirtualBox Sandbox (Bridged Adapter, IP:192.168.1.37).- I edited the hosts file (on my computer, windows) and assigned the domain sandbox-hdp.hortonworks.com to the IP address of 192.168.1.37, but When I start all services via the Ambari UI, I receive the following error message:   safemode: Call From sandbox-hdp.hortonworks.com/172.18.0.2 to sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2019-09-29 18:25:10,968 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://sandbox-hdp.hortonworks.com:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. safemode: Call From sandbox-hdp.hortonworks.com/172.18.0.2 to sandbox-hdp.hortonworks.com:8020 failed on connection exception: java.net.ConnectException: Connection

Hi Experts,To address few of the vulnerabilities present in CDP 7.1.6 we were advised to upgrade to CDP 7.1.7 SP2 version and then we came across with Cloudera releasing the new CDP 7.1.9 version. So, we are confused as to which one should we go for.We are still using HDFS for storage and hive, impala, spark, Tez, oozie as other services in the stack along with JDK 8 and RHEL 7.8, we don't intend to compromise our current flow hope this functionality we can use in CDP 7.1.9. Please share what's your suggestion on this. our architect mostly prefer selecting the -1 version of the current version so technically we should be selecting CDP 7.1.8 which is the -1 version of CDP 7.1.9. The same was done during our last upgrade wherein we moved to CDP 7.1.6 where 7.1.7 was already released. 

I have a requirement to add custom user for allowing impersonation. I can see that CDH 6.3 documentation has some references to it. However, I could not see how I can add similar entries through Cloudera Manager in CDP 7.x. Can someone help with that information ?Essentially, I would like to add the entries like this.  <property> <name>hadoop.proxyuser.super.hosts</name> <value>10.222.0.0/16,10.113.221.221</value></property><property> <name>hadoop.proxyuser.super.users</name> <value>user1,user2</value></property>or something like this.<property> <name>hadoop.proxyuser.alice.hosts</name> <value>host_a,host_b</value></property>  Is there a way to achieve the same through Cloudera Manager in CDP 7.x ? I am not sure if I should be editing the core-site.xml on the nodes directly as those were Auto Generated by the Cloudera Manager.

I deployed nifi on a cluster of 10 servers, I have 5 external zukipirs which are successfully used by Kafka on the same 10 servers, after starting the nifi.service process I see the error The Flow Controller is initializing the Data Flow in the web version and it doesn’t go further than this message I in the web version, Data Flow doesn’t go anywhere. and in nifi-app.log 2023-09-27 16:37:11,645 INFO [main] o.a.n.c.p.AbstractNodeProtocolSender Cluster Coordinator is located at sd-sagn-rtyev:9082. Will send Cluster Connection Request to this address2023-09-27 16:37:11,665 WARN [main] o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: java.net.SocketException: Broken pipe (Write failed)help me!

I don't use column security in ranger. but when I apply any operations to a table consisting of 4000 columns, HIVE checks all privileges on the columns for a very long time. On postgres, I see 1 million selects to get information for each grant id

Hello Team, Facing bellow issues with hive while loading data from HDFS.Following are the error: Error 1ERROR : Vertex failed, vertexName=Reducer 2, vertexId=vertex_1672408587164_0154_2_01, diagnostics=[Task failed, taskId=task_1672408587164_0154_2_01_000001, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1672408587164_0154_2_01_000001_0:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing vector batch (tag=0) (vectorizedVertexNum 1)Error 2Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to 2000 partitions per node, number of dynamic partitions on this node: 2001 Error 3ERROR : Ve

Hi Team, This is Prsanna. I'm new to CDP and trying to create a CDP Trail base cluster for testing purposes, but while creating it I'm facing an error in the Hive Service. Please suggest a possible solution.Please refer to the screenshot and log file.Log file:jar:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/lib/hadoop-mapreduce/ranger-raz-intg-2.1.0.7.1.7.0-551.jar:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.2.14.jre7.jar:/usr/share/java/oracle-connector-java.jar+ [[ -n '' ]]+ [[ create_metastore_tables == webhcat ]]+ [[ create_metastore_tables == hiveserver2 ]]+ [[ create_metastore_tables == updatelocation ]]+ [[ create_metastore_tables == create_metastore_tables ]]+ [[ -z /opt/cloudera/cm ]]+ JDBC_JARS_CLASSPATH='/opt/cloudera/cm/lib/*:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.2.14.jre7.jar:/usr/share/java/oracle-connector-java.jar'+ [[ create_metastore_tables == create_metastore_tables ]]+ '[' 7 -ge 5 ']'+ s