Q&A

Hi Cloudera Community, I am currently using spark1.6 version on cdh5.16.1. Now I have a requirement to upgrade spark from 1.6 to 2.3. So, I want to know which cdh version meets my requirement.

Hi,I want to convert json data to avro.I have used GenerateFlowFile and put dummy json value [{"firstname":"prathik","age":21},{"firstname":"arun","age":22}].I have then used ConvertRecord processor and set JsonTreeReader and AvroRecordSetWriter with AvroSchemaRegistry which has the following schema:{"name": "person","namespace": "nifi","type": "record","fields": [{"name": "firstname", "type": "string" },{ "name": "age", "type": "int" }]}But i am getting this as my output : prathik*arun,where  is a red dotI am new to Apache Nifi.

Hi, I am running into a problem with deploying a custom controller service to NiFi (I am using the latest version of NIfi 1.12 as of writing this). Now what I am trying to do is to create a custom controller service that would allow me to access a Virtuoso DB instance (similarly to DBCPConnectionPool). This controller relies on 2 external jar files (provided by Virtuoso) that I have placed inside a /libs folder under the module that implements the interface of my service. I have pointed to these jars as dependencies in the pom.xml of that same module and maven is able to import them. Furthermore, I am able to successfully compile and run a JUnit test of my implementation of the service which consists of simply getting a connection and checking if it is open.The problem arises when I compile my service into a NAR and then deploy that service in NiFi. NiFi is able to detect it but when I try to enable it, it throws a NoClassDefFoundError that is related to these external j

Searched for this and have not found the solution or I'm having the stupids. From an API I get a JSON file that I then use the EvaluateJsonPath processor to add an attribute to the flow file. One of these attributes is "date_posted" which is in epoch, seconds since 1st Jan 1970. I need to convert this into a date format so using the UpdateAttribute processor. I have tried these${date_posted:multiply(1000):format("yyyy-MM-dd HH:mm:ss","GMT")}${date_posted:toNumber():multiply(1000):format("yyyy-MM-dd HH:mm:ss","GMT")} and even for testing just these:${date_posted:toNumber()}${date_postedmultiply(1000)}${date_posted:toNumber():multiply(1000)} Yet afterwards my date_posted attribute ends up with an empty string:What am I missing?

Hi guys, This is basic question but not able to solve it...I created a scheduled workflow in Oozie to run a Shell script witch runs some impala-shell cmd. Everything worked fine since I added a where clause for one of them. I'm wondering if it's possible to pass a where clause on a query impala-shell ? Here is my basic cmd in shell script :impala-shell -B -q 'select col1,col2,Col3 from MyDB.MyTable where col1 IN ('argument1','argument2','argument3','argument4','argument5') order by col1,col2' --delimited --output_delimiter='|' -o ${DIR_OUT}/MyTable_$(date +"%Y%m%d-%H%M%S").csv --print_header 2>${DIR_LOG}/MyTable_$(date +"%Y%m%d-%H%M%S").log; It works fine without "where col1 IN ('argument1','argument2','argument3','argument4','argument5')" Is there a way to do it without using a file ? (with -f option ???) Thanks for your adviseAlex

CFM 1.0.1Could someone please explain to me why, after reconfiguring to disable SSL, when I try to access the webUI it still is attempting to connect on port 8443? Is this a zookeeper issue? How can I fix this?I had another cluster previously that start out as SSL disabled, but after configuring it to have a cert via nifi toolkit CA, and getting it WORKING with LDAP auth, it would try to request things from port 8080 http, then after vote resolved it will kill the NiFi node.So here I am now. Was unable to get SSL verified while following the 5 year old guide courtesy of @MattWho (you are awesome, i really appreciate all the info you've shared) I followed that guide to a T but the server reporting the cert was not verified.Now I'm trying to just connect to the nifi node via HTTP, so i disabled all related SSL things in the config, yet: Cannot replicate request to Node nifi.domain.net:8443 because the node is not connectedIt still tries to connect to port 8443. Can someone

Hello, when importing data from DB2 via Sqoop to Hive the stored data in /warehouse/tablespace/managed/hive/databasename/tablename/ is steadily growing.For every import (with --hive-import and --hive-overwrite set) there is a new folder: "base_000000n" created. Thus the parent folder is steadily growing. Any way to delete the old folders before importing new data with Sqoop? regards

Hello, I have CDH 5.16.2 with HBase 1.2.0 and a new Cluster with CDH 6.2.0 with HBase 2.1.0. I would like to use CopyTable to copy data into the new cluster. The containers give an error:ERROR [main] org.apache.hadoop.hbase.client.AsyncProcess: Failed to get region location org.apache.hadoop.hbase.exceptions.UnknownProtocolException: org.apache.hadoop.hbase.exceptions.UnknownProtocolException: Is this a pre-hbase-1.0.0 or asynchbase client? Client is invoking getClosestRowBefore removed in hbase-2.0.0 replaced by reverse Scan. at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2452) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.Rp

Hi all   Getting following error when trying to connect via ODBC driver version 2.6.4.1004 (64 bit) :-   [Cloudera][DriverSupport] (1160) Cannot enable SSL for the connection when connecting to a server that has not enabled SSL. If the server has SSL enabled, please check if it has been configured to use a SSL protocol version that is lower than what is allowed for the connection. The minimum SSL protocol version allowed for the connection is: TLS 1.2.   Please suggest.

I am unable to set below parameter when running hive query on spark engine using hive jdbc.   Cannot modify hive.spark.client.server.connect.timeout at runtime. It is in the list of parameters that can't be modified at runtime or is prefixed by a restricted variable.   set hive.spark.client.server.connect.timeout=90000 set hive.spark.client.connect.timeout=900000;   Both parameter is giving error in CDH6.3.3 but it was working in CDH5.16.1. Kindly let us know the option to avoid timeout error.

Hi, As  I learnt that the flow file attributes are held in memory in the nifi process, is it a good idea to clean up the attributes that have been defined earlier and delete them using UpdateAttribute processor if we know that we wouldn't use them anymore ? Or does deleting doesn't matter at all with regards to the memory usage? Please clarify.

I want to host an API in Nifi using HandleHTTP processor. And i wanted to pass some dummy json data. I would like to get some information/resources about how to do it.I am new to Nifi. Thanks in Advance.

Hello guys, has anyone taken the exam since registration of CCA 175 re-opened? If you tried could you tell something about the “new” exam? Is schema of exam different? Thanks in advance Cheers

Hi I've checked this procedure in cloudera on how to validate the encryption of my KMS and HDFS Create a zone and link to the key.su hdfs hdfs crypto -createZone -keyName mykey1 -path /tmp/zone1 Create a file, put it in your zone and ensure the file can be decrypted.su echo "Hello World" > /tmp/helloWorld.txthadoop fs -put /tmp/helloWorld.txt /tmp/zone1hadoop fs -cat /tmp/zone1/helloWorld.txt rm /tmp/helloWorld.txt Just want to ask, i have an existing hdfs directory there that I want to encrypt, my questions are below:1. Can I encrypt and existing hdfs directory using this command hdfs crypto -createZone -keyName mykey1 -path /tmp/zone1?2. if I encrypt the hdfs directory, does the encryption implemented on its sub directories and files under it?  

please suggest if anyone has successfully integrated with AD. I'm trying to use below configs and it's not working  ldapRealm= org.apache.zeppelin.realm.LdapRealmldapRealm.contextFactory.url = ldap://ad.abc.com:389ldapRealm.contextFactory.authenticationMechanism = simpleldapRealm.contextFactory.systemUsername = svc_abcldapRealm.contextFactory.systemPassword = passwdddd#ldapRealm.searchBase = DC=abc,DC=abc,DC=COMldapRealm.userSearchBase = OU=IDM,DC=abc,DC=abc,DC=comldapRealm.userSearchScope = subtreeldapRealm.userSearchAttributeName = sAMAccountNameldapRealm.userSearchFilter = (&(objectclass=person)(sAMAccountName={0}))ldapRealm.authorizationEnabled = trueldapRealm.groupSearchBase = OU=abcGroups,DC=abc,DC=abc,DC=comldapRealm.groupObjectClass = groupldapRealm.memberAttribute= memberldapRealm.groupSearchScope = subtreeldapRealm.groupSearchFilter = (&(objectclass=group)(member={0}))ldapRealm.memberAttributeValueTemplate= $CN=g_app_zep,OU=abcGroups,DC=abc,DC=abc,DC=co

Hi,I am currently facing issue with NiFi 1.7.0 (2 nodes),My flows had been running fine since months. Couple of weeks ago we did a downgrading of the nifi machines. We went from CPU 16 RAM 128G to CPU 4 RAM 32G.After the dowgrading we changed parameters to make sure that nifi take into consideration the new size of the machinesInitial values:Initial memory allocation: 80GMax memory allocation: 112gNew values:Initial memory allocation: 20GMax memory allocation: 26Gwithin nifi we did also change in NIFI Genaral SettingsfromMaximum timer driven thread count: 64Maximum event driven thread count: 16toMaximum timer driven thread count: 2Maximum event driven thread count: 4Now some time when we try to load we get an enexpected error occured with2020-08-18 10:57:51,942 ERROR [NiFi logging handler] org.apache.nifi.StdErr OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000780000000, 218103808, 0) failed; error='Cannot allocate memory' (errno=12)2020-08-18 10:57:51,956 INFO [

I would like to know if there were any perverse effects on using partitioned tables with a single bucket clause. This is to generate only one file in the partition. I use this because the compaction major process doesn't work very good. .I am using this solution to avoid having multiple small files in the partition. In fact, the compaction mechanism (Major) does not always allow only one file in the partition.

NIFI flow.xml.gz is getting cleared out during NIFI restartHDF 3.4NIFI 1.9 no configs have been changes but when restarted the existing flwo.xml.gz of size 70MB is getting overwritten. though restored from a backup and placed back in, again after restarting it's getting cleared.   @MattWho  please advice  

Hi, I created a external table with 'A' as string but stores a decimal value. Loaded data into the table then during report creation I realized my mistake and altered the column 'A' to double. Now I want to access that data but get the below error. I tried  set hive.msck.path.validation=ignore;msck repair table tablename;   But still get the below error.  Bad status for request TFetchResultsReq(fetchType=0, operationHandle=TOperationHandle(hasResultSet=True, modifiedRowCount=None, operationType=0, operationId=THandleIdentifier(secret='\nv\xe01\x14\x7fD\xd2\xb0\xadk\xd1\xbe\x9a\xd2\xd6', guid='\x19\xc2\x13b\x82\x96O\xd1\x93\x04\xf7\xdd\xc5\xcfz\xeb')), orientation=4, maxRows=100): TFetchResultsResp(status=TStatus(errorCode=0, errorMessage='java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.DoubleWritable', sqlState=None, infoMessages=['*or

I have a custom variable in Updatedate processor( state = success), now i want to access those variable in my groovy script which is in invokescriptor processor.def status= flowFile.getAttribute('status')i tried below mentioned both syntax as wellAttemp1:if( status.equals("success")){//}Attemp 2:if(status == "success"){}the above 2 attempts flowFile.getAttribute('status') returnning null value.Please correct my mistake!

We have a user interface based on Java. This application connects Hive as data source. Please find the product version of Apache Hive below. Application is working fine when single user has login with the application to access data from Hive. It gives error while concurrent users login to same application and access the Hive data.. apache-hive-2.3.7db-derby-10.14.2.0hadoop-2.10.0 Further investigation –The authentication mechanism of Hive is NOSASL. We did a small mutli threaded java application for debugging. Please find Java code and log for your analysis.Thanks in advance for your help!! import java.sql.Connection;import java.sql.DriverManager;import java.sql.PreparedStatement;import java.sql.ResultSet;import java.sql.SQLException;public class MapReduceThreadClass implements Runnable {String name;Thread t;String query;MapReduceThreadClass (String thread,String query){name = thread;this.query=query;t = new Thread(this, name);System.out.println("New thread: " + t);t.sta

I can't find location of boostrap.conf of nifi to inrease heap memort, and in the UI it listed as below  

To Schedule the exam, I have to create an account at www.examslocal.com. The step by step guide said that I have to use the same email I used to register on university.cloudera.com. I did it.After that, I have discovered that I'm unable to login because my PSI account is managed by AWS.The PSI support asked me to create a new PSI account, with a different email. Since the email used on niversity.cloudera.com and PSI must be the same, How can I migrate my purchased exam to a newly created Cloudera account?

I have a incoming csv file which I need to route to different nifi flows based on the values in one of the column. Each csv has a column called "Country" which has same country name in all of its rows. What I want is File 1 - 50 rows (All rows will only one value in country column which will be any of the below)England, england, Britain, UK, United kingdom, USA, United States So the file which contain values like England, england, Britain, UK, United kingdom in country column should be routed to one flow and if file contain USA or United States then it should be routed to another flow.Note - One file will only contain one value in all rows. So routing based on first row value of country column for entire file is fine. How to achieve this?

Hi,I am trying to install a node manager in a host. However, I got timeout status via ambari web ui upon the installation of the node manager. All other components successfully installed. Did you guys encounter this error?