Q&A

py4j.protocol.Py4JJavaError: An error occurred while calling o66.save.: java.lang.RuntimeException: java.lang.IllegalArgumentException: Unsupported JDBC protocol: 'null'at com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceWriter.createInternalRowWriterFactory(HiveWarehouseDataSourceWriter.java:102)at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec.doExecute(WriteToDataSourceV2.scala:55)at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecuti

Team ,   I am not able to see any databases when i configure HUE with hive . below is the only error i see in the logs and i cannot make out what exactly is the error .   [15/Apr/2020 11:31:35 -0700] thrift_util DEBUG Thrift call: <class 'TCLIService.TCLIService.Client'>.FetchResults(args=(TFetchResultsReq(fetchType=0, operationHandle=TOperationHandle(hasResultSet=True, modifiedRowCount=None, operationType=3, operationId=THandleIdentifier(secret=7748957ba2246f2a:ede5df3998017cb5, guid=19445a4be1c2c692:e2e7ddb12ee4788f)), orientation=0, maxRows=5000),), kwargs={}) returned in 5ms: TFetchResultsResp(status=TStatus(errorCode=None, errorMessage=None, sqlState=None, infoMessages=None, statusCode=0), results=TRowSet(columnCount=None, binaryColumns=None, rows=[], columns=[TColumn(i32Val=None, byteVal=None, i16Val=None, i64Val=None, stringVal=TStringColumn(nulls='\x00', values=[]), boolVal=None, doubleVal=None, binaryVal=None), TColumn(i32Val=None, byteVal=None, i16Val=None, i6

Had a file system crash, rebuilt Cloudera Manager host with the same everything--except the private TLS key for HTTPS. My psql db is still in tact, so the rebuilt CM host is connecting to it and getting configurations from the previous server. Is this a bad idea? Should I create a new database for the rebuilt CM host, or can I have it connect to the previous db and disable HTTPS manually somehow so I can access the web console and update it from there?Would it be a better idea to just regenerate new keys with the same password? I've never done this before, so I am unclear on any potential conflicts if a new Cloudera Manager server connects to an existing Cloudera Manager database.

Hi, I'm trying to connect to different databases that all have the same structure and execute the same query on them. However, i can't figure out how to give this to nifi execute SQL processor and i'm having to create a processor for every connection to the database. IS there a way for me to create a list of connection strings and then simply loop through them?i.e. i need to dynamically assign the connection pooling service to the processor. How?

Hi,    I've installed Cloudera Manager with Kafka and Zookeeper. I'm trying to build monitoring along with Prometheus and Graffana. Do we have any inbuilt feature to do in an easy way? or any leads, please.    

Continuously my NTP service seems to die on some of my VMs but not others, and I cannot seem to figure out why. I've followed every piece of advice I've been given so far to no avail. NTP service is dying without drifting it seems. When I use `ntpdate` to re-sync to the pool, it's never been more than ~50ms offset. yum install ntp -ysystemctl start ntpdsystemctl enable ntpdntpdate -u pool.ntp.orghwclock --systohcThe VMs are hosted across 3 different ESXi servers that are connected via vCenter. The ESXi hosts all have an NTP service running and enabled, and are syncing to the same external NTP server. Is there anything in `ntp.conf` that should be changed from default in order to prevent complications? I have zero clue why ntp service will randomly die on some hosts, yet I have some hosts that have had the service running without issue for 1+ month, but only if CM doesn't have the proper PID for the ntp service. As I'm writing this I think I may have solved it. By default ntp.conf

This seems to be an issue for many but I didn't see much in light for Linux systems. I've done it all picked up the drivers from https://jdbc.postgresql.org/download.htmlplaced these drives in the nifi-1.11.4/lib/ folder.  I've restarted Nifi, verified the database, verified the filepath, altered the path name with "file:///", altered the path to say just lib/postgresql-42.2.12.jar, attempted to use another database user. I know that the connection information is correct because I connected to the DataGrip UI. I'm completely stumped. Database Connection URL               jdbc:postgresql://localhost:5432/forestdbDatabase Driver Class Name           org/postgresql/Driver.classDatabase Driver Location(s)             /home/fg/Documents/nifi-1.11.4-bin/nifi-1.11.4/lib/postgresql-42.2.12.jarDatabase User              &n

I'm trying to run an incremental import job on SQL Server: sqoop job --create incjob \-- import \--connect 'jdbc:jtds:sqlserver://SQLQP002:1433;useNTLMv2=true;domain=JNL_NT;databaseName=TC31Reporting' \--driver net.sourceforge.jtds.jdbc.Driver \--username lrm0613 \-P \--table JobHistory \--incremental lastmodified \--check-column JobId \--hcatalog-database dataengsandbox \--hcatalog-table JobHistory \-m 1 \--merge-key JobId When I run the job, I get the following error: $ sqoop job --exec incjobWarning: /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.20/04/13 15:58:43 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.14.2Enter password:20/04/13 15:58:48 WARN tool.BaseSqoopTool: Input field/record delimiter options are not used in HCatalog jobs unless the format is text. It is better to use --hive-import in those cases. For text f

Hi Team   Is there any possibility of using SQL server to setup cloudera cluster and it's services. I'm aware that we have options of MYSQL PostgreSql and Oracle. I'm asking this to know is there any way to use of SQL Server.   Can you please share me useful article or link to use sql server.   Thanks and Regards Balu D

HI Team , We are looking for cluster setup to manage 300TB of hdfs volume . With 5% of data increment on weekly basis . how can we calculate and meet the above requirement .?what should be the hdfs and non hdfs space on each data nodes ?what should be the space allocation for edge node .Also the resources  to be allocated to each datanode . it would be great if you post the steps on what basis you will calculating the disk size and resource allocation . Thanks 

Hi Team, My question is : how to transfer licence one organization to another organization. Scenario: Current CDH cluster support roles hand over to other Organization. by this scenario how to get transfer licence related information. because exciting cluster have licence. ThanksDwaraka Y 

Hi, where I can find HiveJDBC41.jar driver for Hortonworks? Hortonworks used simba path "com.simba.hive.jdbc41". The one here: https://www.cloudera.com/downloads/connectors/hive/jdbc/2-6-2.html is with cloudera and can't be used "com.cloudera.hive.jdbc41".   Thanks in advance.

We are trying to implement alerting in our cluster and alerting is setup in Cloudera Manager. So when I stop a service in Cloudera Manager, an alert is sent to my email. Because for some reason I hear that if you stop the service from CM, it is not the same as it crashing on its own. Especially with regards Canary Alerts, which we will not get if we stop a service through Cloudera Manager. So will I not get Canary Alerts for a service if the service is stopped through Cloudera Manager? Also I would like to know how to stop a service manually through Cloudera API. I would appreciate it if some forum member could give the command to stop - say Oozie or HBase - through Cloudera Manager API. Appreciate the help.

Hi, we have installed a "Kafka only" cluster on CDH 6.3 along with ZooKeeper, SMM, Sentry and SchemaRegistry.  During the cluster install, Sentry (from what I can remember) was dependent on HDFS being present.  Now that the cluster is online, does anyone know if it's safe to remove HDFS?   Thank you, Mike

hello, I Know my doubt is silly but i a newbie in bigdata and ambari.I was creating a cluster. In the installation option i m having troubles while naming Target host.I am running ambari server(localhost:8080) on my local machine(laptop). My doubt is how should i name target host or what should i name the target host if i want it to run on my local machine(laptop) itself.If i give "localhost" as the target host name it always fails in the next "Confirm Host" step.Can anyone guide me through on how shouldi name the target host so that i can run everything in a single machine?Thank you!

Hi,I've created hadoop cluster on Azure HDinsight, next I've created some databases on hive. After deleting and re-creating cluster with connection to the same storage account hive is not showing all databases by 'show databases;' command, I see only three default databases (default, information_schema,sys) even though the databases I've created are visible in hdfs.

Hi friends,   After adding my field tags in the schema.xml file like below   <field name="_version_" type="plong" indexed="true" stored="true" multiValued="false" /><field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" /><field name="ip" type="text_general" indexed="true" stored="true"/><field name="request_date" type="date" indexed="true" stored="true"/><field name="request" type="text_general" indexed="true" stored="true"/><field name="department" type="string" indexed="true" stored="true" multiValued="false"/><field name="category" type="string" indexed="true" stored="true" multiValued="false"/><field name="product" type="string" indexed="true" stored="true" multiValued="false"/><field name="action" type="string" indexed="true" stored="true" multiValued="false"/>   And when i tried to create solr collection it is returning error like below   Unable to create core [live

Hi Community.I've created a transactional table on HIVE.I'm able to perform CRUD operations.When I try to query the same table from IMPALA, my query returns 0 rows.I've already invalidated metadata for that table but cannot see any of the existing records. Where am I wrong?

I need to process some XML files that contain some doubled field names as shown is the sample below (Latitude, Longitude and Height). I need to change the 1st ocurrence in Latitude1, Longitude1, Height1 and the 2nd in Latitude2..etc so I can do a JSON conversion after.What would be the best aproach to change these fields?   <?xml version="1.0" encoding="UTF-8"?><vwConnectionLog><ID>18500380</ID><JobID>1336467</JobID><Active>0</Active><StartTimeConnection>2019-12-01 12:09:12</StartTimeConnection><StartTimeSending>2019-12-01 12:09:17</StartTimeSending><EndTimeConnection>2019-12-01 12:25:23</EndTimeConnection><FirstSiteCode>HORE</FirstSiteCode><FirstRefStatID>84</FirstRefStatID><FirstDistanceToStat>41913</FirstDistanceToStat><LastSiteCode>VLC2</LastSiteCode><LastRefStatID>29</LastRefStatID><LastDistanceToStat>35429</LastDis

I have a source table LikeIDUSERDEPT1User1Admin2User1Accounts3User2Finance4User3Sales5User3Finance I want to generate a DataFrame like thisIDUSERDEPARTMENT1User1Admin,Accounts2User2Finance3User3Sales,Finance

Hi, please, how can I get the last day of the month of given date in NiFi?Step 1 could be taking the month out of the given dateStep 2 getting the last day or date of the month taken from step 1 Thanks a lot.

My data is coming in text (ASCII) files, with each line having a fixed number of fields and each field having a fixed length. I want to convert this files to CSV, like this:Input line:abbcccOutput line:a,bb,cccI the example above "a", "bb", "ccc" are fields of fixed length. So, I always know exactly how to split the input line. I looked into the ConvertRecord operator and the ScriptedReader controller service (that can be used as a record reader), but I was not able to find any example of a Python script for ScriptedReader. I found this ExecuteScript Cookbook by @mburgess, but those recipes are much more general, so I cannot use them in SciptedReader (which needs very specific objects for record processing that must be created by the script). Can anyone give a basic example of a Python script that can be used in ScriptedReader to process records? Alternatively, is there another way to accomplish the task (another processor)? Of course, I can use ExecuteScript processor a

My goal is to pass the data coming in as a JSON to my script and update it to UNIX time. using python. but it is saying line 63 is being referenced before called, "json_data". This is at the session writer near the bottomProcessors being used is Convert Record -Execute Script--.The message being passed looks like this:json_data = {"Year":2018,"DOY":12,"Hour":20,"HGI_Lat_of_the_S/C":7.0,"IMF_B_scalar_nT":1.03,"SW_Plasma_Speed_KMs":441.0} and should pop out like this {"HGI_Lat_of_the_S/C": 7.0, "IMF_B_scalar_nT": 1.03, "SW_Plasma_Speed_KMs": 441.0, "Unix_time": 1542074400.0} Error message: ExecuteScript[id=01711026-9fd7-1476-8c67-e730c921e164] Failed to process session due to javax.script.ScriptException: UnboundLocalError: local variable 'json_data' referenced before assignment in <script> at line number 63: org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: UnboundLocalError: local variable 'json_data' referenced before as

when I'm uploading a file to a certain folderNifi processor "get file" read files from this folder before it's finished uploading the fileso the file became corrupted. 

Hello.   I would like to connect using Impala ODBC Driver in CDH 6.3.2 I'm using Windows Server 2016,    What version of the driver do I need to use to use the Impala ODBC Driver?   I referred this page below. https://docs.cloudera.com/documentation/other/connectors/impala-odbc/2-6-9/Cloudera-ODBC-Driver-for-Impala-Install-Guide.pdf And it says "The Cloudera ODBC Driver for Impala is recommended for Impala versions 2.7 through 3.3, and CDH versions 5.9 through 5.16 and 6.0 through 6.2.".   Then, What version of the driver do I need to use? Is it just recommendation? It does not appear to supporting CDH 6.3.2....   Thanks.