Solved

Hi All, We're currently running CDH 6.3.2 and trying to get Python 3.6.9 working on Spark working nodes. So far without success, though... As our cluster is built on RHEL7, default Python version is 2.7, which is EOL and doesn't have all the necessary libraries/modules. Original idea was to install Miniconda3 on all the nodes, create the environment py36 and point PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON to the proper python executable. Apparently this doesn't work as expected, any PySpark job breaks with error message: Fatal Python error: Py_Initialize: Unable to get the locale encoding ModuleNotFoundError: No module named 'encodings'Despite the fact that module "encodings" is out there in py36 environment. If we add spark.executorEnv.PYTHONPATH=/opt/miniconda3/envs/py36/lib/python3.6:/opt/miniconda3/envs/py36/lib/to spark-defaults.conf file in the project root, we becoming different error message:Traceback (most recent call last): File "/opt/miniconda3/en

I wanted to create a cluster in Ambari but unfortunately at the "Confirm hosts" step I could not register my nodes due to following exception:Exception: Registration failed due to: Cannot register host with not supported os type, hostname=ip6-localhost, serverOsType=ubuntu18, agentOsType=ubuntu18Is it because i am using ubuntu 18.04?Thank you.

I am trying to insert a document into mongodb using apache NiFi. I want to insert a document just from Flow File attributes but not the Flow File content.I have tried using PutMongo processor but it insert the content of the Flow file which I don't want it to insert. I need to insert a custom document by using the flow file's attributes.

Attached is the java stack. I am getting this error in the nifi.log whenever i am starting GetFile processor which is first processort in my list.    2020-04-07 01:17:01,371 ERROR [Timer-Driven Process Thread-5] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=51048885-0171-1000-9a41-00f55a11ae71] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.IndexOutOfBoundsException: Index: 0, Size: 0: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0java.lang.IndexOutOfBoundsException: Index: 0, Size: 0at java.util.ArrayList.rangeCheck(ArrayList.java:657)at java.util.ArrayList.get(ArrayList.java:433)at org.apache.nifi.processors.standard.InvokeHTTP.setUpClient(InvokeHTTP.java:665)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:

Hello,I want to do a passwordless ssh login to my localhost.I tried the following steps many times but it asks for password when i try "ssh localhost".1. ssh-keygen2. cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys3. chmod 700 ~/.ssh4. chmod 600 ~/.ssh/authorized_keysThis is screenshot of my terminal.I tried the same steps to login from localhost to udhav.fqdn.com, but even now it promts me for password. For the second case i followed steps provided in this link.https://www.tecmint.com/ssh-passwordless-login-using-ssh-keygen-in-5-easy-steps/I guess i am doing small mistake. Can anyone help me with this?Thank you.  

Hi All My Hive Server 2 is failing to start,Below is the complete log file of the failure,i am unable to figure out whats causing the failure of the service. 2020-03-25 08:33:56: Starting HiveServer2WARNING: Use "yarn jar" to launch YARN applications.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.0.3-1.cdh7.0.3.p0.1635019/jars/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.0.3-1.cdh7.0.3.p0.1635019/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]Exception in thread "main" java.lang.IllegalArgumentException: mr execution engine is not supported! at org.apache.hadoop.hive.conf.HiveConf.validateExecutionEngine(HiveConf.java:5594) at org.apache.hadoop.hiv

Test CaseSetup  create table n1 (id STRING)stored as parquet;insert into n1 values ('-1234');insert into n1 values ('7890');  Resultsimpalad version 2.12.0-cdh5.15.2  SELECT `id`, CAST(`id` AS DECIMAL(38,0)) / 20 * 20 AS D0 , CAST(`id` AS DECIMAL(38,0)) / 20.0 * 20 AS D1FROM n1ORDER BY `id`;+----------+-------+---------+| id | d0 | d1 |+----------+-------+---------+| -1234 | -1220 | -1234.0 || 7890 | 7880 | 7890.0 |+----------+-------+---------+Fetched 4 row(s) in 0.56s  When we are casting the "id" column to a decimal without scale and dividing by an integer, even though the answer should have a scale (-1234 / 20 = -61.7) we lose the scale and the result is truncated to -61. This is then multiplied by 20 to give -1220 which is column d0. This does not get us back to the same value that we started with which was -1234. When we divide by a decimal value (20.0) the result of -61.7 is preserved, which gets

exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:578) at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:518) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:226) at org.apache.hadoop.util.RunJar.main(RunJar.java:141)Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.la

py4j.protocol.Py4JJavaError: An error occurred while calling o66.save.: java.lang.RuntimeException: java.lang.IllegalArgumentException: Unsupported JDBC protocol: 'null'at com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceWriter.createInternalRowWriterFactory(HiveWarehouseDataSourceWriter.java:102)at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec.doExecute(WriteToDataSourceV2.scala:55)at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecuti

Team ,   I am not able to see any databases when i configure HUE with hive . below is the only error i see in the logs and i cannot make out what exactly is the error .   [15/Apr/2020 11:31:35 -0700] thrift_util DEBUG Thrift call: <class 'TCLIService.TCLIService.Client'>.FetchResults(args=(TFetchResultsReq(fetchType=0, operationHandle=TOperationHandle(hasResultSet=True, modifiedRowCount=None, operationType=3, operationId=THandleIdentifier(secret=7748957ba2246f2a:ede5df3998017cb5, guid=19445a4be1c2c692:e2e7ddb12ee4788f)), orientation=0, maxRows=5000),), kwargs={}) returned in 5ms: TFetchResultsResp(status=TStatus(errorCode=None, errorMessage=None, sqlState=None, infoMessages=None, statusCode=0), results=TRowSet(columnCount=None, binaryColumns=None, rows=[], columns=[TColumn(i32Val=None, byteVal=None, i16Val=None, i64Val=None, stringVal=TStringColumn(nulls='\x00', values=[]), boolVal=None, doubleVal=None, binaryVal=None), TColumn(i32Val=None, byteVal=None, i16Val=None, i6

Had a file system crash, rebuilt Cloudera Manager host with the same everything--except the private TLS key for HTTPS. My psql db is still in tact, so the rebuilt CM host is connecting to it and getting configurations from the previous server. Is this a bad idea? Should I create a new database for the rebuilt CM host, or can I have it connect to the previous db and disable HTTPS manually somehow so I can access the web console and update it from there?Would it be a better idea to just regenerate new keys with the same password? I've never done this before, so I am unclear on any potential conflicts if a new Cloudera Manager server connects to an existing Cloudera Manager database.

Hi, I'm trying to connect to different databases that all have the same structure and execute the same query on them. However, i can't figure out how to give this to nifi execute SQL processor and i'm having to create a processor for every connection to the database. IS there a way for me to create a list of connection strings and then simply loop through them?i.e. i need to dynamically assign the connection pooling service to the processor. How?

Hi,    I've installed Cloudera Manager with Kafka and Zookeeper. I'm trying to build monitoring along with Prometheus and Graffana. Do we have any inbuilt feature to do in an easy way? or any leads, please.    

Continuously my NTP service seems to die on some of my VMs but not others, and I cannot seem to figure out why. I've followed every piece of advice I've been given so far to no avail. NTP service is dying without drifting it seems. When I use `ntpdate` to re-sync to the pool, it's never been more than ~50ms offset. yum install ntp -ysystemctl start ntpdsystemctl enable ntpdntpdate -u pool.ntp.orghwclock --systohcThe VMs are hosted across 3 different ESXi servers that are connected via vCenter. The ESXi hosts all have an NTP service running and enabled, and are syncing to the same external NTP server. Is there anything in `ntp.conf` that should be changed from default in order to prevent complications? I have zero clue why ntp service will randomly die on some hosts, yet I have some hosts that have had the service running without issue for 1+ month, but only if CM doesn't have the proper PID for the ntp service. As I'm writing this I think I may have solved it. By default ntp.conf

This seems to be an issue for many but I didn't see much in light for Linux systems. I've done it all picked up the drivers from https://jdbc.postgresql.org/download.htmlplaced these drives in the nifi-1.11.4/lib/ folder.  I've restarted Nifi, verified the database, verified the filepath, altered the path name with "file:///", altered the path to say just lib/postgresql-42.2.12.jar, attempted to use another database user. I know that the connection information is correct because I connected to the DataGrip UI. I'm completely stumped. Database Connection URL               jdbc:postgresql://localhost:5432/forestdbDatabase Driver Class Name           org/postgresql/Driver.classDatabase Driver Location(s)             /home/fg/Documents/nifi-1.11.4-bin/nifi-1.11.4/lib/postgresql-42.2.12.jarDatabase User              &n

I'm trying to run an incremental import job on SQL Server: sqoop job --create incjob \-- import \--connect 'jdbc:jtds:sqlserver://SQLQP002:1433;useNTLMv2=true;domain=JNL_NT;databaseName=TC31Reporting' \--driver net.sourceforge.jtds.jdbc.Driver \--username lrm0613 \-P \--table JobHistory \--incremental lastmodified \--check-column JobId \--hcatalog-database dataengsandbox \--hcatalog-table JobHistory \-m 1 \--merge-key JobId When I run the job, I get the following error: $ sqoop job --exec incjobWarning: /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.20/04/13 15:58:43 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.14.2Enter password:20/04/13 15:58:48 WARN tool.BaseSqoopTool: Input field/record delimiter options are not used in HCatalog jobs unless the format is text. It is better to use --hive-import in those cases. For text f

Hi Team   Is there any possibility of using SQL server to setup cloudera cluster and it's services. I'm aware that we have options of MYSQL PostgreSql and Oracle. I'm asking this to know is there any way to use of SQL Server.   Can you please share me useful article or link to use sql server.   Thanks and Regards Balu D

HI Team , We are looking for cluster setup to manage 300TB of hdfs volume . With 5% of data increment on weekly basis . how can we calculate and meet the above requirement .?what should be the hdfs and non hdfs space on each data nodes ?what should be the space allocation for edge node .Also the resources  to be allocated to each datanode . it would be great if you post the steps on what basis you will calculating the disk size and resource allocation . Thanks 

Hi Team, My question is : how to transfer licence one organization to another organization. Scenario: Current CDH cluster support roles hand over to other Organization. by this scenario how to get transfer licence related information. because exciting cluster have licence. ThanksDwaraka Y 

Hi, where I can find HiveJDBC41.jar driver for Hortonworks? Hortonworks used simba path "com.simba.hive.jdbc41". The one here: https://www.cloudera.com/downloads/connectors/hive/jdbc/2-6-2.html is with cloudera and can't be used "com.cloudera.hive.jdbc41".   Thanks in advance.

We are trying to implement alerting in our cluster and alerting is setup in Cloudera Manager. So when I stop a service in Cloudera Manager, an alert is sent to my email. Because for some reason I hear that if you stop the service from CM, it is not the same as it crashing on its own. Especially with regards Canary Alerts, which we will not get if we stop a service through Cloudera Manager. So will I not get Canary Alerts for a service if the service is stopped through Cloudera Manager? Also I would like to know how to stop a service manually through Cloudera API. I would appreciate it if some forum member could give the command to stop - say Oozie or HBase - through Cloudera Manager API. Appreciate the help.

Hi, we have installed a "Kafka only" cluster on CDH 6.3 along with ZooKeeper, SMM, Sentry and SchemaRegistry.  During the cluster install, Sentry (from what I can remember) was dependent on HDFS being present.  Now that the cluster is online, does anyone know if it's safe to remove HDFS?   Thank you, Mike

hello, I Know my doubt is silly but i a newbie in bigdata and ambari.I was creating a cluster. In the installation option i m having troubles while naming Target host.I am running ambari server(localhost:8080) on my local machine(laptop). My doubt is how should i name target host or what should i name the target host if i want it to run on my local machine(laptop) itself.If i give "localhost" as the target host name it always fails in the next "Confirm Host" step.Can anyone guide me through on how shouldi name the target host so that i can run everything in a single machine?Thank you!

Hi,I've created hadoop cluster on Azure HDinsight, next I've created some databases on hive. After deleting and re-creating cluster with connection to the same storage account hive is not showing all databases by 'show databases;' command, I see only three default databases (default, information_schema,sys) even though the databases I've created are visible in hdfs.

Hi friends,   After adding my field tags in the schema.xml file like below   <field name="_version_" type="plong" indexed="true" stored="true" multiValued="false" /><field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" /><field name="ip" type="text_general" indexed="true" stored="true"/><field name="request_date" type="date" indexed="true" stored="true"/><field name="request" type="text_general" indexed="true" stored="true"/><field name="department" type="string" indexed="true" stored="true" multiValued="false"/><field name="category" type="string" indexed="true" stored="true" multiValued="false"/><field name="product" type="string" indexed="true" stored="true" multiValued="false"/><field name="action" type="string" indexed="true" stored="true" multiValued="false"/>   And when i tried to create solr collection it is returning error like below   Unable to create core [live