Find answers, ask questions, and share your expertise
Recently active
when i tried to running this note%spark2.pysparkimport pysparkfrom pyspark_llap import HiveWarehouseSessionhive = HiveWarehouseSession.session(spark).build()df = hive.execute(" show databases ") the result is File "/tmp/spark-00e8a412-9248-4a78-803f-3ca9d8ddac2d/userFiles-2f95cdc1-3caa-42d7-8356-5b022b67b3b9/pyspark_hwc-1.0.0.3.1.0.0-78.zip/pyspark_llap/sql/session.py", line 228, in session return HiveWarehouseBuilder.session(session) File "/tmp/spark-00e8a412-9248-4a78-803f-3ca9d8ddac2d/userFiles-2f95cdc1-3caa-42d7-8356-5b022b67b3b9/pyspark_hwc-1.0.0.3.1.0.0-78.zip/pyspark_llap/sql/session.py", line 44, in session jvm.com.hortonworks.spark.sql.hive.llap.HiveWarehouseBuilder.session(jsparkSession)) TypeError: 'JavaPackage' object is not callable how do i fix it ? thanks
Hi, I am exploring to learn cloudera, and I am trying to configure cloudera manager.But https://archive.cloudera.com/cm5/ asking me credentials (username and password).Could you please help me to solve this problems please. Thanks,Rahil
Configured new test instance of RangerSettings Binded user has domain admin rights.Now synced only 999 users of 1650 existed in ADWhat problem can be?
Hello. I am using Cloudera Manager trial version 7.1.4. I need to install accumulo but when I tried to install it using parcel I got the error that I need to use version CDH lower than 7. Is there any way to install accumulo using CDH 7?
Hi. I want to know Where is the Jar folder for Spark in Cloudera? In my previous company, we just put all the needed jars inside $SPARK_HOME/jar folder (on every node), so that we don't have to worry much about the --jars, --packages,... when running the spark-submit job. Also, it saves lots of disk space and time since we don't need to include every package when building a jar.But in my new company, which uses Cloudera, I don't know where is this jar folder. I found 2 places (maybe not the right one):- /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars- /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jarsWhere should I put the needed jars file on? It seems like all the jars file on the second one are linked to the first one. Also, I found dozens of lib/jar folders everywhere in Cloudera.Or is there any other way to do this with Cloudera? I read some guides about modified Spark configs on the Cloudera manager on the internet.
we have old hadoop cluster based on HDP from hortonworks version HDP 2.6.4cluster include 2 namenode services when one is the standby namenode and the second is the active namenode , all machines in the cluster are rhel 7.2 version , and we not see any problem on OS level also cluster include 12 workers machines ( worker include the datanode and node manager services ) the story begin when we get alerts from the smoke test script that complain about "`Detected pause in JVM or host machine`" on the standby namenode , so based on that we decided to increase the namenode heap size from 60G to 100G and above setting was based on table that show how much memory to set according to number of files in HDFS and according to the table we decided to set the namenode heapsize to 100G and then we restart the HDFS service after HDFS is completely restarted , we still see the messages about `Detected pause in JVM or host machine` , and this is really strange because we
Hi guys,i am using spark-shell ( spark-shell --master yarn --jars /usr/hdp/current/hive-warehouse-connector/hive-warehouse-connector_2.11-1.0.0.3.1.2.0-4.jar --conf spark.security.credentials.hiveserver2.enabled=false)to read hive tables using spark, i am able to execute commands like create table, create database, show tables, show databases, but i am not able to read data from tables,my code is as belowimport com.hortonworks.hwc.HiveWarehouseSessionval hive = HiveWarehouseSession.session(spark).build()hive.createDatabase("spark_llap01",false)hive.setDatabase("spark_llap01")hive.createTable("hwx_table").column("value", "string").create()hive.executeUpdate("insert into hwx_table values('1')")hive.executeQuery("select * from hwx_table").showi get this error, whenever i try to fetch datajava.lang.AbstractMethodError: Method com/hortonworks/spark/sql/hive/llap/HiveWarehouseDataSourceReader.createBatchDataReaderFactories()Ljava/util/List; is abstracti used beeline to check whether the data
Hello, While browsing through various articles related to Cloudera Manager, I came across an article which talks about exporting Cloudera Manager configs using a script. In this a wget from "${SCM_URL}/cmf/exportCLI" url is doing the trick. Being fairly new with playing APIs, I tried to read across somethings related to CM APIs, however, did not find much. Would be great if someone could help me to understand which exact configurations are exported with the mentioned API. Apologies if seems to be a basic question. However, thought of asking community instead of creating confusions for myself. Thankssnm1523
Hi All,I am using Postgress for Hive metastoreHow can we configure postgress HA with Hive with true HA on the Hive Metastore without changing the connection string manually?
Hi Community ! I am planning on taking the CDP-0011 certification exam soon. Can you recommend any book or online resource for preparation ?Regards,
1:08:30.213 PM ERROR RetryingHMSHandler[pool-9-thread-200]: HMSHandler Fatal error: javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastoreat org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:171)at org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:728)at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)at com.sun.proxy.$Proxy27.commitTransaction(Unknown Source)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1821)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1993)at sun.re
Hi everybody,i'm new to Nifi, and i try to understand how the in/out streams works in groovy. I've created one processor to generate flowfile, one to run a groovy script and a third one to putfile. I've tried many many things but for now my groovy script (not working) look like this: import org.apache.poi.ss.usermodel.*import org.apache.poi.hssf.usermodel.*import org.apache.poi.xssf.usermodel.*import org.apache.poi.ss.util.*import org.apache.poi.ss.usermodel.*import org.apache.poi.hssf.extractor.*import java.nio.charset.*import java.io.*def flowFile = session.get()if(!flowFile) returnflowFile = session.write(flowFile, {inputStream, outputStream -> try {FileOutputStream out = new FileOutputStream("workbook.xls");outputStream.write(out);out.close();//is not working.// def record = "toto"// outputStream.write(record.getBytes(StandardCharsets.UTF_8))//is working and giving me a txt file with toto in it. } catch(e) { log.error("Error during proc
I have a scenario where am triggering API in batches (1000 records in each). from the first API I get the value of total records in attribute list X-Total-Count, i wants to merge them all. Please let me know how i can leverage the wait processor in it to wait until we get the whole data till value of X-Total-Count. Wait Processor properties, i have configured.
We are having a CDP 7.1.7 private cluster with phoenix setup.We had created some phoenix tables long back with certain primary keys, however now when we are trying to check those primary keys on a table via phoenix shell, we are getting empty results.I have tried below commands and no information.!primarykeys <tablename>!column <tablename>!describe <tablename>select column_name from system.catalog where table_name='tablename' Phoenix version is 5.0.0 Please help.
I wanted to wait until 'ExecuteSQL' processor finishes inserting big file into SQL database (table: bu_service_template), then move to other section. 'ExecuteSQL' processor inserts row by row content of a file. I wanted to wait until file is fully inserted and then move to a next file.My current flow:waitflow
We are looking to invoke any https API urls from Nifi using InvokeHTTP processor.Configuration Done: Invokehttp Processor: SSL Context Service:Tried configuring the Java certs file for the Truststore filename. Exception: Processor throws exception saying, InvokeHTTP Routing to Failure due to exception: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target ↳ causes: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target ↳ causes: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCer
Hi, I'm using CDP Public cloud(7.2.15) with CML (2.0.34-b116) and have a Team created. However, we now want to rename the team name but dont have any option in the UI. Is there any option available through API??
Hello All, I have encountered a rather weird bug with NiFi's SplitJson processor. I am using NiFi 1.18 and it's a three node cluster. I think the best way of explaining it will be to use as example, so here is how to recreate this bug: I created this flow with the following sequence of steps:Create a GenerateFlowFile and set the custom text to: { "test": [ { "A": 1 } ], "test2": [ { "B": 1 } ] }Create the top SplitJson and set the path to: $.test.*Create all connections for the top SplitJson (original is terminated), start it and run GenerateFlowFile once -> The output queue should show a single flow file with this content: { "A": 1 }Now select the top SplitJson, Crtl+C, Crtl+V, move the copy down create its connections and set the path to: $.test2.*Start the bottom SplitJson, and run GenerateFlowFile once againBoth output queues have a new flow file and the contents of both flow files are: { "A": 1 }, which is correct for the top SplitJson,
I created my ML workspace , but when I try to create a project and click on the "Workspace" , I get a blank page. Appreciate any inputs.The underlying infra on AWS seems to be fine.Thanks for your feedback.
If a cml app is not used for days, it is getting timeout and application goes from "Running" status to "Failed" status. Upon checking logs it shows - Engine exited with unknown engine status: -1How we can remove timeout option for cml apps deployed ? or any other option to correct this issue.
Give a scenario that data is written to HFile in Hbase. Now a read occurs and result is saved in blockcache. For the data that is there is in block cache an update occurs, which is saved in Memstore. Now if the same data is read it will look for data first in Block Cache and if cache is not yet expired the result is found there. If that data is returned to client then it would be an inconsistent read. Is it possible for an above scenario to occur or is my understanding wrong?
i try to start hive llap but when i start always failed and this logs from task log WARN conf.HiveConf: HiveConf of name hive.heapsize does not existERROR cli.LlapStatusServiceDriver: FAILED: nulljava.lang.NullPointerException: null at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.run(LlapStatusServiceDriver.java:199) [hive-llap-server-3.1.0.jar:3.1.0] at org.apache.hadoop.hive.llap.cli.LlapStatusServiceDriver.main(LlapStatusServiceDriver.java:712) [hive-llap-server-3.1.0.jar:3.1.0]FAILED: nullCommand failed after 1 triesanybody can help ? thanks
I am trying to run an impala shell and I receive the below error:Error connecting: TypeError, __init__() got an unexpected keyword argument 'ssl_version'This happens after the upgrade to 3.4 impala version.
Hi All, When we try to install the Cloudera Management Service we got the below error "login request was rejected" in PostgresSQL. Please advise. PostgresSQL - 10.19Cloudera manager - 7.6.5, andOS - RHEL 8.4. There is no Auto-TLS or Kerberos as it is a POC cluster we trying to install. error message in Cloudera-scm-server.log: 2022-11-23 19:07:36,287 ERROR CommandPusher-1:com.cloudera.cmf.model.DbCommand: Command 1546334432(RepMgrTestDatabaseConnection) has completed. finalstate:FINISHED, success:false, msg:Logon denied for user/password. Able to find the database server and database, but the login request was rejected.2022-11-23 19:07:36,287 INFO CommandPusher-1:com.cloudera.cmf.command.components.CommandStorage: Invoked delete temp files for command:DbCommand{id=1546334432, name=RepMgrTestDatabaseConnection, host=hostname01.company.com} at dir:/var/lib/cloudera-scm-server/temp/commands/1546334432 Kind Regards,Sri
Metadata fetch for Impala on CDP fails with below error:INFO: Connected to metastore.alation.shims.HiveShimsBase$RetryingMetaStoreClient invokeWARNING: MetaStoreClient lost connection. Attempting to reconnect (3 of 3) after 1s for getTableObjectsByName method with arguments: [test_sampling_and_profiling02, [stpsalewithmt1000drows, stpsalewithmt1000drows_view, stpsale_view, stpsalesample_view]]org.apache.thrift.TApplicationException: Internal error processing get_table_objects_by_name at org.apache.thrift.TApplicationException.read(TApplicationException.java:111) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_objects_by_name(ThriftHiveMetastore.java:1496) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_objects_by_name(ThriftHiveMetastore.java:1482) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableObjectsByName(HiveMetaStoreClient.java:143
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.