Solved

Hi Community.I've created a transactional table on HIVE.I'm able to perform CRUD operations.When I try to query the same table from IMPALA, my query returns 0 rows.I've already invalidated metadata for that table but cannot see any of the existing records. Where am I wrong?

I have a source table LikeIDUSERDEPT1User1Admin2User1Accounts3User2Finance4User3Sales5User3Finance I want to generate a DataFrame like thisIDUSERDEPARTMENT1User1Admin,Accounts2User2Finance3User3Sales,Finance

Hi, please, how can I get the last day of the month of given date in NiFi?Step 1 could be taking the month out of the given dateStep 2 getting the last day or date of the month taken from step 1 Thanks a lot.

My data is coming in text (ASCII) files, with each line having a fixed number of fields and each field having a fixed length. I want to convert this files to CSV, like this:Input line:abbcccOutput line:a,bb,cccI the example above "a", "bb", "ccc" are fields of fixed length. So, I always know exactly how to split the input line. I looked into the ConvertRecord operator and the ScriptedReader controller service (that can be used as a record reader), but I was not able to find any example of a Python script for ScriptedReader. I found this ExecuteScript Cookbook by @mburgess, but those recipes are much more general, so I cannot use them in SciptedReader (which needs very specific objects for record processing that must be created by the script). Can anyone give a basic example of a Python script that can be used in ScriptedReader to process records? Alternatively, is there another way to accomplish the task (another processor)? Of course, I can use ExecuteScript processor a

My goal is to pass the data coming in as a JSON to my script and update it to UNIX time. using python. but it is saying line 63 is being referenced before called, "json_data". This is at the session writer near the bottomProcessors being used is Convert Record -Execute Script--.The message being passed looks like this:json_data = {"Year":2018,"DOY":12,"Hour":20,"HGI_Lat_of_the_S/C":7.0,"IMF_B_scalar_nT":1.03,"SW_Plasma_Speed_KMs":441.0} and should pop out like this {"HGI_Lat_of_the_S/C": 7.0, "IMF_B_scalar_nT": 1.03, "SW_Plasma_Speed_KMs": 441.0, "Unix_time": 1542074400.0} Error message: ExecuteScript[id=01711026-9fd7-1476-8c67-e730c921e164] Failed to process session due to javax.script.ScriptException: UnboundLocalError: local variable 'json_data' referenced before assignment in <script> at line number 63: org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: UnboundLocalError: local variable 'json_data' referenced before as

when I'm uploading a file to a certain folderNifi processor "get file" read files from this folder before it's finished uploading the fileso the file became corrupted. 

Hello.   I would like to connect using Impala ODBC Driver in CDH 6.3.2 I'm using Windows Server 2016,    What version of the driver do I need to use to use the Impala ODBC Driver?   I referred this page below. https://docs.cloudera.com/documentation/other/connectors/impala-odbc/2-6-9/Cloudera-ODBC-Driver-for-Impala-Install-Guide.pdf And it says "The Cloudera ODBC Driver for Impala is recommended for Impala versions 2.7 through 3.3, and CDH versions 5.9 through 5.16 and 6.0 through 6.2.".   Then, What version of the driver do I need to use? Is it just recommendation? It does not appear to supporting CDH 6.3.2....   Thanks.        

I am trying to install cloudera 6.3 using postgresql 10 and openjdk 11.0.6When I start the cloudera-scm-server, it fails and i get the below error in the cloudera-scm-server.out. Any ideas?  JAVA_HOME=/usr/lib/jvm/java-openjdkOpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.Unrecognized VM option 'UseParNewGC'Error: Could not create the Java Virtual Machine.Error: A fatal exception has occurred. Program will exit.  Here is the java version  [root@dex cloudera-scm-server]# java -versionopenjdk version "11.0.6" 2020-01-14 LTSOpenJDK Runtime Environment 18.9 (build 11.0.6+10-LTS)OpenJDK 64-Bit Server VM 18.9 (build 11.0.6+10-LTS, mixed mode, sharing) 

Hi All , I am trying to make a directory as a snapshot directory using curl API ,when i run the curl for Allowsnapshot it throws below error  Any help is Appreciated..Thanks in Advance ..!!  curl -i -X PUT "http://internal:50070/webhdfs/v1/dummy/?op=DISALLOWSNAPSHOT"HTTP/1.1 400 Bad RequestCache-Control: no-cacheExpires: Tue, 14 Apr 2020 03:26:30 GMTDate: Tue, 14 Apr 2020 03:26:30 GMTPragma: no-cacheExpires: Tue, 14 Apr 2020 03:26:30 GMTDate: Tue, 14 Apr 2020 03:26:30 GMTPragma: no-cacheContent-Type: application/jsonX-FRAME-OPTIONS: SAMEORIGINTransfer-Encoding: chunkedServer: Jetty(6.1.26.hwx){"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Invalid value for webhdfs parameter \"op\": No enum constant org.apache.hadoop.hdfs.web.resources.PutOpParam.Op.DISALLOWSNAPSHOT"}}[

Is there something about configuring Nifi that needs to be done when setting up?   I ask because I've worked on a few different systems and the one I just set up seems to be having issues with Regex, prepend, and other items. 

Hi Team, While using the envelope tool, I'm getting below error while using the tool to write into hive table:20/04/08 02:54:32 ERROR run.Runner: Pipeline exception occurred: org.apache.spark.sql.AnalysisException: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.readerFor(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/ObjectReader;;20/04/08 02:54:32 ERROR yarn.ApplicationMaster: User class threw exception: java.util.concurrent.ExecutionException: org.apache.spark.sql.AnalysisException: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.readerFor(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/ObjectReader;;java.util.concurrent.ExecutionException: org.apache.spark.sql.AnalysisException: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.readerFor(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/ObjectReader;; Can you please provide any suggestion to overcome this. Regards,Arvind

I am trying into ingest one table from MS SQL Server to hdfs through sqoop and I get following error.        cause:java.io.IOException: Sqoop does not have the splitter for the given SQL data type. Please use either different split column (argument --split-by) or lower the number of mappers to 1. Unknown SQL data type: -15119/10/30 12:42:37 ERROR tool.ImportTool: Import failed: java.io.IOException: Sqoop does not have the splitter for the given SQL data type. Please use either different split column (argument --split-by) or lower the number of mappers to 1. Unknown SQL data type: -151at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:194)at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:305)at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:322)at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:200)at org.apache.hadoop.mapreduce.Job$10.run(Job.

Team ,   I have installed and configured HUE .but i see below error when i try to access HDSF .   Cannot access: /user/admin. HTTPSConnectionPool(host='catl0dlbd00028.corp.cox.com', port=50070): Max retries exceeded with url: /webhdfs/v1/user/admin?op=GETFILESTATUS&user.name=hue&doas=admin (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'SSL23_GET_SERVER_HELLO', 'unknown protocol')],)",),))     I have logged in to HUE using default admin password . DO i have to configure LDAP before accessing HUE ? . I see its going to /user/admin wheer that folder might not be available in hdfs .

Hello, I want to transfer files from hdfs to kudu.  I tried through talend fabric and its components but I have an error : Cannot run anywhere due to node and executor blacklist.   Can you help me please? Thanks a lot.

Hi, we are facing a strange issue with an ORC external Hive table, some lines can't be retrieved. The context is HDP 3.1.0 with ORC files generated with a Spark 2.3.2 job into an HDFS partition with an external table mapped on this HDFS directory. Table partitions were added. When trying to filtering this table with a specific column value, there's no result;but when accessing these files from the Spark interpreter of a Zeppelin notebook, the filter shows the expected lines. Hive LLAP is not used for this issue and have another behaviour (data is retrieved).   The SQL query (JDBC tool or beeline command) that don't return any result : select * from tracabilite.tg1pivot where sscc="330232926251080606" and `application`="LOGUSI" and month="04" and day="01"; Another SQL query that returns 2 results (within the same partition): select * from tracabilite.tg1pivot where sscc="330232926636794272" and `application`="LOGUSI" and month="04" and day="01"; The Zeppelin notebook sample : val l

Hi, I'm attempting to replace values in an ASCII file that im importing from a website. I've added the header with replace text but, I'm having a hard time replacing the spaces with commas so, I can convert the file into csv and later to json. Here is what I'm dealing with.    Edit: I should add that sometimes there is one space other times there are two spaces. As of now I have two spaces in the search attribute and a comma in the replace attribute works for part of this problem.     Value     Search Value       Replacement Value   ,   Character Set   UTF-8   Maximum Buffer Size   1 MB   Replacement Strategy   Regex Replace   Evaluation Mode   Line-by-Line   Line-by-Line Evaluation Mode   All   Before: Year,Day,Hour,Minute,ID_for_SW_Plasma_spacecraft,Percent_of_interpolation,Timeshift,RMS_Timeshift,RMS_Min_v

Team ,   I have installed HUE using the below steps .   https://github.com/steven-dfheinz/HDP2-Hue4-Service.   I was able to install HUE successfully ,but it fails to start . below is the error i am getting .   Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HUE/4.6.0/package/scripts/hue_server.py", line 76, in <module> HueServer().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HUE/4.6.0/package/scripts/hue_server.py", line 26, in start import params File "/var/lib/ambari-agent/cache/common-services/HUE/4.6.0/package/scripts/params.py", line 194, in <module> webhdfs_url = format('http://' + dfs_namenode_http_address + '/webhdfs/v1') File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration p

Hello,   I'm facing an issue with the display and storage of special charactere in hive. I'm using spark for doing a WriteStream like this in Hive,     // Write result in hive val query = trimmedDF.writeStream //.format("console") .format("com.hortonworks.spark.sql.hive.llap.streaming.HiveStreamingDataSource") .outputMode("append") .option("metastoreUri", metastoreUri) .option("database", "dwh_prod") .option("table", "res_idos_0") .option("checkpointLocation", "/tmp/idos_LVD_060420_0") .queryName("test_final") .option("truncate", "false") .option("encoding", "UTF-8") .start() query.awaitTermination()      but when I have a special charactere Hive doesn't display it. I have already fixe encoding UTF8 in the hive table :     select distinct(analyte) from res_idos_0;+--------------------------------------------+| analyte |+---------------------------

Hi May I know what is the difference between CDP 7.0 product and Cloudera Enterprise 6.3? what are the features missing and what are the advantages?

I'm running the latest version of HDP sandbox using Docker. I'm following the instructions here: Getting Started with HDP Sandbox   When I create a table using the DAS UI, the operation completes successfully, and I am able to use the query tool to query the database. However, if I go to the "database" tab, is says there are no tables in the database. If I try to create a new database, it also doesn't appear on the database dropdown.   Also, only the first query I execute is added to the query log, even though subsequent queries complete successfully.   How can I configure sandbox so that I can view tables and databases properly in DAS?   Thanks.

does hive version 1.1.0-cdh5.16.1 supports acid transaction?

Hello!   In my Envelope pipeline, I need to compare two Hive tables. Instead of hardcoding the tables in the .conf file, I would like to pass which tables I'm going to compare. I tried using spark.yarn.appMaster.varName but it doesn't seem to work. I'm running CDH 5.13.3 with Java 1.8 on a Centos VM.   This is what the script that runs the spark job looks like: #!bin/bash sudo -u hdfs spark2-submit \ --master yarn \ --deploy-mode client \ --conf spark.yarn.appMaster.Env.tableA=dbA.tableA \ --conf spark.yarn.appMaster.Env.tableB=dbB.tableB \ envelope-0.7.2.jar comparison.conf    Part of my .conf file: application{name = comparison} steps{ tableA{ type = hive table = ${tableA} } tableB{ type = hive table = ${tableB}} }    

Hi All,   I install Cloudera in the AWS EC2 ubuntu 16.4 version. successfully installed, but not able to access in the browser. can you help me. how to open a browser. Are there any commands to check in Ubuntu, whether it's is running or not.   Regards, BVNP

Until when does Cloudera can cater and support the 6.3 version? until what year or what is the life span? and what is the most extended life span can Cloudera support it?

Hi, I have a Linux box which i configured HDFS client to connect and able to put files in the directories. Note that the machine is not part of the cluster.   1. Is it possible to run hive queries through hive CLI from this client machine? If so what all configuration i need to to.  2. Need one understanding on pushing file to hdfs. If i am pushing the file to hive table under below location and if it is a clustered ENV. will there be any side effect? As of now mine is a single node cluster so using the below. hdfs dfs -rm -r /user/hive/warehouse/schema.db/tablehdfs dfs -copyFromLocal -f /scratch/mano/table.dat /user/hive/warehouse/schema.db/table   Thanks in advance. Manoj