Q&A

Hi,We are getting below warnings while starting hive shell from CLI.$hive19/06/20 07:53:14 WARN conf.HiveConf: HiveConf of name hive.mapred.strict does not exist19/06/20 07:53:14 WARN conf.HiveConf: HiveConf of name hive.mapred.supports.subdirectories does not existlog4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppenderhive>Hive version - 1.2As per I know, we can safely ignore these warnings but, we need to get rid off these.Could you please help on this. Thanks

Hi. On an Oracle BDA system where several network interfaces exist on each host, how can I bind Sentry to wildcard address 0.0.0.0 ? I've already tried by adding:<property>  <name>sentry.service.server.rpcaddress</name>  <value>0.0.0.0</value></property>to Sentry Service Advanced Configuration Snippet (Safety Valve) for sentry-site.xml but it's stil not working (I've double checked that generated sentry-site.xml file contains the property). Any help is appreciated. Thanks in advance. 

Facing this issue with the namenode down. It is a parcel based CDH 5.15.2 installation on AWS.   9:15:18.262 PM ERROR NameNode Failed to start namenode.java.io.FileNotFoundException: File does not exist: /user/spark/spark2ApplicationHistory/.5e9b4c52-032a-4469-b278-80aa6254cfdf at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:429) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:141) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:903) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:756) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:324) at org.apac

Hi,I have a simple question about querying complex types.  The schema, raw table and desired result is listed below.  Could someone help me craft this query in Impala? CDH 5.15 I can get it to work in Hive, but it seems like Impala uses a join-like syntax to access complex columns.  Many Thanks!!schema: snum int items map<string, float>raw parquet table: 1, {"a": 1.3, "b", 9.1} 2, {"c", 84.9, "b", 1.1} 3, {} 4, {"x", 42.5,"p", 0.08}desired result: 1,a,1.3 1,b,9.1 2,c,84.9 2,b,1.1 3,null,null 4,x,42.5 4,p,0.08  

Can anyone tell me the way to install JanusGraph on Ambari. I tried the link https://github.com/JanusGraph/janusgraph-ambari but it didn't work out for me. I have Ubuntu 14.04 and Ambari 2.6 and have already installed Solr 6.6.2 on Ambari. Is there a way to make a connection between the two? Or, is there any other alternative to this (JanusGraph) which might run on Ambari with the same functionality?Actually JanusGraph installed properly on Ambari using the link above. It also starts without any error, but then within 10 seconds it automatically closes. During the final check it says "can't connect to port 8182". What can be the fault in this case?

Hi!I need to execute following awk command on flow file via ExecuteStreamCommand processorawk -F"," '{OFS=","; $4=substr($4,1,19); print $0}'When run it throws an error: Executable command awk ended in an error: awk: fatal: cannot open file `print $0}" for reading (No such file or directory).Flow file comes from ListFTP -> FetchFTP -> ExecuteStreamCommand Could you please help to solve this issue!!Thank you!!When running the same awk command on server it works fine:awk -F"," '{OFS=","; $4=substr($4,1,19); print $0}' myfile.csv

I wrote a java code to insert few thousand records by creating batch, I was not getting good performance from PreparedStatements so I switched to Statements and see if there is some performance difference.However after getting error and reading Impala doc I came to know executebatch() is not supported for statements but I am able to use addBatch() so its kind a strange.Here I am posting some of my code-  private static Connection connectViaDS() throws Exception { Connection connection = null; Class.forName("com.cloudera.impala.jdbc41.DataSource"); connection = DriverManager.getConnection(CONNECTION_URL); return connection; }//inside main--// I have declared all the variables,I have lots of other data as well so it may create more confusion try { connection = connectViaDS(); connection.setAutoCommit(false); Statement statement = connection.createStatement(); Timestamp ed = new Timestamp(e_date);

Hi Experts,   Need your suggestions:   How to enforce a user to submit a job in dedicated resource pool if nothing is mentioned during submission.   Whenever a user submits a  job (without mentioning any queue name) , it should go in a dedicated queue defined by Admins.   Can we control it through YARN ACLs.

Hi, When trying to import table from oracle database to hive from hue qui i get the following error when i am trying to setup the jdbc connection: Connection Failed: Error accessing the database: An error occurred while calling z:java.lang.Class.forName. : java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver at java.net.URLClassLoader$1.run(URLClassLoader.java:360) at java.net.URLClassLoader$1.run(URLClassLoader.java:349) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:348) at java.lang.ClassLoader.loadClass(ClassLoader.java:430) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:323) at java.lang.ClassLoader.loadClass(ClassLoader.java:363) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:195) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMet

Hi there, colleaguesPast several days i tried to implement integration apache ranger and ranger-kafka plugin.My versions:kafka: 2.12_2.2.0ranger: 1.2.0 , ranger-plugin 1.2.0My current problem:as far as i understand, kafka cant find the ranger-created audit files that i made with enable-kafka-plugin.sh script (ranger-kafka-audit.xml and ranger-kafka-security.xml). I sold this files everywhere in file system but this class still cant find it and cause fatal error:[2019-06-06 13:22:44,584] INFO getFilesInDirectory('/usr/lib/kafka/libs/ranger-kafka-plugin-impl'): adding /usr/lib/kafka/libs/ranger-kafka-plugin-impl/guava-17.0.jar (org.apache.ranger.plugin.classloader.RangerPluginClassLoaderUtil)[2019-06-06 13:22:44,584] INFO getFilesInDirectory('/usr/lib/kafka/libs/ranger-kafka-plugin-impl'): adding /usr/lib/kafka/libs/ranger-kafka-plugin-impl/ranger-plugins-common-1.2.0.jar (org.apache.ranger.plugin.classloader.RangerPluginClassLoaderUtil)[2019-06-06 13:22:44,615] INFO [Transaction Marker

so i have a flume agent  with a spooldir channel that got too full (too many files error). the fix was to move the files out of the spooldir, move the checkpoint, and restart the agent, resulting in data flowing again. but now i have some flume-formatted data that i would like to ingest.can i move the files into the spooldir 10% at a time fx?would i have to read the files and add them to the flume source?  additional notes:the reason for the spool dir was a missing sink (another flume agent- turned off)

Hello all,My system administrator wants to check the Impala Query's execution time.I can check the each fragment's total time in[Cloudera Manager > Impala > Query > Query details].But I can't find the total of query execution time. Please anyone tell me where can I check it. Thanks

I am working on a NIFI flow that begins with an InvokeHTTP processor.On the occasion that there is a connection timeout (Failure condition), I wish to assign the InvokeHTTP processor Remote URL value to a new property and have this property passed as data (attribute) through the "Failure" path to another processor. Is it possible to do within the same InvokeHTTP processor?Can anyone help?

My nifi flow is not progressing nor showing any error:Flow is for :fetch a test.json fiel from s3 and populate its content in a table in postgresql db.lists3object-> fetchs3object->JoltTrasformationJSON -> ........ ->PutSQL.The lists3object alone lists key as my test.json in "view status"I'm not able to paste screenshots here.Though all processors got started, none of them are doing any transactions (in & out ) both are 0 bytes and i dont see any errors also.Last message logged in nifi-app.log is:2019-06-14 07:26:48,864 INFO [NiFi Web Server-92] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.aws.s3.ListS32019-06-14 07:26:48,864 INFO [Timer-Driven Process Thread-4] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling ListS3[id=016b1032-cd19-10b1-6d77-5a63389cd903] to run2019-06-14 07:26:49,166 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowCon

I have a relatively large CSV (~80GB) I need to transform into multiple JSON documents/records. I'm using a ConvertRecord processor with a CSVReader and AvroRecordSetWriter and that's where my CSV gets stuck. What's the best approach? Break up the CSV prior to converting it or try to get more horsepower on the server? Server Mem: 16GBCores: 4Maximum Timer Driven Thread Count : 16Java Min/Max Heap: 2GB/10GB

Hi  I am trying to import a single table from sqoop and i get this error:   Warning: /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]19/06/18 16:25:25 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7-cdh6.2.019/06/18 16:25:26 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instea

Hi,Data is in files in the following format 2019-06-15T15:43:12 ( yyyy-MM-ddThh:mm:ss )When I do a select * on the table , it is displayed as NULLHive version - 1.2.1ALTER TABLE table SET SERDEPROPERTIES ("timestamp.formats"="yyyy-MM-dd'T'HH:mm:ss"); - did not helpFollowed the links,https://issues.apache.org/jira/browse/HIVE-9298https://www.ericlin.me/2017/03/alternative-timestamp-support-in-hive-iso-8601/Could you please help on this. Thanks

Hello. We're on version CDH 5.12.2.On Cloudera Manager, Impala -> Query , I see query that is "Executing" but query status is "Finished".query_duration was 42/51 mins. User is running query on Hue. I've searched through official papers, google it, but getting nowhere.Did user left Hue before getting query result? Can anyone give me explanation on this? Thank you! 

I have completed the installation of CDH 5.16.1 on Ubuntu 16.04 LTS with OpenJDK 8.   My employer has developed a UI using Banana (currently packaged with Fusion) that I am required to run on a VM hosted in my machine. The Banana UI needs Solr both to operate and to store data collections that can be visualised. In this case, the Solr instance within the VM uses ports 8983 and 7574. The Banana UI also has a unique security feature developed by my employer (which I have to familirise myself with) that does not yet allow me to host the entire contents of the VM on my server which also hosts the Cloudera CDH instance.   Due to the above arrangement, it is not possible for me to startup Cloudera CDH (though Cloudera Manager is still running). This has proven to be a difficulty since I need to use Solr and also use the various tools (Spark etc).   Could you tell me how I could change the default port for Cloudera Solr (from 8983 to an appropriate port) and for Cloudera Zookee

Hi, I have a dashboard application (using Zoomdata) in which we submit queries to a Impala resource pool with limited number of concurrent requests. While we load the dashboard, some queries might get queued if conc. requests limit is reached.When I change the dashboard time range, I noticed the running queries are properly cancelled, but the queued ones stay in the queue and are going to execute before the new queries submitted. Do you know if there is any Impala JDBC driver property or Impala server setting to be able to cancel queued queries and prevent them from running?

I'm testing Hue - SAML integration.SAML login now works, but all users land in Hue with "default" group.Is there a way to assign Hue groups and permissions based on SAML attributes?

I tried sudo apt-get install lucidworks-hdpsearch. That also gives the same error. My OS is ubuntu 16.04 and I have ambari 2.6 installed.Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/SOLR/6.6.2/package/scripts/solr.py", line 139, in <module> Solr().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/SOLR/6.6.2/package/scripts/solr.py", line 20, in install self.install_packages(env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 821, in install_packages retry_count=agent_stack_retry_count) File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_manag

I am calling an API which returns a JSON object as the body of the response as a flow file. I want to count the total records in the flow file.I tried record processors like CalculateRecordStats needs a record reader and that in-turn needs a schema. But I don't have a schema. So Is there a better way to do it?{"result":[ { "key1": "value1", "key2": "value2" }, { "key1": "value1", "key2": "value2" }]}This is the sample JSON the api response body returns. The records are the items of the list which is the value of the "result" keyIn the above sample there are 2 records (2 items in the list). So that is what I want to calculate.You can also answer here

hi allwe have ambari cluster - HDP version - 2.6.4on one of the master machine we cant start the name nodeand we notice that ls  /hadoop/hdfs/journal/hdfsha/current/ | grep edits_inprogress  edits_inprogress_0000000000018783114.emptywe not have the edits_inprogress_xxxxxx filewhat we have is only the file - edits_inprogress_0000000000018783114.emptyany idea how to recover the - edits_inprogress_xxxxxx file2019-06-13 19:45:42,473 FATAL namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(398)) - Error: recoverUnfinalizedSegments failed for required journal (JournalAndStream(mgr=QJM to java.io.IOException: Timed out waiting 120000ms for a quorum of nodes to respond.        at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:137)        at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createNewUniqueEpoch(QuorumJournalManager.java:183)        at

http://archive.cloudera.com/gplextras5/parcels/5.16.1/  new gplextras parcel has bug.as follows：impala/lib/libimpalalzo.so: undefined symbol: _ZN4base21ThreadCollisionWarner5LeaveEv