Q&A

Hi, Can I create a parameterized view in impala somethig like the below pseudo code: Create View MyView     as     SELECT col1, col2 col3     FROM table_one     WHERE startdate = ${date1}     and enddate         = ${date2}     ...   

After enable Kerberos, all the links (Job history server web UI, Solr UI, etc.), asking for Authorization Kerberos.png

I need to change the property of default_sql_interpreter  to Impala in place of hive so by default Impala query editor get loaded. I check hue.ini but didn't find that. Will that be available in hue database configuration.  

Hi all,I have 6 datanodes on my hortonworks cluster (HDP 2.6.3) and one of them is 91% full. The others are "only" 65% full.Il don't understand why the replication is not homogeneous and how i can fix it ?I check the file system and the same difference is observed :On a safe node :# pwd/grid1/hadoop/hdfs/data/current/BP-332877091-10.136.82.11-1500650625087/current/finalized# du -h . --summarize1.9T    .On the unsafe node :# pwd/grid1/hadoop/hdfs/data/current/BP-332877091-10.136.82.11-1500650625087/current/finalized# du -h . --summarize2.7T    .Same things on each DataNode directories.Thanks for your help.Mathieu

Hi Guys,I am trying to sync ldap users, I am getting below error, any idea, please?====================Review Settings====================Primary LDAP Host : xxxxxxxxxxxxx001Primary LDAP Port (636): 636Secondary LDAP Host <Optional>: xxxxxxxxxxx002Secondary LDAP Port <Optional> (636): 636Use SSL [true/false] (true): trueUser object class (posixUser): posixUserUser ID attribute (uid): uidGroup object class (posixGroup): posixGroupGroup name attribute (cn): cnGroup member attribute (memberUid): memberUidDistinguished name attribute (dn): dnSearch Base (dc=xyzcompany,dc=com): dc=xyzcompany,dc=comReferral method [follow/ignore] (follow): followBind anonymously [true/false] (false): falseHandling behavior for username collisions [convert/skip] for LDAP sync (skip): skipForce lower-case user names [true/false] (true): trueResults from LDAP are paginated when requested [true/false] (false): falseambari.ldap.connectivity.bind_dn: uid=svcTDPlookup,ou=serviceaccounts,ou

Hi.  We have a client who has 2 clusters.  On the security cluster, they have sensitive data that they redact and copy to the analysis cluster.  For security reasons, they would like to minimize the number of open ports on the security cluster.  We have successfully tested using distcp from the shell to copy the data with port 8020 open.  They would now like to automate the process through oozie.  In testing, we have run into an error that port 8042 (Node Manager External Port) is not open.     We do not understand why distcp works fine without port 8042 available when run through the shell but fails when called through Oozie.   Any help would be appreciated.  Thanks.   Henry

we need to partition our Hive Table based on date. Date/Month/Year is it better to use int or string for the partition types.ex:CREATE EXTERNAL TABLE partition (id string, event timestamp and so on)PARTITIONED BY (year INT, month INT, day INT)Stored as Parquetvs CREATE EXTERNAL TABLE partition (id string, event timestamp and so on)PARTITIONED BY (year string, month string, day string)Stored as Parquet Noticed that we couldn't do queries like:... where day > 10 with the string option

The following ODBC download page version has the description "C6 Compatible".　https://www.cloudera.com/downloads/connectors/hive/odbc/2-6-1.html　　Hive ODBC Connector 2.5.25 (C6 Compatible)　https://www.cloudera.com/downloads/connectors/impala/odbc/2-6-2.html　　Impala ODBC Connector 2.5.42 (C6 Compatible)What does "C6 compatible" mean?

We have a highly scalable kafka cluster with 11 nodes and another highly scalable Nifi cluster with 11 nodes that we are currently testing. These are created from our custom blueprint. They are deployed in Azure.We find that everytime we stop and restart the cluster a couple of the nodes go unhealthy and we are unable to recover.Do let us know how to resolve the issue.

Hi All, if we add a few additional DataNodes to our cluster, and then execute a rebalance, do the dependent services (Hive, Impala, SOLR, Flume, etc) require an immediate restart or can they run with a stale configuration for a week or so until we can schedule it?   The DataNodes will include the following roles: HBase RegionServer HDFS DataNode Impala Daemon Solr Server YARN NodeManager Thank you.

Hello,I have an attribute log_source that holds a path like D:\dir1\dir2\dir3I need to get only dir2 but I'm unable to escape the slash with ${log_source:getDelimitedField(3,'\')}I've tried several things unsuccessfully.

I sure hope someone can help me, because otherwise I'm sunk. I'm using the ListS3 processor to feed a FetchS3 processor, but when I list my test directory, which contains one file, I get 4 messages in my queue instead of one. (I also get 4 messages with the directory name, but I filter that out.) The 4 messages have the same filename and I can't find any reason why this is happening.I searched the site but can find no mention of duplicate messages (or flows) from the ListS3 processor. Which is odd because it has been so easy to create this condition. Multiple developers encountered it. It wasn't just one flow. I encountered it when creating flows on a Linux server, version 1.6, and on my local Windows 10 machine, version 1.8. Does it every time with various buckets.If I examine the queue in a connection between processors, I can see the 4 messages and that they all have the same filename. The weirdest thing is that position is the same for each of the 4 messages. When I list the direct

I am seeing below messages and it is messing up the SRC_OFFSET in this case, i have configured UpdateAttribute with 2 concurrent Tasks. mine is a 3 node cluster, Nifi version 1.9.0

The XML that I have is as follows:<?xml version="1.0" encoding="UTF-8"?><dataService>   <dataOutput>      <fltdMessage acid="QXE2828" airline="QXE" arrArpt="KSJC" cdmPart="false" depArpt="KPAE" fdTrigger="HCS_TRACK_MSG" flightRef="97606500" msgType="trackInformation" sensitivity="A" sourceFacility="KZSE" sourceTimeStamp="2019-04-05T16:44:30Z">         <trackInformation>            <qualifiedAircraftId>               <aircraftId>QXE2828</aircraftId>               <computerId>                  <facilityIdentifier>KZSE</facilityIdentifier>                  <idNumber>657</idNumber>               </computer

Hi everyone , This is the Sandbox env.,M trying to load data into Hive table called: dwstg.dummy; here are the steps i followed inside WebShellClientStep1)after connecting to JDBC Hive , below is the command i issuedInput) LOAD DATA INPATH '/mnt/tmp/testme.csv' OVERWRITE INTO TABLE dwstg.dummy;Output) I get following errorError: Error while compiling statement: FAILED: SemanticException Line 1:17 Invalid path ''/mnt/tmp/testme.csv'': No files matching path hdfs://sandbox-hdp.hortonworks.com:8020/mnt/tmp/testme.csv (state=42000,code=40000) Step2)based on the above error , i researched & realized JDBC is not seeing the mount point as it should exist only in HDFS directory for it to see?Step2.1)So i ran below command -->Input) hadoop fs -cp /mnt/tmp/testme.csv hdfs:/ Output: cp: `/mnt/tmp/testme.csv': No such file or directory

Hi Team, I am trying to perform some testing on Impala so that I can analyze the performance of Impala query based on provided configuration.I am using TPCDS queries.I am making jdbc calls to fire queries. In order to change the configuration values for my query at run time for the current session, I am using Impala query options. I am analysing the query attribute values after execution. In one of the jdbc url I am using "mem_limit" query option, I set its value as 3gb (mem_limit=3gb) But I can not see this value is applied to the current session. I am getting below error- "Memory limit exceeded" this is how I am using query option jdbc:impala://host:21050 /tpcds_bin_partitioned_textfile_40;AuthMech=1;KrbRealm=test.com ;KrbHostFQDN=host;KrbServiceName=impala;mem_limit=3gb;"; But when I changed(mem_limit=3gb) the value from clouodera manager->Impala->configuration, it works fiine. What wrong I am doing here.  

Hi.  I am having a "What the heck" moment.  Could someone please explain the theory behind this.  I have always presumed that Sqoop - unlike other MR processes that might require the entire dataset to be in memory to work - should not ever have a OOM issue.  Afterall, it is using its memory as a buffer, copying the data from DB to the staging area in HDFS, and when complete, moving from staging to --target-dir.   So, we were moving a fairly large DB (500GB) but our client would only allow us to use 1 mapper (don't ask why...gulp).  About 90 minutes into the process, it terminated with:   Container is running beyond the 'PHYSICAL' memory limit. Current usage: 1.0Gib of 1 GB physical memory used; 2.7GB of 2.1 GB virtual memory used. Killing container   This is really confusing me.  I suppose I can solve the problem by a) increasing the vmem/pmem ratio (yarn.nodemanager.vmem-pmem-ratio = xyz) OR b) not checking for this error (yarn.nodemanager.

My goal is to take in a large PDF (34MB), split it into smaller, 2-page PDFs, and send each of the new 2-page PDFs out as a new flowfile for further processing. I've provided the code I have so far below, most of which came from the ExecuteScript Cookbook series. It looks as though the upstream flowfile is not even entering the processor (photo attached). Any help would be greatly appreciated!import org.apache.pdfbox.pdmodel.PDDocumentimport org.apache.pdfbox.multipdf.Splitterimport org.apache.commons.StandardCharsetsflowFile = session.get()if(!flowFile) returndef pdf = ""session.read(flowFile, {inputStream -> pdf = IOUtils.toString(inputStream, StandardCharsets.UTF_8)} as InputStreamCallback)def document = PDDocument.load(pdf)def splitter = new Splitter()splitter.setSplitAtPage(2)def forms = splitter.split(document)//Iterator<PDDocument> iterator = forms.listIterator();for(form in forms){ form.close() newFlowFile = session.create() newFlowFile = session.write(n

I am trying to load a dataframe into a Hive table by following the below steps: Read the source table and save the dataframe as a CSV file on HDFS val yearDF = spark.read.format("jdbc").option("url", connectionUrl).option("dbtable", s"(${execQuery}) as year2016").option("user", devUserName).option("password", devPassword).option("partitionColumn","header_id").option("lowerBound", 199199).option("upperBound", 284058).option("numPartitions",10).load() Order the columns as per my Hive table columns My hive table columns are present in a string in the format of: val hiveCols = "col1:coldatatype|col2:coldatatype|col3:coldatatype|col4:coldatatype...col200:datatype"val schemaList = hiveCols.split("\\|")val hiveColumnOrder = schemaList.map(e => e.split("\\:")).map(e => e(0)).toSeqval finalDF = yearDF.selectExpr(hiveColumnOrder:_*) The order of columns that I read in "execQuery" are same as "hiveColumnOrder" and just to make sure of the order, I select the co

We're using cloudera with anaconda parcel on bda production cluster . I tried to execute pyspark code that imports pyarrow package , then i faced with error below . Traceback (most recent call last): File "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera2-1.cdh5.13.3.p0.316101/lib/spark2/python/lib/pyspark.zip/pyspark/sql/utils.py", line 140, in require_minimum_pyarrow_version File "/opt/cloudera/parcels/Anaconda-3.6.5_2/lib/python3.6/site-packages/pyarrow/__init__.py", line 47, in <module>   from pyarrow.lib import cpu_count, set_cpu_countImportError: libboost_system.so.1.66.0: cannot open shared object file: No such file or directoryDuring handling of the above exception, another exception occurred:Traceback (most recent call last): File "lbf_livArea_scr_2.py", line 51, in <module>   @pandas_udf(schema, PandasUDFType.GROUPED_MAP) File "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera2-1.cdh5.13.3.p0.316101/lib/spark2/python/lib/pyspark.zip/py

Hi Team, Greetings, We are using Impala 2.12 along with cdh 5.16.x. We planning to upgrade to Impala 3.1.0. So will there be a need to upgrade to cdh 6.1.0 or higher inorder use Impala 3.1.0 or this can be handled using cdh 5.16.x.   RegardsParth

After a successful fresh Installation of HDP 3.1.0 on a 2 node Ubuntu 18.0.4 instances, we were able to get all the services up and running.But after a night off, Datanode doesn't start. Following is the error:ERROR datanode.DataNode (DataNode.java:secureMain(2883)) - Exception in secureMainjava.io.IOException: The path component: '/' in '/var/lib/hadoop-hdfs/dn_socket' has permissions 0777 uid 0 and gid 0. It is not protected because it is world-writable. This might help: 'chmod o-w /'. For more information: https://wiki.apache.org/hadoop/SocketPathSecurity    at org.apache.hadoop.net.unix.DomainSocket.validateSocketPathSecurity0(Native Method)    at org.apache.hadoop.net.unix.DomainSocket.bindAndListen(DomainSocket.java:193)    at org.apache.hadoop.hdfs.net.DomainPeerServer.<init>(DomainPeerServer.java:40)    at org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:1194)    at org.apache.hadoop.hdfs.s

Hi,  I've changed the Host Monitor Storage Directory (firehose.storage.base.directory) from /var/lib/cloudera-host-monitor to somewhere else in order to make some tests. After that, I turned back to the old place, having never deleted the data from before those directory changes.  Nevertheless, after this directory changes, the data collected before is not used to plot the charts. I can only see on charts data collected after al those changes, despite having much more information in the directory.  Is there anything I could do for the old data to be seen on charts? My current CDH and CDM version is 5.15 Thanks!

hi allAPI + how to set both thrift server in ambari to maintenance modewe need to stop both shrift server services , and to set them to maintenance modesostop both thrift server by APIand set both thrift server to maintenance mode by API

Hi, we use Impala with Parquet and are considering switching to Kudu for having the benefit of a mutable persistence layer.   We currently write the parquet files manually in java code and upload them to HDFS.   For Kudu, i tested switching this to using the KuduClient for inserts but if i insert a Timestamp with "new Date()" and then SELECT the inserted row in Impala, the timestamp is one hour off, so i assume its a timezone issue. Testing the same by executing an INSERT INTO statement via impala jdbc, the result is perfectly fine.   However, it looks like there are serious performance issues when using Batch Inserts via jdbc (70 rows per second) versus using the kuduClient directly (>1000 rows per second). thats why i would prefer the kudu client.   Can someone give me a hint on how to solve the timestamp issue? should that be done in code or can i somehow setup kudu and impala to have the same timezone? I checked already the server dates and they are fine. &nb