Solved

I am seeing below messages and it is messing up the SRC_OFFSET in this case, i have configured UpdateAttribute with 2 concurrent Tasks. mine is a 3 node cluster, Nifi version 1.9.0

The XML that I have is as follows:<?xml version="1.0" encoding="UTF-8"?><dataService>   <dataOutput>      <fltdMessage acid="QXE2828" airline="QXE" arrArpt="KSJC" cdmPart="false" depArpt="KPAE" fdTrigger="HCS_TRACK_MSG" flightRef="97606500" msgType="trackInformation" sensitivity="A" sourceFacility="KZSE" sourceTimeStamp="2019-04-05T16:44:30Z">         <trackInformation>            <qualifiedAircraftId>               <aircraftId>QXE2828</aircraftId>               <computerId>                  <facilityIdentifier>KZSE</facilityIdentifier>                  <idNumber>657</idNumber>               </computer

Hi everyone , This is the Sandbox env.,M trying to load data into Hive table called: dwstg.dummy; here are the steps i followed inside WebShellClientStep1)after connecting to JDBC Hive , below is the command i issuedInput) LOAD DATA INPATH '/mnt/tmp/testme.csv' OVERWRITE INTO TABLE dwstg.dummy;Output) I get following errorError: Error while compiling statement: FAILED: SemanticException Line 1:17 Invalid path ''/mnt/tmp/testme.csv'': No files matching path hdfs://sandbox-hdp.hortonworks.com:8020/mnt/tmp/testme.csv (state=42000,code=40000) Step2)based on the above error , i researched & realized JDBC is not seeing the mount point as it should exist only in HDFS directory for it to see?Step2.1)So i ran below command -->Input) hadoop fs -cp /mnt/tmp/testme.csv hdfs:/ Output: cp: `/mnt/tmp/testme.csv': No such file or directory

Hi Team, I am trying to perform some testing on Impala so that I can analyze the performance of Impala query based on provided configuration.I am using TPCDS queries.I am making jdbc calls to fire queries. In order to change the configuration values for my query at run time for the current session, I am using Impala query options. I am analysing the query attribute values after execution. In one of the jdbc url I am using "mem_limit" query option, I set its value as 3gb (mem_limit=3gb) But I can not see this value is applied to the current session. I am getting below error- "Memory limit exceeded" this is how I am using query option jdbc:impala://host:21050 /tpcds_bin_partitioned_textfile_40;AuthMech=1;KrbRealm=test.com ;KrbHostFQDN=host;KrbServiceName=impala;mem_limit=3gb;"; But when I changed(mem_limit=3gb) the value from clouodera manager->Impala->configuration, it works fiine. What wrong I am doing here.  

Hi.  I am having a "What the heck" moment.  Could someone please explain the theory behind this.  I have always presumed that Sqoop - unlike other MR processes that might require the entire dataset to be in memory to work - should not ever have a OOM issue.  Afterall, it is using its memory as a buffer, copying the data from DB to the staging area in HDFS, and when complete, moving from staging to --target-dir.   So, we were moving a fairly large DB (500GB) but our client would only allow us to use 1 mapper (don't ask why...gulp).  About 90 minutes into the process, it terminated with:   Container is running beyond the 'PHYSICAL' memory limit. Current usage: 1.0Gib of 1 GB physical memory used; 2.7GB of 2.1 GB virtual memory used. Killing container   This is really confusing me.  I suppose I can solve the problem by a) increasing the vmem/pmem ratio (yarn.nodemanager.vmem-pmem-ratio = xyz) OR b) not checking for this error (yarn.nodemanager.

My goal is to take in a large PDF (34MB), split it into smaller, 2-page PDFs, and send each of the new 2-page PDFs out as a new flowfile for further processing. I've provided the code I have so far below, most of which came from the ExecuteScript Cookbook series. It looks as though the upstream flowfile is not even entering the processor (photo attached). Any help would be greatly appreciated!import org.apache.pdfbox.pdmodel.PDDocumentimport org.apache.pdfbox.multipdf.Splitterimport org.apache.commons.StandardCharsetsflowFile = session.get()if(!flowFile) returndef pdf = ""session.read(flowFile, {inputStream -> pdf = IOUtils.toString(inputStream, StandardCharsets.UTF_8)} as InputStreamCallback)def document = PDDocument.load(pdf)def splitter = new Splitter()splitter.setSplitAtPage(2)def forms = splitter.split(document)//Iterator<PDDocument> iterator = forms.listIterator();for(form in forms){ form.close() newFlowFile = session.create() newFlowFile = session.write(n

I am trying to load a dataframe into a Hive table by following the below steps: Read the source table and save the dataframe as a CSV file on HDFS val yearDF = spark.read.format("jdbc").option("url", connectionUrl).option("dbtable", s"(${execQuery}) as year2016").option("user", devUserName).option("password", devPassword).option("partitionColumn","header_id").option("lowerBound", 199199).option("upperBound", 284058).option("numPartitions",10).load() Order the columns as per my Hive table columns My hive table columns are present in a string in the format of: val hiveCols = "col1:coldatatype|col2:coldatatype|col3:coldatatype|col4:coldatatype...col200:datatype"val schemaList = hiveCols.split("\\|")val hiveColumnOrder = schemaList.map(e => e.split("\\:")).map(e => e(0)).toSeqval finalDF = yearDF.selectExpr(hiveColumnOrder:_*) The order of columns that I read in "execQuery" are same as "hiveColumnOrder" and just to make sure of the order, I select the co

We're using cloudera with anaconda parcel on bda production cluster . I tried to execute pyspark code that imports pyarrow package , then i faced with error below . Traceback (most recent call last): File "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera2-1.cdh5.13.3.p0.316101/lib/spark2/python/lib/pyspark.zip/pyspark/sql/utils.py", line 140, in require_minimum_pyarrow_version File "/opt/cloudera/parcels/Anaconda-3.6.5_2/lib/python3.6/site-packages/pyarrow/__init__.py", line 47, in <module>   from pyarrow.lib import cpu_count, set_cpu_countImportError: libboost_system.so.1.66.0: cannot open shared object file: No such file or directoryDuring handling of the above exception, another exception occurred:Traceback (most recent call last): File "lbf_livArea_scr_2.py", line 51, in <module>   @pandas_udf(schema, PandasUDFType.GROUPED_MAP) File "/opt/cloudera/parcels/SPARK2-2.3.0.cloudera2-1.cdh5.13.3.p0.316101/lib/spark2/python/lib/pyspark.zip/py

Hi Team, Greetings, We are using Impala 2.12 along with cdh 5.16.x. We planning to upgrade to Impala 3.1.0. So will there be a need to upgrade to cdh 6.1.0 or higher inorder use Impala 3.1.0 or this can be handled using cdh 5.16.x.   RegardsParth

After a successful fresh Installation of HDP 3.1.0 on a 2 node Ubuntu 18.0.4 instances, we were able to get all the services up and running.But after a night off, Datanode doesn't start. Following is the error:ERROR datanode.DataNode (DataNode.java:secureMain(2883)) - Exception in secureMainjava.io.IOException: The path component: '/' in '/var/lib/hadoop-hdfs/dn_socket' has permissions 0777 uid 0 and gid 0. It is not protected because it is world-writable. This might help: 'chmod o-w /'. For more information: https://wiki.apache.org/hadoop/SocketPathSecurity    at org.apache.hadoop.net.unix.DomainSocket.validateSocketPathSecurity0(Native Method)    at org.apache.hadoop.net.unix.DomainSocket.bindAndListen(DomainSocket.java:193)    at org.apache.hadoop.hdfs.net.DomainPeerServer.<init>(DomainPeerServer.java:40)    at org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:1194)    at org.apache.hadoop.hdfs.s

Hi,  I've changed the Host Monitor Storage Directory (firehose.storage.base.directory) from /var/lib/cloudera-host-monitor to somewhere else in order to make some tests. After that, I turned back to the old place, having never deleted the data from before those directory changes.  Nevertheless, after this directory changes, the data collected before is not used to plot the charts. I can only see on charts data collected after al those changes, despite having much more information in the directory.  Is there anything I could do for the old data to be seen on charts? My current CDH and CDM version is 5.15 Thanks!

hi allAPI + how to set both thrift server in ambari to maintenance modewe need to stop both shrift server services , and to set them to maintenance modesostop both thrift server by APIand set both thrift server to maintenance mode by API

Hi, we use Impala with Parquet and are considering switching to Kudu for having the benefit of a mutable persistence layer.   We currently write the parquet files manually in java code and upload them to HDFS.   For Kudu, i tested switching this to using the KuduClient for inserts but if i insert a Timestamp with "new Date()" and then SELECT the inserted row in Impala, the timestamp is one hour off, so i assume its a timezone issue. Testing the same by executing an INSERT INTO statement via impala jdbc, the result is perfectly fine.   However, it looks like there are serious performance issues when using Batch Inserts via jdbc (70 rows per second) versus using the kuduClient directly (>1000 rows per second). thats why i would prefer the kudu client.   Can someone give me a hint on how to solve the timestamp issue? should that be done in code or can i somehow setup kudu and impala to have the same timezone? I checked already the server dates and they are fine. &nb

Hi, I'm setting up Impala Admission Control.For the user.<username> placement rules, there is "Not recommended" remark "Use the resource pool that matches the username. (Not Recommended)"(https://www.cloudera.com/documentation/enterprise/5-16-x/topics/cm_mc_resource_pools.html)In my use-case, specific limitation for a set (a dozen) of users should be applied. At the same time the group management is relatively hard, therefore, I would prefer the root.[username] approach. I would like to better understand the drawbacks of this approach: are there any technical limitations from Impala point of view, or it is just a bad practice due to it is harder to maintain&support&manage (from Administrator's perspective)?

Hi all,is it possible to move a cluster from a workspace to another workspace?Let's say I have created a cluster in my personal workspace and want to move it to another workspace where more people have access to. Is this possible?Otherwise I have to carefully think about which cluster to spawn in which workspace in advance because it cannot be changed later on.Thanks in advance!Alex

Hi Cloudera Community,  Why Cloudera don't recommended the creation of root.[username] pool on the Resource Pools configuration ? I can't find docummentation realted with this. Best Regards,  

Hi, I foun this error in HUE server each time I pick a query from query history in Hive Editor. Only if the query has NOT expired... Any idea, suggestion?  11:17:02 AMERRORdecoratorsError running fetch_result_sizeTraceback (most recent call last): File "/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/hue/desktop/libs/notebook/src/notebook/decorators.py", line 97, in decorator return func(*args, **kwargs) File "/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/hue/desktop/libs/notebook/src/notebook/api.py", line 264, in fetch_result_size response['result'] = get_api(request, snippet).fetch_result_size(notebook, snippet) File "/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/hue/desktop/libs/notebook/src/notebook/connectors/hiveserver2.py", line 82, in decorator return func(*args, **kwargs) File "/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/hue/desktop/libs/notebook/src/notebook/connectors/hiveserver2.py", line 329, in fetch

Hi all,is there a way to define custom hostnames when deploying a cluster with the WebUI of Cloudbreak or do we have to use the CLI to do that? The documentation just mentions the CLI:https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.9.0/advanced-cluster-options/content/cb_custom-hostnames.html Is it possible to add the corresponding customDomain entries to the request that the Cloudbreak cluster creation wizard creates at the end without having to switch to the CLI?Thanks!Alex

Hello, is it possible for one single user to allocate more than 50% of memory on cdsw because I'm not able to start a session with more than 50% of memory?Actually this makes sense if this is a limitation from cdsw but most of the time I use cdsw alone in our company and I wanted to ask if there is a parameter which I can set to allocate more memory. Thanks.

1. Created HDF cluster in AWS using cloudbreak 2.9. 2. Deleted cloudbreak stack using AWS CloudFormation - I still can use HDF cluster without cloudbreak.Question is can I import existing HDF cluster in AWS to Cloudbreak?BrMargus Roo

The list of supported CentOS versions for CDH clusters managed by Cloudera Altus Director 2.8 and 6.0 do not include CentOS 7.6, however we have several clusters managed by Director 2.8 running CentOS 7.6. Are the docs out of date?

Hello,I am observing quite strange behaviour of anyDelineatedValue expression when I use it with 'OR' operation.If I use 'AND' operation there is no issues.Input attribute value:org.entitlement.list=AA,CC,DD,FFExpression used in RouteOnAttribte/UpdateAttribute processor :${anyDelineatedValue("${org.entitlement.list}", ","):equals('BB'):or(${anyDelineatedValue("${org.entitlement.list}", ","):contains('EE')})}Above expression is giving ArrayIndexOutOfBoundException for OR operation for the given input (where both the values which expression is looking is not present)If I use 'AND' operation expression works fine. What is the difference here? If I pass BB or EE in the input then expression wont give exception !Looks like defect, can some one please suggest if anything wrong in this?ThanksMahendra

We are receiving Json messages from upstream system via Kafka topic. Requirement is to store these messages into HDFS at certain interval. Since we are storing into HDFS we want to merge certain number of these Records in to single file. As per NiFi documentation we are using "MergeRecords" processor for that. About the in coming Records:##These are the multi-line JSon messages with nested structure.Those are based on the same schema (they are picked from single Kafka topic)Those are validated messages and even NiFi processor is able to parse it. so apparently no issues with JSon messages from Schema point of viewPresent ConfigurationBelow is the snapshot of the Processor Configuration. NiFi version: 1.8 Expected behaviorFor the Above configuration its expected that MergeRecords should have weighted for one of the thresholds i.e. Maximum records(100000) or Maximum Bean size(100KBs). Observed BehaviorBut its observed that bean is getting bundled pretty before either of the threshold is

Hi all,  We are going through the hassle of rebuilding our boxes with CentOS7 as there is no upgrade path from CentOS6. Now I have looked over these Kudu documentation to migrating to multiple kudu master but the problem is the first warning it says it is unsafe to do this so I guess my question would be can I take one master making it a 2 node master then follow the documentation on migrating to a 3 node master setup. what process would I follow to remove the 3rd master other than just deleteing the role as I can only imagine how kudu will handle this. Also we are running kudu 1.4 

Hi All,    We have upgrader he VM resources , like CPU and storage is added to each VM in the Cloudera manager.      we have done this one by one VM , and also we have upgraded the edgeserv as well, where we have proxy is installed to access the datanodes. our appliaction is using the impala api URL to access the data from the datanode via the proxy using edgeserver.   from java side we can see the the error  java.sql.SQLException: [Simba][ImpalaJDBCDriver](500310) Invalid operation: Unable to connect to server:;   and checked the edgeserver logs request is recived in proxy and redirct to the data nodes.   in the data nodes logs we can see error logs   E0327 11:57:09.115049 32288 authentication.cc:159] SASL message (Kerberos (external)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Wrong principal in request)I0327 11:57:09.115571 32288 thrift-util.cc:123] TThreadPoolServer: Caught TException: SASL(-13)