Q&A

Hello,We are observing error in Impala Catalouge Logs as belowEI1101 14:52:24.768759 8438 jni-util.cc:308] org.apache.impala.common.InternalException: Error updating the catalog due to lock contention. at org.apache.impala.service.CatalogOpExecutor.updateCatalog(CatalogOpExecutor.java:3474) at org.apache.impala.service.JniCatalog.updateCatalog(JniCatalog.java:308)I1101 14:52:24.768983 8438 status.cc:125] InternalException: Error updating the catalog due to lock contention. @ 0x9812da @ 0xd37e2f @ 0x970728 @ 0x960e7f @ 0xa1f263 @ 0xa1752f @ 0x948d4a @ 0xb420c9 @ 0xb39fc9 @ 0xb3adc2 @ 0xdab24f @ 0xdaba4a @ 0x13346fa @ 0x7fb75bf7cea5 @ 0x7fb75bca59fd CM / CDH - 5.16.2Appericate any assistacne / guidance in fixing this issue. Thanks Wert 

How many workers can CDSW cluster contain ?

Hello! I'm new to Nifi and struggling with what seems like a simple task.I have a file with Header, Body and Trailer in fixed width layout. The header record always starts with an A;The Trailer record always starts with a Z;Body records are identified by B, C, D and so on... (except A and Z) So I first used Route Text to separate Header, Body and Trailer, because my goal here was to replace the fixed width with a delimiter (;) so I can make sense of the info in the file (header, body and trailer will have a different number of columns when separated by a delimiter). Then I used ReplaceText with a regex to create columns delimited by semi colon instead of fixed width. Now I need to regroup the rows and create a single file again, with the header, body and trailer, but this time all separated by semi colons. This is what I'd like to achieve: My template looks like this: Is that possible? I tried using MergeRecord for that but I really don't know how to

2021-10-27 14:10:44,606 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore: 1595e783b53d99cd5eef43b6debb2682/proc created, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE2021-10-27 14:10:44,614 INFO [master/ubuntu22:16000:becomeActiveMaster] regionserver.HRegion: Replaying edits from hdfs://ha-cluster:8020/hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/ubuntu22.mcloud.com%2C16000%2C1635246532227.16352870610082021-10-27 14:10:44,652 INFO [master/ubuntu22:16000:becomeActiveMaster] regionserver.HRegion: Replaying edits from hdfs://ha-cluster:8020/hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/ubuntu22.mcloud.com%2C16000%2C1635246532227.16352875113452021-10-27 14:10:44,657 WARN [master/ubuntu22:16000:becomeActiveMaster] regionserver.HRegion: Failed initialize of region= master:store,,1.1595e783b53d99cd5eef

While trying to add service yarn queue manager the role  " YARN Queue Manager Webapp" not starting and it shows me this error : ERROR ComputeExperienceApp CM Internal user credentials are not available as system environment variables. Please make sure that both CM_USER and CM_PASSWORD are present as environment variables for the application to work correctly.

i use kadmin to generate impala principal keytab(it previous generated by cloudear manager), now impala start failed even though new keytab generated by Cloudera Manager. in cdh admin page error is: unable to login from keytab: Decrypt integrity check faileduse kinit in command line error is: kinit: Password incorrect while getting initial credentials any help will really appreciated.

Hi everyone, I tried to run a spark job on Cloudera by overwriting some basics spark configurations. So I created a spark job with the following Configurations (optional): spark.eventLog.dir = myPathspark.eventLog.enabled = truespark.submit.deployMode = clusterI have executed this job by running a cde cli command. The submitter keeps my overwriting values for these 3 Spark configurations but these values weren't retrieve in my Spark driver. For instance, I found a "default" tmpPath for spark.eventLog.dir instead of myPath and my driver used a client deployMode.Some other configurations (like spark.history.fs.logDirectory) were well overwriting at the same time.  Have you ever met this problem? Thank you 🙂

I am trying to decommission a data node. I started by decommissioning the regionserver, and the nodemanger. I have 7 nodes in the cluster. When I followed the steps to remove the one nodemanager it says it was successful. Even though it says 1 was decommissioned 7 are still showing as active. How do I resolve?  

Hello Team, we are having some performance issue when sending data to Kafka using JSON as format. Performs is very slow. Performance is faster when using DSV parser. How can we fine tune JSON formatter? Regards,Roshan

Hi,   I am getting failure in PutEmail Processor in our organization's nifi server.The error states that "Could not convert socket to TLS:routing to failure: javax.mail.MessagingException: Could not convert socket to TLS:nested exception is:javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake"   I routed the failure relation back to processor and after 30-40(randomly) or more retries it processes successfully.Though trying to figure out why its getting failure in first place. Attaching the log here. 2021-10-25 12:44:36,643 ERROR [Timer-Driven Process Thread-3] o.a.nifi.processors.standard.PutEmail PutEmail[id=bdc0872f-2860-12ce-c193-24605d5b6823] Failed to send email for StandardFlowFileRecord[uuid=844b4761-2a4c-439a-9f3d-5e6f06739c00,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1635165813775-357600, container=default, section=224], offset=3718, length=82370],offset=0,name=SampleFile.csv,size=82370]: Could not convert socke

I've enabled Hive Ssl, Sqoop tries to connect to hive without the ssl information.How do I specify or set the right jdbc URL for Hiveserver2 in Sqoop.I tried using --hs2-url but got 'Unrecognized argument'.

yarn的resourcemanager启动失败，报告文件找不到？这个问题需要怎么分析去解决？Error starting ResourceManagerjava.lang.NoSuchFieldError: id_tc26_gost_3410_12_256 at org.bouncycastle.operator.DefaultSignatureAlgorithmIdentifierFinder.<clinit>(Unknown Source) at org.apache.hadoop.yarn.server.webproxy.ProxyCA.<clinit>(ProxyCA.java:99) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:745) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1149) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:306) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1436) 

Hello, After receiving the necessary resources from yarn (executors), my spark job doesnt start instantly like before.It takes now 15 minute to start.What spark is doing during this time ? And why the time went from a few seconds to 15 minutes ?I use Spark 2.2 with scala on hadoop/yarn

hi i have a following json nested dictionary:{"data" :{"Abc_def_{x_y.z.test_at_here}.json":{"timestamp_start": 0,"timestamp_stop": 0}}}i need to extract the value of 3rd level json key that is from timestamps :{"timestamp_start": 0,"timestamp_stop": 0}I tried using evauatejsonpath with $.data.['Abc_def_{x_y.z.test_at_here}.json']. it worked fine.But i have the content inside curly {x_y.z.test_at_here} in .json key which is dynamic with special characters.Also tried parsing usin updateattribute, but i am unable pass it as a string in eval

Hi Support team,I kindly need your support.I am doing some tests in order to evaluate into Cloudera the Atlas data lineage by importing Hive DB.I executed several tests and also followed entirely this sample: https://community.cloudera.com/t5/Community-Articles/Using-Apache-Atlas-to-view-Data-Lineage/ta-p/246305 In order to migrate all the entities into Atlas, I successfully executed the import-hive.sh script that is located in Atlas hook-bin folder.The result of the migration shows (into Atlas) Hive test DB with all its entities but I cannot see in Atlas any lineage (that should appear as the join between two tables in order to create the third one), simply the message: " No lineage data found ".I also checked that if I create EXTERNAL TABLES rather than TABLE, something appears in Lineage section but is incomplete and not showing exactly the dependencies with source table names.Could you please guide me in the solution of this issue that is hiding the Data Lineage

I have 5 Hadoop/HDFS nodes running RHEL 7.9, Ambari 2.7.5.0, HDP 3.1.5. /hadoop/yarn/local/usercache/hive/filecache/ on all of the Hadoop nodes is growing very large and I want to decrease its size. Is there a YARN setting I can change to lower the amount of filecache specified above?  Do I need to manually delete directories that are X days old to clean this up? Please advise.  Thanks! Mike

Hi All, I have a requirment in my company to use an existing password vault for all the technical accounts used by cloudera CDP. Do you know if there is a way to integrate DCP with any password vault ? in my case HashiCorp ThanksAlexis 

Hi Team,  I have a requirement where I need to schedule an oozie work flow using shell action. Shell action is using a shell script which in turn will be calling a hql file which is available on HDFS location. I am not able to call that hql file from the shell script which I am using in shell action. I know using beeline - f parameters to use to call local files but can you please guide me to call a file which is on HDFS. Any help/suggestions will be highly appreciated. Thanks, Wasim

Hi there, I have an issue on my HBase cluster. HBase version: 2.0.2.3.1.4.0-315There are latency spikes every 10mins on all HBase operations, mostly visible on reads. Please have a look to the first graph below. Metric for the graph is `hbase_table_latency_gettime_max`. I see also spikes every 10mins on `hbase_regionserver_ipc_queuecalltime`, please have a look to the graph below:What I've checked so far:It doesn't look like GC as GC doesn't correlate with spikes time.It is not a major compaction. I see spikes with and without it.It is not replication. I did a test with and without replication.I see nothing suspicious in logs or at least what could bring my attention: DEBUG and TRACE level was enabled.Memstore flushes are happening every hour.Amount of active handlers looks good to me, it is set according to recommendationsThere are scans of meta happening every 5 mins (please have a look to the graph below)There are scans of namespace happening every 10 mins and slightl

Timeline Service V2.0 Reader not startingdetails=row 'prod.timelineservice.entity' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=sandbox-hdp.hortonworks.com,17020,1628801419880, seqNum=-12021-09-29 10:23:01,232 INFO [main] client.RpcRetryingCallerImpl: Call exception, tries=20, retries=36, started=231328 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server sandbox-hdp.hortonworks.com,17020,1632910734969 is not running yetat org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1487)at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2443)at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)at org.apac

Hi all,I have a cluster of which computers are with different hardware spec.Is there a way to make Region Server to hold different number of regions depending on the hardware spec of the host?Or is there other way for performance tuning under this circumstance?The hardware spec for HBase Region Server: CPU (cores)Memory (GB)NumbersComputer 14321Compurer 28642Computer 3242565There are YARN and HBase on this cluster of 8 Computers.

We are using Cloudera 6.2.1 and enabling row and cell level ACLs on our Hbase tables.  After turning on the Hbase authorization toggle, in some environments, everything works perfectly.  We can add new labels and auths from Hbase shell. In a few environments when trying to call the 'add_labels' command in Hbase shell it complains that 'Table hbase:labels does not exist'.  To fix it i ran the following command. create 'hbase:labels','f' Then I was able to call the add_labels command no problem.  I was just curious what actually creates this table and was trying to understand why in some environments it wasn't created. Richard

I am trying to execute a workflow using rest API, but could not proceed due to authentication error: Sample:POST : https://xxxxxxxhostxxxxxxxxx:port/oozie/v2/jobs?action=startContent-Type : application/xmlRequest Payload: <configuration><property><name>oozie.wf.application.path</name><value>hdfs://XXXXXXXXXXXXXXXXXXXXXXX</value></property><property><name>oozie.use.system.libpath</name><value>True</value></property><property><name>user.name</name><value>XXXXXXXXXXXXXXXXXx</value></property><property><name>jobTracker</name><value>yarnRM</value></property><property><name>mapreduce.job.user.name</name><value>XXXXXXXXXXXXXXXX</value></property><property><name>nameNode</name><value>hdfs://XXXXXXXXXXX</value></property></configuration> response

Description:Nifi node will be stuck and all processors won't deal with flow file when one processor is processing a large flow file, like PutFile, ExecuteStreamCommand, ConvertRecord. Nifi node will be very very slow and heavy cpu usage. Sometime the cpu usage rate will over 700%. Then the heartbeat may be Disconnected. Heavy cpu usage also can cause other processors stuck and never consume any flow file until the large flow file processing is completed. As I know, the PutFile or ExecuteStreamCommand shouldn't cost too many cpu, it is just executing a cp command in linux. We have tried to increase the total thread count. But it is not any effect. This issue happened after we upgrade the nifi version from 1.8 to 1.12. We can't understand why these processors can cost so many cpu than nifi node will be disconnected from cluster. Anyone can help me? Thanks a lot. Nifi Version: 1.12Machine capacity:cpu 8 core,memory 64G,Xmx 20G,Nifi node amount 5

Hi I am trying to configure Viewfs with HDFS Federation. I tried this with HDP 3 and I am trying to find the procedure it implement on CDP cluster. May someone please share the procedure to do the same. The objective it to share the data between tow CDP clusters.Thanks