Q&A

Hi, I'm trying to connect nifi with mongodb but facing problems with the SSL authentication. I already checked the connection parameters and it is ok. The controller service I'm using is the StandardRestrictedSSLContextService . Any ideas ? Thanks, Rogerio

Hi all, I have recently setup a new druid/hive stack (and yes I am new to both and to hadoop/tez, etc) (I have used the user admin accross the entire stack for the entire setup via HDP 3.0)I have been able to run some basic queries from hive without an issue, such as:select `__time`,accepteddate,accountid from sms_messages group by accountid, accepteddate, `__time` limit 100;works without issue, but if run (from squirrel)select max(`__time`),accepteddate,accountid from sms_messages group by accountid, accepteddate;(note: this same query works when run on druid directly)it fails with:Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTaskSQLState: 08S01ErrorCode: 1From beeline (we more details):WARN : The session: sessionId=d797cf5f-3b33-49bb-aed9-c57f0e6a30c0, queueName=null, user=admin, doAs=true, isOpen=false, isDefault=false has not been openedINFO : Subscribed to counters: [] for queryId: hive_20190213215439

I have gone through the steps to create an SDX Managed Namespace, created a Data Warehouse Cluster that points to the SDX Managed Namespace, and gotten into the Query editor (HUE) to being testing out queries.  By default the first time I log in there are no permissions set for Sentry so I followed this guide: https://www.cloudera.com/documentation/altus/topics/altsdx_config_namespc_setup.html  create role SentryAdminRoleForAltus;grant all on server server1 to role SentryAdminRoleForAltus;grant role SentryAdminRoleForAltus to group SDXSentryAdminGroup; The only problem is I don't know what group I belong to and its not obvious in the Altus UI.  What value should I input for 'SDXSentryAdimGroup'?

Hello,   I have an HDFS-Hive-Impala regression script that works fine on my kerberized & sentry protected CDH cluster.   Now, I enabled LDAP authentication on HDFS (LdapGroupsMapping), Hive and Impala and the regression script passes HDFS and Hive but fails on the SELECT-INSERT-CREATE Impala actions:   Failure 1 & 2 (similar error for select and insert):   Query: select * from customer.cons limit 10ERROR: AnalysisException: Failed to load metadata for table: 'customer.cons'CAUSED BY: TableLoadingException: Failed to load file metadata for 1 paths for table customer.cons. Table's file metadata could be partially loaded. Check the Catalog server log for more details.    Failure 2:    Query: create table customer.test_141226 (id int)ERROR: ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore: CAUSED BY: MetaException: Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=impala, access=W

Hi All, it may be a trivial question for many, but could you explain what is the difference or relation between classpath defined in yarn.application.classpath and mapreduce.application.classpath? Does the latter overwrite the former for mapreduce applications? There is also variable MR2_CLASSPATH that is included by default in mapreduce.application.classpath. Where is taken from? Is the mapreduce.application.classpath relevant only for gateways from were application is submitted to yarn?

Hello, How do I change data (logs) retention policy for all Cloudera Management Services  like Service Monitor, Host Monitor and Event Server. My target is to retain data (logs) only last 7 days. Cloudera Enterprise 5.11.1 RegardsWert  

Hi, I can find this kind of message in the Impala daemon logs:   impalad.pxlbig04.nz.thenational.com.impala.log.INFO.20190117-104024.29754:I0117 10:42:17.900411 9206 admission-controller.cc:515] Stats: agg_num_running=7, agg_num_queued=0, agg_mem_reserved=525.00 GB, local_host(local_mem_admitted=0, num_admitted_running=0, num_queued=0, backend_mem_reserved=75.00 GB)   Which shows agg_num_running=7   Why is this same kind of metric not available in the CM graphs ?   I want to know how close to the max running queries of 25 that I set in the default queue on Impala Admission Control.

@Sriharsha Chintalapani @Arun Mahadevan @Timothy Spann @pshah@Shu@George Vetticaden @mqureshi @amarnath reddy pappu@sree charan @Suresh Bonam @Sagar Shimpi @Satish Duggana @Sandeep Nemuri @Satish Anjaneyappa @Rishit shah @Balakrishnan Ramasamy I am running a basic Kafka source and sink with one rule in between both have different schemas and i am using a console consumer to read out the sink . While I am adding the test data in the test case it throws the following exception :- Caused by: com.hortonworks.registries.storage.exception.StorageException: java.lang.NoSuchMethodException: com.hortonworks.streamline.streams.catalog.TopologyTestRunCaseSource.setOccurrence(javalang.Long) at com.hortonworks.registries.storage.catalog.AbstractStorable.fromMap(AbstractStorable.java:93) atcom.hortonworks.registries.storage.impl.jdbc.provider.sql.factory.AbstractQueryExecutor$QueryExecution.getStorablesFromResultSet(AbstractQueryExecutor.java:480) at com.hortonworks.registries.storage.i

Is there something wrong with my DB configuration or application config ??I have installed SAM and registry on Postgresql 9.5 !!I have simple kafka source and sink as my topology !! I think i have a problem with the kafka source !! ERROR [2019-01-2214:21:58.894][dw-288- DELETE /api/v1/catalog/topologies/2/testcases/7] c.h.s.s.s.TopologyTestRunResource-Got exception:[StorageException]/ message [com.hortonworks.registries.storage.exception.StorageException: java.lang.NoSuchMethodException: com.hortonworks.streamline.streams.catalog.TopologyTestRunCaseSource.setOccurrence(java.lang.Long)]/ related resource location:[com.hortonworks.streamline.streams.service.TopologyTestRunResource.removeTestRunCase](TopologyTestRunResource.java:548)com.hortonworks.registries.storage.exception.StorageException: com.hortonworks.registries.storage.exception.StorageException: java.lang.NoSuchMethodException: com.hortonworks.streamline.streams.catalog.TopologyTestRunCaseSource.setOccurrence(java.lang

Hello, In my kerberized and Sentry-protected CDH, I started getting the following errors on hive metastore: Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User: hive/master.hadoop.local@HADOOP.LOCAL is not allowed to impersonate sentry/worker1.hadoop.local@HADOOP.LOCALIn core-site.xml I have: hadoop.proxyuser.hive.groups=*hadoop.proxyuser.hive.users=*The error started after I was playing around with LDAP integration, though I rollback my configurations to the previous no-LDAP state. I am trying to figure out what I missed.

Hi All,I am working on HDP sandbox and trying to login into ATLAS UI with user holger_gov and same as password however it give me invalid credentials. Looked for a solution on this community however it did not fix my issuechnaged the authorizer to simple and restarted but did not work so changed it back to ranger and tried again.I see that atlas user is in the OS shown in screenshotsSome of the screenshots detailing my situation are shown belowplease need any helpThanksDhieru

hi allwe want to create an instance of SLIDER viewso we run the following API curl --user admin:admin -i -H 'X-Requested-By: ambari' -X POST http://localhost:8080/api/v1/views/SLIDER/versions/2.0.0/instances/SLIDER_NEWHTTP/1.1 400 Bad RequestX-Frame-Options: SAMEORIGINX-XSS-Protection: 1; mode=blockX-Content-Type-Options: nosniffCache-Control: no-storePragma: no-cacheSet-Cookie: AMBARISESSIONID=hpxbpzq0m16x1xygqwvbu3340;Path=/;HttpOnlyExpires: Thu, 01 Jan 1970 00:00:00 GMTUser: adminContent-Type: text/plainContent-Length: 527{ "status" : 400, "message" : "{\"propertyResults\":{\"ambari.server.password\":{\"valid\":false,\"detail\":\"No property values exist for the required parameter ambari.server.password.\"},\"ambari.server.url\":{\"valid\":false,\"detail\":\"No property values exist for the required parameter ambari.server.url.\"},\"ambari.server.username\":{\"valid\":false,\"detail\":\"No property values exist for the required parameter ambari.server.username.\"}},\"valid\":fals

Hi,I are running a number of HDP clusters on Cloudbreak. The code is residing on GitHub (Spark Python and Scala), we would like to setup a Continuous integration and Delivery pipeline from Github.I have searched quite a bit for recommended practices for this but to no avail. At this point, I am thinking about using Cloudbreak Command Line (CLI) to create the scripts needed and run them from github. 1. Any better recommendations?2. Does that seem like a valid approach? 3. How would I deal with security since I would need to store Cloudbreak credentials on Github to run my scripts?Any input is appreciated.Thx

Hi, I am new to cloudera-hadoop. I am trying to install Cloudera Manager on a single Linux box. is this possible, if so, please point me to directions/how-to docs I have been trying to do the pseudo-distributed install following instructions but I always got stuck at some point or the other. since Cloudera Manager install all required services, figured might be better to try that Regards all

  Execute command Start this NodeManager on role NodeManager  Failed to start role Supervisor returned FATAL. Please check the role log file, stderr, or stdout.   Environment details:   Version: Cloudera Express 5.15.0Java VM Name: Java HotSpot(TM) 64-Bit Server VMJava VM Vendor: Oracle CorporationJava Version: 1.7.0_67 System details:Linux optim-rhel72-uppu.development.unicomglobal.software 3.10.0-327.28.3.el7.x86_64 #1 SMP Fri Aug 12 13:21:05 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux   I have followed the steps under "Configuring TLS/SSL for HDFS, YARN and MapReduce" using the link https://www.cloudera.com/documentation/enterprise/5-15-x/topics/sg_hive_encryption.html Service did not start successfully; not all of the required roles started: only 0/1 roles started. Reasons : Service has only 0 NodeManager roles running instead of minimum required 1 I see below error in the role log:   Error starting NodeManagerorg.apache.hadoop.yarn.exceptio

Hello  Can some one please help me on this issue,My two journal nodes are not in sync and i cant able to restart them and both the namenodes are down.I can see below logs in NN, FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLogError: recoverUnfinalizedSegments failed for required journal (JournalAndStream(mgr=QJM to [10.10.10.10:8485, 10.10.10.11:8485, 10.10.10.09:8485], stream=null))org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size 2/3. 1 successful responses:10.10.10.09:8485: lastPromisedEpoch: 69httpPort: 8480 fromURL: "http://Master_Host(10.10.10.09):8480"  When i use to do restart of JN and not happening,And below are the Journalnode logs, ERROR org.apache.hadoop.hdfs.qjournal.server.JournalNodeRECEIVED SIGNAL 15: SIGTERMWARN org.apache.hadoop.metrics2.impl.MetricsConfigCannot locate configuration: tried hadoop-metrics2-journalnode.properties,hadoop-metrics2.properties  Can some one p

Hello,   I have a  kerberized 5.14 CDH cluster and I want to integrate with LDAP for user authentication (not service). I suppose this is what described in "Local MIT KDC with Active Directory Integration".   I have the following questions about this change, if I have understood the process right:   1. Users now will be defined in AD and not kerberos. This means that current kerberos keytab files will no longer be valid. Right?   2. In case of CLI pipelines, where we first have to do a kinit for the principal, how the authentication will be after LDAP?    3. Which services should be configured to work with LDAP? - Cloudera Manager - Hue - Hive - Impala - ... ?   4. Will groups and Sentry permissions require to be re-configured after enabling LDAP?    Thank you, gerasimos

I configured the hbase thrift server role in order to query data from Hue. After the server is restarted, I tried to query from Hue and got connection error. I looked at hbase thrift log file from /var/log/hbase/hbase-cmf-hbase-HBASETHRIFTSERVER-hostname.log.out and found this error: 2019-01-10 18:19:14,257 INFO org.apache.hadoop.hbase.thrift.ThriftServerRunner: starting TBoundedThreadPoolServer on /0.0.0.0:9090 with readTimeout 60000ms; min worker threads=200, max worker threads=1000, max queued requests=1000 2019-01-10 18:19:46,303 ERROR org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer: Thrift error occurred during processing of message. org.apache.thrift.protocol.TProtocolException: Expected protocol id ffffff82 but got ffffff80 at org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:503) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) at org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThr

Good evening.  We are trying out the SDX Managed Namespace (Beta) with the Data Warehouse cluster and noticed that when we try to use CDH 5.15, we get the following message: "A secure environment must be used with Managed Namespace clusters on CDH515 or later." Understanding that the SDX Managed Namespace is still in beta, any idea why we might be getting that?  It seems to work with CDH 5.14, but we wanted to test out the new Query Editor which looks to only support CDH 5.15. Thanks in advance. Mac 

I am new to Apache atlas and I want to create Table and column level linage. I understood that linage can be created with the entity only using process. For a test purpose I have created custom types and entities under that type.Now I want to show linage for Table and column relations. For Example. - Table 1. Employee Columns : Employee_ID, Employee_Name, Employee_AddressTable 2: DepartmentColumns: Employee_ID, Department_Name, Department_CodeI want to create Employee Table as an Entity and Columns of Employee Table as an Entities too. but columns will go as an attribute of Employee Table entity so i can create linage between Tables and Columns using Process. Same for the Department table.Once it is created then I can create linage for Employee---->Process---->Department and also Employee.Employee_ID----->Process---->Department.Department_NamePlease advice. Is it possible?

Hi, currently I'm dealing with the DistributedMapCache-thing... Found this article which gave me an idea of all this, great! https://community.hortonworks.com/articles/71837/working-with-a-nifi-distributedmapcache.htmlNow I try to involve DistributedMapCache (DMC) in my NiFi-Flow without executing the script. Found old questions here but maybe in 1.7 or 1.8 there are new possibilities? Referring to above article, how can I in NiFi... 1. remove a concrete key (like command "remove")? 2. receive a list of existing keys (like command "keys"), maybe inclusive there values? Further questions: 3. How can I clear all content of the DMC at once? Tested to disable the Controller Service for DMC-Server, but after enabeling data still existing. 4. In practice if one wants to use different tables in DMC would they all be in one DMC? Or would I configure DMC-Server and DMC-ClientService for each table? 5. At the moment I work at local installation. Is all this DMC-thing working without difficulty o

hi allwe have HDP version - 2.6.4 and amabri version - 2.6.1kafka version is 10.1we want to upgrade only the kafka machines to the latest versionso my question is :is it possible to upgrade only the kafka machine to the latest version ?what is the latest kafka version ?

I followed the Hortonworks documentation to install HDF and HDP, and I have already installed the HDF Management Pack. However, I am missing the base URL required for the next steps; did I forget a step in the installation process?

Hi,,   Error thrown while creating table in impala using HUE.   Query Status: ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore: CAUSED BY: MetaException: java.lang.RuntimeException: org.postgresql.util.PSQLException: ERROR: could not extend file "base/24863/24866": No space left on device Hint: Check free disk space.CREATE TABLE temp_enc.e060908_mcc_ecomm_map(mcc_id string,segment string)ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' Thanks Yasmin

Hi All, When I try to run a huge query at Impala that does a lot of aggregation & sorts, it throws up with the following error. The error is vague and of not much help. Our cluster has 32 impala daemons and all the nodes have been configured with 8 disks of 4TB each. We defined scratch directories on all the disks/nodes in Cloudera Manager. I tried looking at the logs as indicated in the error message but I couldn't find anything useful.  Could not create files in any configured scratch directories (--scratch_dirs=) on backend 'data405.prod.com:22000'. See logs for previous errors that may have prevented creating or writing scratch files. Version of impala: 2.12.0-cdh5.15.0  Any pointers would be greatly helpful. Thanks,Ravi