Solved

Hi. I need to import data from non-trivial dbms: SAS (it isn't db at all)...So, i have jdbc-driver (class "com.sas.rio.MVADriver"), and i have socket (host and port). And i have no connection-manager at all. What should i do? How to write working import-string?1. What about connection-manager2. "driver" looks like jdbc:mysql://<ip>:<port>/ So "mysql" is strange part of it. How to replace it?3. Where must i put drives (jar-files) to use it from sqoop?

Hello Everyone, The scenario I'm trying to do is as follows:1- Flume TAILDIR Source reading from a log file and appending a static interceptor to the beginning of the message. The interceptor consists of the host name and the host IP cause it's required with every log message I receive.2- Flume Kafka Producer Sink that take those messages from the file and put them in a Kafka topic. The Flume configuration is as follows: tier1.sources=source1tier1.channels=channel1tier1.sinks =sink1tier1.sources.source1.interceptors=i1tier1.sources.source1.interceptors.i1.type=statictier1.sources.source1.interceptors.i1.key=HostDatatier1.sources.source1.interceptors.i1.value=###HostName###000.00.0.000###tier1.sources.source1.type=TAILDIRtier1.sources.source1.positionFile=/usr/software/flumData/flumeStressAndKafkaFailureTestPos.jsontier1.sources.source1.filegroups=f1tier1.sources.source1.filegroups.f1=/usr/software/flumData/flumeStressAndKafkaFailureTest.txttier1.sources.so

the server and the agents are located on container(LXC) basd on proxmox virtual env 5.3 with the latest versioni have some issues with ambari metricstail /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out File "/usr/lib/python2.6/site-packages/resource_monitoring/core/metric_collector.py", line 45, in process_event self.process_host_collection_event(event) File "/usr/lib/python2.6/site-packages/resource_monitoring/core/metric_collector.py", line 80, in process_host_collection_event metrics.update(self.host_info.get_combined_disk_io_counters()) File "/usr/lib/python2.6/site-packages/resource_monitoring/core/host_info.py", line 276, in get_combined_disk_io_counters rawdict = psutil.disk_io_counters(True) File "/usr/lib/python2.6/site-packages/resource_monitoring/psutil/build/lib.linux-x86_64-2.7/psutil/__init__.py", line 1726, in disk_io_counters raise RuntimeError("couldn't find any physical disk")RuntimeError: couldn't find any physical disk

Hi,  I got an error during an attempt to access a remote cluster secured by Kerberos and I dont know why the client is trying to find out the hdfs principal in the local KDC.  The setup is as follows (intentionally ommit full domain names and host names to keep it tidy):   - each cluster (CLUSTERDEV and CLUSTERPROD) has its own KDC (DEVREALM and PRODREALM) - the KDC trusts each other (verified by kvno hdfs/<namenodehost>@REMOTE_REALM from boths sides) - both clusters are running NameNode in HA mode   I have configured the Trusted realm in ClouderaManager for CLUSTERDEV set to CLUSTERPROD. (This triggered the RULE change in auth_to_local for core-site.xml). I have done the same for CLUSTERPROD and set trusted CLUSTERDEV.   - each krb5.conf in CLUSTERDEV has also a PRODREALM in [realms] (I can kinit with "remote" account) - each krb5.conf has [capaths]  DEVREALM  = { PRODREALM = . } - and vice versa, in CLUSTERPROD each krb5.conf the DEVRELAM is a

Hello, I would like to know the various activities performed by respective users on Cloudera Manager. For example, which user restarts a service or moves a cluster / service to maintenance mode. Would be great if any one could share some information on this. Thankssnm1523

Hi,Is it possible to perform Namenode and Resource Manager failover through rest api in Ambari?Any help and suggestions will be greatly appreciated.Thank you.

HDP-2.6.5, RHEL 7, Kerberized, Ambari-2.6 upgraded to Ambari-2.7.3Oozie Service Check fails:STDERRTraceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 139, in <module> OozieServiceCheck().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 52, in service_check OozieServiceCheckDefault.oozie_smoke_shell_file(smoke_test_file_name, prepare_hdfs_file_name) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/service_check.py", line 70, in oozie_smoke_shell_file raise Fail(format(NO_DOCS_FOLDER_MESSAGE)) NameError: global name 'Fail' is not definedSTDOUT2019-02-15 19:02:10,245 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.5.74-2 -> 2.

Hey guys, I have already asked this on multiple forums but never got a reply, so I thought that I might get one here. I have an about 1 gig dataset, and it's got a "cityid" column of which there are 324 unique values, so after partitioning I should get 324 folders in hdfs. But whenever I partition, it fails, you can look at the exception messages here https://community.hortonworks.com/questions/238893/notenoughreplicasexception-when-writing-into-a-par.html It's definitely an HDFS issue, because everything worked out on MapR. What could possible be the problem? Btw, I tried this on a fresh install of hortonworks and cloudera and with default settings, so nothing was compromised. If you need any more details please ask. Could this be a setup issue or something? Like maybe I need to increase memory somewhere in the HDFS or something?

Hi Team,I have two questions, can anyone help me 1) How will I get the sysdate -1 updated records using NiFi(Ex: I have a timestamp 09:00 today but when I get incremental the processor need to check since 09:01 yesterday or else less than 2 hours of timestamp)?2) When delete happens in the source how will affect in target? and I need to store delete records in another table in the target how will I achieve using NiFi?Thank you.

does sharelib jars like hive-exechive-commonhive-jdbchive-serivcelibthriftneeds to be included explicitly in java action <file> tag or those are auto included?From this document i see they should be auto included if i am not wrong? how-to-use-the-sharelib-in-apache-oozie-cdh-5 

Context: Hive3, HDP 3.1. Tests done with Python/odbc (official HDP driver) under Windows and Linux.I ran the following queries: "select ? as lic, ? as cpg""select * from (select ? as lic, ? as cpg) as t""with init as (select ? as lic, ? as cpg) select * from init",1) and 2) work fine, and give me the expected result. 3 gives me a ParseException : Error while compiling statement: FAILED: ParseException line 1:21 cannot recognize input near '?' 'as' 'lic' in select clause (80) (SQLPrepare)")The exact same statements ran with java/jdbc work fine. Note that 2) looks like is a workaround for 3) but it works for this tiny example, not for bigger queries.Is there something I can do to have ODBC working as expected? Alternatively, where can I find the limits of the ODBC driver?For full context, the full test code is as follow:cnxnstr = 'DSN=HiveProd'cnxn = pyodbc.connect(cnxnstr, autocommit=True)cursor = cnxn.cursor()queries = [ "with init as (select ? as lic, ? as cpg) select * from init"

I am trying to toggle another remote port through the Remote Proc Group however it only lets me toggle the first 'input port' and the others are greyed out. Does anyone know how to make the remote proc group have the ability to select the other input ports on the remote instance?thanks in advance.

Hi, I'm trying to connect nifi with mongodb but facing problems with the SSL authentication. I already checked the connection parameters and it is ok. The controller service I'm using is the StandardRestrictedSSLContextService . Any ideas ? Thanks, Rogerio

Hi all, I have recently setup a new druid/hive stack (and yes I am new to both and to hadoop/tez, etc) (I have used the user admin accross the entire stack for the entire setup via HDP 3.0)I have been able to run some basic queries from hive without an issue, such as:select `__time`,accepteddate,accountid from sms_messages group by accountid, accepteddate, `__time` limit 100;works without issue, but if run (from squirrel)select max(`__time`),accepteddate,accountid from sms_messages group by accountid, accepteddate;(note: this same query works when run on druid directly)it fails with:Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTaskSQLState: 08S01ErrorCode: 1From beeline (we more details):WARN : The session: sessionId=d797cf5f-3b33-49bb-aed9-c57f0e6a30c0, queueName=null, user=admin, doAs=true, isOpen=false, isDefault=false has not been openedINFO : Subscribed to counters: [] for queryId: hive_20190213215439

I have gone through the steps to create an SDX Managed Namespace, created a Data Warehouse Cluster that points to the SDX Managed Namespace, and gotten into the Query editor (HUE) to being testing out queries.  By default the first time I log in there are no permissions set for Sentry so I followed this guide: https://www.cloudera.com/documentation/altus/topics/altsdx_config_namespc_setup.html  create role SentryAdminRoleForAltus;grant all on server server1 to role SentryAdminRoleForAltus;grant role SentryAdminRoleForAltus to group SDXSentryAdminGroup; The only problem is I don't know what group I belong to and its not obvious in the Altus UI.  What value should I input for 'SDXSentryAdimGroup'?

Hello,   I have an HDFS-Hive-Impala regression script that works fine on my kerberized & sentry protected CDH cluster.   Now, I enabled LDAP authentication on HDFS (LdapGroupsMapping), Hive and Impala and the regression script passes HDFS and Hive but fails on the SELECT-INSERT-CREATE Impala actions:   Failure 1 & 2 (similar error for select and insert):   Query: select * from customer.cons limit 10ERROR: AnalysisException: Failed to load metadata for table: 'customer.cons'CAUSED BY: TableLoadingException: Failed to load file metadata for 1 paths for table customer.cons. Table's file metadata could be partially loaded. Check the Catalog server log for more details.    Failure 2:    Query: create table customer.test_141226 (id int)ERROR: ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore: CAUSED BY: MetaException: Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=impala, access=W

Hi All, it may be a trivial question for many, but could you explain what is the difference or relation between classpath defined in yarn.application.classpath and mapreduce.application.classpath? Does the latter overwrite the former for mapreduce applications? There is also variable MR2_CLASSPATH that is included by default in mapreduce.application.classpath. Where is taken from? Is the mapreduce.application.classpath relevant only for gateways from were application is submitted to yarn?

Hello, How do I change data (logs) retention policy for all Cloudera Management Services  like Service Monitor, Host Monitor and Event Server. My target is to retain data (logs) only last 7 days. Cloudera Enterprise 5.11.1 RegardsWert  

Hi, I can find this kind of message in the Impala daemon logs:   impalad.pxlbig04.nz.thenational.com.impala.log.INFO.20190117-104024.29754:I0117 10:42:17.900411 9206 admission-controller.cc:515] Stats: agg_num_running=7, agg_num_queued=0, agg_mem_reserved=525.00 GB, local_host(local_mem_admitted=0, num_admitted_running=0, num_queued=0, backend_mem_reserved=75.00 GB)   Which shows agg_num_running=7   Why is this same kind of metric not available in the CM graphs ?   I want to know how close to the max running queries of 25 that I set in the default queue on Impala Admission Control.

@Sriharsha Chintalapani @Arun Mahadevan @Timothy Spann @pshah@Shu@George Vetticaden @mqureshi @amarnath reddy pappu@sree charan @Suresh Bonam @Sagar Shimpi @Satish Duggana @Sandeep Nemuri @Satish Anjaneyappa @Rishit shah @Balakrishnan Ramasamy I am running a basic Kafka source and sink with one rule in between both have different schemas and i am using a console consumer to read out the sink . While I am adding the test data in the test case it throws the following exception :- Caused by: com.hortonworks.registries.storage.exception.StorageException: java.lang.NoSuchMethodException: com.hortonworks.streamline.streams.catalog.TopologyTestRunCaseSource.setOccurrence(javalang.Long) at com.hortonworks.registries.storage.catalog.AbstractStorable.fromMap(AbstractStorable.java:93) atcom.hortonworks.registries.storage.impl.jdbc.provider.sql.factory.AbstractQueryExecutor$QueryExecution.getStorablesFromResultSet(AbstractQueryExecutor.java:480) at com.hortonworks.registries.storage.i

Is there something wrong with my DB configuration or application config ??I have installed SAM and registry on Postgresql 9.5 !!I have simple kafka source and sink as my topology !! I think i have a problem with the kafka source !! ERROR [2019-01-2214:21:58.894][dw-288- DELETE /api/v1/catalog/topologies/2/testcases/7] c.h.s.s.s.TopologyTestRunResource-Got exception:[StorageException]/ message [com.hortonworks.registries.storage.exception.StorageException: java.lang.NoSuchMethodException: com.hortonworks.streamline.streams.catalog.TopologyTestRunCaseSource.setOccurrence(java.lang.Long)]/ related resource location:[com.hortonworks.streamline.streams.service.TopologyTestRunResource.removeTestRunCase](TopologyTestRunResource.java:548)com.hortonworks.registries.storage.exception.StorageException: com.hortonworks.registries.storage.exception.StorageException: java.lang.NoSuchMethodException: com.hortonworks.streamline.streams.catalog.TopologyTestRunCaseSource.setOccurrence(java.lang

Hello, In my kerberized and Sentry-protected CDH, I started getting the following errors on hive metastore: Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User: hive/master.hadoop.local@HADOOP.LOCAL is not allowed to impersonate sentry/worker1.hadoop.local@HADOOP.LOCALIn core-site.xml I have: hadoop.proxyuser.hive.groups=*hadoop.proxyuser.hive.users=*The error started after I was playing around with LDAP integration, though I rollback my configurations to the previous no-LDAP state. I am trying to figure out what I missed.

Hi All,I am working on HDP sandbox and trying to login into ATLAS UI with user holger_gov and same as password however it give me invalid credentials. Looked for a solution on this community however it did not fix my issuechnaged the authorizer to simple and restarted but did not work so changed it back to ranger and tried again.I see that atlas user is in the OS shown in screenshotsSome of the screenshots detailing my situation are shown belowplease need any helpThanksDhieru

hi allwe want to create an instance of SLIDER viewso we run the following API curl --user admin:admin -i -H 'X-Requested-By: ambari' -X POST http://localhost:8080/api/v1/views/SLIDER/versions/2.0.0/instances/SLIDER_NEWHTTP/1.1 400 Bad RequestX-Frame-Options: SAMEORIGINX-XSS-Protection: 1; mode=blockX-Content-Type-Options: nosniffCache-Control: no-storePragma: no-cacheSet-Cookie: AMBARISESSIONID=hpxbpzq0m16x1xygqwvbu3340;Path=/;HttpOnlyExpires: Thu, 01 Jan 1970 00:00:00 GMTUser: adminContent-Type: text/plainContent-Length: 527{ "status" : 400, "message" : "{\"propertyResults\":{\"ambari.server.password\":{\"valid\":false,\"detail\":\"No property values exist for the required parameter ambari.server.password.\"},\"ambari.server.url\":{\"valid\":false,\"detail\":\"No property values exist for the required parameter ambari.server.url.\"},\"ambari.server.username\":{\"valid\":false,\"detail\":\"No property values exist for the required parameter ambari.server.username.\"}},\"valid\":fals

Hi,I are running a number of HDP clusters on Cloudbreak. The code is residing on GitHub (Spark Python and Scala), we would like to setup a Continuous integration and Delivery pipeline from Github.I have searched quite a bit for recommended practices for this but to no avail. At this point, I am thinking about using Cloudbreak Command Line (CLI) to create the scripts needed and run them from github. 1. Any better recommendations?2. Does that seem like a valid approach? 3. How would I deal with security since I would need to store Cloudbreak credentials on Github to run my scripts?Any input is appreciated.Thx