Re: ClouderaDirector 2.2.0 failed with local-repos...

MrBee · ‎08-09-2017

Glad to hear this is still a useful workaround 🙂

MrBee · ‎08-03-2017

This was back in 2016, nowadays I would go for Nifi (open source) or StreamSets (free to use, pay for support) Flume is deprecated in Hortonworks now and will be removed from in future releases 3.*: deprecations_HDP.

MrBee · ‎01-31-2017

I had a similar problem. I had enabled the agent_tls, but the keystore field was not filled or the file was on a different location. Now the server did not start anymore. I needed to rollback the setting, thx for your post. I used mysql tool on the command-line to connect as root to MySQL db, and executed an update: use scm; update CONFIGS set VALUE='false' where ATTR='agent_tls'; Query OK, 1 row affected (0.05 sec) After a restart of cloudera-scm-server, the server was working again and I could enter the UI.

MrBee · ‎01-31-2017

When I used the FullyQualifiedDomainName (with a '.' in it) the repo is working fine! parcelRepositories: ["http://localrepo.cdh-cluster.internal/parcels/cdh5/", "http://localrepo.cdh-cluster.internal/parcels/spark2/"]

MrBee · ‎12-14-2016

I’ll try that out this week. And let you know! Thx for your advice.

MrBee · ‎12-13-2016

Localrepo synced latest version from: - ClouderaDirector - ClouderaManager Also serving parcels: - CDH - spark2 Bootstrap config: cloudera-manager { ... repository: "http://localrepo/cloudera-manager/" repositoryKeyUrl: "http:// localrepo /cloudera-manager/RPM-GPG-KEY-cloudera" } ... cluster { products { CDH : 5 } parcelRepositories : ["http://localrepo/parcels/cdh5/", "http://localrepo/parcels/spark2/"] ... } We start with cloudera-director-client bootstrap-remote with the config file. The ClouderaDirector provisioning: ClouderaManager, datanodes, masters are created. But script failes at around step 870/900. No errors in ClouderaManager logs, error appears in ClouderaDirector log, getting something from an empty-collection when building some Repo-list. Bootstrap remote with a config file end with failed state: /var/log/cloudera-director-server/application.log [2016-12-13 10:00:53] INFO [pipeline-thread-31] - c.c.l.pipeline.util.PipelineRunner: >> BootstrapClouderaManagerAgent$HostInstall/4 [DeploymentContext{environment=Environment{n ame='DataLake-devtst', provider=InstanceProviderConfig{t ... [2016-12-13 10:00:53] ERROR [pipeline-thread-31] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed java.util.NoSuchElementException: null at com.google.common.collect.AbstractIterator.next(AbstractIterator.java:154) at com.google.common.collect.Iterators.getOnlyElement(Iterators.java:307) at com.google.common.collect.Iterables.getOnlyElement(Iterables.java:284) at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent.getRepoUrl(BootstrapClouderaManagerAgent.java:325) at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent.newApiHostInstallArguments(BootstrapClouderaManagerAgent.java:307) at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent.access$200(BootstrapClouderaManagerAgent.java:63) at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall.run(BootstrapClouderaManagerAgent.java:162) at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall.run(BootstrapClouderaManagerAgent.java:112) Is this a bug? Or am I doing somthing wrong? Local repo looks like this, and works fine for installing ClouderaDirector: [root@localrepo mirror]# ls -ARls | grep / ./cloudera-director: ./cloudera-director/repodata: ./cloudera-director/RPMS: ./cloudera-director/RPMS/x86_64: ./cloudera-director/RPMS/x86_64/repodata: ./cloudera-manager: ./cloudera-manager/repodata: ./cloudera-manager/RPMS: ./cloudera-manager/RPMS/x86_64: ./cloudera-manager/RPMS/x86_64/repodata: ./parcels: ./parcels/cdh5: ./parcels/spark2:

MrBee · ‎09-07-2016

As @Jean-Philippe Player mentions read Parquet directory as tables its not yet supported by Hive. Source: http://www.cloudera.com/documentation/archive/impala/2-x/2-0-x/topics/impala_parquet.html. You are able to do it in Impala: # Using Impala: CREATE EXTERNAL TABLE ingest_existing_files LIKE PARQUET '/user/etl/destination/datafile1.dat' STORED AS PARQUET LOCATION '/user/etl/destination'; With some spark/scala code you can generate the create table statement based on a parquet file: spark.read.parquet("/user/etl/destination/datafile1.dat").registerTempTable("mytable") val df = sqlContext.sql("describe mytable") // "colname (space) data-type" val columns = df.map(row => row(0) + " " + row(1)).collect() // Print the Hive create table statement: println("CREATE EXTERNAL TABLE mytable") println(s" (${columns.mkString(", ")})") println("STORED AS PARQUET ") println("LOCATION '/user/etl/destination/datafile1.dat';")

MrBee · ‎07-27-2016

Hi @Junichi Oda, We have the same error in the Ranger log, even when the groupnames are filled: ERROR LdapUserGroupBuilder [UnixUserSyncThread] - sink.addOrUpdateUser failed with exception: org/apache/commons/httpclient/URIException, for user: userX, groups: [groupX, groupY] I have inspected the sourcecode from ranger-0.6 which is part of HDP-2.4.3.0 our current version of the stack. Interesting enough all calls to remote server inside LdapUserGroupBuilder.addOrUpdateUser(user, groups) are wrapped in a try-catch(Exception e). There is addUser, addUserGroupInfo and delXUserGroupInfo. But we don't see that in the log. The addOrUpdateUser is wrapped with try-catch(Throwable t). Looks like its an Error not an Exception! I found this RANGER-804 ticket revering to missing classes. I copied the jars in '/usr/hdp/current/ranger-usersync/lib' from another folder. The code runs but I have a Certificate PKI error at the moment because we use LDAPS, but looks like this might get you further. Greetings, Alexander

MrBee · ‎07-26-2016

Hi @Zaher, Depending on your data you should care about the channel you choose. The memory-channel is simple and easy, but data is lost when the Flume-agent crashes (OutOfMemory) most likely, or power/hardware-issues also likely... There are channels with higher durability for your data. The filechannel is very durable when underlaying storage is redundant as well. Take a look at the flume-channels and there configuration options. For your OutOfMem-problem you can decrease the transaction and batch capacity and increase the heap in the flume-env config in Ambari as @Michael Miklavcic suggests.

MrBee · ‎07-26-2016

We manage our Flume-agents in Ambari. We have 3 'data-ingres'-nodes of many nodes. These nodes are bundled in a ConfigGroup, which are located at the top in Ambari > Flume > config with the name 'dataLoaders'. The default flume.conf is empty, for the config-group 'dataLoaders' we override the default and add 2 agents: Pulling data from a Queue and put it in Kafka + HDFS Receiving JSON and placing it on a Kafka-topic. Each host in the config-group will run the 2 agents, which can be restarted separately from the Ambari-flume summary page. When you have changed the config, it is traceable/audited in Ambari. A restart from Ambari will place the new config file for the flumes. Ambari-agent on the Flume host will inspect if the process is running and Alarm you when its dead. Ambari will help you when upgrading stack to latest version(s). notes: You cannot put a host in multiple config groups. (don't mix responsibilities) The configuration is in plain text and no validation at all. (start and check /var/log/flume/**.log) Rolling restart for a config group is not supported (restart flume-agents 1 by 1) Ambari 'alive'-checks are super simple, locked-up agent is running, but not working... Ambari Flume data insight charts are too simple, (Grafana coming, or use JMXExporter -> Prometheus)

MrBee · ‎05-24-2016

Hi @Jonas Straub, we configured a secure SolrCloud cluster, with success. There is one MAJOR issue: https://issues.apache.org/jira/browse/RANGER-678 The ranger plugins (hive, hdfs, kafka, hbase, solr) generating audit logs, are not able to send the audit-logs to a secure Solr. The bug was reported 06/Oct/15, but not yet addressed. How do we get it addressed so people can start using a secure Solr for audit logging? Greetings, Alexander

MrBee · ‎05-24-2016

Great article, When testing the connection to Solr from Ranger as @Jonas Straub mentions the /var/log/ranger/admin/xa_portal.log shows the URL. It tries to access ${Solr URL}/admin/collections. So you should enter an URL ending with /solr. Than the log gives an Authentication Required 401. Now Solr is Kerbors-secured the request from Ranger to fetch collections should also use a kerberos-ticket... Did someone manage to make the lookup from Ranger to Solr (/w kerberos) work?

MrBee · ‎05-02-2016

I assume you are using the ambari-metrics-system to collect statistics. You need to add a jar to the flume classpath in order to make the charts work. Edit the 'Advanced flume-env' config in Ambari. Make sure that the flume-env template contains: ... if [ -e "/usr/lib/flume/lib/ambari-metrics-flume-sink.jar" ]; then export FLUME_CLASSPATH=$FLUME_CLASSPATH:/usr/lib/flume/lib/ambari-metrics-flume-sink.jar fi ... Restart flume, now you should be able to see the collected metrics.

MrBee · ‎05-02-2016

Depending on your OS the setting might be different then you expect. To check the actual value become root and switch to the user hbase and print the actual limits. # on Hbase Region Server: sudo -i su hbase # print limits for the user hbase: ulimit -a On our RedHat 6 system, there was a file 90-nproc.conf in /etc/security/limits.d/ deployed. This limits the nr of processes for users to 1024. The user ambari received these limits and when starting hbase from ambari the limits are passed over somehow. As @rmaruthiyodan mentions you can check the running process limits. grep 'open files' /proc/<Ambari Agent PID>/limits grep 'open files' /proc/<Region Server PID>/limits Hbase book config suggests: 'Set it to north of 10k'

MrBee · ‎04-29-2016

You can locate them through ambari. When you (re)start a service you can click on the operations > operation > tasks and inspect the commands: If you look closely the script being executed for restarting the nodemanager is at 08:53:13,592. The script is located in /usr/hdp/current/hadoop-yarn-nodemanager/sbin/yarn-daemon.sh. This file is shipped with the distribution. Before executing this file users are created and config is pushed. The preparation of these steps happen on the AmbariServer. You can search for the python scripts. For example the nodemanager in the /var/lib/ambari-server/resources/common-services/YARN/2.1.0.2.0/package/scripts/. If you change one these files, don't forget to restart the ambari-server, because the files are cached. After an ambari-server upgrade these changes will be overridden reverted. Hope this helps.

MrBee · ‎04-21-2016

Hi Stefan Kupstaitis-Dunkler, We are using HDP-2.3.4.0 and use Kafka en SparkStreaming (Scala & Python) on a (Kerberos + Ranger) secured Cluster. You need to add a jaas config location to the spark-sumbit command. We are using it in yarn-client mode. The kafka_client_jaas.conf file is send as a resource with the --files option and available in the yarn-container. We did not get ticket renewal working yet... spark-submit (all your stuff) \ --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.conf=kafka_client_jaas.conf" \ --files "your_other_files,kafa_client_jaas.conf,serviceaccount.headless.keytab" \ (rest of your stuff) # --principal and --keytab does not work and conflict with --files keytab. # The jaas file will be placed in the yarn-containers by Spark. # The file contains the reference to the keytab-file and the principal for Kafka and ZooKeeper: KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=false useKeyTab=true principal="serviceaccount@DOMAIN.COM" keyTab="serviceaccount.headless.keytab" renewTicket=true storeKey=true serviceName="kafka"; }; Client { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=false useKeyTab=true principal="serviceaccount@DOMAIN.COM" keyTab="serviceaccount.headless.keytab" renewTicket=true storeKey=true serviceName="zookeeper"; } If you need more info, feel free to ask. Greetings, Alexander

MrBee · ‎02-18-2015

Reset authorized_proxy_user_config to default (hue=*) still works.

MrBee · ‎02-18-2015

Hi, thx for your quick response! This solution did indeed solve the problem. I was also tried to change the setting in the Clusters > Impala authorized_proxy_user_config (default: hue=*) I have changed this to hue=*;yarn=*. Let me reset this to default and test, without my modifications.

MrBee · ‎02-17-2015

Using Cloudera Manager we want to enable Impala on YARN. We did so by adding the service llama ApplicationMaster and changing the min cores/mem to 0 and enabling cgroups. We restarted the whole cluster. (HDFS works, Hive on YARN works) Problem: (Shell build version: Impala Shell v2.1.0-cdh5 (e48c2b4) built on Tue Dec 16 19:00:35 PST 2014) [Not connected] > connect data01; Error connecting: TTransportException, TSocket read 0 bytes Kerberos ticket found in the credentials cache, retrying the connection with a secure transport. Connected to data01:21000 Server version: impalad version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa83ff7ca6e2) [data01:21000] > use mydb; Query: use mydb [data01:21000] > select * from mytable limit 10; Query: select * from mytable limit 10 ERROR: com.cloudera.llama.util.LlamaException: AM_CANNOT_REGISTER - cannot register AM 'application_1424245272359_0001' for queue 'root.alexanderbij' : java.lang.reflect.UndeclaredThrowableException, com.cloudera.llama.util.LlamaException: AM_CANNOT_REGISTER - cannot register AM 'application_1424245272359_0001' for queue 'root.alexanderbij' : java.lang.reflect.UndeclaredThrowableException, at com.cloudera.llama.am.yarn.YarnRMConnector.register(YarnRMConnector.java:270), at com.cloudera.llama.am.cache.CacheRMConnector.register(CacheRMConnector.java:178), at com.cloudera.llama.am.impl.NormalizerRMConnector.register(NormalizerRMConnector.java:107), at com.cloudera.llama.am.impl.PhasingOutRMConnector.register(PhasingOutRMConnector.java:139), at com.cloudera.llama.am.impl.SingleQueueLlamaAM.start(SingleQueueLlamaAM.java:158), at com.cloudera.llama.am.impl.ThrottleLlamaAM.start(ThrottleLlamaAM.java:164), at com.cloudera.llama.am.impl.MultiQueueLlamaAM.getSingleQueueAMInfo(MultiQueueLlamaAM.java:169), at com.cloudera.llama.am.impl.MultiQueueLlamaAM.reserve(MultiQueueLlamaAM.java:286), at com.cloudera.llama.am.impl.GangAntiDeadlockLlamaAM.reserve(GangAntiDeadlockLlamaAM.java:205), at com.cloudera.llama.am.impl.ExpansionReservationsLlamaAM.reserve(ExpansionReservationsLlamaAM.java:131), at com.cloudera.llama.am.impl.APIContractLlamaAM.reserve(APIContractLlamaAM.java:144), at com.cloudera.llama.am.LlamaAMServiceImpl.Reserve(LlamaAMServiceImpl.java:132), at com.cloudera.llama.am.MetricLlamaAMService.Reserve(MetricLlamaAMService.java:140), at com.cloudera.llama.thrift.LlamaAMService$Processor$Reserve.getResult(LlamaAMService.java:512), at com.cloudera.llama.thrift.LlamaAMService$Processor$Reserve.getResult(LlamaAMService.java:497), at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39), at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39), at com.cloudera.llama.server.ClientPrincipalTProcessor.process(ClientPrincipalTProcessor.java:47), at com.cloudera.llama.server.AuthzTProcessor.process(AuthzTProcessor.java:89), at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206), at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145), at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615), at java.lang.Thread.run(Thread.java:745), Caused by: java.lang.reflect.UndeclaredThrowableException, at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1655), at com.cloudera.llama.am.yarn.YarnRMConnector.register(YarnRMConnector.java:239), ... 22 more, Caused by: com.cloudera.llama.util.LlamaException: AM_TIMED_OUT_STARTING_STOPPING - AM 'application_1424245272359_0001' timed out ('30000' ms) in state 'FAILED' transitioning to '[ACCEPTED]' while 'starting', at com.cloudera.llama.am.yarn.YarnRMConnector._monitorAppState(YarnRMConnector.java:429), at com.cloudera.llama.am.yarn.YarnRMConnector._initYarnApp(YarnRMConnector.java:294), at com.cloudera.llama.am.yarn.YarnRMConnector.access$400(YarnRMConnector.java:83), at com.cloudera.llama.am.yarn.YarnRMConnector$4.run(YarnRMConnector.java:243), at com.cloudera.llama.am.yarn.YarnRMConnector$4.run(YarnRMConnector.java:240), at java.security.AccessController.doPrivileged(Native Method), at javax.security.auth.Subject.doAs(Subject.java:415), at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642), ... 23 more [data01:21000] > Looking at log in Cloudera Manager (Diagnostics) PriviledgedActionException as:llama (auth:PROXY) via yarn/master01.mydomain.int@MYDOMAIN (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: yarn/master01.mydomain.int@MYDOMAIN is not allowed to impersonate llama In the configuration of YARN Service-Wide > Proxy: all services including llama have a *. Looking at the YARN ResourceManager on master01 running process, inspecting the core-site.xml. I can confirm that these values are applied. Do you have any clue where the problem might be?

MrBee · ‎02-17-2015

The suggestion from Daisuke could be teh solution, When you forget to install the JCE you will see messages like: INFO org . apache . hadoop . ipc . Server : IPC Server listener on 8022 : readAndProcess threw exception javax . security . sasl . SaslException : GSS initiate failed [ Caused by GSSException : Failure unspecified at GSS - API level ( Mechanism level : Encryption type AES256 CTS mode with HMAC SHA1 - 96 is not supported / enabled )] from client 127.0.0.1 . Count of bytes read : 0 javax . security . sasl . SaslException : GSS initiate failed [ Caused by GSSException : Failure unspecified at GSS - API level ( Mechanism level : Encryption type AES256 CTS mode with HMAC SHA1 - 96 is not supported / enabled )] at com . sun . security . sasl . gsskerb . GssKrb5Server . evaluateResponse ( GssKrb5Server . java : 159 ) You can see more details in the kerberos security logs using this startup parameter: HADOOP_OPTS = "-Dsun.security.krb5.debug=true" Greetings, Alexander Bij

Online	Offline
Last Visited	‎07-29-2017 03:13 AM

Date Registered	‎02-17-2015 07:53 AM
Last Visited	‎07-29-2017 03:13 AM
Total Messages Posted	40
Total Kudos Received	25

Re: ClouderaDirector 2.2.0 failed with local-repos...

Re: Flume in Production - To Ambari or not to Amb...

Re: Too many open files in region server logs

Re: parquet snappy file loading into hive

Re: Flume in Production - To Ambari or not to Amb...

Re: how to rollback cloudera manager tls configura...

Re: ClouderaDirector 2.2.0 failed with local-repos...

Re: ClouderaDirector 2.2.0 failed with local-repos...

ClouderaDirector 2.2.0 failed with local-repositor...

Re: Create Hive table to read parquet files from p...

Re: How does the gid associate the groupname in th...

Re: java.lang.OutOfMemoryError: Java heap space wi...

Re: Flume in Production - To Ambari or not to Amb...

Re: Securing Solr Collections with Ranger + Kerber...

Re: Securing Solr Collections with Ranger + Kerber...

Re: Can't start flume agent via Ambari.

Re: Too many open files in region server logs

Re: Where can I get start up scripts for nodemanag...

Re: How to read from a Kafka topic using Spark (st...

Re: CDH 5.3 Impala Llama Kerberos: yarn is not all...

Re: CDH 5.3 Impala Llama Kerberos: yarn is not all...

CDH 5.3 Impala Llama Kerberos: yarn is not allowed...

Re: kerberos authentication failure: GSSAPI Failur...