Starting with HDP 3.0 release, Hortonworks has stopped CentOS 6 as a supported platform. This would mean existing cluster nodes needs to be migrated to CentOS 7 before install/upgrade to HDP 3.0. However, CentOS has no official support for upgrading OS from CentOS 6 to CentOS 7. This article provides an unofficial guide for migrating cluster nodes to CentOS 7 without losing data. This is downtime migration. We would strongly recommend trying these steps on test clusters. These steps are applicable only for a smaller cluster ( <= 10 nodes ) and larger nodes we would need different strategy + automation for the same. Below are some of the assumptions or pre-requisite to be performed before upgrading the OS. Pre-Requisite Visit all service configuration for e.x. HDFS, YARN etc make sure none of the configurations are pointing to non-root disk. If the services are pointing to root disk then upgrading the OS will format the root disk and we will lose data. Please make sure to migrate all configuration pointing to root disk to non-root disk path before upgrading the OS. For .e.x. below service configs. HDFS Data Node Directory ( dfs.datanode.data.dir) is pointing to non-root disk ( i.e. on mounted disk other than where OS is installed ) HDFS Name Node Directory (dfs.namenode.name.dir) is pointing to non-root disk Yarn Local Directories config (yarn.nodemanager.local-dirs) is pointing to non-root disk. Zookeeper dataDir is pointing to non-root disk. Oozie Data Directory is pointing Make sure none of the databases for services like Hive, Oozie, Ranger, and Ambari are within the cluster. If they are part of the clusters being migrated, please migrates the DBs to external hosts before upgrading the OS. Take a backup of cluster blueprint. Please refer to this link on exporting cluster blueprint Backup all required data in the cluster root disk on all nodes. Migrating Slaves Nodes Pick a slave node in the cluster for upgrading the OS. Please note each slave nods needs to be upgraded one at a time. Please do not attempt upgrading multiple slave node in a single go. Follow below steps to upgrade the slave node from CentOS 6 to CentOS 7. If the selected slave node has DataNode installed, please Decommission the DataNode. Please refer to this link for decommissioning. Stop all services on the node from Ambari. ssh to the stop ambari-agent on the node using below command. ambari-agent stop Double check that required files on root disk is backed up. Now upgrade the OS to CentOS 7. Steps to upgrade the OS is out of scope for this article. Once the OS is upgraded, make sure hostname and IP address are same as before the upgrade. Update /etc/hosts with all the cluster nodes IP address and hostname. Disable firewall. Add the required ambari repo to /etc/yum.repos.d and install ambari agent using below command yum install ambari-agent Install required JDK version pointing to the path configured in ambari.properties Reset the certificates and configure ambari-agent to connect to ambari-server using below command. ambari-agent reset <ambari-server-host-name> ambari-agent had issues with connecting ambari-server when we were performing and below steps helped us in solving the problem. Under /var/lib/ambari-server/keys on ambari-server node, move the existing csr and crt files for the upgraded host being registered. Add below entry under [security] section of /etc/ambari-agent/conf/ambari-agent.ini force_https_protocol=PROTOCOL_TLSv1_2 Now start the ambari-agent using below command. ambari-agent start Once the host is ambari-agent is up and running, goto the Ambari and for the upgraded host run "Recover Host" option from the Host Actions. Recover Host reinstalls all services on the upgraded node with the required yum package. Once the recover host is complete, recommission the DataNode. Start all the services on the host. Migrating Master Nodes Upgrading the Master Nodes to CentOS 7 is similar to that of slaves node. Since this article covers upgrade with downtime, inline upgrading is out of scope. In case we are performing inline upgrade we would need to have HA for all master and move the components to different hosts before upgrading the same, however, this is out of scope from the article. Migrating Ambari Node Run below pre-requisite before upgrading the Ambari Node. Backup the key directories in ambari by running below command. ambari-server backup Backup the ambari.properties under /etc/ambari-server/conf Back up the ambari-server DB if it is embedded postgres DB using below command pg_dump -U {ambari.db.username} -f ambari.sql Preferably restore the DB on an external host by running below command psql -d ambari -f /tmp/dbdumps/ambari.sql Post the pre-requisite is complete follow below steps for the actual upgrade. Upgrade the OS from CentOS 6 to CentOS 7 after taking required backup on the root disk. Update /etc/hosts with all the cluster nodes IP address and hostname. Disable firewall. Configure ambari repo under /etc/yum.repos.d Install ambari-server using below command yum install ambari-server yum install ambari-agent Install required JDK version pointing to the path configured in ambari.properties Reset the certificates and configure ambari-agent to connect to ambari-server using below command. ambari-agent reset <ambari-server-host-name> Run below setup command to point to the external DB. Also, set up similar to how the original ambari server is configured. ambari-server setup Start ambari-agent using below command ambari-agent start Start ambari-server using below command ambari-server start Migrating MIT KDC If the cluster is configured with MIT KDC and is installed within the cluster, follow below steps to backup and restore kerberos database. Please note kdc needs to be installed on the same host where it was installed before the upgrade. prerequisite Backup the keytab from the HDP cluster under /etc/security/keytabs from all nodes. Note down your kdc admin principal and password Backup /etc/krb5.conf Backup /var/kerberos directory Backup backup Take the kerberos database dump using below command ( to be executed on the node running Kerberos ) kdb5_util dump kdb5_dump.txt Safely backup the kdb5_dump.txt. restore Restore the kerberos database execute below command kdb5_util load kdb5_dump.txt Restore the /etc/krb5.conf from backup. Restore /var/kerberos/krb5kdc/kdc.conf from backup. Restore /var/kerberos/krb5kdc/kadm5.acl from backup. Run below command to store master principal in stash file ( kdc admin password is required ) kdb5_util stash Start KDC server using below command service krb5kdc start ... View more

There are at times we would need to move kerberos database to different nodes or upgrade the OS of KDC node ( for e.x CentOS6 to CentOS7 ). Obviously you would not want to lose you the kdc users especially if your HDP cluster is configured to use this kdc. Follow below steps to backup and restore kerberos database. prerequisite * Backup the keytab from the HDP cluster under /etc/security/keytabs from all nodes. * Note down your kdc admin principal and password * Backup /etc/krb5.conf * Backup /var/kerberos directory Backup * Take the kerberos database dump using below command ( to be executed on node running kerberos ) kdb5_util dump kdb5_dump.txt * Safely backup the kdb5_dump.txt. Restore * Restore the kerberos database execute below command kdb5_util load kdb5_dump.txt * Restore the /etc/krb5.conf from backup * Restore /var/kerberos/krb5kdc/kdc.conf from backup * Restore /var/kerberos/krb5kdc/kadm5.acl from backup * Run below command to store master principal in stash file ( kdc admin password is required ) kdb5_util stash * Start KDC server using below command service krb5kdc start ... View more

If you looking to upload template using NIFI REST API, below command curl command upload the template as multipart form post curl -XPOST -H "Authorization: Bearer {{ token }}" https://localhost:9091/nifi-api/process-groups/10c263d0-0160-1000-0000-00006a157654/templates/upload -k -v -F template=@NifiTemplate.xml ... View more

While this article provides a mechanism through which we could setup Spark with HiveContext, there are some limitation that when using Spark with HiveContext. For e.x Hive support writing query result to HDFS using the "INSERT OVERWRITE DIRECTORY" i.e INSERT OVERWRITE DIRECTORY 'hdfs://cl1/tmp/query' SELECT * FROM REGION Above command will result is writing the result of above query to HDFS. However if the same query is passed to Spark with HiveContext, this will fail since "INSERT OVERWRITE DIRECTORY" is not a supported feature when using Spark. This is tracked via this jira. If the same needs to be achieved via spark -- it could achieved by using the Spark CSV library ( required in case of Spark1 ). Below is the code snippet on how to achieve the same. DataFrame df = hiveContext.sql("SELECT * FROM REGION"); df.write() .format("com.databricks.spark.csv") .option("delimiter", "\u0001") .save("hdfs://cl1/tmp/query"); Above command will save the result in HDFS under dir /tmp/query. Please note the delimiter which is used, this is same as what hive currently supports. Also below depedency needs to be added to pom.xml <dependency> <groupId>com.databricks</groupId> <artifactId>spark-csv_2.10</artifactId> <version>1.5.0</version> </dependency> ... View more

Storm 1.1.X provide an external storm kafka client that we could use to build storm topology. Please note this is support for Kafka 0.10 onwards. Below is the step by step guide on how to use the API's. Add below dependency to your pom.xml <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-kafka-client</artifactId> <version>1.1.1-SNAPSHOT</version> </dependency> The kafka spout implementation for the topology is configured using KafkaSpoutConfig. Below is a sample config object creation. KafkaSpoutConfig spoutConf = KafkaSpoutConfig.builder(bootStrapServers, topic) .setGroupId(consumerGroupId) .setOffsetCommitPeriodMs(10_000) .setFirstPollOffsetStrategy(UNCOMMITTED_LATEST) .setMaxUncommittedOffsets(1000000) .setRetry(kafkaSpoutRetryService) .setRecordTranslator (new TupleBuilder(), outputFields, topic ) .build(); Above class follows builder pattern. bootStrapServers is the Kafka broker end point from where the consumer records are to be polled. topic is the kafka topic name. It can be a collection of kafka topic ( multiple topic or a Pattern ( regular expression ) as well. consumerGroupId would set the kafka consumer group id ( group.id). setFirstPollOffsetStrategy allows you to set from where the consumer records should be fetched. This takes an enum as input and below is the description for the same. EARLIEST - spout will fetch the first offset of the partition, irrespective of commit LATEST - spout will fetch records greater than the last offset in partition, irrespective of commit. UNCOMMITTED_EARLIEST - spout will fetch the first offset of the parition, if there is no commit UNCOMMITTED_LATEST - spout will fetch records from the last offset, if there is no commit. kafkaSpoutRetryService impl is provided below. This is making use of ExponentialBackOff. This setRetry provides a pluggable interface if in case you would want to have failed tuples retry differently. KafkaSpoutRetryService kafkaSpoutRetryService = new KafkaSpoutRetryExponentialBackoff(KafkaSpoutRetryExponentialBackoff.TimeInterval.microSeconds(500), KafkaSpoutRetryExponentialBackoff.TimeInterval.milliSeconds(2), Integer.MAX_VALUE, KafkaSpoutRetryExponentialBackoff.TimeInterval.seconds(10)); setRecordTranslator provides a mechanism through which we can specify how the kafka consumer records should be converted to tuples. In the above given e.x the TupleBuilder is implementing Func interface. Below is the sample impl of apply method that needs to be overridden. OutputFields is the list of the fields that will be emitted in tuple. Please note there are multiple ways to set translate records to tuple. Please go through storm kafka client documentation for more details. public List<Object> apply(ConsumerRecord<String, String> consumerRecord) { try { String records[] = consumerRecord.value().split('|') return Arrays.asList(records); } catch (Exception e) { LOGGER.debug("Failed to Parse {}. Throwing Exception {}", consumerRecord.value() , e.getMessage() ); e.printStackTrace(); } return null; } Once the above step is complete, topology can include above created spoutConf as below. TopologyBuilder builder = new TopologyBuilder(); Config conf = new Config(); conf.setNumWorkers(1); builder.setSpout(KAFKA_SPOUT, new KafkaSpout(spoutConf), 1); Reference: https://github.com/apache/storm/blob/1.x-branch/docs/storm-kafka-client.md ... View more

Problem There are time we would want to remove a ZK node in a secure cluster which is ACL protected. Something as below ACLs [zk: xyz.com:2181(CONNECTED) 0] getAcl /infra-solr 'sasl,'infra-solr : cdrwa 'world,'anyone : r [zk: xyz.com:2181(CONNECTED) 0] rmr /test Authentication is not valid : /test Here only read privilege is available to rest. Soln Goto zookeeper home. for e.x cd /usr/hdp/current/zookeeper-server Run below command java -cp "./zookeeper.jar:lib/slf4j-api-1.6.1.jar" org.apache.zookeeper.server.auth.DigestAuthenticationProvider super:password SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. super:password->super:DyNYQEQvajljsxlhf5uS4PJ9R28= Copy the super:DyNYQEQvajljsxlhf5uS4PJ9R28= text and login to Ambari and goto zookeeper config. Add below to zookeeper-env template config export SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Dzookeeper.DigestAuthenticationProvider.superDigest=super:DyNYQEQvajljsxlhf5uS4PJ9R28=" Save and Restart Zookeeper Launch zookeeper cli ( /usr/hdp/current/zookeeper-client/bin/zkCli.sh -server xyz.com ) addauth as below addauth digest super:password Now try rmr /test -- This should work. Note Please be careful while running these on production systems. ... View more

Ever come across below error while starting HDFS datanode in unsecure cluster ? 016-12-22 09:00:41,045 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2016-12-22 09:00:41,152 INFO datanode.DataNode (DataNode.java:shutdown(1915)) - Shutdown complete. 2016-12-22 09:00:41,153 ERROR datanode.DataNode (DataNode.java:secureMain(2630)) - Exception in secureMain java.net.SocketException: Permission denied at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:463) at sun.nio.ch.Net.bind(Net.java:455) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125) at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:475) at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1021) at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:455) at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:440) at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:844) at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:194) at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:340) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:745) 2016-12-22 09:00:41,157 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 2016-12-22 09:00:41,159 INFO datanode.DataNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG: This error means DataNode is unable to bind to provided socket. Reason ? This is most like because the http port or other port of DataNode are listening on port < 1024. For e.x http port property is set to as below. dfs.datanode.http.address=1022 Solution: Change the port something as 50075 which is greater than 1024. DataNode should be able to start with this solution. ... View more

Ambari 2.4.0.0 officially supports LogSearch Component [ Tech Preview] . To learn more about Log Search component please refer to link Ambari LogSearch . While LogSearch component does support pushing the logs to Kafka topic [ based on which real time log analytics can be performed ] this is official not supported in Ambari 2.4.0.0. This might get addressed in Ambari 2.5.0.0 probably. This article provides the details on how to configure LogSearch [ LogFeeder component ] to push to Kafka topic if there is a need to capture and perform real time analytics based on logs in your cluster. 1.After installing LogSearch component from Ambari 2.4.0.0, go to the LogSearch component config screen and add below property under "Advanced logfeeder-properties" to property "logfeeder.config.files" {default_config_files},kafka-output.json 2. Create the kafka-output.json file with below content under directory /etc/ambari-logsearch-logfeeder/conf/ on the nodes which has logfeeder [ ideally all the nodes in your cluster ] { "output": [ { "is_enabled": "true", "destination": "kafka", "broker_list": "ctr-e25-1471039652053-0001-01-000006.test.domain:6667", "topic": "log-streaming", "conditions": { "fields": { "rowtype": [ "service" ] } } } ] } 3. Configure Kafka PLAINTEXT listener as below if the cluster is Kerberozied because workaround to push to PLAINTEXTSASL is not available.Make sure broker endpoint configured in Step#2 is PLAINTEXT PLAINTEXT://localhost:6667,PLAINTEXTSASL://localhost:6668 4. Create Kafka topic and provide ACLs to ANONYMOUS user. Below command help in the same. ./bin/kafka-topics.sh --zookeeper zookeeper-node:2181 --create --topic log-streaming --partitions 1 --replication-factor 1 ./bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=zookeeper-node:2181 --add --allow-principal User:ANONYMOUS --operation Read --operation Write --operation Describe --topic log-streaming 5. Restart LogSearch service from Ambari and thats it ! logs should be pushing by now. Below is the command to check the same. /bin/kafka-console-consumer.sh --zookeeper zookeeper-node:2181 --topic log-streaming --from-beginning --security-protocol PLAINTEXT ... View more

Kafka from 0.9 onwards started support SASL_PLAINTEXT ( authentication and non-encrypted) for communication b/w brokers and consumer/produce r with broker. To know more about SASL, please refer to this link. Maven Dependency Add below maven dependency to your pom.xml <dependency><br> <groupId>org.apache.kafka</groupId><br> <artifactId>kafka-clients</artifactId><br> <version>0.10.0.0</version><br></dependency> 2. Kerberos Setup Configure JAAS configuration file with contents as below KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true principal="user@EXAMPLE.COM" useKeyTab=true serviceName="kafka" keyTab="/etc/security/keytabs/user.headless.keytab"; }; Above configuration is set to use key tab and ticket cache. The Kakfa Client Producer use above info to get TGT and authenticates with Kafka broker. Note: a] Make sure the /etc/krb5.conf has realms mapping for "EXAMPLE.COM" and also the default_realm is set to "EXAMPLE.COM" under [libdefaults] section. Please refer this link for more information. b] Run below command and make sure it is successful kinit -kt /etc/security/keytabs/user.headless.keytab user@EXAMPLE.COM 3. Initialization Kafka Producer The Kafka Producer Client needs certain information to initialize itself. This can be provided either as a property file input or as a HashMap as below. Properties properties = new Properties(); properties.put("bootstrap.servers","comma-seperated-list-of-brokers"); properties.put("key.serializer","org.apache.kafka.common.serialization.StringSerializer");// key serializer properties.put("value.serializer","org.apache.kafka.common.serialization.StringSerializer"); //value serializer properties.put("acks","1"); //message durability -- 1 mean ack after writing to leader is success. value of "all" means ack after replication. properties.put("security.protocol","SASL_PLAINTEXT"); // Security protocol to use for communication. properties.put("batch.size","16384");// maximum size of message KafkaProducer<String,String> producer = new KafkaProducer<String, String>(properties);<br> 3. Push Message producer.send(new ProducerRecord<String, String>(", "key", "value"), new Callback() { public void onCompletion(RecordMetadata metadata, Exception e) { if (e != null) { LOG.error("Send failed for record: {}", metadata); } else { LOG.info("Message delivered to topic {} and partition {}. Message offset is {}",metadata.topic(),metadata.partition(),metadata.offset()); } } }); } producer.close(); Above code pushes message to kafka broker and on completion( acked ) the method "onCompletion" is invoked. 4. Run While running this code add below VM params -Djava.security.auth.login.config=<PATH_TO_JAAS_FILE_CREATED_IN_STEP2> -Djava.security.krb5.conf=/etc/krb5.conf ... View more