Member since
07-30-2019
333
Posts
356
Kudos Received
76
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
9632 | 02-17-2017 10:58 PM | |
2193 | 02-16-2017 07:55 PM | |
7779 | 12-21-2016 06:24 PM | |
1695 | 12-20-2016 01:29 PM | |
1202 | 12-16-2016 01:21 PM |
10-30-2015
01:47 PM
Ambari will overwrite HDP.repo file by design, as soon as it detects the file deviated from a reference version in configuration database. If you're trying an upgrade, register a new version in Ambari instead of trying to replace a repo file. Ambari will create a HDP-<version>.repo file behind the scenes.
... View more
10-30-2015
01:35 PM
From the HDF/NiFi standpoint, the only difference would be in a configuration switch for PutSolrContentStream: Standalone connects to a Solr node directly (e.g. port 9893) SolrCloud goes through a Zookeeper quorum (e.g. port 2181) and can talk to multiple nodes
... View more
10-30-2015
01:08 PM
1 Kudo
Wade, this is a documentation bug. The now() statement returns a Date object, not a long epoch, one more step required. E.g. consider this expression which gets 'yesterday': ${now():toNumber():minus(86400000):format('yyyy-MM-dd')}
You can follow https://issues.apache.org/jira/browse/NIFI-1078
... View more
10-29-2015
10:28 PM
1 Kudo
Chris, Companies with strict security controls will require HTTP SPNEGO, however, they also must provide the infrastructure for the client. The biggest burden is setting up clients and browsers for Kerberized access. The only reason Ambari doesn't do it automatically is dev resources - this is already tracked internally. Ideally, Ambari will have an enhanced security wizard and prompt a user if she wants to also secure Hadoop web UIs.
... View more
10-29-2015
09:44 PM
First, you have to ask, what's a flowfile? It may not even be a file to add to, but just some binary chunk of data, an event, or an encrypted 100GB db dump, anything, really. Next, where would you want to add the line to? Body/payload? Attribute? Prepend or append? Various processors will be used based on those answers. Sometimes it's easier to invoke ExecuteCommand and run your favorite linux command-line hack, too 🙂 Finally, if there's complicated logic on where and how to add the line - consider a custom processor: https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
... View more
10-27-2015
09:46 PM
Cassandra, We have an official documentation on switching JDKs in a cluster, please refer to http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.0/bk_ambari_reference_guide/content/ch_changing_the_jdk_version_on_an_existing_cluster.html
... View more
10-27-2015
05:51 PM
In 2.3 it has been deprecated and not recommended for production deployments.
... View more
10-27-2015
03:38 PM
I noticed this change doesn't persist after restart, any pointers here?
... View more
10-27-2015
03:19 PM
For the record, this made it into the 0.4.0 release now. Thanks everyone!
... View more
10-27-2015
01:02 PM
3 Kudos
Let's clarify some confusion - we're not talking about Kafka data logs, but rather logging of the Kafka broker process itself. So, your logs are getting big? Several solutions, based on your appetite for coding and linux admin automation. Reduce Logging Output to WARN Kafka broker is quite chatty about client connections, and that fills up logs quickly. Update the logger level for 'server.log' to write only WARN and above. E.g. in Ambari, go to Kafka -> Config -> Advanced kafka-log4j section. Scroll down and find the log4j.logger.kafka entry, modify the level to be WARN: Change hourly logs to daily and rotate By default, those logs in question will create hourly files. If you pair it with some external rotation/deletion policy, might want to switch to daily logs instead of hourly. In the same section above, find and update the log4j.appender.kafkaAppender.DatePattern . See the reference docs for syntax: https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html Use LogRotate Research http://linuxcommand.org/man_pages/logrotate8.html . Powerful, but given that you already have log4j in Kafka, might be redundant. An option if you want to go more on the admin side than app dev/ops. Use an enhanced Log4j rolling appender with MaxBackupIndex The version of log4j shipped by default doesn't support MaxBackupIndex attribute on the DailyRollingFileAppender. You can, however, find it in many libraries on the internet or quickly compile it yourself from e.g. this: http://wiki.apache.org/logging-log4j/DailyRollingFileAppender . Once you drop the extra jar in Kafka's lib directory, you could add the log4j.logger.kafka.MaxBackupIndex attribute in the config to specify how many of those log files to keep around. WARNING: using the MaxBackupIndex also means logs will be lost if not picked up in time.
... View more