Here is a useful related read: http://www.ngdata.com/the-hbase-side-effect-processor-and-hbase-replication-monitoring/ ... View more

Looks like you are missing a loadSolr command in your morphline, for example as shown here: see http://www.cloudera.com/documentation/enterprise/latest/topics/search_batch_index_use_mapreduce.html?scroll=csug_topic_4_3   (FYI, with MapReduceIndexerTool the SOLR_LOCATOR is substituted from whatever is specified on the CLI with --zk-host option) ... View more

On yarn the params are called mapreduce.map.java.opts and mapreduce.reduce.java.opts. Wolfgang. ... View more

You can plug a morphline into hbase-indexer to do some mini ETL on the fly during indexing from HBase into Solr. See the docs:   http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/search_hbase_batch_indexer.html   and   http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/search_etl_morphlines.html ... View more

The SOLR_LOCATOR is a variable that works via simple text substitution (ala unix shell scripts). You can define as many variables as you like within the same morphline config file. For example along these lines: SOLR_LOCATOR_1 : { collection : collection1, zkHost : ${ZK_HOST} } SOLR_LOCATOR_2 : { collection : collection2, zkHost : ${ZK_HOST} } morphlines : [ { id : morphline1 ... { loadSolr { solrLocator : ${SOLR_LOCATOR_1} } } } { id : morphline2 ... { loadSolr { solrLocator : ${SOLR_LOCATOR_2} } } } ] Wolfgang ... View more

Try to use the sanitizeUnkownSolrFields command per http://kitesdk.org/docs/current/morphlines/morphlines-reference-guide.html#sanitizeUnknownSolrFields Wolfgang. ... View more

You found the recommended solution already 🙂 ... View more

The toAvro command expects a java.util.Map as input on conversion to a nested Avro record, per   https://github.com/kite-sdk/kite/blob/master/kite-morphlines/kite-morphlines-avro/src/main/java/org/kitesdk/morphline/avro/AvroConversions.java#L73-L87   However, your input data contains a (nested) Jackson JSON object, not a java.util.Map. Hence the conversion can't succeed.   Consider writing a custom morphline command that implements whatever conversion rules you wish, per http://kitesdk.org/docs/current/morphlines/morphlines-reference-guide.html#Implementing_your_own_Custom_Command    ... View more

Arrange it such that morphlineA pipe records into morphlineB:   Command morphlineB = new Compiler().compile(morphlineFileB, morphlineIdB, morphlineContextB, null); Command morphlineA = new Compiler().compile(morphlineFileA, morphlineIdA, morphlineContextA, morphlineB); ... View more

That’s the expected behavior per the doc at kitesdk.org/docs/current/kite-morphlines/morphlinesReferenceGuide.html#xquery: "The XPath string value of the attribute or child is filled into the record field”. The “XPath string value” is the concatenation of the *text nodes* and it does not include the element names or attribute names, per www.w3.org/TR/xquery-operators/#func-string Also, Solr/Lucene wouldn’t know that to do with those tag names anyway. A Lucene/Solr field holds primitive type such as a string, it doesn’t work with nested structures. P.S. If you really need this, I believe saxon (and hence the xquery command) has an extension function to emit the serialization of an XML document into a string, but unfortunately that extension function is probably not available in the free "Saxon-HE" version that we ship with kite-morphlines-saxon: www.saxonica.com/documentation9.4-demo/html/extensions/functions/serialize.html Alternatively, you could write your own custom morphline command that implements whatever xquery serialization logic you like, of course. The code would be be a copy n’ paste of the existing xquery command expect for adjusting this part: github.com/kite-sdk/kite/blob/master/kite-morphlines/kite-morphlines-saxon/src/main/java/org/kitesdk/morphline/saxon/XQueryBuilder.java#L196-L198 ... View more

The "if" command and "equals" command and indeed all morphline commands know nothing about hbase colunmns or hbase qualifiers, except for the extractHBaseCells command. Use  extractHBaseCells to extract whatever hbase columns you want into whatever morphline record fields you want, then subsequently use "if", "equals" or similar to act on the morphline record fields (not on hbase columns or qualifiers direcly). ... View more

The xquery command expects a byte[] rather than a string as input, and that input must be in the outputField : “_attachment_body” field rather than the "data" field. Try changing the extractHBaseCells command to use type : "byte[]” and outputField : “_attachment_body"   Also you need to change your xquery command to wrap your XML output into yet another XML element (e.g. “record”).   For example, in order to generate a morphline record with a “myFoo" field that contains “foo", as well as a “myBar" field that contains “bar", your xquery command should be formulated such that it outputs an XML fragment like this:   <record> <myFoo>foo</myFoo> <myBar>bar</myBar> </record>   Wolfgang. ... View more