question Re: hbase/solar indexing feature Lily HBase NRT Indexer Service in Archives of Support Questions (Read Only)

hbase/solar indexing feature Lily HBase NRT Indexer Service

akhettar — Fri, 16 Sep 2022 09:11:38 GMT

I am a complete newbie to Cloudera search. I went through the example described here: hbase indexer all seems to be good, the data field is being indexed successfully in apache solar. However when I add simple step like adding timestap field - see below, the indexer stops working, any idea as to why this happening? ndication to where I can find the indexers logs will be very much appreciated.

My ultimate goad is to store an xml in the 'data' filed and apply Xpath to index only the element I want its value to be indexed. Can anyone me give me a clue how to achieve this? are there any examples which uses either xpath or xquery?

kind regards,

akhettar

morphlines : [
{
id : morphline1
importCommands : ["org.kitesdk.morphline.**", "com.ngdata.**"]

commands : [
{
extractHBaseCells {
mappings : [
{
inputColumn : "data&colon;*"
outputField : "data"
type : string
source : value
}

#{
# inputColumn : "data&colon;item"
# outputField : "_attachment_body"
# type : "byte[]"
# source : value
#}
]
}
}
{ addCurrentTime { field : ts } }

#for avro use with type : "byte[]" in extractHBaseCells mapping above
#{ readAvroContainer {} }
#{
# extractAvroPaths {
# paths : {
# data &colon; /user_name
# }
# }
#}

{ logTrace { format : "output record: {}", args : ["@{}"] } }
]
}
]

Re: hbase/solar indexing feature Lily HBase NRT Indexer Service

whosch — Tue, 04 Nov 2014 08:49:02 GMT

The solr schema.xml config file needs to conform to the documents that you are trying to insert. Try adjusting schema.xml accordingly and tell solr about it via the solrctl CLI.

Also see http://kitesdk.org/docs/current/kite-morphlines/morphlinesReferenceGuide.html#sanitizeUnknownSolrFields

XPath and XQuery docs are here: http://kitesdk.org/docs/current/kite-morphlines/morphlinesReferenceGuide.html#xquery

The log files of the Solr server and MapReduce tasks, etc can be displayed in the Cloudera Manager GUI.

Wolfgang.

Re: hbase/solar indexing feature Lily HBase NRT Indexer Service

akhettar — Tue, 04 Nov 2014 12:59:47 GMT

Many thanks Wolfgang for your input. As for the logs, I am not running Cloudera manager, the VM requires at least 8 GB or RAM and it's freezing my machine. I was hoping to be pointed to log file in the file system so I could tail?

Regards,

Ayache

Re: hbase/solar indexing feature Lily HBase NRT Indexer Service

whosch — Tue, 04 Nov 2014 20:33:02 GMT

Try /var/log/solr

Re: hbase/solar indexing feature Lily HBase NRT Indexer Service

akhettar — Tue, 04 Nov 2014 22:02:58 GMT

Thanks for the hint. /var/log/solr/solr.out is for solar server, but was interesed in hbase-indexer server. Here is the link for the benefit of all: tail -f /var/log/hbase-solr/hbase-solr-hbase.out

Re: hbase/solar indexing feature Lily HBase NRT Indexer Service

akhettar — Wed, 05 Nov 2014 09:39:35 GMT

I got the above morphline conf working...the 'ts' varialbe wasn't declared in the schema.xml. I have experimented with some other commands, however the xquery command isn't working properly.

Here is the morophline conf:

morphlines : [
{
id : morphline1
importCommands : ["org.kitesdk.morphline.**", "com.ngdata.**"]

commands : [
{
extractHBaseCells {
mappings : [
{
inputColumn : "data&colon;*"
outputField : "data"
type : string
source : value
}

#{
# inputColumn : "data&colon;item"
# outputField : "_attachment_body"
# type : "byte[]"
# source : value
#}
]
}
}

{ xquery {
fragments : [
{
fragmentPath : "/"
externalVariables : {
myVariable : "hello world"
}
queryString : """
declare variable $myVariable as xs:string external;
(: Example xquery 🙂
let $name := /employees/employee/name/text()
return
<ts> { $name } </ts>
"""
}
]
}
}

#for avro use with type : "byte[]" in extractHBaseCells mapping above
#{ readAvroContainer {} }
#{
# extractAvroPaths {
# paths : {
# data &colon; /user_name
# }
# }
#}

{ logTrace { format : "output record: {}", args : ["@{}"] } }
]
}
]

morphline-hbase-mapper.xml

<?xml version="1.0"?>
<indexer table="record" mapper="com.ngdata.hbaseindexer.morphline.MorphlineResultToSolrMapper">

<param name="morphlineFile" value="/etc/hbase-solr/conf/morphlines.conf"/>

</indexer>

The input to hbase is: put 'record', 'row12', 'data', '<employees><employee><name>ayache</name><age>29</age></employee></employees>'

All I can see in the log is the following:

14/11/05 07:29:22 WARN morphline.LocalMorphlineResultToSolrMapper: Morphline /etc/hbase-solr/conf/morphlines.conf@null failed to process record: {data=[<employees><employee><name>ayache</name><age>29</age></employee></employees>]}

Any idea as to why it's failing to index the above input?

Many thanks

Ayache

Re: hbase/solar indexing feature Lily HBase NRT Indexer Service

akhettar — Wed, 05 Nov 2014 18:00:04 GMT

I've debugged this further but unfortunately got no where. What version of kite-sdk is the QuickStartVM is running if I may ask?

I've moved away from using xquery and started to expirement with xslt instead. Running the example shown in the morpheline reference guide - see below my morphline sample

morphlines : [
{
id : morphline1
importCommands : ["org.kitesdk.**", "com.ngdata.**"]

commands : [

{
xslt {
fragments : [
{
fragmentPath : "/"
parameters : {
myVariable : "hello world"
}
queryString : """

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">

<xsl:template match="@*|node()">
<xsl:copy>
<xsl:apply-templates select="@*|node()"/>
</xsl:copy>
</xsl:template>

</xsl:stylesheet>
"""
}
]
}
}

{ logTrace { format : "output record: {}", args : ["@{}"] } }
]
}
]

If push the following into hbase: put 'record', 'row41', 'data', '<employees><employee><name>ayache</name><age>29</age></employee></employees>'

I would expect an index to be created with the same xml ingested into hbase. I can see in the logs that the import is done - see below. But when I search on Solar web console I can't see any entry.

Can anyone help me at least to run one of the xquery or xslt examples?

Thanks

Ayache

Re: hbase/solar indexing feature Lily HBase NRT Indexer Service

whosch — Wed, 05 Nov 2014 18:42:22 GMT

The xquery command expects a byte[] rather than a string as input, and that input must be in the outputField : “_attachment_body” field rather than the "data" field. Try changing the extractHBaseCells command to use type : "byte[]” and outputField : “_attachment_body"

Also you need to change your xquery command to wrap your XML output into yet another XML element (e.g. “record”).

For example, in order to generate a morphline record with a “myFoo" field that contains “foo",

as well as a “myBar" field that contains “bar", your xquery command should be formulated such

that it outputs an XML fragment like this:

</record>

Wolfgang.

Re: hbase/solar indexing feature Lily HBase NRT Indexer Service

akhettar — Thu, 06 Nov 2014 11:08:33 GMT

Many thanks Wolfgang for the suggestion, they really did the trick. For future references are these mentioned anywere in the reference guide?

Re: hbase/solar indexing feature Lily HBase NRT Indexer Service

whosch — Thu, 06 Nov 2014 11:39:33 GMT

It’s mentioned in the ref guide for the next upcoming kite version per https://github.com/kite-sdk/kite/blob/master/kite-morphlines/src/site/confluence/morphlinesReferenceGuide.confluence#L2879-L2889

Re: hbase/solar indexing feature Lily HBase NRT Indexer Service

akhettar — Thu, 06 Nov 2014 12:00:07 GMT

Many thanks for your help.

Ayache

Re: hbase/solar indexing feature Lily HBase NRT Indexer Service

akhettar — Sun, 09 Nov 2014 09:51:44 GMT

Just one last question regarding the above implementation. How do I make the xquery step only get triggered for certain column 'family name'. Here is the scenario

put 'record', 'row1', 'data', '<employees><employee><name>ayache</name><age>29</age></employee></employees>'

The above input into hbase will be handled by xquery step.

put 'record', 'row2', 'context', 'business' ===> for this input I don't want to call the xquery strep, rather the following mapping will suffice (extracting cell value

{
inputColumn : "context"
outputField : "context"
type : string
source : value
}

I thought about declaring two morphlines, one handling fields that require xquery step and others just one to oine mapping.

I tried something like this:

morphlines : [
{
id : morphline1
importCommands : ["org.kitesdk.**", "com.ngdata.**"]

commands : [

{
extractHBaseCells {
mappings : [
{
inputColumn : "data"
outputField : "_attachment_body"
type : "byte[]"
source : value
}
]
}
}
{
xquery {
fragments : [
{
fragmentPath : "/"

queryString : """

(: All namespace declarations go here 🙂

declare namespace inps = "http://inps.co.uk/";

(: Extracting all the fieleds that need indexing 🙂

let $name := /employees/employee/name/text()

(: Returning the list of the fields that needs to be indexed. These fields are defined in solar schema.xml file. 🙂

return
<fieldsToIndex>
<name>{$name}</name>

</fieldsToIndex>
"""
}
]
}
}

{ logTrace { format : "output record: {}", args : ["@{}"] } }
]
}
{
id : morphline2
importCommands : ["org.kitesdk.**", "com.ngdata.**"]

commands : [

]{
extractHBaseCells {
mappings : [

{
inputColumn : "context"
outputField : "context"
type : string
source : value
}
]
}
}

{ logTrace { format : "output record: {}", args : ["@{}"] } }
]
}
]

Looks like the second morphline is ignored. First, is this a sensible solution? If so, how do I make the second morphline known.

kind regards,

Ayache

Re: hbase/solar indexing feature Lily HBase NRT Indexer Service

whosch — Sun, 09 Nov 2014 20:35:28 GMT

You can express it all in a single morphline. Consider using if-then-else command or the tryRules command or similar in order to check which case applies and execute whatever corresponding logic is appropriate for that case. You can have multiple extractHBaseCells commands in a single morphline, e.g. one in each branch of the tryRules command.

Wolfgang.

Re: hbase/solar indexing feature Lily HBase NRT Indexer Service

akhettar — Mon, 10 Nov 2014 11:23:13 GMT

Thanks again Wolfgang for your prompet respon, this is very helpful. I've looked at teh if/els and tryRules. Both use the 'contains' command for conditions. The contain however only matches the value of the field, my use case is about asserting a presence of the fied regardless of the value. So if the filed is 'context' I want to apply xquery step if not, just one to one mapping from hbase cell. Is there a command to check for the presence of a field?

Thanks

Re: hbase/solar indexing feature Lily HBase NRT Indexer Service

whosch — Mon, 10 Nov 2014 11:32:56 GMT

Try equals { id : [] } for example as shown here: http://kitesdk.org/docs/current/kite-morphlines/morphlinesReferenceGuide.html#if

In a morphline record there is no difference between a field with zero values and a field that doesn't exist.

Re: hbase/solar indexing feature Lily HBase NRT Indexer Service

akhettar — Mon, 10 Nov 2014 14:20:27 GMT

I tried equals earlier, but I am afraid it's not working form me. Here is snippet for morphline:

if {
conditions : [
{ equals { p : [] } }
]

then : [

# handling headers

{logInfo { format : "processing headers..." } }

]

else : [

# handling payload

{logInfo { format : "processing payload..." } }

]

So with the follwing entry into hbase:

put 'payload', 'row1', 'p', '<employees><employee><name>ayache</name><age>29</age></employee></employees>'

still going into the 'handling headers' close.

My use case mandate that the field 'p' is qualified with qualifier 'in': So not usre how to reference qualifier in the 'equals' command. I've tried this equals { "p:in" : [] } . This is the next step really, It's not matching withouth the qualifier anyway, any idea what I have missed?

Regards,

Ayache

Re: hbase/solar indexing feature Lily HBase NRT Indexer Service

whosch — Mon, 10 Nov 2014 15:10:01 GMT

The "if" command and "equals" command and indeed all morphline commands know nothing about hbase colunmns or hbase qualifiers, except for the extractHBaseCells command. Use extractHBaseCells to extract whatever hbase columns you want into whatever morphline record fields you want, then subsequently use "if", "equals" or similar to act on the morphline record fields (not on hbase columns or qualifiers direcly).

Re: hbase/solar indexing feature Lily HBase NRT Indexer Service

akhettar — Mon, 10 Nov 2014 15:48:41 GMT

Got you, all working perfectly now. Thank you so much for your help, I now understand more about how morphline / hbase-indexer work.