Reply
Highlighted
Explorer
Posts: 23
Registered: ‎01-30-2014
Accepted Solution

extractAvroPaths nested structure

I'm having trouble extracting a nested structure from my avro data.

 

_attachment_body=[

{

    "username" : "alex"

    "date" : "21-08-2014"

    "attachments" : [

        "documents" : [

              {

                  "title": "test"

                  "tags" : [ "a", "b", "c" ]

              },

              {

                  "optional1" : "test2"

                  "title" : "test2"

              } ],

        "context" : "school"

    ]

}

 

Extracting the paths with Avro:

 

...

{ extractAvroPaths {

     flatten : true

     paths : {

         /my_user : /username       # this works fine

         # all three of these result in the same error message, with different patemeters

         /my_attachments : /attachments[]

         /my_documents : /attachments[]/documents[]

         /my_contexts : /attachments[]/documents[]/context

     }

  }

}

 

Results in the following error message:

com.typesafe.config.ConfigExceptionWrongType: morph-solr.conf: 30: Cannot concatenate object or list with a non-object-or-list, ConfigString("/my_attachments") and SimpleConfigList([]) are not compatible.

 

Eventualy I would like to map the fields to a solr index. So if its possible to extract the nested structures, the followup question would be how to map those to a solr schema, but lets take it one step at a time :)

Explorer
Posts: 23
Registered: ‎01-30-2014

Re: extractAvroPaths nested structure

Srry too quick on the post trigger: this was just a matter of quoting the mapping:

 

 /my_attachments : "/attachments[]" 

 

The question remains how to map the structure to a solr index, but will post that in the appropriate section.

Cloudera Employee
Posts: 146
Registered: ‎08-21-2013

Re: extractAvroPaths nested structure

Try to use syntax with quotes, like so :

/my_contexts : "/attachments[]/documents[]/context"

Wolfgang.

Announcements
The Kite SDK is a collection of docs, sample code, APIs, and tools to make Hadoop application development faster. Learn more at http://kitesdk.org.