$\"index-pipeline\"$ <\/div>\n
Out-of-the-box, Lucidworks Fusion<\/a>\u00ae does a great many tasks remarkably well. Every now and then, however, you come across an issue that may take a little extra effort to index. What I\u2019ll describe below, in this particular case, is a way to circumvent the Fusion parser and spin up your own custom PipelineDocument<\/a> in an Index Pipeline Stage<\/a> in Fusion.
\nSo, first things first. We need to override the Fusion parser framework. We\u2019ll do this by changing two things in our POST to the index pipeline.
\nFirst, we\u2019ll change the Content-Type<\/b>
\nFrom:<\/p>\n
application\/vnd.lucidworks-document<\/pre>\nTo:<\/p>\napplication\/octet-stream<\/pre>\nThis tells Fusion that what we\u2019re sending is a raw stream, instead of a PipelineDocument JSON object. This is critical, because otherwise the parser will try to do something with your input, which we want to avoid in this case.
\nNext, we set the \u2018skipParsing\u2019 query param to \u2018true\u2019. Our POST url will then look like this:<\/p>\npipelines\/A_Nested_Documents\/collections\/_test_\/index?echo=true&skipParsing=true<\/pre>\n<\/div>\nYou\u2019ll notice that I\u2019ve also added \u2018echo=true\u2019. This means a successful result will return 200 rather than 204 (No Content). This is entirely optional. You can also \u201csimulate=true\u201d if you want to test \u2014 but not actually add \u2014 your documents to the collection.<\/p>\nNow to the code\u2026<\/h3>\nFrom the code side, the first thing you\u2019ll want to do is create a new Index Pipeline in Fusion. You\u2019ll give it a name (ID) and then you\u2019ll add a JavaScript stage.<\/p>\n<\/div>\nNext, you\u2019ll want to declare the Java classes you\u2019ll be using:
\n
\nif (doc !== null && doc.getFirstFieldValue(\"_raw_content_\") !== null) { <\/code>
\nvar ex = java.lang.Exception;
\nvar pipelineDoc = com.lucidworks.apollo.common.pipeline.PipelineDocument;
\nvar outdocs = java.util.ArrayList;
\nvar ObjectMapper = org.codehaus.jackson.map.ObjectMapper;
\nvar SerializationConfig = org.codehaus.jackson.map.SerializationConfig;
\nvar String = java.lang.String;
\nvar e = java.lang.Exception;
\nvar base64 = java.util.Base64;
\nvar String = java.lang.String;
\nvar ArrayList = java.util.ArrayList;
\nvar mapper = new ObjectMapper();
\nvar pretty = true;
\nvar result = new String(\u201c\u201d);
\noutdocs = new ArrayList();
\npipelineDoc = new com.lucidworks.apollo.common.pipeline.PipelineDocument();
\nNotice that I check the doc for \u2018null\u2019 and check for the _raw_content_ field to be there BEFORE my declarations. This is because there is no reason to declare anything if the document isn\u2019t there, or your content isn\u2019t there.
\n
\ntry {
\nvar rawlist = doc.getFieldValues(\"_raw_content_\");
\nraw = rawlist[0]; <\/code>
\n\/\/ logger.info(\u201cRaw class: \u201d + raw.getClass().getSimpleName());
\n\/\/ will be a byte[]
\nvar stdin = new String(raw, \u201cUTF8\u201d); \/\/ turn the stream into a string.
\nvar json = JSON.parse(stdin); \/\/ parse with the JavaScript JSON parser.
\n\/\/ logger.info(\u201cJSON: \u201d + stdin);
\nmapper.configure(SerializationConfig.Feature.FAIL_ON_EMPTY_BEANS, false);
\nvar children = json[0].fields; \/\/ get fields for the parent.
\n\/\/ set our PipelineDocument ID
\npipelineDoc.setId(json[0].id);
\nlogger.info(\u201cdocument ID: \u201d + doc.getId());
\nHere we\u2019re taking the UTF8 encoded byte array, and turning it into a java.lang.String, then using the native JSON parser to create a JavaScript JSON object. In this case, there was an issue with the Java Jackson JSON parser and polymorphic documents, so we wanted to avoid the parser, and handle it ourselves. At the end of it all, we set the ID for our new PipelineDocument
\nNow comes the heavy lifting part. We have our JSON object, and we\u2019re going to plug it into our PipelineDocument. In this case, we\u2019re adding child documents denoted by the \u201c_childDocuments_\u201d field. You can store these in a variety of ways, so you\u2019ll want to consider that before you set out to code your project. In this instance, I\u2019m embedding the PipelineDoucments into my parent, but you could create separate documents and bind them together using the Taxonomy API in Fusion as well, but I digress.
\n
\nif (children !== null) { <\/code>
\n\/**
\n* Here is the heavy lifting. Now that we have a JSON object,
\n* We will transform it into a PipelineDocuemnt. This
\n* document will be fasioned after our collection schema.
\n*\/
\nfor (var i = 0; i
\n} catch (ex) {
\nlogger.error(ex.getLocalizedMessage());
\n}
\n\/\/ if all goes well, we\u2019ll return our new doc right here.
\nlogger.info(\u201cRETURN BRANCH 1\u201d);
\noutdocs.add(pipelineDoc);
\nreturn outdocs;
\n} else {
\nlogger.info(\u201cDoc ID was null\u201d);
\n\/\/ if something was wrong with the doc, or it was null, we return here.
\nreturn doc;
\n}
\n\/\/ otherwise, we return here.
\nreturn doc;
\n}<\/p>\nAnd there it is! Your document has now been indexed into Fusion\/Solr
\nYou can find the complete code for this blog here: CustomIndexStageParsing.<\/a>, and sample JSON here: Nested.json<\/a>
\nHappy Searching!<\/p>\n<\/body>","protected":false},"excerpt":{"rendered":"
Out-of-the-box, Lucidworks Fusion\u00ae does a great many tasks remarkably well. Every now and then, however, you come across an issue that may take a little extra effort to index. What I\u2019ll describe below, in this particular case, is a way to circumvent the Fusion parser and spin up your own custom PipelineDocument in an Index […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"pagelayer_contact_templates":[],"_pagelayer_content":"","footnotes":""},"categories":[],"tags":[],"class_list":["post-1430","post","type-post","status-publish","format-standard","hentry"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/capture.club\/portal\/wp-json\/wp\/v2\/posts\/1430","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/capture.club\/portal\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/capture.club\/portal\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/capture.club\/portal\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/capture.club\/portal\/wp-json\/wp\/v2\/comments?post=1430"}],"version-history":[{"count":0,"href":"https:\/\/capture.club\/portal\/wp-json\/wp\/v2\/posts\/1430\/revisions"}],"wp:attachment":[{"href":"https:\/\/capture.club\/portal\/wp-json\/wp\/v2\/media?parent=1430"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/capture.club\/portal\/wp-json\/wp\/v2\/categories?post=1430"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/capture.club\/portal\/wp-json\/wp\/v2\/tags?post=1430"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}