Sometimes user wants to index a document when some value is present or absent in the page. The typical example is robots meta tag. The same is also hold for following the outlinks. This plugin can support this feature by having two predefined fields index and follow and based on their values it decides to index or follow the document or not. Something like this might be do the job:
<document>
<extract-to field="robots">
<normalize normalizer="lower">
<first skipEmptyItems="true">
<attribute name="content">
<expr value="meta[name=robots]" />
</attribute>
</first>
</normalize>
</extract-to>
<extract-to field="index">
<not>
<matches pattern="noindex">
<field-value field="robots" />
</matches>
</not>
</extract-to>
<extract-to field="follow">
<not>
<matches pattern="nofollow">
<field-value field="robots" />
</matches>
</not>
</extract-to>
</document>
Sometimes user wants to index a document when some value is present or absent in the page. The typical example is robots meta tag. The same is also hold for following the outlinks. This plugin can support this feature by having two predefined fields index and follow and based on their values it decides to index or follow the document or not. Something like this might be do the job: