Searching full text is wonderful. Can we add a search by 'regex' feature?

[j-j-v]

Opt in.

Would be very handy even were we only able to search titles... but can now search full text. Be sure to allow ranging over new lines ⸻ birds(.*\n)*bees for example ⸻ were you to add this feature.

Thanks!

Joe

[Jaifroid]

@j-j-v It wouldn't be possible to search full text by regex, because the full-text search system is a Xapian database constructed from key words and terms found in the full text. In other words, the full text of each article is not actually extracted and available during search (to run a regex on). Full-text search is effectively just a database query.

However, running a regular expression on title search is not only feasible, it's already available in the PWA. See here go to About (info icon top-right) and go to the "Title search syntax" section. There is info on how to use a regular expression in your search at the bottom of that section (begins "For the geeks..."). Is that the kind of thing you're interested in? Note this isn't supported here in the Browser Extension.

[parthdagia05]

Hi! I’m new to contributing to Kiwix JS and I’d love to try helping with this feature.
From what I understood, regex-based search for full text isn’t possible because of how the Xapian index works, but regex on titles is already supported in the PWA version. I was thinking maybe I could extend that same regex title search feature to the browser extension and Electron builds, so it works consistently everywhere?

Does that sound like a good place for a new contributor to start? I can look into where the title search logic lives and try adding regex detection there.

[Jaifroid]

@badassscoder The Electron builds already have it, because they are based on the PWA.

The main reason I developed that for the PWA back in the day was because we didn't (then) have a way to support full-text search. It was a kind of substitute to allow people to search through a limited subset of titles starting with a specific string. It's not full regex search over the full title index: that would be impossible as there can be millions and millions of titles in a full Wikipedia. What it is is a way to search through up to around 50,000 titles that START with a specific text string.

The bottom line is, this feature was a kludge which has been superseded by our support of full-text search. I'd have to have a very good use case for introducing it here.

If you're looking for an easier first issue, I'd suggest #1375.

[parthdagia05]

Got it, thanks a lot for explaining that! I understand now why the regex title search approach wouldn’t add much value.
I’ll take a look at issue #1375 instead seems like a good place to start.