Building a Custom Search Engine

Building a Better Search for the Forum: The Tale of One Developer, One Upgrade, and One Very Stubborn Query

Upgrading my forum package to Umbraco 17 felt a bit like moving into a freshly renovated house: everything was sleek, modern, and full of potential… and absolutely none of my old furniture fit anymore. My search logic — once a loyal, predictable companion — immediately started behaving like it had been out all night and forgotten how to do its job.

Queries that used to return sensible results were now wandering off into the void, returning nothing, or occasionally coming back with something so bizarre I wondered if I’d accidentally summoned a different index entirely. At first, I blamed Examine. Then Lucene. Then the alignment of the planets.

Eventually, I realised the truth:

My old fluent query wasn’t broken — it just wasn’t built for the new world I’d moved into.

And so began my little adventure.

The Old Query: A Once‑Faithful Friend

Here’s the query that had served me well for years:

            if (_examineManager.TryGetIndex("ForumIndex", out var index))
            {
                var searcher = index.Searcher;
                var search = searcher.CreateQuery(IndexTypes.Content)
                    .Field("__NodeTypeAlias","forumPost").And().Field("status", 1)
                    .And()
                    .GroupedOr(textFields.ToArray(), query.Boost(2.0f))
                    .Or()
                    .GroupedOr(textFields.ToArray(), query.MultipleCharacterWildcard());

                results = search.Execute();
            }

It wasn’t elegant, but it worked. It was like that old IKEA bookshelf you’ve moved through five apartments — a bit wobbly, but still standing.

Then Umbraco 17 arrived, and suddenly the bookshelf collapsed under the weight of a single paperback.

My Descent Into Raw Lucene (or: How I Learned Spaces Mean AND)

In a moment of equal parts curiosity and desperation, I dove into raw Lucene. It felt like opening the hood of a car you’ve been driving for years and realising you’ve never actually looked at the engine.

And that’s when I discovered the thing that changed everything:

Spaces mean AND

A query like: (term1) (term2) (term3) means: term1 AND term2 AND term3

Which meant my “OR block” — the one I lovingly crafted — was actually behaving like an AND block. No wonder the results were acting like a grumpy librarian refusing to hand over books unless you said the magic words.

Raw Lucene didn’t fix my problem, but it did shine a very bright light on it.

The Realisation: I Didn’t Need a Clever Query — I Needed a Proper Search Service

Once I stopped trying to outsmart the fluent API, the solution became obvious: centralise everything.

Instead of sprinkling Lucene logic across controllers like confetti, I built a dedicated ForumSearchService — a place where all the logic could live, breathe, and behave like a grown‑up.

This new service gave me:

clean DI registration
a predictable, testable API
boosting for subject vs message
synonym expansion
wildcard and split‑term matching
author / forum / date filtering
scoring explanations
a debug mode that tells you exactly what’s happening under the hood

Suddenly, search wasn’t a mysterious creature lurking in the shadows — it was a well‑trained, well‑documented, very cooperative part of the system.

The Search Pipeline

Readable. Predictable. Debuggable. Everything happens in one place, in a clear order, with no magic.

Before vs After: A Tale of Two Approaches

Before: Fluent API Chaos

Everything was scattered. Boolean logic was fragile. Analyzer mismatches caused silent failures. Debugging felt like trying to catch smoke with your hands.

.Field("__NodeTypeAlias", "forumPost")
.And(q => q.GroupedOr(["subject", "message"], exactTerms))
.Or(q => q.GroupedOr(["subject", "message"], wildcardTerms))
.Or(q => q.GroupedOr(["subject", "message"], splitWildcardTerms));

It worked… until it didn’t.

After: A Proper Search Service

Everything lives in one place.
Everything is predictable.
Everything is testable.
Everything is beginner‑friendly.

using Examine;
using Examine.Lucene.Providers;
using Examine.Lucene.Search;
using Examine.Search;
using StackExchange.Profiling.Internal;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.RegularExpressions;

public class ForumSearchService
{
    private readonly BaseLuceneSearcher _searcher;
    private readonly ISynonymProvider _synonyms;
    private readonly bool _debug;
    private readonly Dictionary<string, int> _termCounts = new();
    private readonly List<string> _zeroResultQueries = new();

    public ForumSearchService(ISearcher searcher, ISynonymProvider synonyms, bool debugMode = false)
    {
        _searcher = (BaseLuceneSearcher)searcher;
        _synonyms = synonyms;
        _debug = debugMode;
    }

    public IEnumerable<ForumSearchResult> SearchPosts(
        string[] fields,
        string queryText,
        string author = null,
        string forum = "All",
        PhraseType phrasetype = PhraseType.Any, //Any, All,Exact
        WhereTopic? wheretopic = null, //Open, Closed, Solved
        DateTime? from = null, //after
        DateTime? to = null, //before
        int page = 1,
        int pageSize = 20)
    {
        if (string.IsNullOrWhiteSpace(queryText))
            return Enumerable.Empty<ForumSearchResult>();

        TrackTermUsage(queryText);

        var terms = Split(queryText).ToArray();

        // Expand synonyms
        var allTerms = terms
            .Distinct(StringComparer.OrdinalIgnoreCase)
            .ToArray();

        switch (phrasetype)
        {
            case PhraseType.Phrase:
                allTerms = [queryText.Trim()];
                break;
            case PhraseType.All:
                allTerms = terms;
                break;
            case PhraseType.Any:
                allTerms = terms;
                break;
            default:
                allTerms = terms.Concat(terms.SelectMany(t => _synonyms.GetSynonyms(t))).Distinct(StringComparer.OrdinalIgnoreCase).ToArray();
                break;
        }
        var lucene = BuildLuceneQuery(fields, allTerms, author, from, to,phrasetype,wheretopic,forum);

        if (_debug)
            Console.WriteLine($"[ForumSearch DEBUG] Lucene Query:\n{lucene}\n");
        var query = _searcher.CreateQuery("content", BooleanOperation.And, _searcher.LuceneAnalyzer, new LuceneSearchOptions() { AllowLeadingWildcard = true })
            .NativeQuery(lucene);

        var results = query.Execute();

        if (!results.Any())
            TrackZeroResultQuery(queryText);

        return results
            .Skip((page - 1) * pageSize)
            .Take(pageSize)
            .Select(hit => new ForumSearchResult
            {
                NodeId = int.TryParse(hit.Id, out var id) ? id : 0,
                Url = hit.Values.TryGetValue("url", out var url) ? url : "",
                Subject = hit.Values.TryGetValue("subject", out var subject) ? subject : "",
                MessageSnippet = HighlightTerms(hit.Values.TryGetValue("message", out var msg) ? MessageParser.ExtractMerged(MessageParser.StripMarkup(msg)) : "", allTerms),
                Score = hit.Score,
                ScoreExplanation = ExplainScore(hit, allTerms)
            });
    }

    public IEnumerable<ForumSuggestion> GetSuggestions(string term, int max = 10)
    {
        if (string.IsNullOrWhiteSpace(term))
            return Enumerable.Empty<ForumSuggestion>();

        var escaped = Escape(term);

        var lucene = $@"
            +(__NodeTypeAlias:forumPost)
            (subject:*{escaped}*)
            ";

        if (_debug)
            Console.WriteLine($"[ForumSearch DEBUG] Suggest Query:\n{lucene}\n");

        var query = _searcher.CreateQuery("content", BooleanOperation.And, _searcher.LuceneAnalyzer, new LuceneSearchOptions() { AllowLeadingWildcard = true })
            .NativeQuery(lucene);

        var results = query.Execute();

        return results
            .OrderByDescending(r => r.Score)
            .Select(r => new ForumSuggestion
            {
                Subject = r.Values.TryGetValue("subject", out var subject) ? subject : "",
                Url = r.Values.TryGetValue("url", out var url) ? url : ""
            })
            .Where(s => !string.IsNullOrWhiteSpace(s.Subject))
            .DistinctBy(s => s.Subject)
            .Take(max);
    }
	
    public static string HighlightTerms(string text, IEnumerable<string> terms, string tag = "mark", int maxLength = 300)
    {
        if (string.IsNullOrWhiteSpace(text))
            return string.Empty;

        var raw = text;
        foreach (var term in terms.Where(t => !string.IsNullOrWhiteSpace(t)))
        {
            var pattern = Regex.Escape(term);
            raw = Regex.Replace(
                raw,
                $@"(\b{pattern}\b|{pattern})",
                $"<{tag}>$1</{tag}>",
                RegexOptions.IgnoreCase
            );
        }

        if (raw.Length <= maxLength)
            return raw;

        return raw.Substring(0, maxLength) + "...";
    }
	
    public IReadOnlyDictionary<string, int> GetTopTerms(int count = 20) =>
        _termCounts
            .OrderByDescending(kvp => kvp.Value)
            .Take(count)
            .ToDictionary(k => k.Key, v => v.Value);

    public IReadOnlyList<string> GetZeroResultQueries() =>
        _zeroResultQueries.AsReadOnly();	
		
    private string BuildLuceneQuery(string[] fields, IEnumerable<string> terms, string author, DateTime? from, DateTime? to, 
        PhraseType phrasetype, WhereTopic? whereTopic, string forum)
    {
        const int primaryBoost = 10;
        const int secondaryBoost = 5;

        var clauses = new List<string>();

        foreach (var term in terms)
        {
            var esc = Escape(term);

            clauses.Add($"{fields[0]}:{esc}^{primaryBoost}");// Exact
            clauses.Add($"{fields[0]}:*{esc}*^{primaryBoost - 1}");// Wildcard
            clauses.Add($"{fields[0]}:{esc}~2^{primaryBoost - 2}");// Fuzzy
            for (var i = 0; i < fields.Length; i++)
            {
                var field = fields[i];
                
                clauses.Add($"{field}:{esc}^{secondaryBoost}");// Exact
                clauses.Add($"{field}:*{esc}*^{secondaryBoost - 1}");// Wildcard
                clauses.Add($"{field}:{esc}~2^{secondaryBoost - 2}");// Fuzzy
            }
        }

        // Phrase match
        var joined = string.Join(" ", terms.Select(Escape));
        if (terms.Count() > 1)
        {
            clauses.Add($"{fields[0]}:\"{joined}\"~2^{primaryBoost + 2}");// Exact
            for (var i = 0; i < fields.Length; i++)
            {
                clauses.Add($"{fields[i]}:\"{joined}\"~2^{secondaryBoost + 1}");
            }
        }

        string AndFields(string[] fields, IEnumerable<string> values)
        {
            //if we are doing an all then we need to OR subject and message but then AND the terms
            //(subject:wiz AND subject:forum) OR (message:wiz AND message:forum)

            var test = "(";
            for(int i = 0; i < fields.Length; i++)
            {
                var boost = i == 0 ? $"^{primaryBoost}"  : $"^{secondaryBoost}";
                test = test + string.Join(" AND ",
                from val in values
                select $"{fields[i]}:{val}{boost}");

                if (i < fields.Length-1)
                {
                    test = test + ") OR (";
                }
                else
                {
                    test = test + ")";
                }
            }
            return test;
        }

        var orBlock = string.Join(" OR ", clauses.Select(c => $"({c})"));

        // Author filter
        var authorClause = !string.IsNullOrWhiteSpace(author)
            ? $"+(author:{Escape(author)})"
            : "";
        // Forum filter
        var forumClause = "";
        if (forum.HasValue() && forum != "All")
        {
            if (int.TryParse(forum, out var forumIdInt))
            {
                forumClause = $"+(forumid:{forumIdInt})";
            }
        }
        // Date range filter
        var dateClause = "";
        if (from.HasValue || to.HasValue)
        {
            if (from.HasValue)//before
            {
                var fromStr = ((DateTimeOffset)from).Ticks;// ?? long.MinValue;
                var toStr = long.MaxValue;

                dateClause = $"+(lastTicks:[{fromStr} TO {toStr}])";
            }
            if(to.HasValue)
            {
                var fromStr = long.MinValue;
                var toStr = ((DateTimeOffset)to).Ticks; //?? long.MaxValue;
                dateClause = $"+(lastTicks:[{fromStr} TO {toStr}])";
            }

        }
        var topicClause = "";
        if (whereTopic.HasValue)
        {
            if (whereTopic == WhereTopic.Open)
            {
                topicClause += "+(status:1)";
            }
            else if (whereTopic == WhereTopic.Closed)
            {
                topicClause += "+(status:0)";
            }
            else if (whereTopic == WhereTopic.Solved)
            {
                topicClause += "+(answered:true)";
            }
        }

        return $@"
			+(__NodeTypeAlias:forumPost)
			{topicClause}
			{forumClause}
			{authorClause}
			{dateClause}
			+({(phrasetype == PhraseType.All ? AndFields(fields, terms.ToArray()) : orBlock)})
			";
    }

    private void TrackTermUsage(string query)
    {
        foreach (var term in Split(query))
        {
            if (_termCounts.ContainsKey(term))
                _termCounts[term]++;
            else
                _termCounts[term] = 1;
        }
    }

    private void TrackZeroResultQuery(string query)
    {
        _zeroResultQueries.Add(query);
    }

    private static string[] Split(string input) =>
        input.Split(' ', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries);

    private static string Escape(string term) =>
        Regex.Replace(term, @"([+\-!(){}\[\]^""~*?:\\])", @"\$1");

    private static string ExplainScore(ISearchResult hit, IEnumerable<string> terms)
    {
        var matched = terms
            .Where(t => hit.Values.Values.Any(v => v.Contains(t, StringComparison.OrdinalIgnoreCase)))
            .ToArray();

        return matched.Any()
            ? $"Matched terms: {string.Join(", ", matched)}"
            : "No direct term matches found";
    }
}

The difference is like switching from a tangled box of cables to a neatly labelled drawer system.

Registering the Service in Umbraco 17

builder.Services.AddSingleton<ISynonymProvider, InMemorySynonymProvider>();

builder.Services.AddSingleton<ForumSearchService>(sp =>
{
    var examine = sp.GetRequiredService<IExamineManager>();

    if (!examine.TryGetIndex("ForumIndex", out var index))
        throw new InvalidOperationException("ForumIndex not found in Examine");

    var searcher = (BaseLuceneSearcher)index.Searcher;

    return new ForumSearchService(
        searcher,
        sp.GetRequiredService<ISynonymProvider>(),
        debugMode: true
    );
});

Clean. Predictable. No surprises.

Using It in Controllers

public class ForumController : SurfaceController
{
    private readonly IForumSearchService _search;

    public ForumController(IForumSearchService search)
    {
        _search = search;
    }

    public IActionResult Search(string q)
    {
        var results = SearchService.SearchPosts(
            fields : ["subject","message"],
            queryText: q,
            phrasetype : PhraseType.Phrase,
            //wheretopic: null
            //author: Request.Query["author"],
            //forum: Request.Query["forumid"],
            // from: DateTime.TryParse(Request.Query["from"], out var f) ? f : null,
            // to: DateTime.TryParse(Request.Query["to"], out var t) ? t : null,
            page: 1,
            pageSize: 20
        );
        return View(results);
    }
}

This is the part where you lean back, sip your coffee, and enjoy how tidy everything looks.

And Yes — It Works Better Than Before

Once everything moved into a proper service, everything clicked:

wiz matches
wiz* matches
*wiz* matches
"wiz forum" matches
fuzzy matches work
synonyms work
boosting works
highlighting works
pagination works
analytics works

It’s like giving your search engine a proper breakfast and watching it suddenly become energetic and helpful.

Final Thoughts

If you’re upgrading to Umbraco 17 and your Examine queries start acting like they’ve forgotten who they are, don’t panic. Nothing’s broken — the rules have just changed.

Stop fighting the fluent API.
Stop trying to duct‑tape your old queries back together.
Build a proper search service.

Raw Lucene helped me understand the problem.A dedicated ForumSearchService solved it.

And honestly? It feels good to have search logic that behaves like it’s on your side again.

Example search results for "Surface Control"