Vector & Graph Store Design

[jdrew1303]

Hi,

I've been working on adding the sqlite-vec vector store and while going through it theres a lot of bleed through in the different apis. There are times where I want to use a custom store or a store that would not be accepted into core (such as kuzudb, even with it being archived) and at the moment its not really possible to add a store without committing to core. For most of my stuff I don't mind committing it (other than it needs to be a bit more robust than I might need in prod 😬) but for some stuff it just doesnt make sense. Theres also a maintenance burden on semantica for accepting these contributions. People will expect these to be up to date and work if theyre in core.

If the store system moved to a plugin registry model, then someone could register their own store and just use it. You can still have a core set of support DBs but if theres more niche stuff then a user can use a third party lib or roll their own. Maybe a dev api something like this:

# 1. Explicitly register the new store
GraphStore.register_backend("my_db", MyNewGraphDB)

# 2. Now it can be used by name
store = GraphStore(backend="my_db", uri="localhost:1234")

So if you wanted to implement the above api it would look something like below:

class GraphStore:
    # Remove hardcoded backend initializers
    _registered_backends = {}

    @classmethod
    def register_backend(cls, name: str, backend_class: type):
        """Explicitly register a new graph backend."""
        cls._registered_backends[name] = backend_class

    def __init__(self, backend: str, **config):
        if backend not in self._registered_backends:
            raise ValueError(f"Backend '{backend}' not registered. Call GraphStore.register_backend first.")
        
        # Instantiate the explicitly registered class
        backend_class = self._registered_backends[backend]
        self._store_backend = backend_class(**config)
        self._manager = GraphManager(self._store_backend)

Currently, GraphStore has a hardcoded list of backends like neo4j and falkordb. So that would be replaced with the registry. Its the same for the VectorStore.

It wouldn't be a particularly complex change, but it would be a breaking change. You could do fancier things like auto plugin discovery but its overkill and comes with its own set of problems.

Anyway its just a bit of an idea I had while going through it and I thought it might be food for thought? In the mean time I'm still working away on the SQLite-vec store. Should have it in over the next few days. 👍

[jdrew1303]

To stop it being a breaking change you could actually just auto register the currently supported backends:

class GraphStore:
    _registered_backends = {}
    
    @classmethod
    def _register_core_backends(cls):
        """Auto-register core backends for backward compatibility."""
        from .backends import Neo4jBackend, FalkorDBBackend
        cls.register_backend("neo4j", Neo4jBackend)
        cls.register_backend("falkordb", FalkorDBBackend)
    
    def __init__(self, backend: str, **config):
        if not self._registered_backends:
            self._register_core_backends()
        
        if backend not in self._registered_backends:
            raise ValueError(
                f"Backend '{backend}' not registered. "
                f"Available: {list(self._registered_backends.keys())}. "
                f"Register custom backends with GraphStore.register_backend()"
            )

You could then remove these for the next major release.

[KaifAhmad1]

Hey @jdrew1303 👋

Sorry for the late reply on this! This is a great idea! I took a look at the codebase and you're spot on - we definitely have a hardcoded backend problem.

Right now:

• GraphStore has if/elif chains for neo4j, falkordb, and neptune
• VectorStore has a SUPPORTED_BACKENDS set but same issue
• Adding new stores means touching core code every time

Your registry approach fixes this perfectly:

GraphStore.register_backend("sqlite_vec", SQLiteVecBackend)
store = GraphStore(backend="sqlite_vec", path="./data")

What I like:

• ✅ No breaking changes with your auto-registration idea
• ✅ Users can add custom backends without forking
• ✅ We don't have to maintain every possible integration
• ✅ Clean, explicit API

Implementation would be straightforward:

class GraphStore:
    _registered_backends = {}
    
    @classmethod
    def register_backend(cls, name: str, backend_class: type):
        cls._registered_backends[name] = backend_class
    
    @classmethod
    def _register_core_backends(cls):
        # Auto-register neo4j, falkordb, neptune for backward compatibility
        if not cls._registered_backends:
            cls.register_backend("neo4j", Neo4jStore)
            cls.register_backend("falkordb", FalkorDBStore)
            # etc...

Same pattern would work for VectorStore.

Next steps:
Your sqlite-vec PR would be perfect to validate this approach! Once we see it working, we can refactor both stores to use the registry pattern.

Excited to see the PR! 🚀