Skip to content

feat: pass IndexConfig to repositories with scalability options - #981

Merged
lyingbug merged 1 commit into
Tencent:mainfrom
ochanism:enhancement/1205-3-2
Apr 17, 2026
Merged

feat: pass IndexConfig to repositories with scalability options#981
lyingbug merged 1 commit into
Tencent:mainfrom
ochanism:enhancement/1205-3-2

Conversation

@ochanism

@ochanism ochanism commented Apr 16, 2026

Copy link
Copy Markdown
Contributor

Overview

Pass VectorStore's IndexConfig to all repository constructors so that custom index/collection names and shard/replica settings take effect. Previously engine_factory ignored store.IndexConfig, causing all DB stores to use the same env-var defaults — breaking the core value of VectorStore separation.

Part of #921

Changes

IndexConfig struct (5 new scalability fields):

  • ShardNumber, ReplicationFactor (Qdrant)
  • ShardsNum, ReplicaNumber (Milvus — shards at CreateCollection, replicas at LoadCollection)
  • DesiredShardCount (Weaviate)

Repository constructors (all accept optional *IndexConfig, nil = env fallback):

  • ES v8: index name + shards/replicas in createIndexIfNotExists
  • ES v7: index name + shards/replicas + new createIndexIfNotExists (was missing)
  • Qdrant: collection prefix + shard/replication in ensureCollection
  • Milvus: collection name + WithShardNum + WithReplica
  • Weaviate: collection prefix + ShardingConfig + ReplicationConfig

Validation & security:

  • ValidateIndexConfig: name regex + numeric bounds (shards ≤ 64, replicas ≤ 10)
  • Auto-detect server version on CreateStore (prevents ES v7/v8 SDK mismatch)
  • Sanitize ES v7 error messages to avoid cluster info disclosure

Other:

  • Nil-safe getter helpers (pointer receivers) + ResolveIndexName/ResolveCollectionName with env fallback
  • OptionalUint32 with negative overflow guard
  • /types API updated with all scalability index_fields
  • 30+ unit tests

Core

Key files:

  • internal/types/vectorstore.go — IndexConfig fields, getters, validation
  • internal/container/engine_factory.go — passes &store.IndexConfig to each repository
  • internal/container/container.go — passes nil to env path (preserves existing behavior)
  • Repository files: elasticsearch/v7/, elasticsearch/v8/, qdrant/, milvus/, weaviate/

Why

  • Without this change, users can set index/collection names in the UI but they are silently ignored
  • Scalability options (shards, replicas) are essential for production vector DB deployments
  • Auto version detect prevents the wrong ES SDK from being used (v8 SDK → v7 server = 406 error)

Test Plan

  • Getter helpers: nil receiver, zero, positive, negative values
  • ResolveIndexName/ResolveCollectionName: nil fallback, env var, precedence
  • OptionalUint32: zero, negative, positive
  • ValidateIndexConfig: valid configs, name injection, numeric bounds (15 cases)
  • Scalability fields JSON round-trip serialization
  • Go build + go test + go vet clean
  • Manual E2E: ES v7 index created with custom name + shards/replicas confirmed

Related

@ochanism ochanism changed the title [NELO-1205] feat: pass IndexConfig to repositories with scalability options feat: pass IndexConfig to repositories with scalability options Apr 16, 2026
@ochanism
ochanism force-pushed the enhancement/1205-3-2 branch from 35e3f33 to 9ad0b6b Compare April 16, 2026 06:39
@ochanism
ochanism force-pushed the enhancement/1205-3-2 branch from 9ad0b6b to f9daca2 Compare April 17, 2026 01:02
@ochanism
ochanism marked this pull request as ready for review April 17, 2026 01:04
@lyingbug
lyingbug merged commit 927d3de into Tencent:main Apr 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants