Skip to content

fix: SQLite/Lite mode compatibility for FAQ import and chunk operations#968

Merged
lyingbug merged 1 commit intoTencent:mainfrom
swim2sun:main
Apr 15, 2026
Merged

fix: SQLite/Lite mode compatibility for FAQ import and chunk operations#968
lyingbug merged 1 commit intoTencent:mainfrom
swim2sun:main

Conversation

@swim2sun
Copy link
Copy Markdown
Contributor

@swim2sun swim2sun commented Apr 14, 2026

Pull Request

描述 (Description)

Fix multiple SQLite/Lite mode compatibility issues that cause FAQ import to crash or produce incorrect results. These bugs affect all users running WeKnora in Lite mode (SQLite + no Redis).

变更类型 (Type of Change)

  • 🐛 Bug 修复 (Bug fix)
  • 🧪 测试相关 (Test related)
  • 🔧 配置变更 (Configuration change)

影响范围 (Scope)

  • 后端 API (Backend API)
  • 数据库 (Database)
  • 配置文件 (Configuration)

测试 (Testing)

  • 集成测试 (Integration tests)
  • 手动测试 (Manual testing)

测试步骤 (Test Steps)

  1. Build Lite version on Linux: SKIP_FRONTEND=1 make build-lite
  2. Start with .env.lite config, create a FAQ knowledge base
  3. Import FAQ entries via CSV — verify no panic, entries created with correct seq_id
  4. Clear FAQ entries (soft delete), re-import — verify no UNIQUE constraint error
  5. Run tests: go test -tags sqlite_fts5 -v ./internal/application/repository/ -run SQLite

检查清单 (Checklist)

  • 代码遵循项目的编码规范
  • 已进行自我代码审查
  • 代码变更已添加适当的注释
  • 变更不会产生新的警告
  • 已添加测试用例证明修复有效或功能正常

相关 Issue

Fixes #963

数据库迁移 (Database Migration)

  • 不需要数据库迁移

部署说明 (Deployment Notes)

No special deployment steps. Changes are backward-compatible with PostgreSQL (Standard mode). The AssignChunkSeqIDs function is a no-op when seq_ids are already set by PostgreSQL's autoIncrement.

其他信息 (Additional Information)

Bugs Fixed

1. Redis nil pointer panic on FAQ import
UpsertFAQEntries calls redisClient.Set()/Get() without nil checks. In Lite mode Redis is disabled (redisClient == nil), causing panic.
Fix: Add in-memory sync.Map fallback for progress tracking and task locks.

2. seq_id always NULL in SQLite
GORM's autoIncrement tag on non-primary-key columns (SeqID) doesn't work in SQLite. All chunks and tags get seq_id = NULL, breaking FAQ entry listing and tag deletion.
Fix: Add AssignChunkSeqIDs() pre-assignment before batch insert; add BeforeCreate hook for KnowledgeTag.
3. UNIQUE constraint violation after soft-delete + re-import
AssignChunkSeqIDs uses GORM's default query which filters out soft-deleted records (deleted_at IS NULL). New seq_ids collide with soft-deleted ones.
Fix: Use Unscoped() when querying MAX(seq_id).

4. no such function: NOW in SQLite
Raw SQL uses PostgreSQL's NOW() function which doesn't exist in SQLite.
Fix: Replace with time.Now() (GORM updates) or datetime('now') (raw SQL), with dialector check where needed.

5. FAQ knowledge base accepts file uploads
CreateKnowledgeFromFile doesn't check kb.Type, allowing file uploads to FAQ knowledge bases which creates duplicate document-type chunks alongside FAQ entries.
Fix: Add kb.Type == "faq" guard returning descriptive error.

6. Linux build failure
Makefile uses macOS-only linker flag -Wl,-no_warn_duplicate_libraries unconditionally.
Fix: Conditionally apply based on uname.

New Tests (6 cases)

  • TestCreateChunks_SQLite_SeqIDAutoAssigned
  • TestCreateChunks_SQLite_SeqIDContinuesFromExisting
  • TestCreateChunks_SQLite_SeqIDUniqueAcrossKBs
  • TestCreateChunks_SQLite_SeqIDAfterSoftDelete
  • TestKnowledgeTag_SQLite_SeqIDAutoAssigned
  • TestUpdateChunk_SQLite_NoNOWError

Multiple fixes for WeKnora Lite mode (SQLite, no Redis):

1. Redis nil pointer panic on FAQ import (knowledge.go)
   - Add in-memory fallback (sync.Map) for FAQ import progress tracking
   - Add nil guards for running task locks and progress queries

2. seq_id not auto-incrementing in SQLite (chunk.go, tag.go)
   - GORM autoIncrement on non-PK columns doesn't work in SQLite
   - Add AssignChunkSeqIDs() to pre-assign seq_ids before batch insert
   - Add BeforeCreate hook for KnowledgeTag
   - Use Unscoped() to include soft-deleted records in MAX(seq_id) query

3. NOW() function not available in SQLite (chunk.go)
   - Replace raw SQL NOW() with time.Now() or datetime('now')
   - Use dialector check for raw SQL that must differ between PG and SQLite

4. FAQ KB should reject file uploads (knowledge.go)
   - Add kb.Type == "faq" check in CreateKnowledgeFromFile

5. Linux/macOS build compatibility (Makefile)
   - Conditionally apply macOS-only linker flag based on uname

6. Add SQLite integration tests (chunk_sqlite_test.go)
   - TestCreateChunks_SQLite_SeqIDAutoAssigned
   - TestCreateChunks_SQLite_SeqIDContinuesFromExisting
   - TestCreateChunks_SQLite_SeqIDUniqueAcrossKBs
   - TestCreateChunks_SQLite_SeqIDAfterSoftDelete
   - TestKnowledgeTag_SQLite_SeqIDAutoAssigned
   - TestUpdateChunk_SQLite_NoNOWError

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@lyingbug lyingbug merged commit d2c6814 into Tencent:main Apr 15, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: FAQ import panics with nil pointer in Lite mode (Redis disabled)

2 participants