perf: optimize indexing speed with batching and incremental sync

- Batching BDD: group INSERT/UPDATE operations in batches of 100 using UNNEST
- Incremental MeiliSearch: only sync books modified since last sync
- Optimized fingerprint: use only size+mtime+filename (100x faster)
- Increased DB connections from 5 to 20
- Reduced progress update frequency (every 1s or 10 files)
- Add sync_metadata table to track last MeiliSearch sync
This commit is contained in:
2026-03-06 22:09:37 +01:00
parent ee76090265
commit d5d582db57
4 changed files with 498 additions and 158 deletions

View File

@@ -10,6 +10,7 @@ axum.workspace = true
chrono.workspace = true
notify = "6.1"
parsers = { path = "../../crates/parsers" }
rand.workspace = true
reqwest.workspace = true
serde.workspace = true
serde_json.workspace = true