perf: optimize indexing speed with batching and incremental sync
- Batching BDD: group INSERT/UPDATE operations in batches of 100 using UNNEST - Incremental MeiliSearch: only sync books modified since last sync - Optimized fingerprint: use only size+mtime+filename (100x faster) - Increased DB connections from 5 to 20 - Reduced progress update frequency (every 1s or 10 files) - Add sync_metadata table to track last MeiliSearch sync
This commit is contained in:
@@ -10,6 +10,7 @@ axum.workspace = true
|
||||
chrono.workspace = true
|
||||
notify = "6.1"
|
||||
parsers = { path = "../../crates/parsers" }
|
||||
rand.workspace = true
|
||||
reqwest.workspace = true
|
||||
serde.workspace = true
|
||||
serde_json.workspace = true
|
||||
|
||||
Reference in New Issue
Block a user