perf: optimize indexing speed with batching and incremental sync
- Batching BDD: group INSERT/UPDATE operations in batches of 100 using UNNEST - Incremental MeiliSearch: only sync books modified since last sync - Optimized fingerprint: use only size+mtime+filename (100x faster) - Increased DB connections from 5 to 20 - Reduced progress update frequency (every 1s or 10 files) - Add sync_metadata table to track last MeiliSearch sync
This commit is contained in:
6
infra/migrations/0007_add_sync_metadata.sql
Normal file
6
infra/migrations/0007_add_sync_metadata.sql
Normal file
@@ -0,0 +1,6 @@
|
||||
CREATE TABLE IF NOT EXISTS sync_metadata (
|
||||
id INTEGER PRIMARY KEY,
|
||||
last_meili_sync TIMESTAMPTZ
|
||||
);
|
||||
|
||||
INSERT INTO sync_metadata (id, last_meili_sync) VALUES (1, NULL) ON CONFLICT DO NOTHING;
|
||||
Reference in New Issue
Block a user