Initial commit: YouTube Studio Flow (backend, frontend, infrastructure, docs)
This commit is contained in:
+88
@@ -0,0 +1,88 @@
|
||||
# 2026-06-29 01 — Scheduled Remote Conflict Detection
|
||||
|
||||
**Status:** Shipped
|
||||
**Scope:** Backend, frontend Settings UI, ops env, docs
|
||||
|
||||
---
|
||||
|
||||
## What we set out to do
|
||||
|
||||
Resolve the "Dead Code — `YouTubeSyncService.detectConflict()`" backlog entry. Before this session, the method was fully implemented but never called; conflict handling as a user-facing flow was effectively broken:
|
||||
|
||||
- **Detection** only happened on manual "Refresh from YouTube" per video.
|
||||
- **Resolution** either meant running a full channel refresh (clobbers *every* video's local fields) or re-pushing local (silently overwrites remote). Neither offered a per-video accept/reject choice and neither showed a diff.
|
||||
|
||||
Decision: wire `detectConflict()` up as a scheduled sweep AND build a proper per-video accept-remote path. Direction "push local" reuses the existing `POST /videos/:id/sync`.
|
||||
|
||||
## Design decisions (locked in during the session)
|
||||
|
||||
| Question | Decision |
|
||||
|---|---|
|
||||
| Granularity | Per-video accept/reject (v1). Field-level merge deferred. |
|
||||
| Settings location | Per-team, with cadence global via env var |
|
||||
| Skip already-conflicted videos in the sweep? | Yes — no point re-detecting an unresolved conflict |
|
||||
| Add a `POST /videos/:id/force-local` endpoint? | No — reuse existing `POST /videos/:id/sync` |
|
||||
| Scheduler mechanism | BullMQ repeatable job (no new dependency) |
|
||||
| Selection strategy | Stalest-first per team, filtered by `lastSyncedAt < now - minAgeDays`, `remoteConflict = false`, `youtubeDeletedAt = null` |
|
||||
|
||||
## What changed
|
||||
|
||||
### Backend
|
||||
|
||||
- **Schema** — new migration `20260629000000_add_conflict_detection_settings`:
|
||||
- `Team.conflictDetectionEnabled Boolean @default(false)`
|
||||
- `Team.conflictDetectionBatchSize Int @default(50)`
|
||||
- `Team.conflictDetectionMinAgeDays Int @default(7)`
|
||||
- `Video.pendingRemoteSnapshot Json?`
|
||||
- `Video.pendingRemoteDescription String?`
|
||||
- **`YouTubeSyncService.detectConflict()`** (`backend/src/modules/youtube-sync/youtube-sync.service.ts`) — on hash mismatch now persists `pendingRemoteSnapshot` (mirrors `youtubeSnapshot` shape) and `pendingRemoteDescription`. On hash match with a stale flag, self-heals (clears flag + pending fields). No more dead code.
|
||||
- **New queue** `CONFLICT_DETECTION` in `backend/src/queues/queues.constants.ts`.
|
||||
- **New processor** `backend/src/queues/processors/conflict-detection.processor.ts` — iterates enabled teams, stalest-first selection, per-call quota guard, per-team batch cap. Logs summary. Stops the whole sweep on quota exhaustion.
|
||||
- **New scheduler** `backend/src/queues/schedulers/conflict-detection.scheduler.ts` — `OnModuleInit` registers a BullMQ repeatable driven by `CONFLICT_DETECTION_CRON` (default `0 3 * * *`), gated by `CONFLICT_DETECTION_ENABLED`. Wipes stale repeatables on boot so config changes take effect.
|
||||
- **Wired both** into `backend/src/worker.module.ts`.
|
||||
- **New endpoint** `POST /videos/:id/accept-remote` (EDITOR role) in `backend/src/modules/videos/videos.controller.ts` + `.service.ts`. Zero YouTube API calls — promotes stored `pendingRemoteSnapshot` into live columns and `youtubeSnapshot`, sets `renderedDescription = youtubeDescription = pendingRemoteDescription`, recomputes `lastSyncedHash`, sets `lastSyncedAt = now`, clears pending + flag. Emits audit log entry `action: 'accept-remote'`.
|
||||
- **Team settings** — three new fields added to `GET`/`PATCH /teams/:teamId/settings` with bounds validation (batch 1–500, min-age ≥ 0).
|
||||
|
||||
### Frontend
|
||||
|
||||
- `TeamSettings` interface (`frontend/src/lib/api.ts`) extended with the three new fields.
|
||||
- New "Remote Conflict Detection" section on `/settings`: Enable toggle, "Videos per run" (number, 1–500), "Only check videos older than (days)" (number, ≥0). Admin-only edit. Own Save button (kept separate from the schedule section to avoid mixed-scope saves).
|
||||
|
||||
### Ops / env
|
||||
|
||||
- `CONFLICT_DETECTION_ENABLED` and `CONFLICT_DETECTION_CRON` added to:
|
||||
- `backend/.env` (locally set to `true` / default cron)
|
||||
- `backend/.env.example` (default `false`)
|
||||
- `infrastructure/.env.example`
|
||||
- `infrastructure/docker-compose.yml` worker service (uses `:-` defaults)
|
||||
|
||||
### Documentation
|
||||
|
||||
- **Backlog entries removed** from `06 - Backlog/01 - Technical Debt and Future Work.md`:
|
||||
- "Dead Code — `YouTubeSyncService.detectConflict()`"
|
||||
- "Improvements Worth Considering — Automatic Remote Conflict Detection"
|
||||
- **Docs brought back in sync** (11 findings across 8 files identified by an audit agent, all applied):
|
||||
- Rewrote the two aggressively-wrong sections (Gotchas `remoteConflict Is Not Automatically Detected`, Schema `remoteConflict` paragraph) which literally claimed the opposite of the new behavior.
|
||||
- Filled content gaps in `04 - Database Schema`, `05 - Queue System`, `13 - Team Settings`, `02 - Videos API`, `09 - Teams API`, `02 - Environment Variables`, `05 - Deployment and Operations`.
|
||||
- Vault gained a new `07 - Daily Notes/` section (this note is the first entry).
|
||||
|
||||
## Verification
|
||||
|
||||
- Backend `tsc --noEmit` clean after `prisma generate`.
|
||||
- Frontend `tsc --noEmit` clean for the touched files (`settings/page.tsx`, `api.ts`); pre-existing errors elsewhere untouched.
|
||||
- Migration file present but not yet applied on any prod DB — user ran `prisma migrate` locally.
|
||||
|
||||
## Follow-ups worth flagging
|
||||
|
||||
- **UI for the conflict itself** — the video editor should show a per-field diff between the current local state and `pendingRemoteSnapshot` when `remoteConflict === true`, with "Accept remote" and "Keep local" buttons. Backend is ready; frontend diff view doesn't exist yet.
|
||||
- **Field-level merge** — declined for v1; if requested later, the storage layer already supports it (fields are individually addressable in `pendingRemoteSnapshot`).
|
||||
- **Multi-team frequency** — if teams later want different cadences, `frequency` needs to move from env-var to a per-team column and the scheduler must manage a repeatable-per-team.
|
||||
|
||||
## Related
|
||||
|
||||
- [[04 - Database Schema]] — new Team + Video columns
|
||||
- [[05 - Queue System]] — processor details, quota model
|
||||
- [[13 - Team Settings]] — per-team knobs, validation
|
||||
- [[02 - Videos API]] — `accept-remote` endpoint
|
||||
- [[04 - Gotchas]] — resolution options and self-heal
|
||||
- [[02 - Environment Variables]] — `CONFLICT_DETECTION_*`
|
||||
+55
@@ -0,0 +1,55 @@
|
||||
# 2026-07-04 01 — Batched Conflict Detection
|
||||
|
||||
**Status:** Shipped
|
||||
**Scope:** Backend refactor + docs
|
||||
|
||||
---
|
||||
|
||||
## What we set out to do
|
||||
|
||||
The scheduled conflict-detection sweep from [[2026-06-29 01 Scheduled Remote Conflict Detection]] was doing one `videos.list` API call per video. YouTube's `videos.list` accepts up to 50 IDs per call and costs the same 1 quota unit either way — the per-video pattern was wasting ~98% of the quota it consumed.
|
||||
|
||||
Goal: batch, without changing observable behavior for users or the API surface.
|
||||
|
||||
## Design decisions locked in
|
||||
|
||||
| Question | Decision |
|
||||
|---|---|
|
||||
| Refactor `detectConflict` in place or add a new method? | Add `detectConflictsForVideos(videoIds[])` and delete the single-video `detectConflict()` — nothing else called it |
|
||||
| Grouping strategy for batches | Group by `channelId` first (batch API needs a per-channel OAuth client), then chunks of 50 within each channel |
|
||||
| Where does the quota check live? | Moved from the processor's per-video loop into the service's per-batch loop — `canSpend(1)` before each `getVideosBatch` call |
|
||||
| Extend `getVideosBatch` to fetch `recordingDetails`? | Yes — the batch endpoint previously fetched only `snippet` + `status`, which would have dropped `recordingDate` from the hash and produced false positives. Adding `recordingDetails` is free (quota is per method-call, not per part) |
|
||||
|
||||
## What changed
|
||||
|
||||
- **`backend/src/modules/youtube-sync/youtube-api.client.ts:106`** — added `recordingDetails` to `getVideosBatch()`'s `part` array. No behavior change for existing callers (channel-import already falls back to DB values for missing fields).
|
||||
- **`backend/src/modules/youtube-sync/youtube-sync.service.ts`** — removed `detectConflict(videoId)`, added `detectConflictsForVideos(videoIds[])`, extracted the per-video hash+persist logic into a private `applyConflictDetection(video, remoteItem)` helper. The batch method returns `{ scanned, conflicts, quotaExhausted }` so the processor can stop the sweep cleanly.
|
||||
- **`backend/src/queues/processors/conflict-detection.processor.ts`** — inner loop replaced with a single service call per team. `QuotaService` no longer injected (moved into the service). Error handling now per-team (not per-video), which changes granularity of the `errors` counter — an API failure aborts one team but doesn't leak quota.
|
||||
|
||||
## Cost math — before vs after
|
||||
|
||||
| Team `batchSize` | Old quota per team per run | New quota per team per run |
|
||||
|---:|---:|---:|
|
||||
| 50 | 50 | 1 |
|
||||
| 250 | 250 | 5 |
|
||||
| 500 | 500 | 10 |
|
||||
|
||||
Numbers assume all videos on one channel. Extra API call per channel boundary within a batch, so multi-channel teams pay slightly more (still ~50× cheaper than before).
|
||||
|
||||
## Follow-ups worth flagging
|
||||
|
||||
- **Default `conflictDetectionBatchSize = 50` is now overly conservative.** Existing team settings unchanged out of caution. Users could safely raise to 250–500. Worth a mention in release notes if you ship an announcement.
|
||||
- **`applyConflictDetection` is `private` on the service** — if a manual `POST /videos/:id/detect-conflict` endpoint ever becomes a thing, it should call `detectConflictsForVideos([id])` rather than making a single-video sibling method reappear.
|
||||
- **Batch API doesn't return `contentDetails`.** Not needed by conflict detection but referenced by `refreshFromYouTube`. That path still uses the single-video `getVideoMetadata` — no change needed.
|
||||
|
||||
## Verification
|
||||
|
||||
- Backend `tsc --noEmit` clean
|
||||
- No migration required — this is a service-level refactor only
|
||||
- Frontend untouched
|
||||
|
||||
## Related
|
||||
|
||||
- [[2026-06-29 01 Scheduled Remote Conflict Detection]] — original feature
|
||||
- [[05 - Queue System]] — updated processor description with new cost model
|
||||
- [[04 - Gotchas]] — updated `remoteConflict` cost section
|
||||
Reference in New Issue
Block a user