Initial commit: YouTube Studio Flow (backend, frontend, infrastructure, docs)
This commit is contained in:
@@ -0,0 +1,94 @@
|
||||
# Local Setup
|
||||
|
||||
Step-by-step guide to running YouTube Studio Flow on a local machine for development.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Node.js 18+** — required by both backend and frontend
|
||||
- **Docker** — used to run Postgres and Redis locally
|
||||
- **Git** — for cloning the repository
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — Start Infrastructure
|
||||
|
||||
Start Postgres and Redis using Docker Compose:
|
||||
|
||||
```bash
|
||||
cd infrastructure
|
||||
docker compose up -d postgres redis
|
||||
```
|
||||
|
||||
> **Important:** Redis must run with `--maxmemory-policy noeviction`. BullMQ silently drops jobs if Redis uses `allkeys-lru` eviction. This policy is already pre-configured in `infrastructure/docker-compose.yml`. Do not change it. See [[04 - Gotchas]] for more detail.
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Backend Setup
|
||||
|
||||
In a terminal, set up and start the NestJS API:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npm install
|
||||
cp .env.example .env
|
||||
# Fill in all required values in .env — see [[02 - Environment Variables]]
|
||||
npx prisma generate
|
||||
npx prisma migrate deploy
|
||||
npm run start:dev
|
||||
```
|
||||
|
||||
The API is now running on **http://localhost:3001**.
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — Queue Worker
|
||||
|
||||
Open a **second terminal** and start the BullMQ queue processor:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npx ts-node src/worker.ts
|
||||
```
|
||||
|
||||
The worker runs from the same codebase as the API but through a separate entry point (`src/worker.ts` → `WorkerModule`). It handles all background jobs: YouTube sync, description rendering, linting, bulk operations, and CSV imports.
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — Frontend Setup
|
||||
|
||||
Open a **third terminal** and start the Next.js frontend:
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm install
|
||||
# Create frontend/.env.local with the following content:
|
||||
# NEXT_PUBLIC_API_URL=http://localhost:3001/api/v1
|
||||
npm run dev
|
||||
```
|
||||
|
||||
The app is now running on **http://localhost:3000**.
|
||||
|
||||
---
|
||||
|
||||
## Verify the Setup
|
||||
|
||||
1. Navigate to **http://localhost:3000**
|
||||
2. Click **"Sign in with Google"**
|
||||
3. Complete the Google OAuth flow
|
||||
4. You should be redirected to the **Overview** page
|
||||
|
||||
If the redirect fails, check that `GOOGLE_CALLBACK_URL` and `FRONTEND_URL` are set correctly in `backend/.env`. See [[02 - Environment Variables]].
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- All three processes must run simultaneously for full functionality:
|
||||
- `npm run start:dev` — HTTP API on :3001
|
||||
- `npx ts-node src/worker.ts` — BullMQ job processor
|
||||
- `npm run dev` — Next.js frontend on :3000
|
||||
- Database migrations run automatically with `npx prisma migrate deploy`
|
||||
- After any schema change: stop the backend → `npx prisma generate` → `npx prisma migrate deploy` → restart both the API and worker
|
||||
- For non-obvious behaviors and known traps, see [[04 - Gotchas]]
|
||||
@@ -0,0 +1,68 @@
|
||||
# Environment Variables
|
||||
|
||||
Complete reference for all environment variables used by the backend and frontend.
|
||||
|
||||
---
|
||||
|
||||
## Backend (`backend/.env`)
|
||||
|
||||
Copy `backend/.env.example` to `backend/.env` and fill in all required values before starting the API or worker. See [[01 - Local Setup]] for the full setup sequence.
|
||||
|
||||
| Variable | Required | Description |
|
||||
|---|---|---|
|
||||
| `DATABASE_URL` | Yes | PostgreSQL connection string. Format: `postgresql://user:pass@localhost:5432/dbname` |
|
||||
| `REDIS_URL` | Yes | Redis connection string. Format: `redis://:password@localhost:6379` |
|
||||
| `JWT_SECRET` | Yes | Secret for signing access tokens. Use a long random string. |
|
||||
| `JWT_REFRESH_SECRET` | Yes | Secret for signing refresh tokens. Must differ from `JWT_SECRET`. |
|
||||
| `GOOGLE_CLIENT_ID` | Yes | Google OAuth app client ID |
|
||||
| `GOOGLE_CLIENT_SECRET` | Yes | Google OAuth app client secret |
|
||||
| `GOOGLE_CALLBACK_URL` | Yes | OAuth redirect URL. Local: `http://localhost:3001/api/v1/auth/google/callback` |
|
||||
| `TOKEN_ENCRYPTION_KEY` | Yes | Exactly 32 characters. AES-256 key for encrypting YouTube OAuth tokens in the DB. **If this changes, all channel connections break.** |
|
||||
| `FRONTEND_URL` | Yes | Frontend origin for OAuth redirect. Local: `http://localhost:3000` |
|
||||
| `PORT` | No | API port. Default: `3001` |
|
||||
| `NODE_ENV` | No | `development` or `production` |
|
||||
| `CONFLICT_DETECTION_ENABLED` | No | Global kill switch for the scheduled remote-conflict sweep. Default: `false`. Only takes effect on the worker process; the API doesn't read it. Per-team opt-in still required via `Team.conflictDetectionEnabled`. See [[05 - Queue System]]. |
|
||||
| `CONFLICT_DETECTION_CRON` | No | Cron pattern (BullMQ format) for the sweep. Default: `0 3 * * *` (daily at 03:00 UTC). Only read when `CONFLICT_DETECTION_ENABLED=true`. |
|
||||
|
||||
### Example `backend/.env`
|
||||
|
||||
```env
|
||||
DATABASE_URL=postgresql://studioflow:yourpassword@localhost:5432/studioflow
|
||||
REDIS_URL=redis://:yourpassword@localhost:6379
|
||||
JWT_SECRET=a-very-long-random-string-for-access-tokens
|
||||
JWT_REFRESH_SECRET=a-different-very-long-random-string-for-refresh-tokens
|
||||
GOOGLE_CLIENT_ID=123456789-abc.apps.googleusercontent.com
|
||||
GOOGLE_CLIENT_SECRET=GOCSPX-yourGoogleSecret
|
||||
GOOGLE_CALLBACK_URL=http://localhost:3001/api/v1/auth/google/callback
|
||||
TOKEN_ENCRYPTION_KEY=exactly32characterslongkeyhere!!
|
||||
FRONTEND_URL=http://localhost:3000
|
||||
PORT=3001
|
||||
NODE_ENV=development
|
||||
CONFLICT_DETECTION_ENABLED=false
|
||||
CONFLICT_DETECTION_CRON=0 3 * * *
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Frontend (`frontend/.env.local`)
|
||||
|
||||
Create `frontend/.env.local` manually (it is not committed to git and has no `.example` counterpart).
|
||||
|
||||
| Variable | Required | Description |
|
||||
|---|---|---|
|
||||
| `NEXT_PUBLIC_API_URL` | Yes | Backend API base URL. Local: `http://localhost:3001/api/v1` |
|
||||
|
||||
### Example `frontend/.env.local`
|
||||
|
||||
```env
|
||||
NEXT_PUBLIC_API_URL=http://localhost:3001/api/v1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Notes
|
||||
|
||||
- Never commit `.env` or `.env.local` files to git — both are already listed in `.gitignore`
|
||||
- `TOKEN_ENCRYPTION_KEY` must remain stable for the entire lifetime of the database. Rotating it invalidates all stored YouTube OAuth tokens, requiring every channel to be re-authenticated. See [[04 - Gotchas]] for more detail.
|
||||
- Use distinct values for `JWT_SECRET` and `JWT_REFRESH_SECRET` — reusing the same secret across both token types weakens the separation between access and refresh token validation
|
||||
- In production, use a secrets manager or CI/CD secret injection rather than plain `.env` files
|
||||
@@ -0,0 +1,209 @@
|
||||
# Recipes
|
||||
|
||||
Step-by-step instructions for common development tasks. For module structure conventions and shared service patterns, see the [[02 - Backend]] architecture reference.
|
||||
|
||||
---
|
||||
|
||||
## Recipe: Add a New Backend Module
|
||||
|
||||
1. Create the module directory: `backend/src/modules/my-feature/`
|
||||
|
||||
2. Create `my-feature.module.ts`:
|
||||
|
||||
```typescript
|
||||
import { Module } from '@nestjs/common';
|
||||
import { PrismaModule } from '../../shared/prisma/prisma.module';
|
||||
import { AuditModule } from '../../shared/audit/audit.module';
|
||||
import { MyFeatureController } from './my-feature.controller';
|
||||
import { MyFeatureService } from './my-feature.service';
|
||||
|
||||
@Module({
|
||||
imports: [PrismaModule, AuditModule],
|
||||
controllers: [MyFeatureController],
|
||||
providers: [MyFeatureService],
|
||||
})
|
||||
export class MyFeatureModule {}
|
||||
```
|
||||
|
||||
3. Create `my-feature.controller.ts` with the required decorators:
|
||||
|
||||
```typescript
|
||||
import { Controller, UseGuards, Request } from '@nestjs/common';
|
||||
import { ApiTags, ApiBearerAuth } from '@nestjs/swagger';
|
||||
import { JwtAuthGuard } from '../auth/guards/jwt-auth.guard';
|
||||
|
||||
@ApiTags('my-feature')
|
||||
@ApiBearerAuth()
|
||||
@UseGuards(JwtAuthGuard)
|
||||
@Controller('my-feature')
|
||||
export class MyFeatureController {
|
||||
constructor(private readonly myFeatureService: MyFeatureService) {}
|
||||
}
|
||||
```
|
||||
|
||||
4. Create `my-feature.service.ts` with `PrismaService` injection. Always scope DB queries to `req.user.teamId`:
|
||||
|
||||
```typescript
|
||||
import { Injectable } from '@nestjs/common';
|
||||
import { PrismaService } from '../../shared/prisma/prisma.service';
|
||||
|
||||
@Injectable()
|
||||
export class MyFeatureService {
|
||||
constructor(private readonly prisma: PrismaService) {}
|
||||
}
|
||||
```
|
||||
|
||||
5. Register the module in `AppModule` imports array at `backend/src/app.module.ts`
|
||||
|
||||
6. If the worker also needs access to this module's services (e.g. for use in a queue processor), register it in `WorkerModule` at `backend/src/worker.module.ts` as well
|
||||
|
||||
> **Audit logging:** If the module performs user-facing mutations (create/update/delete), it must import `AuditModule` and inject `AuditService`. Call `AuditService.log(actorId, entityType, entityId, action, before, after)` in every mutation. See [[04 - Backend Architecture Patterns]] for the full audit pattern.
|
||||
|
||||
---
|
||||
|
||||
## Recipe: Add a New Lint Rule
|
||||
|
||||
1. Create `backend/src/modules/linting/rules/my-rule.rule.ts`:
|
||||
|
||||
```typescript
|
||||
import { LintRule, LintIssue } from './base.rule';
|
||||
import { LintSeverity } from '@prisma/client';
|
||||
|
||||
export class MyRule implements LintRule {
|
||||
code = 'MY_RULE_CODE';
|
||||
severity = LintSeverity.WARNING;
|
||||
|
||||
check(video: any): LintIssue | null {
|
||||
if (/* condition */) {
|
||||
return {
|
||||
targetField: 'title',
|
||||
message: 'Describe the problem clearly',
|
||||
fixSuggestion: 'Explain how to fix it',
|
||||
};
|
||||
}
|
||||
return null;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Open `backend/src/modules/linting/linting.service.ts`, import the new rule class, and add an instance to the `rules[]` array:
|
||||
|
||||
```typescript
|
||||
private readonly rules: LintRule[] = [
|
||||
new ExistingRule(),
|
||||
new MyRule(), // add here
|
||||
];
|
||||
```
|
||||
|
||||
3. The rule runs automatically on all subsequent lint jobs. It can be disabled per-team in the Settings page (lint rule disabling is stored on the `Team` model). See [[03 - Metadata Linting]] for the full lint system overview.
|
||||
|
||||
---
|
||||
|
||||
## Recipe: Add a New API Endpoint to the Frontend
|
||||
|
||||
1. If the endpoint returns a new response shape, add a TypeScript interface to `frontend/src/lib/api.ts`:
|
||||
|
||||
```typescript
|
||||
export interface MyResponse {
|
||||
id: string;
|
||||
name: string;
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
2. Add the API function in the same file:
|
||||
|
||||
```typescript
|
||||
export const myNewAction = (id: string, data: MyData): Promise<MyResponse> =>
|
||||
apiClient.post<MyResponse>(`/my-feature/${id}/action`, data).then(r => r.data);
|
||||
```
|
||||
|
||||
3. Use it in a component with TanStack Query:
|
||||
|
||||
```typescript
|
||||
const qc = useQueryClient(); // must be called at component level, not inside a callback
|
||||
|
||||
const mut = useMutation({
|
||||
mutationFn: () => myNewAction(id, data),
|
||||
onSuccess: () => {
|
||||
qc.invalidateQueries({ queryKey: ['affected-key'] });
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
> See [[03 - Component Patterns]] for TanStack Query conventions, including `placeholderData` usage and multi-key invalidation.
|
||||
|
||||
---
|
||||
|
||||
## Recipe: Add a New Sidebar Navigation Item
|
||||
|
||||
1. Open `frontend/src/components/shared/Sidebar.tsx`
|
||||
|
||||
2. Import the icon from `lucide-react`:
|
||||
|
||||
```typescript
|
||||
import { MyIcon } from 'lucide-react';
|
||||
```
|
||||
|
||||
3. Add an entry to the `navItems` array:
|
||||
|
||||
```typescript
|
||||
{ label: 'My Page', href: '/my-page', icon: MyIcon, group: 'WORKSPACE' },
|
||||
```
|
||||
|
||||
Available groups and their current members:
|
||||
|
||||
| Group | Used for |
|
||||
|---|---|
|
||||
| `WORKSPACE` | Overview, Videos, Calendar |
|
||||
| `OPERATIONS` | Bulk Jobs, Import/Export |
|
||||
| `PEOPLE` | Collaborators, Teams |
|
||||
| `LOGGING` | Audit Log, Quota |
|
||||
|
||||
4. Create the corresponding route at `frontend/src/app/(dashboard)/my-page/page.tsx`
|
||||
|
||||
---
|
||||
|
||||
## Recipe: Add a New Prisma Field
|
||||
|
||||
1. Add the field to the appropriate model in `backend/prisma/schema.prisma`
|
||||
|
||||
2. Stop the backend (and worker if running)
|
||||
|
||||
3. Create and apply the migration:
|
||||
|
||||
```bash
|
||||
npx prisma migrate dev --name add-my-field
|
||||
```
|
||||
|
||||
4. Regenerate the Prisma client:
|
||||
|
||||
```bash
|
||||
npx prisma generate
|
||||
```
|
||||
|
||||
5. Restart the backend and worker
|
||||
|
||||
> **Never remove enum values** from the Prisma schema. PostgreSQL enum removal requires a raw SQL migration and risks data loss if any rows reference the removed value. Mark deprecated values in the UI instead. See [[04 - Gotchas]] for detail.
|
||||
|
||||
---
|
||||
|
||||
## Recipe: Add a New System Token (e.g. `{video.myField}`)
|
||||
|
||||
System tokens are built-in substitution tokens like `{video.title}` or `{collab.youtube}` that are handled by dedicated resolvers rather than team variable lookup.
|
||||
|
||||
1. Add the new token definition to `SYSTEM_VARIABLES[]` in:
|
||||
`backend/src/shared/system-variables/system-variables.registry.ts`
|
||||
|
||||
2. Add the token string to the `SYSTEM_VARIABLE_TOKENS` Set in the same file:
|
||||
|
||||
```typescript
|
||||
export const SYSTEM_VARIABLE_TOKENS = new Set([
|
||||
// existing tokens...
|
||||
'{video.myField}',
|
||||
]);
|
||||
```
|
||||
|
||||
3. Implement the resolution logic in `RenderEngineService` (`backend/src/shared/render-engine/render-engine.service.ts`) in the appropriate resolver method
|
||||
|
||||
> **Critical:** Without step 2, `resolveVariables()` will treat the token as a team variable key, find nothing, and silently produce an empty string in the rendered description. This is a common source of invisible rendering bugs. See [[07 - Render Engine]] for the full token resolution pipeline.
|
||||
@@ -0,0 +1,196 @@
|
||||
# Gotchas
|
||||
|
||||
Non-obvious behaviors, known traps, and decisions that have caused bugs or confusion during development. Read this before debugging anything that "should work."
|
||||
|
||||
---
|
||||
|
||||
## CONDITIONAL Block — Unknown Rule Types and Operators Silently Default to `true`
|
||||
|
||||
`evaluateCondition()` in `render-engine.service.ts` evaluates each rule in a CONDITIONAL block's `condition.rules` array via a `switch` statement. The `default` branch returns `true` for any unrecognised `rule.type`. A second `default: return true` exists inside the `collab_count` case for unrecognised operators.
|
||||
|
||||
Practical consequences:
|
||||
- A typo in `rule.type` (e.g. `"variable_fille"` instead of `"variable_filled"`) silently makes the rule pass, causing the block to render unconditionally.
|
||||
- An unrecognised `operator` on a `collab_count` rule (e.g. `"neq"` instead of `"eq"`) also silently passes.
|
||||
- No error is thrown, no warning is logged, and no lint rule checks condition JSON for valid rule types.
|
||||
|
||||
Known valid `rule.type` values: `"variable_filled"`, `"variable_empty"`, `"collab_count"`.
|
||||
Known valid `collab_count` operators: `"eq"`, `"gt"`, `"lt"`, `"gte"`, `"lte"`.
|
||||
|
||||
If a CONDITIONAL block appears to render even when its condition should not be met, check the condition JSON for typos in `type` or `operator`.
|
||||
|
||||
---
|
||||
|
||||
## Campaign.status Must Be Exactly `"active"` (Lowercase)
|
||||
|
||||
`Campaign.status` is a free-form `String` column with no enum constraint and no API-level validation. The `DESC_OUTDATED_SPONSOR_COPY` lint rule (`linting/rules/desc-outdated-sponsor.rule.ts`) checks `status !== 'active'` — an exact case-sensitive string match.
|
||||
|
||||
Any value other than the lowercase string `"active"` is treated as inactive:
|
||||
- `"ACTIVE"` → inactive (lint ERROR fires)
|
||||
- `"paused"`, `"disabled"`, `"inactive"` → inactive (lint ERROR fires)
|
||||
- Any typo → inactive (lint ERROR fires)
|
||||
|
||||
When a campaign is incorrectly treated as inactive, its CAMPAIGN blocks are excluded from all rendered descriptions and every video that references them shows `DESC_OUTDATED_SPONSOR_COPY`. If campaigns appear to have stopped working for no obvious reason, check `Campaign.status` for a case mismatch or typo.
|
||||
|
||||
---
|
||||
|
||||
## remoteConflict — Detection Paths and Resolution
|
||||
|
||||
`Video.remoteConflict` can be set by three paths:
|
||||
|
||||
1. **Manual refresh** — `POST /videos/:id/refresh` fetches YouTube-side metadata and recomputes `lastSyncedHash`. If the new hash differs from the stored one, the flag flips on.
|
||||
2. **Scheduled sweep** — the `CONFLICT_DETECTION` BullMQ queue runs on a cron pattern (`CONFLICT_DETECTION_CRON`, default 03:00 daily) when the operator sets `CONFLICT_DETECTION_ENABLED=true` on the worker AND the team opts in via `Team.conflictDetectionEnabled`. See [[05 - Queue System]].
|
||||
3. **Implicit via full channel refresh** — a channel-level re-import overwrites local fields wholesale, which is not conflict *detection* but effectively resolves any conflict by clobbering local state.
|
||||
|
||||
When path 1 or 2 detects a mismatch, `pendingRemoteSnapshot` (JSON) and `pendingRemoteDescription` (String) capture the freshly fetched remote state so the user can review and resolve it without a second YouTube call.
|
||||
|
||||
**Resolution options:**
|
||||
|
||||
- `POST /videos/:id/accept-remote` — adopts the stored pending snapshot as the new local state. Zero YouTube API calls.
|
||||
- `POST /videos/:id/sync` — pushes local over remote (standard sync). Clears the flag on success.
|
||||
|
||||
**Self-heal:** if the sweep re-checks a video that was previously flagged but the remote now matches `lastSyncedHash` again (e.g. the creator reverted their out-of-band edit), the flag and pending fields are cleared automatically on the next pass.
|
||||
|
||||
**Cost:** the scheduled sweep uses `videos.list` batched up to 50 IDs at a time — **1 quota unit per batch**, not per video. A `batchSize` of 250 costs roughly 5 units per team per run, plus one extra call whenever a batch spans a channel boundary (batches are grouped by channel first because the OAuth client is per-channel). `QuotaService.canSpend(1)` guards each batch and the sweep stops early on exhaustion.
|
||||
|
||||
---
|
||||
|
||||
## lintStatus Is a Denormalized Cache
|
||||
|
||||
`Video.lintStatus` is not computed on read — it is a stored value written when lint jobs complete. Any code path that deletes `LintResult` rows (e.g. re-importing a channel, removing a lint rule) must recompute it afterward. Use `POST /lint/team/recompute-status` to heal all stale statuses across the team. The linting page calls this automatically on load.
|
||||
|
||||
---
|
||||
|
||||
## Redis Eviction Policy
|
||||
|
||||
BullMQ silently drops jobs if Redis is configured with `allkeys-lru` eviction. Redis must run with `--maxmemory-policy noeviction`. This is already set in `infrastructure/docker-compose.yml`. Do not change it. If jobs seem to disappear without being processed, check the Redis eviction policy first.
|
||||
|
||||
---
|
||||
|
||||
## TOKEN_ENCRYPTION_KEY Rotation
|
||||
|
||||
YouTube OAuth tokens are AES-256 encrypted in the database using `TOKEN_ENCRYPTION_KEY`. If this key is changed or lost, all stored tokens become unreadable and every connected channel breaks. Channels must then be fully re-authenticated. The key must remain stable for the entire lifetime of the database. See [[02 - Environment Variables]] for the full variable reference.
|
||||
|
||||
---
|
||||
|
||||
## Recording Date Is Never Imported from YouTube
|
||||
|
||||
The YouTube API does not return `recordingDate` in video list or playlist responses. It always imports as `null`. If a user sets a recording date locally, the video will show as "Push pending" (because the hash changes) and will remain so until the change is pushed to YouTube. This is expected behavior, not a bug.
|
||||
|
||||
---
|
||||
|
||||
## freetext: IDs in blockOrder
|
||||
|
||||
`VideoConfig.blockOrder` and `Template.defaultBlocks` are JSON string arrays that may contain IDs with the prefix `freetext:` (e.g. `"freetext:abc123"`). These do not correspond to any `DescriptionBlock` row in the database. Their content comes entirely from `blockOverrides[id].content`. Always check for this prefix before performing a DB lookup on a block ID. See [[02 - Description Engine]] for the full block rendering model.
|
||||
|
||||
---
|
||||
|
||||
## hashMetadata Fields
|
||||
|
||||
`hashMetadata()` (in `backend/src/shared/render-engine/hash.ts`) hashes exactly these fields:
|
||||
|
||||
- `title`
|
||||
- `description`
|
||||
- `tags`
|
||||
- `categoryId`
|
||||
- `privacyStatus`
|
||||
- `defaultLanguage`
|
||||
- `defaultAudioLanguage`
|
||||
- `selfDeclaredMadeForKids`
|
||||
- `embeddable`
|
||||
- `license`
|
||||
- `recordingDate`
|
||||
|
||||
All of these fields are included in the YouTube push payload. A local change to any of them will flip `hasPendingChanges` to true and trigger a sync. `privacyStatus` was added to the hash after it was discovered that privacy-only changes were silently dropped (the hash did not change, so the push was skipped).
|
||||
|
||||
---
|
||||
|
||||
## Date Formatting Single-Pass Regex
|
||||
|
||||
`applyDateFormat()` in `backend/src/shared/render-engine/render-engine.service.ts` uses a single-pass regex replacement. Do not refactor it to use chained `.replace()` calls. Chained replacements cause re-substitution bugs — for example, the `M` token in a date format would match again inside the already-substituted word "March", corrupting the output. The single-pass approach was introduced specifically to fix this class of bug.
|
||||
|
||||
---
|
||||
|
||||
## collab.youtube Resolves to a Full URL
|
||||
|
||||
The token `{collab.youtube}` resolves to the full YouTube channel URL, for example `https://www.youtube.com/@handle`. It does not resolve to just the handle string. If you need only the handle portion, that is not currently a supported token.
|
||||
|
||||
---
|
||||
|
||||
## {video.publishedAt} Does Not Exist
|
||||
|
||||
`{video.publishedAt}` is not a supported render token and has never been registered in `VIDEO_RESOLVERS` or `SYSTEM_VARIABLE_TOKENS`. It was present in earlier versions of this documentation in error.
|
||||
|
||||
If any description block contains `{video.publishedAt}`, the placeholder will remain unreplaced in the rendered output — it resolves to nothing, leaving the literal string `{video.publishedAt}` in the description. The `DESC_EMPTY_PLACEHOLDER` lint rule will flag it.
|
||||
|
||||
The correct tokens for dates are `{video.scheduledAt}` and `{video.recordingDate}`. `publishedAt` is a database field accessible via the API but has no corresponding render token.
|
||||
|
||||
---
|
||||
|
||||
## collab.handle Is Deprecated
|
||||
|
||||
The token `{collab.handle}` was renamed to `{collab.youtube}`. Any description block content still containing `{collab.handle}` will not resolve — it will remain as an unresolved placeholder in the rendered output. The `DESC_EMPTY_PLACEHOLDER` lint rule will catch this and flag it. Update affected blocks manually by replacing `{collab.handle}` with `{collab.youtube}`.
|
||||
|
||||
---
|
||||
|
||||
## Prisma Enum Values Cannot Be Removed
|
||||
|
||||
Removing a value from a PostgreSQL enum requires a raw SQL migration and risks data corruption if any existing rows reference the removed value. Never remove values from Prisma enums. Instead, mark them as deprecated in the UI so they are hidden from users but remain valid in the database. The `BlockType` values `GLOBAL` and `REPEATABLE` are the current examples of this pattern.
|
||||
|
||||
---
|
||||
|
||||
## Two Separate Entry Points
|
||||
|
||||
The backend has two separate NestJS entry points that must both be running:
|
||||
|
||||
- `src/main.ts` → `AppModule` → HTTP API on port 3001
|
||||
- `src/worker.ts` → `WorkerModule` → BullMQ queue processor (no HTTP)
|
||||
|
||||
When you add a new module, register it in `AppModule`. If the worker's queue processors also need to use services from that module, register it in `WorkerModule` as well. Forgetting the `WorkerModule` registration causes runtime errors in background jobs that are invisible until a relevant job is actually processed.
|
||||
|
||||
---
|
||||
|
||||
## Collaborator IDs Live on Video, Not VideoConfig
|
||||
|
||||
Collaborators are assigned via `Video.collaboratorIds` — a JSON string array on the `Video` row itself, not on `VideoConfig`. There is no `VideoCollaborator` join table. This single field is used by the render engine to resolve `{collab.*}` tokens, by the video list filter (`collaboratorId` query param), and by the calendar.
|
||||
|
||||
Use the `array_contains` Prisma operator when querying this JSON field.
|
||||
|
||||
---
|
||||
|
||||
## SiYoutube Does Not Exist
|
||||
|
||||
`react-icons/si` v5 does not export `SiYoutube`. Attempting to import it will cause a build error. Use `FaYoutube` from `react-icons/fa` instead for the YouTube icon. All other platform icons (`SiTwitch`, `SiInstagram`, `SiTiktok`, `SiX`, `SiBluesky`, `SiDiscord`) are available in `react-icons/si`.
|
||||
|
||||
---
|
||||
|
||||
## Middle-Click on Windows
|
||||
|
||||
Windows browsers intercept `mousedown` for middle-click before the `auxclick` event fires, entering autoscroll mode instead. Using `onAuxClick` or `window.open()` to handle middle-click will not work reliably on Windows. To support middle-click navigation on table rows or cards, use a real `<a>` element as an absolutely positioned overlay over the clickable area. The row overlay pattern in `VideoTable.tsx` is the reference implementation for this.
|
||||
|
||||
---
|
||||
|
||||
## min-width: 0 on Flex and Grid Children
|
||||
|
||||
Flex and grid children default to `min-width: auto`, meaning they cannot shrink below their content's natural size. This causes horizontal overflow on any flex or grid child that contains long text, wide tables, or deeply nested content. Add `min-width: 0` to every flex/grid child at every nesting level that might contain wide content. This applies in CSS Modules and must be repeated at each level — the parent setting does not propagate. See [[02 - CSS Conventions]] for the full CSS pattern reference.
|
||||
|
||||
---
|
||||
|
||||
## Deleted Blocks Are Silently Skipped at Render Time
|
||||
|
||||
If a block ID in `VideoConfig.blockOrder` no longer has a corresponding `DescriptionBlock` row (because the block was deleted), the render engine silently skips it — `render-engine.service.ts` line 196: `if (!block) continue;`. No error is thrown, no warning is logged, and no lint result is produced.
|
||||
|
||||
The deleted block simply disappears from the rendered description without any indication that the output is incomplete. A video can silently lose description content with no user-facing signal.
|
||||
|
||||
There is no lint rule that checks for orphaned block IDs in `blockOrder`. If a block that is referenced by many videos is deleted, all of those videos will render incomplete descriptions until their `blockOrder` is manually cleaned up.
|
||||
|
||||
---
|
||||
|
||||
## BullMQ Job Deduplication
|
||||
|
||||
BullMQ deduplicates jobs by `jobId` — if a job with the same ID already exists in the queue and has not yet run, the new submission is silently ignored. Using a static ID like `lint-{videoId}` is intentional for normal lint enqueueing (prevents duplicate lint jobs from piling up). For forced reruns — such as "rerun all lint checks" — use a timestamp suffix to bypass deduplication:
|
||||
|
||||
```typescript
|
||||
jobId: `lint-${videoId}-${Date.now()}`
|
||||
```
|
||||
|
||||
Without the suffix, the "rerun" submits a job that is immediately deduplicated against the existing queued job and never actually runs.
|
||||
@@ -0,0 +1,99 @@
|
||||
# Deployment and Operations
|
||||
|
||||
---
|
||||
|
||||
## Production Stack
|
||||
|
||||
The production setup is a single-host Docker Compose deployment using **Traefik** as a reverse proxy. There is no Kubernetes, cloud-managed infrastructure, or horizontal scaling configuration.
|
||||
|
||||
**Services** (`infrastructure/docker-compose.yml`):
|
||||
|
||||
| Service | Image | Role |
|
||||
|---|---|---|
|
||||
| `migrate` | backend Dockerfile | Runs `prisma migrate deploy` once before API starts |
|
||||
| `api` | backend Dockerfile | NestJS HTTP API on port 3001 |
|
||||
| `worker` | backend Dockerfile | BullMQ queue processor (`node dist/worker.js`) |
|
||||
| `frontend` | frontend Dockerfile | Next.js on port 3000 |
|
||||
| `postgres` | postgres:16-alpine | Database — bound to `127.0.0.1:5432` (not public) |
|
||||
| `redis` | redis:7-alpine | BullMQ queues — `noeviction` policy, AOF persistence |
|
||||
|
||||
**SSL / routing**: Traefik handles TLS termination with automatic Let's Encrypt certificates. Both API (`/api` prefix) and frontend run on the same domain — Traefik routes by path prefix. The `traefik-network` external network must exist before deploy.
|
||||
|
||||
**Build**: Multi-stage Dockerfile (`node:22-alpine`). Builder compiles TypeScript; runner installs prod-only deps. The Prisma CLI is copied from builder stage so the `migrate` service can run schema migrations.
|
||||
|
||||
---
|
||||
|
||||
## Secrets Management
|
||||
|
||||
All secrets are passed as environment variables from `infrastructure/.env` (based on `.env.example`). There is no secrets vault, no encrypted secret store, and no runtime secret injection. The `.env` file must be present on the host before `docker compose up`.
|
||||
|
||||
Secrets that must be set:
|
||||
|
||||
| Variable | Notes |
|
||||
|---|---|
|
||||
| `TOKEN_ENCRYPTION_KEY` | Exactly 32 characters. AES-256 key for YouTube OAuth tokens. See rotation note below. |
|
||||
| `JWT_SECRET` / `JWT_REFRESH_SECRET` | Generate with `openssl rand -base64 48` |
|
||||
| `POSTGRES_PASSWORD` / `REDIS_PASSWORD` | Strong random passwords |
|
||||
| `GOOGLE_CLIENT_ID` / `GOOGLE_CLIENT_SECRET` | From Google Cloud Console |
|
||||
|
||||
---
|
||||
|
||||
## Health Check
|
||||
|
||||
`GET /api/v1/health` returns `{ "status": "ok" }`. This is a **shallow** check — it only confirms the Node process is responding; it does not verify database connectivity or Redis availability. Docker healthcheck polls this endpoint every 15 s; the `frontend` service waits for the API to be healthy before starting.
|
||||
|
||||
---
|
||||
|
||||
## Logging
|
||||
|
||||
No structured logging is configured. The application uses NestJS's default logger (`console.log`/`console.error`). In production, logs go to stdout and are captured by Docker's logging driver (default: `json-file`). There is no log aggregation, no Sentry integration, and no OpenTelemetry instrumentation.
|
||||
|
||||
---
|
||||
|
||||
## Job Failure Handling
|
||||
|
||||
BullMQ jobs do not have automatic retry configured except where noted:
|
||||
|
||||
| Queue | Retry | Failed job retention |
|
||||
|---|---|---|
|
||||
| `youtube-sync` | None | Last 5 failed jobs kept (`removeOnFail: { count: 5 }`); completed jobs removed |
|
||||
| `render` | None | BullMQ default (kept until manually cleared) |
|
||||
| `lint` | None | BullMQ default |
|
||||
| `bulk-metadata` | None | BullMQ default |
|
||||
| `import` | None | BullMQ default |
|
||||
| `conflict-detection` | None | Last 10 failed jobs kept (`removeOnFail: 10`); last 10 completed kept (`removeOnComplete: 10`). Registered as a BullMQ repeatable by `ConflictDetectionScheduler` when `CONFLICT_DETECTION_ENABLED=true`. |
|
||||
|
||||
There is no dead-letter queue and no mechanism to notify users when a background job fails. A failed sync job means the user's video was not pushed to YouTube — the UI will continue to show "push pending" with no error indication. Failed jobs can be inspected directly in Redis or via a BullMQ dashboard (none is currently deployed).
|
||||
|
||||
---
|
||||
|
||||
## Rate Limiting
|
||||
|
||||
There is no rate limiting on any API endpoint. `@nestjs/throttler` is not installed. All routes are unthrottled.
|
||||
|
||||
---
|
||||
|
||||
## Database Backups
|
||||
|
||||
No backup strategy is configured. PostgreSQL data lives in the `postgres_data` Docker volume on the host. There is no automated backup job, no pg_dump schedule, and no offsite backup.
|
||||
|
||||
To back up manually:
|
||||
|
||||
```bash
|
||||
docker exec <postgres_container> pg_dump -U $POSTGRES_USER $POSTGRES_DB > backup.sql
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## TOKEN_ENCRYPTION_KEY Rotation
|
||||
|
||||
YouTube OAuth tokens are AES-256 encrypted in the database using `TOKEN_ENCRYPTION_KEY`. **Changing this key breaks all channel connections** — all stored tokens become unreadable and every connected channel must be fully re-authenticated by its owner.
|
||||
|
||||
There is no tooling for key rotation. If rotation is required (e.g. key compromise), the procedure is:
|
||||
|
||||
1. Take the application offline.
|
||||
2. Write a one-off migration script that decrypts every `Channel.accessToken` and `Channel.refreshToken` with the old key and re-encrypts them with the new key.
|
||||
3. Update `TOKEN_ENCRYPTION_KEY` in `.env`.
|
||||
4. Bring the application back online.
|
||||
|
||||
Until such a script is written, **key rotation = forced re-authentication for all channel owners**. See also the [[01 - Technical Debt and Future Work]] backlog.
|
||||
@@ -0,0 +1,82 @@
|
||||
# Verifying Changes
|
||||
|
||||
Commands to check your work after making changes. Run these before considering a task done.
|
||||
|
||||
---
|
||||
|
||||
## Backend
|
||||
|
||||
Run from the `backend/` directory.
|
||||
|
||||
### Type check + compile
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
`nest build` compiles all TypeScript and surfaces type errors. This is the primary way to verify backend changes are type-correct. Fix all errors before finishing.
|
||||
|
||||
### Lint
|
||||
|
||||
```bash
|
||||
npm run lint
|
||||
```
|
||||
|
||||
Runs ESLint with auto-fix on `src/**/*.ts`. Run after making changes to catch style violations. If auto-fix changes files, review the diff.
|
||||
|
||||
### Tests
|
||||
|
||||
```bash
|
||||
npm run test
|
||||
```
|
||||
|
||||
Runs the Jest test suite. Run when modifying shared services, the render engine, or any logic that has existing tests.
|
||||
|
||||
---
|
||||
|
||||
## Frontend
|
||||
|
||||
Run from the `frontend/` directory.
|
||||
|
||||
### Type check + build
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
`next build` compiles TypeScript and runs the full Next.js production build. It catches type errors, missing imports, and invalid JSX. The most thorough check available for the frontend.
|
||||
|
||||
### Lint
|
||||
|
||||
```bash
|
||||
npm run lint
|
||||
```
|
||||
|
||||
Runs `next lint` (ESLint). Run after making changes to catch import order violations, unused variables, and React-specific issues.
|
||||
|
||||
---
|
||||
|
||||
## After schema changes
|
||||
|
||||
After any modification to `backend/prisma/schema.prisma`:
|
||||
|
||||
```bash
|
||||
# Stop the backend and worker first, then:
|
||||
npx prisma generate # regenerate the Prisma client
|
||||
npx prisma migrate deploy # apply pending migrations
|
||||
# Restart both backend processes
|
||||
```
|
||||
|
||||
If you used `migrate dev` locally to create a migration file, commit the generated file in `backend/prisma/migrations/`.
|
||||
|
||||
---
|
||||
|
||||
## Checklist
|
||||
|
||||
| Change type | Commands to run |
|
||||
|---|---|
|
||||
| Backend logic change | `npm run build` → `npm run lint` |
|
||||
| Render engine / shared service | `npm run build` → `npm run test` |
|
||||
| Prisma schema change | `npx prisma generate` → `npx prisma migrate deploy` → `npm run build` |
|
||||
| Frontend component change | `npm run build` → `npm run lint` |
|
||||
| Both frontend and backend | Run both sets above |
|
||||
Reference in New Issue
Block a user