feat: upload current kino project
This commit is contained in:
12
.dockerignore
Normal file
12
.dockerignore
Normal file
@@ -0,0 +1,12 @@
|
||||
.venv/
|
||||
**/.venv/
|
||||
__pycache__/
|
||||
**/__pycache__/
|
||||
.pytest_cache/
|
||||
**/.pytest_cache/
|
||||
*.pyc
|
||||
*.pyo
|
||||
backend/*.egg-info/
|
||||
backend/**/*.egg-info/
|
||||
plugin.video.xstream/
|
||||
script.module.xstreamscraper/
|
||||
13
.env.example
Normal file
13
.env.example
Normal file
@@ -0,0 +1,13 @@
|
||||
KINOPROJEKT_DB=/data/kino.sqlite3
|
||||
KINOPROJEKT_MEDIA_ROOT=/jellyfin
|
||||
KINOPROJEKT_TMP=/data/tmp
|
||||
JELLYFIN_URL=http://jellyfin:8096
|
||||
# JELLYFIN_API_KEY belongs in the protected target-host .env or Vaultwarden, never in Nextcloud notes.
|
||||
JELLYFIN_API_KEY=
|
||||
DEFAULT_MAX_HEIGHT=1080
|
||||
MAX_DOWNLOAD_BYTES=15000000000
|
||||
KINOPROJEKT_ADMIN_USER=admin
|
||||
# Generate with: python -c "from app.services.auth import hash_password; import getpass; print(hash_password(getpass.getpass()))"
|
||||
KINOPROJEKT_ADMIN_PASSWORD_HASH=
|
||||
# Generate with: python -c "import secrets; print(secrets.token_urlsafe(48))"
|
||||
KINOPROJEKT_SESSION_SECRET=
|
||||
37
.gitignore
vendored
Normal file
37
.gitignore
vendored
Normal file
@@ -0,0 +1,37 @@
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
.pytest_cache/
|
||||
.mypy_cache/
|
||||
.ruff_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
*.egg-info/
|
||||
.venv/
|
||||
venv/
|
||||
|
||||
# Runtime/local data
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
data/
|
||||
backend/data/
|
||||
*.sqlite
|
||||
*.sqlite3
|
||||
*.db
|
||||
|
||||
# Media/download staging
|
||||
backend/app/web/downloads/*
|
||||
!backend/app/web/downloads/.gitkeep
|
||||
|
||||
# Node/build artifacts
|
||||
node_modules/
|
||||
dist/
|
||||
build/
|
||||
|
||||
# OS/editor
|
||||
.DS_Store
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
530
Botomir-Status.md
Normal file
530
Botomir-Status.md
Normal file
@@ -0,0 +1,530 @@
|
||||
# Botomir-Status
|
||||
|
||||
Aktualisiert: 2026-06-28 20:39 CEST
|
||||
|
||||
## Update 2026-06-28 20:39 CEST
|
||||
|
||||
- Vor Umbau lokales Fallback-Backup erstellt:
|
||||
|
||||
```text
|
||||
/root/botomir-work/backups/kino-projekt-current-before-extension-20260628-202959.tgz
|
||||
sha256: fe623994047e59e6db886a3637098244342ffa16e813c1b0d8c1d81228222255
|
||||
```
|
||||
|
||||
- Neue generische Browser-Extension `Kino Capture` implementiert, orientiert an der Architektur von Video DownloadHelper:
|
||||
- Manifest V3
|
||||
- beobachtet `webRequest`/Response-Header im aktiven Tab
|
||||
- sammelt `.m3u8`, `.mpd`, `.mp4`, `.webm`, `.mkv`, `.mov`, `.m4v` und `video/*`/MPEGURL/DASH Content-Types
|
||||
- ergänzt DOM-/Performance-Scan per Content-Script
|
||||
- Popup zeigt Kandidaten und öffnet die Kino-App mit `?capture=...`
|
||||
- App-URL ist im Popup konfigurierbar, Default `http://192.168.178.222:8099/`
|
||||
- keine hoster-spezifischen Bypässe/DRM-/CAPTCHA-/Token-Umgehung
|
||||
- Kino-App-UI um Download-Link und Installationshinweise ergänzt.
|
||||
- Extension-ZIP gebaut und statisch ausgeliefert:
|
||||
|
||||
```text
|
||||
/static/downloads/kino-capture-extension.zip
|
||||
sha256 lokal: 6c7bbceac8c8b85cc144fdd9f1d7de68284fd0eee7ecb2e5d6b60c5e7b62f3a9
|
||||
```
|
||||
|
||||
- Checks:
|
||||
|
||||
```text
|
||||
node --check service_worker.js/content.js/popup.js
|
||||
python3 -m json.tool manifest.json
|
||||
pytest -q
|
||||
# 38 passed, 1 warning in 0.75s
|
||||
```
|
||||
|
||||
- Deployment auf `192.168.178.222` durchgeführt.
|
||||
- Verifikation:
|
||||
|
||||
```text
|
||||
GET http://127.0.0.1:8099/health
|
||||
# {"status":"ok"}
|
||||
|
||||
GET http://192.168.178.222:8099/static/downloads/kino-capture-extension.zip
|
||||
# HTTP/1.1 200 OK
|
||||
# content-type: application/zip
|
||||
# content-length: 6535
|
||||
|
||||
GET http://192.168.178.222:8099/
|
||||
# enthält: Kino-Capture Extension herunterladen
|
||||
```
|
||||
|
||||
## Update 2026-06-28 20:02 CEST
|
||||
|
||||
- Browser-Capture-Bookmarklet robuster gemacht:
|
||||
- wenn `window.open(...)` durch Popup-Blocker/Browser-Schutz nicht öffnet, navigiert es jetzt als Fallback den aktuellen Tab zur Kino-App
|
||||
- wenn im aktuellen Tab keine Medien-URL gefunden wurde, erscheint ein Hinweis/Confirm statt scheinbar „nichts passiert“
|
||||
- Regex akzeptiert nun Query **und Fragment** nach Medien-Endungen (`.m3u8?…`, `.mp4#…`)
|
||||
- Tests weiterhin grün:
|
||||
|
||||
```text
|
||||
pytest -q
|
||||
# 38 passed, 1 warning in 0.75s
|
||||
```
|
||||
|
||||
- Deployment auf `192.168.178.222` durchgeführt.
|
||||
- Verifikation:
|
||||
|
||||
```text
|
||||
GET http://127.0.0.1:8099/health
|
||||
# {"status":"ok"}
|
||||
|
||||
GET /static/app.js
|
||||
# enthält: if(!opened)location.href=dest
|
||||
# enthält: Keine Medien-URL
|
||||
```
|
||||
|
||||
## Update 2026-06-28 19:47 CEST
|
||||
|
||||
- Analyzer generisch erweitert, ohne hoster-/schutzspezifische Umgehungen einzubauen:
|
||||
- erkennt jetzt auch JS-escaped Medien-URLs wie `https:\/\/...\/master.m3u8`
|
||||
- erkennt unquoted absolute Medien-URLs in Inline-JS/Config-Objekten
|
||||
- normalisiert HTML-Entities und escaped slashes vor der Kandidatenprüfung
|
||||
- Fehlertext verbessert: Wenn eine HTML-Seite erreichbar ist, aber automatisch kein importierbarer Medienkandidat gefunden wird, kommt jetzt ein Hinweis auf `Seite analysieren` bzw. Browser-Capture statt nur `text/html`.
|
||||
- Regressionstest ergänzt für HTML/JS-Extraktion.
|
||||
|
||||
```text
|
||||
cd /root/botomir-work/kino-projekt-current/backend
|
||||
. .venv/bin/activate
|
||||
pytest -q
|
||||
# Ergebnis: 38 passed, 1 warning in 0.76s
|
||||
```
|
||||
|
||||
- Deployment auf Jellyfin-LXC `192.168.178.222` durchgeführt.
|
||||
- Zielhost-Verifikation:
|
||||
|
||||
```text
|
||||
extract_media_urls(...)
|
||||
# ['https://test-streams.mux.dev/x36xhzz/x36xhzz.m3u8?token=1',
|
||||
# 'https://cdn.example.org/movie/manifest.mpd?x=1']
|
||||
|
||||
GET http://127.0.0.1:8099/health
|
||||
# {"status":"ok"}
|
||||
|
||||
POST /api/probe mit öffentlicher HLS-Test-URL
|
||||
# provider=mediathek, title=x36xhzz.m3u8, allowed=true
|
||||
```
|
||||
|
||||
## Update 2026-06-28 19:31 CEST
|
||||
|
||||
- Fehler `URL is reachable, but content type/extension is not accepted: text/html` behoben.
|
||||
- Ursache: Beim Import/Probe wurde teilweise noch die HTML-Playerseite selbst an den Import-Provider gegeben. HTML-Seiten sind keine Mediendatei; der Provider lehnte sie korrekt ab, statt den zuvor gefundenen `.m3u8`/Medienkandidaten zu nutzen.
|
||||
- Fix: `/api/probe` und `/api/imports` versuchen bei `text/html` jetzt automatisch die Seitenanalyse, wählen den besten erlaubten Kandidaten (`direct_video`, `hls_manifest`, `dash_manifest`) und speichern/importieren dessen Medien-URL. Die ursprüngliche Seiten-URL wird in den Job-Metadaten als `source_page_url` erhalten.
|
||||
- Regressionstest ergänzt: HTML-Seite mit erkanntem HLS-Kandidaten wird zu dessen `.m3u8`-URL aufgelöst.
|
||||
|
||||
```text
|
||||
cd /root/botomir-work/kino-projekt-current/backend
|
||||
. .venv/bin/activate
|
||||
pytest -q
|
||||
# Ergebnis: 37 passed, 1 warning in 0.75s
|
||||
```
|
||||
|
||||
- Deployment auf Jellyfin-LXC `192.168.178.222` durchgeführt und Dienst geprüft:
|
||||
|
||||
```text
|
||||
importable_kinds ['dash_manifest', 'direct_video', 'hls_manifest']
|
||||
systemctl is-active kino-projekt.service
|
||||
# active
|
||||
GET http://127.0.0.1:8099/health
|
||||
# {"status":"ok"}
|
||||
|
||||
POST /api/probe mit öffentlicher HLS-Test-URL
|
||||
# Ergebnis: provider=mediathek, title=x36xhzz.m3u8, allowed=true
|
||||
```
|
||||
|
||||
## Update 2026-06-28 19:20 CEST
|
||||
|
||||
- Fehler beim Import von HLS/DASH-Kandidaten behoben: `.m3u8`/`.mpd` sind nur Manifeste/Playlists und dürfen nicht per HTTP direkt als Datei nach Jellyfin kopiert werden.
|
||||
- `MediathekProvider.download()` erkennt jetzt HLS/DASH-Manifeste und lädt sie über `yt-dlp` + ffmpeg, mit `--remux-video mp4`, statt die wenige-KB-Manifestdatei zu speichern.
|
||||
- Direkte Medienlinks wie `.mp4` bleiben beim bisherigen HTTP-Streaming-Pfad.
|
||||
- Regressionstests ergänzt:
|
||||
|
||||
```text
|
||||
cd /root/botomir-work/kino-projekt-current/backend
|
||||
. .venv/bin/activate
|
||||
pytest -q
|
||||
# Ergebnis: 36 passed, 1 warning in 0.75s
|
||||
```
|
||||
|
||||
- Deployment auf Jellyfin-LXC `192.168.178.222` durchgeführt. Verifikation auf dem Ziel:
|
||||
|
||||
```text
|
||||
yt-dlp command: /opt/kino-projekt/venv/bin/python -m yt_dlp
|
||||
ffmpeg: vorhanden
|
||||
m3u8_detected: True
|
||||
systemctl is-active kino-projekt.service
|
||||
# active
|
||||
GET http://127.0.0.1:8099/health
|
||||
# {"status":"ok"}
|
||||
|
||||
/opt/kino-projekt/venv/bin/python -m yt_dlp --simulate ... x36xhzz.m3u8
|
||||
# m3u8_native mp4
|
||||
```
|
||||
|
||||
## Update 2026-06-28 19:08 CEST
|
||||
|
||||
- Jellyfin-API-Key aus dem bestehenden Vaultwarden-Item `Kino-Projekt App-Login` Feld `jellyfin-api` gelesen, ohne den Wert auszugeben.
|
||||
- `/etc/kino-projekt.env` auf Jellyfin-LXC `192.168.178.222` ergänzt/aktualisiert:
|
||||
- `JELLYFIN_URL=http://127.0.0.1:8096`
|
||||
- `JELLYFIN_API_KEY=[REDACTED]`
|
||||
- Vorheriges Env-Backup angelegt: `/etc/kino-projekt.env.bak-jellyfin-*`.
|
||||
- Jellyfin Refresh API direkt mit dem Key geprüft:
|
||||
|
||||
```text
|
||||
POST http://127.0.0.1:8096/Library/Refresh
|
||||
# Ergebnis: HTTP 204
|
||||
```
|
||||
|
||||
- `kino-projekt.service` neugestartet und geprüft:
|
||||
|
||||
```text
|
||||
systemctl is-active kino-projekt.service
|
||||
# active
|
||||
|
||||
GET http://127.0.0.1:8099/health
|
||||
# {"status":"ok"}
|
||||
```
|
||||
|
||||
- Systemd nutzt weiterhin `EnvironmentFile=/etc/kino-projekt.env`; vorhandene Env-Keys wurden geprüft, ohne Werte auszugeben.
|
||||
|
||||
## Update 2026-06-28 18:56 CEST
|
||||
|
||||
- Login-Problem untersucht: der Login über `https://kino.dasposchi.de` funktionierte, aber beim direkten LAN-Zugriff `http://192.168.178.222:8099` setzte die App immer ein `Secure`-Cookie. Browser speichern/senden `Secure`-Cookies über HTTP nicht zuverlässig; dadurch wirkte der Login im LAN technisch defekt.
|
||||
- Fix umgesetzt: Session-Cookie ist nur noch `Secure`, wenn der Request selbst HTTPS nutzt oder `X-Forwarded-Proto: https` gesetzt ist. Direkter LAN-HTTP-Zugriff bekommt ein nicht-`Secure` HttpOnly/SameSite=Lax-Cookie, Mesh/HTTPS bleibt `Secure`.
|
||||
- Tests ergänzt und ausgeführt:
|
||||
|
||||
```text
|
||||
cd /root/botomir-work/kino-projekt-current/backend
|
||||
. .venv/bin/activate
|
||||
pytest -q
|
||||
# Ergebnis: 34 passed, 1 warning in 0.73s
|
||||
```
|
||||
|
||||
- Deployment auf Jellyfin-LXC `192.168.178.222` durchgeführt, Dienst neugestartet und geprüft.
|
||||
|
||||
### Verifikation
|
||||
|
||||
```text
|
||||
GET http://192.168.178.222:8099/health
|
||||
# Ergebnis: HTTP 200 {"status":"ok"}
|
||||
|
||||
Login über http://192.168.178.222:8099/api/auth/login
|
||||
# Ergebnis: 204; Set-Cookie ohne Secure; /api/auth/status danach authenticated=true
|
||||
|
||||
Login über https://kino.dasposchi.de/api/auth/login via Mesh
|
||||
# Ergebnis: 204; Set-Cookie mit Secure; /api/auth/status danach authenticated=true
|
||||
```
|
||||
|
||||
## Update 2026-06-28 18:41 CEST
|
||||
|
||||
- Analyzer-Code aus dem separaten lokalen Stand in den aktuellen, deployed Provider-/Import-Stand gemergt, ohne die bestehende Auth-/Import-Architektur zurückzurollen.
|
||||
- Neu im deployed Dienst:
|
||||
- `POST /api/analyze` kombiniert HTML-Scan, `yt-dlp`-Analyse und Headless-Browser-Netzwerkbeobachtung.
|
||||
- `POST /api/browser-capture` validiert Kandidaten aus dem Bookmarklet/echten Browser.
|
||||
- WebUI enthält jetzt „Seite analysieren“, Browser-Capture-Bookmarklet und Kandidatenliste mit Dateigrößenanzeige.
|
||||
- SSRF-Schutz für Seitenanalyse/Downloads wurde ergänzt; private/interne Hosts werden geblockt.
|
||||
- uBlock-Origin-Pfadlogik ist aktiv; Browser-Analyse meldete `ublock_origin=true` im Smoke-Test.
|
||||
- Deployment auf Jellyfin-LXC `192.168.178.222` abgeschlossen. Vor dem Austausch wurde ein Backup unter `/opt/kino-projekt/backups/deploy-20260628-*/app-src-before-analyzer.tgz` angelegt.
|
||||
|
||||
### Verifikation in diesem Lauf
|
||||
|
||||
```text
|
||||
cd /root/botomir-work/kino-projekt-current/backend
|
||||
. .venv/bin/activate
|
||||
pytest -q
|
||||
# Ergebnis: 32 passed, 1 warning in 0.48s
|
||||
|
||||
GET http://192.168.178.222:8099/health
|
||||
# Ergebnis: HTTP 200 {"status":"ok"}
|
||||
|
||||
POST http://192.168.178.222:8099/api/browser-capture
|
||||
# Payload: filesamples MP4-Kandidat, Login-Cookie aus lokaler Passwortdatei
|
||||
# Ergebnis: HTTP 200, 1 Kandidat, source=browser-capture, kind=direct_video
|
||||
|
||||
POST http://192.168.178.222:8099/api/analyze
|
||||
# Payload: https://filesamples.com/samples/video/mp4/sample_640x360.mp4
|
||||
# Ergebnis: HTTP 200, 1 Kandidat, source=browser, kind=direct_video, ublock_origin=true
|
||||
|
||||
curl --interface tailscale0 --resolve kino.dasposchi.de:443:100.64.0.9 https://kino.dasposchi.de/health
|
||||
# Ergebnis: HTTP 200 {"status":"ok"}
|
||||
|
||||
POST https://kino.dasposchi.de/api/analyze über Mesh
|
||||
# Ergebnis: Login 204; Analyze 1 Kandidat, source=browser, kind=direct_video, ublock_origin=true
|
||||
```
|
||||
|
||||
### Hinweise
|
||||
|
||||
- Remote-Produktionsvenv hat keine Test-Extras (`pytest`) installiert; Tests wurden lokal im Arbeitsstand ausgeführt und der deployte Dienst per HTTP-Smoke-Test verifiziert.
|
||||
- Ein echter Import in `/jellyfin` wurde weiterhin nicht ausgelöst; verifiziert wurden Analyse, Kandidatenvalidierung, Auth und Dienstgesundheit.
|
||||
|
||||
## Update 2026-06-25 06:20 CEST
|
||||
|
||||
- Reverse-Proxy-Blocker für `kino.dasposchi.de` behoben: Nginx Proxy Manager auf dem Public-VPS hat jetzt einen eigenen Proxy-Host `kino.dasposchi.de` mit Let’s-Encrypt-Zertifikat und Headscale-Mesh-only Access List.
|
||||
- Für den Upstream wurde ein dedizierter Reverse-SSH-Tunnel vom Jellyfin-LXC zum VPS eingerichtet:
|
||||
- `kino-vps-tunnel.service` auf CT `100` (`jellyfin`) ist `active`.
|
||||
- VPS/NPM-Container `headscale-tunnel-sshd:8892` leitet auf `127.0.0.1:8099` im Jellyfin-LXC weiter.
|
||||
- NPM-Datenbank/Configs wurden vor Änderung gesichert: `/opt/infra/npm/backups/botomir-kino-20260625-060320/`.
|
||||
- Neues Zertifikat: `/etc/letsencrypt/live/npm-29/`, SAN `DNS:kino.dasposchi.de`, gültig bis `2026-09-23 03:05:22 UTC`.
|
||||
- Secret-Hygiene: App-Passwort wurde nicht ausgegeben; lokale Passwortdatei ist jetzt `0600`.
|
||||
|
||||
### Verifikation in diesem Lauf
|
||||
|
||||
```text
|
||||
# Public ohne Mesh: TLS/SNI repariert, aber bewusst blockiert
|
||||
GET http://kino.dasposchi.de/
|
||||
# Ergebnis: HTTP 301 -> https://kino.dasposchi.de/
|
||||
|
||||
GET https://kino.dasposchi.de/
|
||||
# Ergebnis: HTTP 403 openresty (Headscale-Mesh-only Access List greift)
|
||||
|
||||
# Mesh-Zugriff über public Hostname und VPS-Mesh-IP
|
||||
curl --interface tailscale0 --resolve kino.dasposchi.de:443:100.64.0.9 https://kino.dasposchi.de/
|
||||
# Ergebnis: HTTP 200, 2217 bytes, WebUI erreichbar
|
||||
|
||||
curl --interface tailscale0 --resolve kino.dasposchi.de:443:100.64.0.9 https://kino.dasposchi.de/health
|
||||
# Ergebnis: HTTP 200 {"status":"ok"}
|
||||
|
||||
curl --interface tailscale0 --resolve kino.dasposchi.de:443:100.64.0.9 https://kino.dasposchi.de/api/auth/status
|
||||
# Ergebnis: HTTP 200 {"enabled":true,"authenticated":false,"username":null}
|
||||
|
||||
# App-Login ohne Secret-Ausgabe
|
||||
POST https://kino.dasposchi.de/api/auth/login (über Mesh, Passwort aus root-only lokaler Datei)
|
||||
# Ergebnis: HTTP 204
|
||||
|
||||
GET https://kino.dasposchi.de/api/auth/status (mit Session-Cookie)
|
||||
# Ergebnis: HTTP 200 {"enabled":true,"authenticated":true,"username":"admin"}
|
||||
|
||||
POST https://kino.dasposchi.de/api/probe {"url":"https://archive.org/details/BigBuckBunny_328"} (mit Session-Cookie)
|
||||
# Ergebnis: HTTP 200, provider=internet_archive, title="Big Buck Bunny", external_id="BigBuckBunny_328", allowed=True
|
||||
|
||||
# Zielhost lokal
|
||||
systemctl is-active kino-projekt.service
|
||||
# Ergebnis: active
|
||||
systemctl is-active kino-vps-tunnel.service
|
||||
# Ergebnis: active
|
||||
curl http://127.0.0.1:8099/health
|
||||
# Ergebnis: HTTP 200 {"status":"ok"}
|
||||
|
||||
# VPS/NPM
|
||||
nginx -t im npm-server-app Container
|
||||
# Ergebnis: syntax is ok; test is successful
|
||||
proxy_host DB-Zeile:
|
||||
# id=15, domain=["kino.dasposchi.de"], upstream=headscale-tunnel-sshd:8892, access_list_id=4, certificate_id=29, ssl_forced=1, enabled=1
|
||||
```
|
||||
|
||||
### Aktueller Stand
|
||||
|
||||
- Akzeptanzkriterien `public blockiert`, `Mesh erreichbar`, `App-Login aktiv`, `Probe legaler Quelle`, `Jellyfin-Dienst gesund` sind erfüllt.
|
||||
- Nicht durchgeführt: echter Download/Import in die Jellyfin-Bibliothek. Grund: Das würde reale Mediendateien in `/jellyfin` schreiben; ohne explizite Auswahl/Bestätigung wurde nur eine Probe gegen eine legale Quelle durchgeführt.
|
||||
|
||||
## Update 2026-06-25 02:00 CEST
|
||||
|
||||
- Zielhost-Deployment auf dem Jellyfin-LXC `jellyfin` (`192.168.178.222`) durchgeführt: bestehendes `/opt/kino-projekt/app-src` wurde gesichert und durch den aktuellen legal-source FastAPI-Stand ersetzt.
|
||||
- Vor dem Austausch wurde ein Backup erstellt: `/opt/kino-projekt/backups/deploy-20260625-015600/` mit `app-src.tar.gz` und der geschützten `/etc/kino-projekt.env`.
|
||||
- Der bestehende systemd-Dienst `kino-projekt.service` wurde neu gestartet und läuft weiter auf Port `8099`.
|
||||
- `/jellyfin/YouTube-Mediathek` ist vorhanden und passend gesetzt: `2775 kino:media`.
|
||||
- Kompatibilität ergänzt: `backend/app/config.py` akzeptiert jetzt das bereits auf dem Zielhost vorhandene Legacy-Setting `KINOPROJEKT_DB_URL=sqlite:////...`, ohne neue Secrets oder `.env`-Werte in Nextcloud abzulegen.
|
||||
|
||||
### Verifikation in diesem Lauf
|
||||
|
||||
```text
|
||||
cd /root/botomir-work/kino-projekt-current/backend
|
||||
. .venv/bin/activate
|
||||
pytest -q
|
||||
# Ergebnis: 27 passed, 1 warning in 0.45s
|
||||
|
||||
cd /root/botomir-work/kino-projekt-current
|
||||
docker build -t kino-projekt-botomir:cron-20260625-deploy .
|
||||
# Ergebnis: Image gebaut: sha256:e78ca77fb8434ea10ff4622489f4543a31f2d682c688ff72f71f4c429b832669
|
||||
|
||||
Container-Smoke-Test:
|
||||
GET http://127.0.0.1:18085/health
|
||||
# Ergebnis: HTTP 200 {"status":"ok"}
|
||||
GET http://127.0.0.1:18085/api/auth/status
|
||||
# Ergebnis: HTTP 200 {"enabled":false,"authenticated":true,"username":null}
|
||||
POST http://127.0.0.1:18085/api/probe {"url":"http://localhost/video.mp4"}
|
||||
# Ergebnis: HTTP 400, private/loopback Ziele werden abgelehnt
|
||||
|
||||
Zielhost:
|
||||
systemctl is-active kino-projekt.service
|
||||
# Ergebnis: active
|
||||
GET http://192.168.178.222:8099/health
|
||||
# Ergebnis: HTTP 200 {"status":"ok"}
|
||||
GET http://192.168.178.222:8099/api/auth/status
|
||||
# Ergebnis: HTTP 200 {"enabled":true,"authenticated":false,"username":null}
|
||||
GET http://192.168.178.222:8099/
|
||||
# Ergebnis: HTTP 200, 2217 bytes, neue WebUI erreichbar
|
||||
POST http://192.168.178.222:8099/api/probe ohne Login
|
||||
# Ergebnis: HTTP 401 {"detail":"Login required"}
|
||||
stat /jellyfin/YouTube-Mediathek
|
||||
# Ergebnis: 2775 kino:media /jellyfin/YouTube-Mediathek
|
||||
Jellyfin Health: GET http://192.168.178.222:8096/health
|
||||
# Ergebnis: HTTP 200 Healthy
|
||||
```
|
||||
|
||||
### Noch offen / Blocker
|
||||
|
||||
- Öffentliches `https://kino.dasposchi.de/` ist noch nicht korrekt auf den Dienst geroutet: aktueller Probe-Fehler `TLS connect error ... tlsv1 unrecognized name`/HTTP `000`. Der lokale Dienst auf dem Jellyfin-LXC läuft; als nächstes muss der Nginx-Proxy-Manager-/Edge-VHost für `kino.dasposchi.de` auf `192.168.178.222:8099` mit Mesh-only Access List eingerichtet bzw. repariert werden.
|
||||
- Ein echter Import mit App-Login/Jellyfin-Refresh wurde noch nicht ausgelöst, weil kein Klartext-App-Passwort in Nextcloud liegt und Secrets nicht ausgegeben wurden. Die Auth-Schicht ist aktiv (`enabled=true`) und blockiert unauthentifizierte API-Aufrufe korrekt.
|
||||
|
||||
### Dateien mit Änderungen in diesem Lauf
|
||||
|
||||
- `backend/app/config.py` — Legacy-Kompatibilität für `KINOPROJEKT_DB_URL` ergänzt.
|
||||
- Zielhost `/opt/kino-projekt/app-src/` — aktueller App-Stand deployed.
|
||||
- `Botomir-Status.md` — diese Statusnotiz aktualisiert.
|
||||
|
||||
## Update 2026-06-24 21:41 CEST
|
||||
|
||||
- Jellyfin-Zielprofile umgesetzt: `film`, `serie` und `clip` erzeugen jetzt serverseitig die zur verifizierten Bibliotheksstruktur passenden Zielordner unter `/jellyfin/Filme`, `/jellyfin/Serien/.../Staffel<N>` und `/jellyfin/YouTube-Mediathek`.
|
||||
- WebUI erweitert: Zielprofil-Radios sowie optionale Felder für Titel, Jahr, Serientitel, Staffel, Episode und Episodentitel werden beim Import an `POST /api/imports` gesendet.
|
||||
- Import-Request-Metadaten werden in SQLite mit der Source gespeichert, damit der Hintergrundjob die Zielpfad-Korrekturen ohne zusätzliche Secrets/Nextcloud-Daten verwenden kann.
|
||||
- README und Deployment-Doku beschreiben die Zielprofile und den anzulegenden `/jellyfin/YouTube-Mediathek`-Ordner.
|
||||
|
||||
### Verifikation in diesem Lauf
|
||||
|
||||
```text
|
||||
cd /root/botomir-work/kino-projekt-current/backend
|
||||
. .venv/bin/activate
|
||||
pytest -q
|
||||
# Ergebnis: 27 passed, 1 warning in 0.49s
|
||||
|
||||
cd /root/botomir-work/kino-projekt-current
|
||||
docker build -t kino-projekt-botomir:cron-20260624-profiles .
|
||||
# Ergebnis: Image gebaut: sha256:574c9013385541f774a160eb99b14ab139cfd99c8a7e0aec4119daf41b4a18f8
|
||||
# Build-Kontext: 26.66 kB
|
||||
|
||||
docker run -d --rm -p 127.0.0.1:18084:8080 -e KINOPROJEKT_DB=/tmp/kino.sqlite3 -e KINOPROJEKT_MEDIA_ROOT=/tmp/jellyfin kino-projekt-botomir:cron-20260624-profiles
|
||||
curl http://127.0.0.1:18084/health
|
||||
# Ergebnis: HTTP 200 {"status":"ok"}
|
||||
GET /
|
||||
# Ergebnis: HTTP 200, 2217 bytes, Zielprofil-Steuerelemente vorhanden
|
||||
GET /static/app.js
|
||||
# Ergebnis: HTTP 200, 4973 bytes, `currentImportPayload` sendet `target_profile`
|
||||
```
|
||||
|
||||
### Dateien mit Änderungen in diesem Lauf
|
||||
|
||||
- `backend/app/services/paths.py` — Jellyfin-Zielprofil-Pfade (`film`, `serie`, `clip`) ergänzt.
|
||||
- `backend/app/schemas.py` — ImportRequest um Zielprofil- und Korrekturfelder erweitert.
|
||||
- `backend/app/main.py` — Import-Request-Metadaten werden in `Source.metadata_json` persistiert; Jobs verwenden das Zielprofil.
|
||||
- `backend/app/services/downloader.py` — Hintergrundimport nutzt die Zielprofil-Pfade statt freier Bibliotheksnamen.
|
||||
- `backend/app/web/index.html`, `backend/app/web/app.js`, `backend/app/web/style.css` — WebUI-Zielprofil-Auswahl und optionale Korrekturfelder.
|
||||
- `backend/tests/test_path_safety.py` — Regressionstests für Film-/Serien-/Clip-Pfade.
|
||||
- `README.md`, `docs/deployment.md`, `Botomir-Status.md` — Doku/Status aktualisiert.
|
||||
|
||||
### Offene nächste Schritte
|
||||
|
||||
1. Zielhost-Deployment vorbereiten: geschützte `.env`/Vaultwarden-Werte für App-Login und Jellyfin setzen; keine Secrets in Nextcloud ablegen.
|
||||
2. Auf dem Zielhost `/jellyfin/YouTube-Mediathek` mit Owner/Gruppe analog zu `/jellyfin/Filme` und `/jellyfin/Serien` anlegen.
|
||||
3. Mesh-only Veröffentlichung `kino.dasposchi.de` und danach echten legalen Testimport plus Jellyfin-Refresh verifizieren.
|
||||
|
||||
## Update 2026-06-24 17:34 CEST
|
||||
|
||||
- Internet-Archive-Provider ergänzt: `archive.org/details/<identifier>`, `archive.org/download/<identifier>/...` und `archive.org/metadata/<identifier>` werden jetzt vor dem generischen Direktlink-Provider erkannt.
|
||||
- Metadaten werden über die öffentliche Archive-Metadata-API abgefragt; Downloads wählen nur plausible öffentliche Medien-Dateien und laufen danach durch den bestehenden SSRF-geschützten Downloadpfad.
|
||||
- Provider-Policy und README wurden um Internet Archive als erlaubte Quelle erweitert.
|
||||
|
||||
### Verifikation in diesem Lauf
|
||||
|
||||
```text
|
||||
cd /root/botomir-work/kino-projekt-current/backend
|
||||
. .venv/bin/activate
|
||||
pytest -q
|
||||
# Ergebnis: 23 passed, 1 warning in 0.43s
|
||||
|
||||
python - <<'PY'
|
||||
import asyncio
|
||||
from app.providers.internet_archive import InternetArchiveProvider
|
||||
async def main():
|
||||
meta = await InternetArchiveProvider().probe('https://archive.org/details/BigBuckBunny_328')
|
||||
print(meta.model_dump())
|
||||
asyncio.run(main())
|
||||
PY
|
||||
# Ergebnis: provider=internet_archive, title='Big Buck Bunny', external_id='BigBuckBunny_328'
|
||||
|
||||
docker build -t kino-projekt-botomir:cron-20260624-ia .
|
||||
# Ergebnis: Image gebaut: sha256:4c45d54599dcde1a5d212213c637190000cd2086e9b0671e6499b55ffe723135
|
||||
# Build-Kontext: 19.41 kB
|
||||
|
||||
docker run -d --rm -p 127.0.0.1:18083:8080 -e KINOPROJEKT_DB=/tmp/kino.sqlite3 kino-projekt-botomir:cron-20260624-ia
|
||||
curl http://127.0.0.1:18083/health
|
||||
# Ergebnis: HTTP 200 {"status":"ok"}
|
||||
curl -H 'content-type: application/json' -d '{"url":"https://archive.org/details/BigBuckBunny_328"}' http://127.0.0.1:18083/api/probe
|
||||
# Ergebnis: provider=internet_archive, title='Big Buck Bunny', external_id='BigBuckBunny_328', allowed=True
|
||||
```
|
||||
|
||||
### Dateien mit Änderungen in diesem Lauf
|
||||
|
||||
- `backend/app/providers/internet_archive.py` — neuer Provider für öffentliche Internet-Archive-Items.
|
||||
- `backend/app/main.py` — Provider registriert, vor dem generischen Direktlink-Provider.
|
||||
- `backend/app/providers/mediathek.py` — Archive-URLs vom generischen Direktlink-Provider ausgeschlossen.
|
||||
- `backend/tests/test_internet_archive_provider.py` — Unit-Tests für Erkennung, Identifier-Parsing, Medienauswahl und Fehlfall ohne Medien.
|
||||
- `backend/tests/test_provider_detection.py` — Detection-Regression für Internet Archive ergänzt.
|
||||
- `README.md`, `docs/provider-policy.md`, `Botomir-Status.md` — Dokumentation aktualisiert.
|
||||
|
||||
## Aktueller Stand
|
||||
|
||||
- Projekt wurde aus `Projekte/ToDo/` nach `Projekte/Angefangen/` verschoben.
|
||||
- Initialer, rechtlich sauberer MVP-Code wurde ergänzt. Der hochgeladene xstream-Code bleibt nur Analyse-Anhang; Piracy-Scraper/Hoster werden nicht implementiert.
|
||||
- Umgesetzt: FastAPI-Healthcheck, statische Minimal-WebUI, Provider-Abstraktion, YouTube-Erkennung/Probe über `yt-dlp` ohne Cookies/Umgehungsoptionen, konservative Mediathek-/Direktlink-Prüfung mit SSRF-Schutz, SQLite-Modelle, Pfadsicherheitshelfer, Dockerfile, Compose-Datei, `.dockerignore` und Doku.
|
||||
- Import-Job-MVP: `POST /api/imports`, `GET /api/imports`, `GET /api/imports/{id}`, `GET /api/sources`, Hintergrund-Runner, sichere Zielpfad-Kopie, YouTube-/Direktlink-Download-Wrapper, optionaler Jellyfin-Library-Refresh über `JELLYFIN_API_KEY`, UI-Button zum Starten eines Imports und Downloader-Test.
|
||||
- App-Login: Argon2-Passwort-Hash, signiertes Session-Cookie (`HttpOnly`, `Secure`, `SameSite=Lax`), `POST /api/auth/login`, `POST /api/auth/logout`, `GET /api/auth/status`, UI-Login-Formular, `.env.example` und Tests. API-Routen sind geschützt, sobald `KINOPROJEKT_ADMIN_PASSWORD_HASH` und `KINOPROJEKT_SESSION_SECRET` gesetzt sind.
|
||||
- Neu in diesem Lauf: Import-Fortschritt verbessert. `yt-dlp` läuft mit `--newline`; Download-Prozentzeilen werden geparst und in den Job-Fortschritt gemappt. Direkte Mediathek-/HTTP-Downloads melden Fortschritt über `content-length`, wenn verfügbar. Die WebUI pollt `GET /api/imports/{id}` alle 2 Sekunden und zeigt Status, Prozent, Zielpfad sowie Hinweise/Fehler live an.
|
||||
- Secrets wurden nicht in Nextcloud abgelegt. `.env.example` enthält nur leere Platzhalter und Befehle zum lokalen Erzeugen von Hash/Secret.
|
||||
|
||||
## Verifikation lokal im Hermes-Container
|
||||
|
||||
```text
|
||||
cd /root/botomir-work/kino-projekt-current/backend
|
||||
. .venv/bin/activate
|
||||
pytest -q
|
||||
# Ergebnis: 17 passed, 1 warning in 0.41s
|
||||
```
|
||||
|
||||
Docker-Build:
|
||||
|
||||
```text
|
||||
cd /root/botomir-work/kino-projekt-current
|
||||
docker build -t kino-projekt-botomir:cron-20260624-progress .
|
||||
# Ergebnis: Image gebaut: sha256:f6b1fc960f939d2cda658aefc1df7e0c29f5d73bc2a4ca04a2e27fbbea897b0b
|
||||
# Build-Kontext: 21.22 kB; lokale .venv/__pycache__/egg-info-Artefakte wurden durch .dockerignore ausgeschlossen.
|
||||
```
|
||||
|
||||
Container-Smoke-Test ohne Zielhost-Secrets:
|
||||
|
||||
```text
|
||||
docker run -d --rm -p 127.0.0.1:18082:8080 \
|
||||
-e KINOPROJEKT_DB=/tmp/kino.sqlite3 \
|
||||
kino-projekt-botomir:cron-20260624-progress
|
||||
|
||||
curl http://127.0.0.1:18082/health
|
||||
# Ergebnis: HTTP 200 {"status":"ok"}
|
||||
|
||||
curl http://127.0.0.1:18082/api/auth/status
|
||||
# Ergebnis: HTTP 200 {"enabled":false,"authenticated":true,"username":null}
|
||||
|
||||
curl -o /tmp/kino_index.html -w '%{http_code} bytes=%{size_download}\n' http://127.0.0.1:18082/
|
||||
# Ergebnis: 200 bytes=1212
|
||||
|
||||
curl http://127.0.0.1:18082/static/app.js | grep -q 'pollJob'
|
||||
# Ergebnis: yes
|
||||
```
|
||||
|
||||
## Dateien mit Änderungen in diesem Lauf
|
||||
|
||||
- `backend/app/providers/base.py` — Provider-Download-Signatur um optionalen Fortschritts-Callback erweitert.
|
||||
- `backend/app/providers/youtube.py` — `yt-dlp --newline`, Fortschrittsparser und laufendes Stream-Auslesen statt nur finalem `communicate()`.
|
||||
- `backend/app/providers/mediathek.py` — direkter HTTP-Download meldet Fortschritt anhand `content-length`.
|
||||
- `backend/app/services/downloader.py` — Provider-Fortschritt wird auf `progress=0.15..0.75` gemappt und in SQLite gespeichert.
|
||||
- `backend/app/web/app.js` — Job-Status wird live gepollt und gerendert.
|
||||
- `backend/tests/test_downloader.py` — Regressionstest für `yt-dlp`-Fortschrittsparser ergänzt.
|
||||
- `Botomir-Status.md` — diese Statusnotiz aktualisiert.
|
||||
|
||||
## Offene nächste Schritte
|
||||
|
||||
1. Zielhost-Deployment vorbereiten: `KINOPROJEKT_ADMIN_PASSWORD_HASH`, `KINOPROJEKT_SESSION_SECRET`, `JELLYFIN_API_KEY` und Zielpfade aus Vaultwarden/geschützter `.env` setzen; keine Secrets in Nextcloud ablegen.
|
||||
2. Mesh-only Deployment von `kino.dasposchi.de` vorbereiten/umsetzen, sobald Zielhost und Reverse-Proxy-Pfad eindeutig feststehen.
|
||||
3. Danach echten Import gegen einen legalen Testlink sowie Jellyfin-Refresh auf dem Zielhost verifizieren.
|
||||
7
Dockerfile
Normal file
7
Dockerfile
Normal file
@@ -0,0 +1,7 @@
|
||||
FROM python:3.12-slim
|
||||
WORKDIR /app/backend
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg ca-certificates && rm -rf /var/lib/apt/lists/*
|
||||
COPY backend/ ./
|
||||
RUN pip install --no-cache-dir .[test] yt-dlp
|
||||
EXPOSE 8080
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080"]
|
||||
632
Plan-Botomir.md
Normal file
632
Plan-Botomir.md
Normal file
@@ -0,0 +1,632 @@
|
||||
# Kino-Projekt Implementation Plan
|
||||
|
||||
> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Eine interne WebUI namens „Kino-Projekt“, mit der YouTube-Videos sowie Links aus legalen Mediatheken/öffentlich erlaubten Quellen geprüft, heruntergeladen/importiert, sauber für Jellyfin abgelegt und anschließend in Jellyfin aktualisiert werden.
|
||||
|
||||
**Architecture:** Das System besteht aus einer Mesh-only WebUI, einem FastAPI-Backend, einer SQLite-Datenbank, einer Job-Queue für Imports und einem Provider-System. Provider sind strikt getrennt: YouTube, Mediathek-/Direktlink-Import, Internet Archive und später weitere legale Quellen. Downloads laufen nur über geprüfte Provider und definierte Zielpfade; Jellyfin wird danach per API aktualisiert.
|
||||
|
||||
**Tech Stack:** Python 3.12+, FastAPI, SQLite, SQLModel oder SQLAlchemy, yt-dlp, httpx, Docker Compose, Jellyfin API, Nginx Proxy Manager/Headscale-Mesh.
|
||||
|
||||
---
|
||||
|
||||
## Scope
|
||||
|
||||
Der Projektname bleibt **Kino-Projekt**.
|
||||
|
||||
Der initiale, rechtlich saubere Scope ist:
|
||||
|
||||
1. YouTube-Videos, Playlists und Kanäle importieren, soweit öffentlich erreichbar und nicht DRM-/Login-/Paywall-geschützt.
|
||||
2. Links aus legalen Mediatheken oder erlaubten Quellen hinzufügen, z. B. öffentlich verfügbare ARD/ZDF/Arte/3sat-/Mediathek-URLs, Internet Archive, eigene Download-URLs oder andere Quellen mit klar erlaubter Nutzung.
|
||||
3. Downloads in Jellyfin-kompatibler Struktur ablegen.
|
||||
4. Jellyfin-Library-Refresh auslösen.
|
||||
5. UI zeigt Status, Quelle, Zielpfad und Fehler transparent an.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
Nicht implementieren:
|
||||
|
||||
- Scraping/Download von Piracy-Seiten wie Kinox/Kinoger/xstream-Hostern.
|
||||
- Umgehung von DRM, Paywalls, Logins, Geoblocking, Altersprüfungen oder Zugriffsbeschränkungen.
|
||||
- Cookie-Import oder Browser-Session-Nutzung als Standardfunktion.
|
||||
- Beliebige Shell-Parameter aus der UI.
|
||||
- Freie Schreibpfade außerhalb der konfigurierten Medienordner.
|
||||
|
||||
---
|
||||
|
||||
## Ziel-UX
|
||||
|
||||
Die WebUI soll sich wie ein internes „Kino-Dashboard“ anfühlen:
|
||||
|
||||
- URL oder Suchbegriff eingeben.
|
||||
- Quelle erkennen: YouTube, Mediathek, Direktlink, Internet Archive.
|
||||
- Metadaten anzeigen: Titel, Kanal/Sender, Dauer, Beschreibung, Thumbnail, Quelle.
|
||||
- Qualität/Zielbibliothek auswählen.
|
||||
- Import starten.
|
||||
- Fortschritt beobachten.
|
||||
- Nach Abschluss Link zu Jellyfin bzw. Zielpfad anzeigen.
|
||||
|
||||
Beispiel-Ziel-URL:
|
||||
|
||||
```text
|
||||
https://kino.dasposchi.de
|
||||
```
|
||||
|
||||
Die WebUI wird nur über das Headscale-Mesh erreichbar gemacht.
|
||||
|
||||
---
|
||||
|
||||
## Datenmodell
|
||||
|
||||
### Tabelle: `sources`
|
||||
|
||||
```python
|
||||
class Source(SQLModel, table=True):
|
||||
id: int | None = Field(default=None, primary_key=True)
|
||||
kind: str # youtube, mediathek, direct_url, internet_archive
|
||||
url: str
|
||||
title: str | None = None
|
||||
provider: str | None = None
|
||||
external_id: str | None = None
|
||||
metadata_json: str = "{}"
|
||||
created_at: datetime
|
||||
```
|
||||
|
||||
### Tabelle: `import_jobs`
|
||||
|
||||
```python
|
||||
class ImportJob(SQLModel, table=True):
|
||||
id: int | None = Field(default=None, primary_key=True)
|
||||
source_id: int = Field(foreign_key="source.id")
|
||||
status: str # queued, probing, downloading, postprocessing, refreshing, done, failed
|
||||
target_library: str
|
||||
target_path: str | None = None
|
||||
progress: float = 0.0
|
||||
error: str | None = None
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Provider-Konzept
|
||||
|
||||
Alle Quellen implementieren ein gemeinsames Interface.
|
||||
|
||||
```python
|
||||
class Provider(Protocol):
|
||||
name: str
|
||||
|
||||
def can_handle(self, url: str) -> bool: ...
|
||||
|
||||
async def probe(self, url: str) -> MediaMetadata: ...
|
||||
|
||||
async def download(self, url: str, target_dir: Path, options: DownloadOptions) -> DownloadResult: ...
|
||||
```
|
||||
|
||||
### Provider 1: YouTube
|
||||
|
||||
- Erkennung über `youtube.com`, `youtu.be`, Playlist-/Channel-URLs.
|
||||
- Metadaten via `yt-dlp --dump-json`.
|
||||
- Download via `yt-dlp`, aber ohne Cookies und ohne Umgehungsoptionen.
|
||||
- Qualitätsprofil: maximal 1080p als Default.
|
||||
|
||||
### Provider 2: Mediathek/Direktlink
|
||||
|
||||
- Für legale Mediathek-Links und direkte Mediendateien.
|
||||
- Keine feste Domain-Allowlist, weil diese zu aufwändig zu pflegen wäre.
|
||||
- Stattdessen konservative Sicherheitsprüfung pro URL:
|
||||
- nur `http`/`https`
|
||||
- DNS-Auflösung und Redirect-Kette prüfen
|
||||
- keine privaten, loopback-, link-local- oder sonstigen internen Ziel-IP-Ranges
|
||||
- Content-Type muss Video/Audio oder ein bekannter öffentlicher Stream-Typ sein
|
||||
- Dateiendung/Container plausibilisieren
|
||||
- Maximalgröße, Timeout und Rate-Limits erzwingen
|
||||
- keine Cookies, keine Login-Flows, keine Umgehungsflags
|
||||
- Wenn `yt-dlp` eine Mediathek-URL direkt unterstützt, darf derselbe sichere yt-dlp Wrapper genutzt werden.
|
||||
- Keine DRM-/Paywall-/Login-/Geoblocking-Umgehung.
|
||||
|
||||
### Provider 3: Internet Archive
|
||||
|
||||
- API-/URL-basierter Import öffentlich verfügbarer Inhalte.
|
||||
- Später als eigener Provider, wenn MVP stabil ist.
|
||||
|
||||
---
|
||||
|
||||
## Konfiguration
|
||||
|
||||
Datei: `.env`
|
||||
|
||||
```env
|
||||
KINOPROJEKT_DB=/data/kino.sqlite3
|
||||
KINOPROJEKT_MEDIA_ROOT=/jellyfin
|
||||
KINOPROJEKT_TMP=/data/tmp
|
||||
JELLYFIN_URL=http://jellyfin:8096
|
||||
JELLYFIN_API_KEY=change-me
|
||||
DEFAULT_MAX_HEIGHT=1080
|
||||
MAX_DOWNLOAD_BYTES=15000000000
|
||||
```
|
||||
|
||||
Secrets wie `JELLYFIN_API_KEY` kommen nicht nach Nextcloud, sondern in Vaultwarden oder eine geschützte `.env` auf dem Zielhost.
|
||||
|
||||
---
|
||||
|
||||
## Ziel-Dateistruktur
|
||||
|
||||
```text
|
||||
kino-projekt/
|
||||
backend/
|
||||
app/
|
||||
main.py
|
||||
config.py
|
||||
db.py
|
||||
models.py
|
||||
schemas.py
|
||||
providers/
|
||||
base.py
|
||||
youtube.py
|
||||
mediathek.py
|
||||
internet_archive.py
|
||||
services/
|
||||
downloader.py
|
||||
jellyfin.py
|
||||
paths.py
|
||||
metadata.py
|
||||
web/
|
||||
index.html
|
||||
app.js
|
||||
style.css
|
||||
tests/
|
||||
test_provider_detection.py
|
||||
test_path_safety.py
|
||||
test_youtube_probe.py
|
||||
test_mediathek_probe.py
|
||||
pyproject.toml
|
||||
docker-compose.yml
|
||||
README.md
|
||||
docs/
|
||||
provider-policy.md
|
||||
deployment.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Tasks
|
||||
|
||||
### Task 1: Projektgerüst anlegen
|
||||
|
||||
**Objective:** Repository-Struktur, Python-Projekt und minimale FastAPI-App erstellen.
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/pyproject.toml`
|
||||
- Create: `backend/app/main.py`
|
||||
- Create: `backend/app/config.py`
|
||||
- Create: `backend/tests/test_health.py`
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. FastAPI-Abhängigkeiten eintragen.
|
||||
2. `/health` Endpoint erstellen.
|
||||
3. Test für `/health` schreiben.
|
||||
4. `pytest` ausführen.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
pytest -q
|
||||
uvicorn app.main:app --host 0.0.0.0 --port 8080
|
||||
curl http://localhost:8080/health
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
```json
|
||||
{"status":"ok"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Konfiguration und Pfadsicherheit
|
||||
|
||||
**Objective:** Medien-/Temp-Pfade sicher konfigurieren und Path Traversal verhindern.
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/app/services/paths.py`
|
||||
- Modify: `backend/app/config.py`
|
||||
- Create: `backend/tests/test_path_safety.py`
|
||||
|
||||
**Rules:**
|
||||
|
||||
- Alle Downloads nur unter `KINOPROJEKT_TMP`.
|
||||
- Alle finalen Dateien nur unter `KINOPROJEKT_MEDIA_ROOT`.
|
||||
- Keine `../`-Escape-Möglichkeiten.
|
||||
- Titel/Kanalnamen werden für Dateisysteme normalisiert.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
pytest backend/tests/test_path_safety.py -q
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Datenbankmodelle erstellen
|
||||
|
||||
**Objective:** SQLite-Modelle für Quellen und Import-Jobs anlegen.
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/app/db.py`
|
||||
- Create: `backend/app/models.py`
|
||||
- Create: `backend/tests/test_models.py`
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
pytest backend/tests/test_models.py -q
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Provider-Interface definieren
|
||||
|
||||
**Objective:** Gemeinsame Provider-Abstraktion für YouTube, Mediathek und zukünftige Quellen schaffen.
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/app/providers/base.py`
|
||||
- Create: `backend/tests/test_provider_detection.py`
|
||||
|
||||
**Implementation Notes:**
|
||||
|
||||
- `MediaMetadata` enthält Titel, Beschreibung, Thumbnail, Dauer, Provider, external_id.
|
||||
- `DownloadOptions` enthält Qualität, Zielbibliothek und Audio-only-Flag.
|
||||
- `DownloadResult` enthält Ausgabedateien und Metadatenpfade.
|
||||
|
||||
---
|
||||
|
||||
### Task 5: YouTube Provider Probe
|
||||
|
||||
**Objective:** YouTube-URLs erkennen und Metadaten über `yt-dlp --dump-json` abrufen.
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/app/providers/youtube.py`
|
||||
- Create: `backend/tests/test_youtube_provider.py`
|
||||
|
||||
**Security Constraints:**
|
||||
|
||||
- Kein Cookie-Import.
|
||||
- Keine Login-/DRM-/Paywall-Umgehung.
|
||||
- Subprocess-Aufruf mit fester Argumentliste, keine Shell.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
python -m app.providers.youtube 'https://www.youtube.com/watch?v=PUBLIC_TEST_ID'
|
||||
```
|
||||
|
||||
Expected: JSON-Metadaten oder sauberer Fehler.
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Mediathek-/Direktlink Provider Probe
|
||||
|
||||
**Objective:** Legale Mediathek- und Direktlinks ohne feste Domain-Allowlist prüfen und Metadaten grob erfassen.
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/app/providers/mediathek.py`
|
||||
- Create: `backend/tests/test_mediathek_provider.py`
|
||||
|
||||
**Rules:**
|
||||
|
||||
- Keine feste Domain-Allowlist, weil diese zu aufwändig zu pflegen wäre.
|
||||
- Stattdessen Sicherheitsprüfung pro URL:
|
||||
- nur `http`/`https`
|
||||
- keine privaten/loopback/link-local IP-Ziele nach DNS-Auflösung
|
||||
- Redirect-Kette prüfen
|
||||
- Content-Type/Dateiendung plausibilisieren
|
||||
- Maximalgröße/Timeout/Rate-Limits erzwingen
|
||||
- keine Cookies, keine Login-Flows, keine Umgehungsflags
|
||||
- Direkte Medien-URLs müssen erlaubte Content-Types liefern.
|
||||
- yt-dlp darf als Extractor genutzt werden, wenn Quelle öffentlich und ohne Umgehung erreichbar ist.
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Probe API erstellen
|
||||
|
||||
**Objective:** UI kann eine URL einreichen und Metadaten anzeigen.
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/app/main.py`
|
||||
- Create: `backend/app/schemas.py`
|
||||
- Create: `backend/tests/test_probe_api.py`
|
||||
|
||||
**Endpoint:**
|
||||
|
||||
```http
|
||||
POST /api/probe
|
||||
Content-Type: application/json
|
||||
|
||||
{"url":"https://www.youtube.com/watch?v=..."}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"provider": "youtube",
|
||||
"title": "...",
|
||||
"thumbnail": "...",
|
||||
"duration_seconds": 123,
|
||||
"allowed": true
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Minimal-WebUI bauen
|
||||
|
||||
**Objective:** Eine einfache interne WebUI zum Einfügen und Prüfen von Links erstellen.
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/app/web/index.html`
|
||||
- Create: `backend/app/web/style.css`
|
||||
- Create: `backend/app/web/app.js`
|
||||
- Modify: `backend/app/main.py`
|
||||
|
||||
**UI:**
|
||||
|
||||
- Titel: „Kino-Projekt“
|
||||
- URL-Eingabe
|
||||
- Button: „Quelle prüfen“
|
||||
- Metadatenkarte
|
||||
- Button: „Import starten“ zunächst disabled, bis Task 10.
|
||||
|
||||
---
|
||||
|
||||
### Task 9: Download Wrapper bauen
|
||||
|
||||
**Objective:** Sicheren Download-Service mit yt-dlp und festen Optionen implementieren.
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/app/services/downloader.py`
|
||||
- Create: `backend/tests/test_downloader_args.py`
|
||||
|
||||
**YouTube Default Command:**
|
||||
|
||||
```bash
|
||||
yt-dlp \
|
||||
--no-playlist \
|
||||
--write-thumbnail \
|
||||
--write-info-json \
|
||||
--merge-output-format mkv \
|
||||
-f "bv*[height<=1080]+ba/b[height<=1080]/b" \
|
||||
-o "<safe-target-template>" \
|
||||
"<url>"
|
||||
```
|
||||
|
||||
**Important:** In Python `subprocess.run([...], shell=False)` nutzen.
|
||||
|
||||
---
|
||||
|
||||
### Task 10: Import-Job API
|
||||
|
||||
**Objective:** Import-Jobs anlegen und ausführen.
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/app/main.py`
|
||||
- Create: `backend/app/services/jobs.py`
|
||||
- Create: `backend/tests/test_jobs.py`
|
||||
|
||||
**Endpoints:**
|
||||
|
||||
```http
|
||||
POST /api/imports
|
||||
GET /api/imports/{job_id}
|
||||
GET /api/imports
|
||||
```
|
||||
|
||||
MVP darf Jobs synchron oder mit einfachem BackgroundTask ausführen. Später kann Redis/RQ ergänzt werden.
|
||||
|
||||
---
|
||||
|
||||
### Task 11: Jellyfin Refresh Service
|
||||
|
||||
**Objective:** Nach erfolgreichem Import Jellyfin-Library aktualisieren.
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/app/services/jellyfin.py`
|
||||
- Create: `backend/tests/test_jellyfin.py`
|
||||
|
||||
**Endpoint intern:**
|
||||
|
||||
```http
|
||||
POST {JELLYFIN_URL}/Library/Refresh
|
||||
X-Emby-Token: {JELLYFIN_API_KEY}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
|
||||
- Mit Test-API-Key gegen Jellyfin prüfen.
|
||||
- Danach in Jellyfin sichtbar machen.
|
||||
|
||||
---
|
||||
|
||||
### Task 12: Docker Compose
|
||||
|
||||
**Objective:** Dienst als Container betreiben.
|
||||
|
||||
**Files:**
|
||||
- Create: `Dockerfile`
|
||||
- Create: `docker-compose.yml`
|
||||
- Create: `.env.example`
|
||||
|
||||
**Services:**
|
||||
|
||||
```yaml
|
||||
services:
|
||||
kino-projekt:
|
||||
build: .
|
||||
ports:
|
||||
- "127.0.0.1:8099:8080"
|
||||
volumes:
|
||||
- ./data:/data
|
||||
- /jellyfin:/jellyfin
|
||||
env_file:
|
||||
- .env
|
||||
```
|
||||
|
||||
|
||||
### Task 13: App-Login einbauen
|
||||
|
||||
**Objective:** Zusätzlich zu Mesh-only einen einfachen App-Login mit sicherem Passwort-Hash und Session-Cookie einbauen.
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/app/services/auth.py`
|
||||
- Modify: `backend/app/models.py`
|
||||
- Modify: `backend/app/main.py`
|
||||
- Modify: `backend/app/web/app.js`
|
||||
- Create: `backend/tests/test_auth.py`
|
||||
|
||||
**Rules:**
|
||||
|
||||
- Passwort nie im Klartext speichern.
|
||||
- Hashing mit `argon2-cffi` oder `passlib[bcrypt]`.
|
||||
- Session-Cookie `HttpOnly`, `Secure`, `SameSite=Lax`.
|
||||
- Initialer Admin-User wird über `.env`/Setup-Befehl erstellt, nicht im Repo.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
pytest backend/tests/test_auth.py -q
|
||||
curl -kI https://kino.dasposchi.de/ # ohne Session: Redirect/Login oder 401
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 14: Mesh-only Reverse Proxy
|
||||
|
||||
**Objective:** WebUI nur im Headscale-Netz verfügbar machen.
|
||||
|
||||
**NPM Config:**
|
||||
|
||||
- Domain: `kino.dasposchi.de`
|
||||
- Forward: Zielhost Port `8099`
|
||||
- Access List: `headscale-mesh-only`
|
||||
- SSL: Let’s Encrypt
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
curl -4 -kI https://kino.dasposchi.de/ # public: 403
|
||||
curl --interface tailscale0 -kI https://kino.dasposchi.de/ # mesh: 200
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 15: Dokumentation
|
||||
|
||||
**Objective:** Nutzung und Grenzen dokumentieren.
|
||||
|
||||
**Files:**
|
||||
- Create: `README.md`
|
||||
- Create: `docs/provider-policy.md`
|
||||
- Create: `docs/deployment.md`
|
||||
|
||||
**README enthält:**
|
||||
|
||||
- Projektname bleibt Kino-Projekt.
|
||||
- Unterstützte Quellen: YouTube, legale Mediatheken, Direktlinks, Internet Archive später.
|
||||
- Keine feste Mediathek-Domain-Allowlist; stattdessen URL-/Redirect-/IP-/Content-Type-/Größen-/Timeout-Sicherheitschecks.
|
||||
- Nicht unterstützte Quellen: Piracy-/Hoster-Scraper, DRM, Paywalls, Login-geschützte Inhalte.
|
||||
- Jellyfin-Konfiguration mit Medienwurzel `/jellyfin`.
|
||||
- App-Login plus Mesh-only Deployment.
|
||||
|
||||
---
|
||||
|
||||
## Getroffene Entscheidungen
|
||||
|
||||
1. **Host:** Neue kleine Debian-VM oder Debian-LXC/CT auf Proxmox. Empfehlung: LXC/CT, wenn Jellyfin-Medienpfade sauber per Mount/Berechtigung eingebunden werden können; sonst VM.
|
||||
2. **Zielbibliothek/Zielordner:** Inhalte werden unter `/jellyfin` in passende Jellyfin-Ordner geladen. Die WebUI bietet dafür ein Zielprofil an, z. B. Film, Serie, YouTube/Mediathek. „Passender Ordner“ wird technisch so konkretisiert:
|
||||
- Filme: `/jellyfin/Filme/<Titel> (<Jahr>)/...`
|
||||
- Serien: `/jellyfin/Serien/<Serientitel>/Staffel<Nummer>/...` (bestehende Struktur nutzt z. B. `Staffel1`, `Staffel2`, nicht `Season 01`)
|
||||
- YouTube/Mediathek-Clips ohne Serienstruktur: `/jellyfin/YouTube-Mediathek/<Kanal oder Sender>/<Titel> [<id>].mkv`
|
||||
- `/jellyfin/YouTube-Mediathek` existiert aktuell noch nicht und soll beim Deployment mit passenden Rechten angelegt werden.
|
||||
3. **Medienwurzel:** `/jellyfin`.
|
||||
4. **Login:** Zusätzlich zu Mesh-only wird ein Login eingebaut. MVP: lokaler App-Login mit Passwort-Hash und Session-Cookie; optional später OIDC/Authelia.
|
||||
5. **Mediathek-Allowlist:** Keine feste Allowlist. Stattdessen URL-Sicherheitsprüfungen, Content-Type-Prüfung, Redirect-Prüfung, Size-/Timeout-Limits und Verbot privater Ziel-IP-Ranges.
|
||||
|
||||
Empfohlener MVP:
|
||||
|
||||
- Host: neue kleine Debian-VM oder Debian-LXC/CT auf Proxmox.
|
||||
- Domain: `kino.dasposchi.de`.
|
||||
- Medienwurzel: `/jellyfin`.
|
||||
- Bibliothek/Zielprofil: automatisch auswählbare Zielprofile für Film, Serie und YouTube/Mediathek.
|
||||
- Provider zuerst: YouTube + generischer Mediathek-/Direktlink-Provider ohne Domain-Allowlist, aber mit strikten Sicherheitschecks.
|
||||
- Zugriff: Headscale-Mesh-only plus App-Login.
|
||||
|
||||
---
|
||||
|
||||
## Konkretisierung: „passender Jellyfin-Ordner“
|
||||
|
||||
Die WebUI soll nicht nach einer abstrakten Jellyfin-Bibliothek fragen, sondern nach einem **Zielprofil**. Das Zielprofil bestimmt den Ordner unter `/jellyfin` und die Dateibenennung.
|
||||
|
||||
MVP-Zielprofile:
|
||||
|
||||
1. **Film**
|
||||
- Ziel: `/jellyfin/Filme/<Titel> (<Jahr>)/<Titel> (<Jahr>).mkv`
|
||||
- Nutzer kann Jahr optional manuell korrigieren.
|
||||
2. **Serie/Episode**
|
||||
- Ziel: `/jellyfin/Serien/<Serientitel>/Staffel<SS>/<Serientitel> - S<SS>E<EE> - <Episodentitel>.mkv`
|
||||
- Bestehende Ordner nutzen `Staffel1`, `Staffel2`, `Staffel3` ohne Leerzeichen/führende Null.
|
||||
- Staffel/Episode können manuell gesetzt werden, falls Metadaten fehlen.
|
||||
3. **YouTube/Mediathek-Clip**
|
||||
- Ziel: `/jellyfin/YouTube-Mediathek/<Kanal oder Sender>/<Titel> [<id>].mkv`
|
||||
- Default für einzelne YouTube-/Mediathek-Videos.
|
||||
|
||||
Die vorhandene Jellyfin-Ordnerstruktur muss vor der Umsetzung einmal geprüft werden; falls die realen Ordnernamen anders sind, werden die Zielprofile entsprechend angepasst.
|
||||
|
||||
---
|
||||
|
||||
## Verifizierte Jellyfin-Struktur
|
||||
|
||||
Geprüft auf Proxmox CT `100` (`jellyfin`, IP `192.168.178.222`). Jellyfin ist aktiv und lauscht auf `8096`.
|
||||
|
||||
Jellyfin-Bibliotheken laut Konfiguration:
|
||||
|
||||
- Filme: `/jellyfin/Filme`
|
||||
- Serien: `/jellyfin/Serien`
|
||||
|
||||
Rechte/Owner:
|
||||
|
||||
- `/jellyfin`: `jellyfinuser:media`, Modus `775`
|
||||
- `/jellyfin/Filme`: `jellyfinuser:media`, setgid-Gruppe `media`
|
||||
- `/jellyfin/Serien`: `jellyfinuser:media`, setgid-Gruppe `media`
|
||||
- Jellyfin-Prozessuser `jellyfin` ist Mitglied der Gruppe `media`
|
||||
|
||||
Beispiele aus der vorhandenen Struktur:
|
||||
|
||||
```text
|
||||
/jellyfin/Filme/Der Super Mario Bros. Film (2023)/Der Super Mario Bros. Film (2023).mp4
|
||||
/jellyfin/Serien/The White Lotus/Staffel1/The-White-Lotus-S01E02-German-720p-WEB-h264-WvF.mp4
|
||||
```
|
||||
|
||||
Für das Kino-Projekt bedeutet das:
|
||||
|
||||
- Die Annahme `/jellyfin` als Medienwurzel ist korrekt.
|
||||
- Filme und Serien sollen direkt in die vorhandenen Ordner `/jellyfin/Filme` und `/jellyfin/Serien` geschrieben werden.
|
||||
- Serien-Staffelordner sollen zur vorhandenen Struktur passend `Staffel1`, `Staffel2`, ... heißen.
|
||||
- Für YouTube-/Mediathek-Einzelclips soll neu `/jellyfin/YouTube-Mediathek` angelegt werden, idealerweise ebenfalls `jellyfinuser:media` und setgid.
|
||||
|
||||
---
|
||||
|
||||
## Akzeptanzkriterien MVP
|
||||
|
||||
- `https://kino.dasposchi.de` ist öffentlich blockiert und im Mesh erreichbar.
|
||||
- Zusätzlich ist ein App-Login aktiv.
|
||||
- YouTube-URL kann geprüft werden.
|
||||
- Mediathek-/Direktlink kann ohne feste Domain-Allowlist geprüft werden, sofern URL-, Redirect-, IP-, Content-Type-, Größen- und Timeout-Sicherheitschecks bestehen.
|
||||
- Import-Job lädt Datei unter `/jellyfin` in das gewählte Zielprofil bzw. den passenden Jellyfin-Ordner.
|
||||
- Jellyfin Refresh wird ausgelöst.
|
||||
- Status und Fehler sind in der UI sichtbar.
|
||||
- Passwort-Hashes/Secrets liegen nicht in Nextcloud oder im Repository.
|
||||
- Keine nicht erlaubten Quellen/Umgehungsmechanismen sind implementiert.
|
||||
7
Projektbeschreibung.md
Normal file
7
Projektbeschreibung.md
Normal file
@@ -0,0 +1,7 @@
|
||||
Folgendes soll umgesetzt werden;:
|
||||
|
||||
Eine WebUI mit der man in der Lage ist, Serien und Filme zu suchen.
|
||||
Durchsucht werden dabei gängige Hosting-Seiten, wie kinox.to, kinoger.com und co - orientieren an das kodi-plugin xstream.
|
||||
Zusätzlich soll ein Download stattfinden und die Datei auf den jellyfin-server übertragen und die bibliothek dort refresht werden.
|
||||
|
||||
Die Repository zu xstream habe ich zur analyse ebenfalls hochgeladen.
|
||||
51
README.md
Normal file
51
README.md
Normal file
@@ -0,0 +1,51 @@
|
||||
# Kino-Projekt
|
||||
|
||||
Interne WebUI zum Prüfen und Importieren von Inhalten aus legalen Quellen in Jellyfin.
|
||||
|
||||
## Sicherheitsgrenzen
|
||||
|
||||
Dieses Projekt implementiert bewusst **keine** Scraper für Piracy-Seiten, keine DRM-/Paywall-/Login-Umgehung und keinen Cookie-Import. Unterstützt werden YouTube-Metadaten via `yt-dlp`, öffentliche Internet-Archive-Items sowie konservativ geprüfte öffentliche Medien-/Mediathek-URLs.
|
||||
|
||||
## Entwicklung
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
python -m venv .venv
|
||||
. .venv/bin/activate
|
||||
pip install -e '.[test]'
|
||||
pytest -q
|
||||
uvicorn app.main:app --host 0.0.0.0 --port 8080
|
||||
```
|
||||
|
||||
Healthcheck: `curl http://localhost:8080/health`
|
||||
|
||||
## API-MVP
|
||||
|
||||
- `POST /api/probe` prüft eine URL und liefert Metadaten, ohne einen Download zu starten.
|
||||
- `POST /api/imports` legt Source + Import-Job in SQLite an und startet den Import im Hintergrund.
|
||||
- `GET /api/imports` und `GET /api/imports/{id}` zeigen Status/Fortschritt/Fehler an.
|
||||
- `GET /api/sources` listet bekannte Quellen.
|
||||
|
||||
## Zielprofile
|
||||
|
||||
Die WebUI bietet drei Jellyfin-Zielprofile an:
|
||||
|
||||
- `film` → `/jellyfin/Filme/<Titel> (<Jahr>)/`
|
||||
- `serie` → `/jellyfin/Serien/<Serientitel>/Staffel<N>/<SxxExx - Episodentitel>/`
|
||||
- `clip` → `/jellyfin/YouTube-Mediathek/<Kanal oder Quelle>/<Titel> [<id>]/`
|
||||
|
||||
Optionale Titel-/Jahr-/Serienfelder dienen nur zur Zielpfad-Korrektur; Pfade werden serverseitig normalisiert und bleiben unter `KINOPROJEKT_MEDIA_ROOT`.
|
||||
|
||||
## App-Login
|
||||
|
||||
Wenn `KINOPROJEKT_ADMIN_PASSWORD_HASH` und `KINOPROJEKT_SESSION_SECRET` gesetzt sind, schützt ein lokaler App-Login die API. Das Passwort wird nur als Argon2-Hash gespeichert; die Session liegt in einem `HttpOnly`, `Secure`, `SameSite=Lax` Cookie.
|
||||
|
||||
Hash/Secret für die geschützte Zielhost-`.env` erzeugen:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
python -c "from app.services.auth import hash_password; import getpass; print(hash_password(getpass.getpass()))"
|
||||
python -c "import secrets; print(secrets.token_urlsafe(48))"
|
||||
```
|
||||
|
||||
Jellyfin wird nur aktualisiert, wenn `JELLYFIN_API_KEY` in einer geschützten Zielhost-Umgebung gesetzt ist; Secrets gehören nicht in Nextcloud.
|
||||
0
backend/app/__init__.py
Normal file
0
backend/app/__init__.py
Normal file
53
backend/app/config.py
Normal file
53
backend/app/config.py
Normal file
@@ -0,0 +1,53 @@
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from urllib.parse import unquote, urlparse
|
||||
|
||||
from pydantic import Field, model_validator
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
def sqlite_url_to_path(value: str | None) -> Path | None:
|
||||
if not value:
|
||||
return None
|
||||
parsed = urlparse(value)
|
||||
if parsed.scheme != "sqlite":
|
||||
return None
|
||||
if parsed.netloc and parsed.netloc != "localhost":
|
||||
return None
|
||||
return Path(unquote(parsed.path))
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
db_path: Path = Field(default=Path("/data/kino.sqlite3"), alias="KINOPROJEKT_DB")
|
||||
legacy_db_url: str | None = Field(default=None, alias="KINOPROJEKT_DB_URL")
|
||||
media_root: Path = Field(default=Path("/jellyfin"), alias="KINOPROJEKT_MEDIA_ROOT")
|
||||
tmp_dir: Path = Field(default=Path("/data/tmp"), alias="KINOPROJEKT_TMP")
|
||||
jellyfin_url: str = Field(default="http://jellyfin:8096", alias="JELLYFIN_URL")
|
||||
jellyfin_api_key: str | None = Field(default=None, alias="JELLYFIN_API_KEY")
|
||||
default_max_height: int = Field(default=1080, alias="DEFAULT_MAX_HEIGHT")
|
||||
max_download_bytes: int = Field(default=15_000_000_000, alias="MAX_DOWNLOAD_BYTES")
|
||||
request_timeout: float = Field(default=15.0, alias="KINOPROJEKT_REQUEST_TIMEOUT")
|
||||
admin_username: str = Field(default="admin", alias="KINOPROJEKT_ADMIN_USER")
|
||||
admin_password_hash: str | None = Field(default=None, alias="KINOPROJEKT_ADMIN_PASSWORD_HASH")
|
||||
session_secret: str | None = Field(default=None, alias="KINOPROJEKT_SESSION_SECRET")
|
||||
rsync_target: str | None = Field(default=None, alias="KINOPROJEKT_RSYNC_TARGET")
|
||||
rsync_ssh_key: Path | None = Field(default=None, alias="KINOPROJEKT_RSYNC_SSH_KEY")
|
||||
rsync_remote_root: Path = Field(default=Path("/jellyfin"), alias="KINOPROJEKT_RSYNC_REMOTE_ROOT")
|
||||
|
||||
@model_validator(mode="after")
|
||||
def apply_legacy_db_url(self) -> "Settings":
|
||||
# The first target-host deployment used KINOPROJEKT_DB_URL=sqlite:////...
|
||||
# Keep it working unless the newer KINOPROJEKT_DB path is explicitly set.
|
||||
fields_set = getattr(self, "__pydantic_fields_set__", set())
|
||||
if "db_path" not in fields_set:
|
||||
legacy_path = sqlite_url_to_path(self.legacy_db_url)
|
||||
if legacy_path is not None:
|
||||
self.db_path = legacy_path
|
||||
return self
|
||||
|
||||
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
|
||||
|
||||
|
||||
@lru_cache
|
||||
def get_settings() -> Settings:
|
||||
return Settings()
|
||||
22
backend/app/db.py
Normal file
22
backend/app/db.py
Normal file
@@ -0,0 +1,22 @@
|
||||
from pathlib import Path
|
||||
|
||||
from sqlmodel import SQLModel, Session, create_engine
|
||||
|
||||
from app.config import get_settings
|
||||
|
||||
|
||||
def get_engine(db_path: Path | None = None):
|
||||
settings = get_settings()
|
||||
path = db_path or settings.db_path
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
return create_engine(f"sqlite:///{path}", connect_args={"check_same_thread": False})
|
||||
|
||||
|
||||
def init_db(engine=None) -> None:
|
||||
SQLModel.metadata.create_all(engine or get_engine())
|
||||
|
||||
|
||||
def get_session():
|
||||
engine = get_engine()
|
||||
with Session(engine) as session:
|
||||
yield session
|
||||
304
backend/app/main.py
Normal file
304
backend/app/main.py
Normal file
@@ -0,0 +1,304 @@
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import BackgroundTasks, Depends, FastAPI, HTTPException, Request
|
||||
from fastapi.responses import FileResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from sqlmodel import Session, select
|
||||
|
||||
from app.config import get_settings
|
||||
from app.db import get_engine, get_session, init_db
|
||||
from app.models import ImportJob, Source
|
||||
from app.providers.base import ProviderError
|
||||
from app.providers.internet_archive import InternetArchiveProvider
|
||||
from app.providers.mediathek import MediathekProvider
|
||||
from app.providers.youtube import YouTubeProvider
|
||||
from app.schemas import (
|
||||
AnalyzeRequest,
|
||||
AnalyzeResponse,
|
||||
BrowserCaptureRequest,
|
||||
ImportJobResponse,
|
||||
ImportRequest,
|
||||
InteractiveAnalyzeClickRequest,
|
||||
InteractiveAnalyzeKeyRequest,
|
||||
InteractiveAnalyzeResponse,
|
||||
InteractiveAnalyzeStartRequest,
|
||||
MediaCandidateResponse,
|
||||
ProbeRequest,
|
||||
ProbeResponse,
|
||||
SourceResponse,
|
||||
)
|
||||
from app.services.downloader import metadata_json, run_import_job
|
||||
from app.services.interactive_browser import interactive_browser_manager
|
||||
from app.services.page_analyzer import analyze_browser_capture, analyze_page
|
||||
from app.services.url_safety import UnsafeUrlError
|
||||
from app.services.auth import LoginPayload, auth_enabled, clear_session_cookie, login_response, require_user
|
||||
|
||||
providers = [YouTubeProvider(), InternetArchiveProvider(), MediathekProvider()]
|
||||
engine = get_engine()
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
app.state.settings = get_settings()
|
||||
init_db(engine)
|
||||
yield
|
||||
|
||||
|
||||
app = FastAPI(title="Kino-Projekt", version="0.2.0", lifespan=lifespan)
|
||||
app.state.settings = get_settings()
|
||||
|
||||
|
||||
def pick_provider(url: str):
|
||||
return next((p for p in providers if p.can_handle(url)), None)
|
||||
|
||||
|
||||
IMPORTABLE_CANDIDATE_KINDS = {"direct_video", "hls_manifest", "dash_manifest"}
|
||||
|
||||
|
||||
def candidate_import_score(candidate) -> tuple[int, int]:
|
||||
kind_priority = {"direct_video": 3, "hls_manifest": 2, "dash_manifest": 2}.get(candidate.kind, 0)
|
||||
return kind_priority, int(candidate.content_length or 0)
|
||||
|
||||
|
||||
async def resolve_importable_url(url: str):
|
||||
"""Return a provider/metadata for an importable media URL.
|
||||
|
||||
Users often paste the player/page URL first. If that page is reachable as
|
||||
text/html, the generic provider cannot import it directly; analyze it and
|
||||
switch to the best discovered media candidate before creating the job.
|
||||
"""
|
||||
provider = pick_provider(url)
|
||||
if provider is None:
|
||||
raise HTTPException(status_code=400, detail="No legal provider accepted this URL")
|
||||
try:
|
||||
metadata = await provider.probe(url)
|
||||
return url, provider, metadata, None
|
||||
except ProviderError as exc:
|
||||
original_error = exc
|
||||
if "text/html" not in str(exc).lower():
|
||||
raise
|
||||
|
||||
candidates = await analyze_page(url)
|
||||
allowed = [
|
||||
candidate
|
||||
for candidate in candidates
|
||||
if candidate.allowed and candidate.kind in IMPORTABLE_CANDIDATE_KINDS
|
||||
]
|
||||
if not allowed:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=(
|
||||
"HTML page is reachable, but no importable media URL was found automatically. "
|
||||
"Use 'Seite analysieren' or the Browser-Capture bookmarklet after playing the video, "
|
||||
"then import one of the discovered media candidates."
|
||||
),
|
||||
) from original_error
|
||||
candidate = max(allowed, key=candidate_import_score)
|
||||
provider = pick_provider(candidate.url)
|
||||
if provider is None:
|
||||
raise HTTPException(status_code=400, detail="No legal provider accepted the discovered media URL")
|
||||
metadata = await provider.probe(candidate.url)
|
||||
return candidate.url, provider, metadata, candidate
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
def health() -> dict[str, str]:
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@app.get("/api/auth/status")
|
||||
def auth_status(request: Request) -> dict[str, bool | str | None]:
|
||||
settings = request.app.state.settings
|
||||
user = None
|
||||
if auth_enabled(settings):
|
||||
try:
|
||||
current = require_user(request)
|
||||
user = current.username if current else None
|
||||
except HTTPException:
|
||||
user = None
|
||||
return {"enabled": auth_enabled(settings), "authenticated": user is not None or not auth_enabled(settings), "username": user}
|
||||
|
||||
|
||||
@app.post("/api/auth/login", status_code=204)
|
||||
def login(payload: LoginPayload, request: Request):
|
||||
return login_response(payload, request, request.app.state.settings)
|
||||
|
||||
|
||||
@app.post("/api/auth/logout", status_code=204)
|
||||
def logout(request: Request):
|
||||
return clear_session_cookie(request)
|
||||
|
||||
|
||||
@app.post("/api/probe", response_model=ProbeResponse)
|
||||
async def probe(payload: ProbeRequest, _: object = Depends(require_user)) -> ProbeResponse:
|
||||
url = str(payload.url)
|
||||
provider = pick_provider(url)
|
||||
if provider is None:
|
||||
raise HTTPException(status_code=400, detail="No legal provider accepted this URL")
|
||||
try:
|
||||
resolved_url, provider, metadata, candidate = await resolve_importable_url(url)
|
||||
except ProviderError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
response = metadata.model_dump()
|
||||
if candidate is not None:
|
||||
response["title"] = candidate.title or metadata.title
|
||||
response["external_id"] = resolved_url
|
||||
response["description"] = "Automatisch aus der HTML-Seite erkannte Medienquelle"
|
||||
return ProbeResponse(**response, allowed=True)
|
||||
|
||||
|
||||
|
||||
|
||||
@app.post("/api/analyze", response_model=AnalyzeResponse)
|
||||
async def analyze(payload: AnalyzeRequest, _: object = Depends(require_user)) -> AnalyzeResponse:
|
||||
try:
|
||||
candidates = await analyze_page(str(payload.url))
|
||||
except UnsafeUrlError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
return AnalyzeResponse(
|
||||
page_url=str(payload.url),
|
||||
candidates=[MediaCandidateResponse(**candidate.__dict__) for candidate in candidates],
|
||||
)
|
||||
|
||||
|
||||
@app.post("/api/browser-capture", response_model=AnalyzeResponse)
|
||||
async def browser_capture(payload: BrowserCaptureRequest, _: object = Depends(require_user)) -> AnalyzeResponse:
|
||||
try:
|
||||
candidates = await analyze_browser_capture(
|
||||
[item.model_dump() for item in payload.candidates],
|
||||
str(payload.page_url) if payload.page_url else None,
|
||||
)
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
return AnalyzeResponse(
|
||||
page_url=str(payload.page_url) if payload.page_url else "browser-capture",
|
||||
candidates=[MediaCandidateResponse(**candidate.__dict__) for candidate in candidates],
|
||||
)
|
||||
|
||||
|
||||
async def interactive_response(session) -> InteractiveAnalyzeResponse:
|
||||
candidates = await interactive_browser_manager.candidates(session)
|
||||
return InteractiveAnalyzeResponse(
|
||||
session_id=session.id,
|
||||
page_url=session.page.url or session.page_url,
|
||||
screenshot=await interactive_browser_manager.screenshot_data_url(session),
|
||||
viewport={"width": 1280, "height": 720},
|
||||
candidates=[MediaCandidateResponse(**candidate.__dict__) for candidate in candidates],
|
||||
ublock_origin=bool(session.ublock_path),
|
||||
)
|
||||
|
||||
|
||||
@app.post("/api/interactive-analyze", response_model=InteractiveAnalyzeResponse)
|
||||
async def start_interactive_analyze(payload: InteractiveAnalyzeStartRequest, _: object = Depends(require_user)) -> InteractiveAnalyzeResponse:
|
||||
try:
|
||||
session = await interactive_browser_manager.start(str(payload.url))
|
||||
return await interactive_response(session)
|
||||
except UnsafeUrlError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@app.post("/api/interactive-analyze/{session_id}/click", response_model=InteractiveAnalyzeResponse)
|
||||
async def click_interactive_analyze(session_id: str, payload: InteractiveAnalyzeClickRequest, _: object = Depends(require_user)) -> InteractiveAnalyzeResponse:
|
||||
try:
|
||||
session = await interactive_browser_manager.click(session_id, payload.x, payload.y)
|
||||
return await interactive_response(session)
|
||||
except KeyError as exc:
|
||||
raise HTTPException(status_code=404, detail="Interactive analysis session not found") from exc
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@app.post("/api/interactive-analyze/{session_id}/key", response_model=InteractiveAnalyzeResponse)
|
||||
async def key_interactive_analyze(session_id: str, payload: InteractiveAnalyzeKeyRequest, _: object = Depends(require_user)) -> InteractiveAnalyzeResponse:
|
||||
try:
|
||||
session = await interactive_browser_manager.press(session_id, payload.key)
|
||||
return await interactive_response(session)
|
||||
except KeyError as exc:
|
||||
raise HTTPException(status_code=404, detail="Interactive analysis session not found") from exc
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@app.delete("/api/interactive-analyze/{session_id}", status_code=204)
|
||||
async def stop_interactive_analyze(session_id: str, _: object = Depends(require_user)) -> None:
|
||||
await interactive_browser_manager.stop(session_id)
|
||||
|
||||
|
||||
@app.post("/api/imports", response_model=ImportJobResponse, status_code=202)
|
||||
async def create_import(
|
||||
request: ImportRequest,
|
||||
background_tasks: BackgroundTasks,
|
||||
_: object = Depends(require_user),
|
||||
session: Session = Depends(get_session),
|
||||
) -> ImportJob:
|
||||
url = str(request.url)
|
||||
provider = pick_provider(url)
|
||||
if provider is None:
|
||||
raise HTTPException(status_code=400, detail="No legal provider accepted this URL")
|
||||
try:
|
||||
resolved_url, provider, metadata, candidate = await resolve_importable_url(url)
|
||||
except ProviderError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
|
||||
source = Source(
|
||||
kind=provider.name,
|
||||
provider=provider.name,
|
||||
url=resolved_url,
|
||||
title=request.title or (candidate.title if candidate is not None else None) or metadata.title,
|
||||
external_id=metadata.external_id,
|
||||
metadata_json=metadata_json(
|
||||
metadata,
|
||||
import_request={
|
||||
"source_page_url": url if resolved_url != url else None,
|
||||
"discovered_media_url": resolved_url if resolved_url != url else None,
|
||||
"discovered_candidate_kind": candidate.kind if candidate is not None else None,
|
||||
"target_profile": request.target_profile,
|
||||
"title": request.title,
|
||||
"year": request.year,
|
||||
"series_title": request.series_title,
|
||||
"season": request.season,
|
||||
"episode": request.episode,
|
||||
"episode_title": request.episode_title,
|
||||
},
|
||||
),
|
||||
)
|
||||
session.add(source)
|
||||
session.commit()
|
||||
session.refresh(source)
|
||||
|
||||
job = ImportJob(source_id=source.id, target_library=request.target_profile, status="queued")
|
||||
session.add(job)
|
||||
session.commit()
|
||||
session.refresh(job)
|
||||
background_tasks.add_task(run_import_job, job.id, engine, providers, get_settings())
|
||||
return job
|
||||
|
||||
|
||||
@app.get("/api/imports", response_model=list[ImportJobResponse])
|
||||
def list_imports(_: object = Depends(require_user), session: Session = Depends(get_session)) -> list[ImportJob]:
|
||||
return list(session.exec(select(ImportJob).order_by(ImportJob.created_at.desc())).all())
|
||||
|
||||
|
||||
@app.get("/api/imports/{job_id}", response_model=ImportJobResponse)
|
||||
def get_import(job_id: int, _: object = Depends(require_user), session: Session = Depends(get_session)) -> ImportJob:
|
||||
job = session.get(ImportJob, job_id)
|
||||
if job is None:
|
||||
raise HTTPException(status_code=404, detail="Import job not found")
|
||||
return job
|
||||
|
||||
|
||||
@app.get("/api/sources", response_model=list[SourceResponse])
|
||||
def list_sources(_: object = Depends(require_user), session: Session = Depends(get_session)) -> list[Source]:
|
||||
return list(session.exec(select(Source).order_by(Source.created_at.desc())).all())
|
||||
|
||||
|
||||
app.mount("/static", StaticFiles(directory="app/web"), name="static")
|
||||
|
||||
|
||||
@app.get("/", include_in_schema=False)
|
||||
def index() -> FileResponse:
|
||||
return FileResponse("app/web/index.html", headers={"Cache-Control": "no-store"})
|
||||
30
backend/app/models.py
Normal file
30
backend/app/models.py
Normal file
@@ -0,0 +1,30 @@
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlmodel import Field, SQLModel
|
||||
|
||||
|
||||
def utcnow() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
class Source(SQLModel, table=True):
|
||||
id: int | None = Field(default=None, primary_key=True)
|
||||
kind: str
|
||||
url: str
|
||||
title: str | None = None
|
||||
provider: str | None = None
|
||||
external_id: str | None = None
|
||||
metadata_json: str = "{}"
|
||||
created_at: datetime = Field(default_factory=utcnow)
|
||||
|
||||
|
||||
class ImportJob(SQLModel, table=True):
|
||||
id: int | None = Field(default=None, primary_key=True)
|
||||
source_id: int = Field(foreign_key="source.id")
|
||||
status: str = "queued"
|
||||
target_library: str
|
||||
target_path: str | None = None
|
||||
progress: float = 0.0
|
||||
error: str | None = None
|
||||
created_at: datetime = Field(default_factory=utcnow)
|
||||
updated_at: datetime = Field(default_factory=utcnow)
|
||||
0
backend/app/providers/__init__.py
Normal file
0
backend/app/providers/__init__.py
Normal file
46
backend/app/providers/base.py
Normal file
46
backend/app/providers/base.py
Normal file
@@ -0,0 +1,46 @@
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from typing import Protocol
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class ProviderError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class MediaMetadata(BaseModel):
|
||||
provider: str
|
||||
title: str | None = None
|
||||
description: str | None = None
|
||||
thumbnail: str | None = None
|
||||
duration_seconds: int | None = None
|
||||
external_id: str | None = None
|
||||
|
||||
|
||||
class DownloadOptions(BaseModel):
|
||||
max_height: int = 1080
|
||||
target_library: str = "movies"
|
||||
audio_only: bool = False
|
||||
max_bytes: int = 15_000_000_000
|
||||
|
||||
|
||||
class DownloadResult(BaseModel):
|
||||
output_files: list[Path]
|
||||
metadata_files: list[Path] = []
|
||||
|
||||
|
||||
class Provider(Protocol):
|
||||
name: str
|
||||
|
||||
def can_handle(self, url: str) -> bool: ...
|
||||
|
||||
async def probe(self, url: str) -> MediaMetadata: ...
|
||||
|
||||
async def download(
|
||||
self,
|
||||
url: str,
|
||||
target_dir: Path,
|
||||
options: DownloadOptions,
|
||||
progress_callback: Callable[[float], None] | None = None,
|
||||
) -> DownloadResult: ...
|
||||
122
backend/app/providers/internet_archive.py
Normal file
122
backend/app/providers/internet_archive.py
Normal file
@@ -0,0 +1,122 @@
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from urllib.parse import quote, urlparse
|
||||
|
||||
import httpx
|
||||
|
||||
from app.providers.base import DownloadOptions, DownloadResult, MediaMetadata, ProviderError
|
||||
from app.providers.mediathek import MEDIA_EXTENSIONS, MediathekProvider
|
||||
|
||||
ARCHIVE_HOSTS = {"archive.org", "www.archive.org"}
|
||||
METADATA_BASE = "https://archive.org/metadata"
|
||||
DOWNLOAD_BASE = "https://archive.org/download"
|
||||
|
||||
|
||||
class InternetArchiveProvider:
|
||||
name = "internet_archive"
|
||||
|
||||
def can_handle(self, url: str) -> bool:
|
||||
parsed = urlparse(url)
|
||||
return parsed.scheme in {"http", "https"} and (parsed.hostname or "").lower() in ARCHIVE_HOSTS and self._identifier_from_url(url) is not None
|
||||
|
||||
async def probe(self, url: str) -> MediaMetadata:
|
||||
identifier = self._identifier_from_url(url)
|
||||
if not identifier:
|
||||
raise ProviderError("Internet Archive URL must contain an item identifier")
|
||||
metadata = await self._metadata(identifier)
|
||||
meta = metadata.get("metadata") or {}
|
||||
files = metadata.get("files") or []
|
||||
selected = self._select_media_file(files)
|
||||
if selected is None:
|
||||
raise ProviderError("Internet Archive item has no accepted public media file")
|
||||
title = self._string(meta.get("title")) or identifier
|
||||
description = self._string(meta.get("description"))
|
||||
return MediaMetadata(
|
||||
provider=self.name,
|
||||
title=title,
|
||||
description=description,
|
||||
external_id=identifier,
|
||||
)
|
||||
|
||||
async def download(
|
||||
self,
|
||||
url: str,
|
||||
target_dir: Path,
|
||||
options: DownloadOptions,
|
||||
progress_callback: Callable[[float], None] | None = None,
|
||||
) -> DownloadResult:
|
||||
identifier = self._identifier_from_url(url)
|
||||
if not identifier:
|
||||
raise ProviderError("Internet Archive URL must contain an item identifier")
|
||||
metadata = await self._metadata(identifier)
|
||||
selected = self._select_media_file(metadata.get("files") or [])
|
||||
if selected is None:
|
||||
raise ProviderError("Internet Archive item has no accepted public media file")
|
||||
file_name = selected["name"]
|
||||
safe_url = f"{DOWNLOAD_BASE}/{quote(identifier, safe='')}/{quote(file_name, safe='/')}"
|
||||
return await MediathekProvider().download(safe_url, target_dir, options, progress_callback=progress_callback)
|
||||
|
||||
async def _metadata(self, identifier: str) -> dict:
|
||||
url = f"{METADATA_BASE}/{quote(identifier, safe='')}"
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(15, read=20), follow_redirects=True) as client:
|
||||
response = await client.get(url)
|
||||
if response.status_code == 404:
|
||||
raise ProviderError(f"Internet Archive item not found: {identifier}")
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
if not isinstance(data, dict):
|
||||
raise ProviderError("Internet Archive metadata response was not an object")
|
||||
return data
|
||||
|
||||
def _identifier_from_url(self, url: str) -> str | None:
|
||||
parsed = urlparse(url)
|
||||
parts = [part for part in parsed.path.split("/") if part]
|
||||
if not parts:
|
||||
return None
|
||||
if parts[0] in {"details", "download", "metadata"} and len(parts) >= 2:
|
||||
return parts[1]
|
||||
return parts[0]
|
||||
|
||||
def _select_media_file(self, files: list[dict]) -> dict | None:
|
||||
candidates: list[tuple[int, dict]] = []
|
||||
for item in files:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
name = item.get("name")
|
||||
if not isinstance(name, str) or not name or name.endswith("/") or ".." in Path(name).parts:
|
||||
continue
|
||||
lower = name.lower()
|
||||
ext = Path(lower).suffix
|
||||
fmt = str(item.get("format") or "").lower()
|
||||
source = str(item.get("source") or "").lower()
|
||||
if ext not in MEDIA_EXTENSIONS and "mpeg4" not in fmt and "h.264" not in fmt:
|
||||
continue
|
||||
if source in {"metadata", "derivative"} and ext not in {".mp4", ".m4v", ".mp3", ".m4a", ".ogg", ".webm"}:
|
||||
continue
|
||||
size = self._int_or_zero(item.get("size"))
|
||||
# Prefer original/public mp4-ish media, then larger files as likely higher quality.
|
||||
score = 0
|
||||
if source == "original":
|
||||
score += 1000
|
||||
if ext in {".mp4", ".m4v", ".webm"}:
|
||||
score += 500
|
||||
if size:
|
||||
score += min(size // 1_000_000, 400)
|
||||
candidates.append((score, item))
|
||||
if not candidates:
|
||||
return None
|
||||
candidates.sort(key=lambda pair: pair[0], reverse=True)
|
||||
return candidates[0][1]
|
||||
|
||||
def _int_or_zero(self, value: object) -> int:
|
||||
try:
|
||||
return int(value) # type: ignore[arg-type]
|
||||
except (TypeError, ValueError):
|
||||
return 0
|
||||
|
||||
def _string(self, value: object) -> str | None:
|
||||
if isinstance(value, str):
|
||||
return value.strip() or None
|
||||
if isinstance(value, list) and value and isinstance(value[0], str):
|
||||
return value[0].strip() or None
|
||||
return None
|
||||
193
backend/app/providers/mediathek.py
Normal file
193
backend/app/providers/mediathek.py
Normal file
@@ -0,0 +1,193 @@
|
||||
import asyncio
|
||||
import ipaddress
|
||||
import shutil
|
||||
import socket
|
||||
import sys
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path, PurePosixPath
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
|
||||
from app.providers.base import DownloadOptions, DownloadResult, MediaMetadata, ProviderError
|
||||
from app.providers.youtube import parse_yt_dlp_progress
|
||||
|
||||
MEDIA_CONTENT_TYPES = (
|
||||
"video/",
|
||||
"audio/",
|
||||
"application/vnd.apple.mpegurl",
|
||||
"application/x-mpegurl",
|
||||
"application/dash+xml",
|
||||
)
|
||||
MEDIA_EXTENSIONS = {
|
||||
".mp4",
|
||||
".mkv",
|
||||
".webm",
|
||||
".mov",
|
||||
".m4v",
|
||||
".mp3",
|
||||
".m4a",
|
||||
".aac",
|
||||
".ogg",
|
||||
".flac",
|
||||
".m3u8",
|
||||
".mpd",
|
||||
}
|
||||
MANIFEST_CONTENT_TYPES = {"application/vnd.apple.mpegurl", "application/x-mpegurl", "application/dash+xml"}
|
||||
MANIFEST_EXTENSIONS = {".m3u8", ".mpd"}
|
||||
|
||||
|
||||
def is_stream_manifest(url: str, content_type: str | None = None) -> bool:
|
||||
clean_content_type = (content_type or "").split(";", 1)[0].lower()
|
||||
ext = PurePosixPath(urlparse(url).path).suffix.lower()
|
||||
return clean_content_type in MANIFEST_CONTENT_TYPES or ext in MANIFEST_EXTENSIONS
|
||||
|
||||
|
||||
def ytdlp_command() -> list[str]:
|
||||
binary = shutil.which("yt-dlp")
|
||||
if binary:
|
||||
return [binary]
|
||||
return [sys.executable, "-m", "yt_dlp"]
|
||||
|
||||
|
||||
class MediathekProvider:
|
||||
name = "mediathek"
|
||||
|
||||
def can_handle(self, url: str) -> bool:
|
||||
parsed = urlparse(url)
|
||||
host = parsed.hostname or ""
|
||||
excluded_hosts = {"youtube.com", "www.youtube.com", "m.youtube.com", "youtu.be", "archive.org", "www.archive.org"}
|
||||
return parsed.scheme in {"http", "https"} and host not in excluded_hosts
|
||||
|
||||
async def probe(self, url: str) -> MediaMetadata:
|
||||
parsed = urlparse(url)
|
||||
if parsed.scheme not in {"http", "https"} or not parsed.hostname:
|
||||
raise ProviderError("Only http/https URLs are allowed")
|
||||
self._reject_private_targets(parsed.hostname)
|
||||
async with httpx.AsyncClient(follow_redirects=True, timeout=httpx.Timeout(10, read=10), max_redirects=5) as client:
|
||||
response = await client.head(url)
|
||||
if response.status_code in {405, 403}:
|
||||
response = await client.get(url, headers={"Range": "bytes=0-0"})
|
||||
for hop in [*response.history, response]:
|
||||
host = hop.url.host
|
||||
if host:
|
||||
self._reject_private_targets(host)
|
||||
content_type = response.headers.get("content-type", "").split(";", 1)[0].lower()
|
||||
ext = PurePosixPath(urlparse(str(response.url)).path).suffix.lower()
|
||||
if not (content_type.startswith(MEDIA_CONTENT_TYPES) or ext in MEDIA_EXTENSIONS):
|
||||
raise ProviderError(f"URL is reachable, but content type/extension is not accepted: {content_type or ext or 'unknown'}")
|
||||
title = PurePosixPath(urlparse(str(response.url)).path).name or str(response.url.host)
|
||||
return MediaMetadata(provider=self.name, title=title, external_id=str(response.url))
|
||||
|
||||
async def download(
|
||||
self,
|
||||
url: str,
|
||||
target_dir: Path,
|
||||
options: DownloadOptions,
|
||||
progress_callback: Callable[[float], None] | None = None,
|
||||
) -> DownloadResult:
|
||||
metadata = await self.probe(url)
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
source_url = metadata.external_id or url
|
||||
if is_stream_manifest(source_url):
|
||||
return await self._download_with_ytdlp(source_url, target_dir, options, progress_callback)
|
||||
|
||||
name = PurePosixPath(urlparse(source_url).path).name or "download.bin"
|
||||
output_path = target_dir / name
|
||||
written = 0
|
||||
async with httpx.AsyncClient(follow_redirects=True, timeout=httpx.Timeout(20, read=60), max_redirects=5) as client:
|
||||
async with client.stream("GET", source_url) as response:
|
||||
response.raise_for_status()
|
||||
for hop in [*response.history, response]:
|
||||
host = hop.url.host
|
||||
if host:
|
||||
self._reject_private_targets(host)
|
||||
total = int(response.headers.get("content-length") or 0)
|
||||
with output_path.open("wb") as fh:
|
||||
async for chunk in response.aiter_bytes():
|
||||
written += len(chunk)
|
||||
if written > options.max_bytes:
|
||||
raise ProviderError("Download exceeds configured maximum size")
|
||||
fh.write(chunk)
|
||||
if total > 0 and progress_callback is not None:
|
||||
progress_callback(min(written / total, 1.0))
|
||||
if progress_callback is not None:
|
||||
progress_callback(1.0)
|
||||
return DownloadResult(output_files=[output_path])
|
||||
|
||||
async def _download_with_ytdlp(
|
||||
self,
|
||||
url: str,
|
||||
target_dir: Path,
|
||||
options: DownloadOptions,
|
||||
progress_callback: Callable[[float], None] | None = None,
|
||||
) -> DownloadResult:
|
||||
"""Download HLS/DASH manifests as real media, not the tiny manifest file.
|
||||
|
||||
A .m3u8/.mpd is only a playlist/manifest. Saving it directly produces a
|
||||
few-KB text file in Jellyfin instead of the actual media stream. Let
|
||||
yt-dlp drive ffmpeg so segments are downloaded and remuxed into MP4.
|
||||
"""
|
||||
command = ytdlp_command()
|
||||
output_template = str(target_dir / "%(title,playlist_index,id).120s.%(ext)s")
|
||||
height_expr = f"bestvideo[height<={options.max_height}]+bestaudio/best[height<={options.max_height}]/best"
|
||||
if options.audio_only:
|
||||
height_expr = "bestaudio/best"
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*command,
|
||||
"--newline",
|
||||
"--no-playlist",
|
||||
"--no-warnings",
|
||||
"--restrict-filenames",
|
||||
"--format",
|
||||
height_expr,
|
||||
"--remux-video",
|
||||
"mp4",
|
||||
"--max-filesize",
|
||||
str(options.max_bytes),
|
||||
"--output",
|
||||
output_template,
|
||||
url,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
captured: list[str] = []
|
||||
|
||||
async def consume(stream: asyncio.StreamReader | None) -> None:
|
||||
if stream is None:
|
||||
return
|
||||
while True:
|
||||
line = await stream.readline()
|
||||
if not line:
|
||||
break
|
||||
text = line.decode(errors="replace").strip()
|
||||
if not text:
|
||||
continue
|
||||
captured.append(text)
|
||||
if len(captured) > 80:
|
||||
del captured[:40]
|
||||
progress = parse_yt_dlp_progress(text)
|
||||
if progress is not None and progress_callback is not None:
|
||||
progress_callback(progress)
|
||||
|
||||
await asyncio.wait_for(asyncio.gather(consume(proc.stdout), consume(proc.stderr), proc.wait()), timeout=60 * 60)
|
||||
if proc.returncode != 0:
|
||||
detail = "\n".join(captured).strip() or "yt-dlp manifest download failed"
|
||||
raise ProviderError(detail[:1000])
|
||||
files = [p for p in target_dir.iterdir() if p.is_file()]
|
||||
media_files = [p for p in files if p.suffix.lower() not in {".json", ".jpg", ".jpeg", ".png", ".webp", ".m3u8", ".mpd"}]
|
||||
if not media_files:
|
||||
raise ProviderError("yt-dlp finished without producing a media file")
|
||||
if progress_callback is not None:
|
||||
progress_callback(1.0)
|
||||
return DownloadResult(output_files=media_files)
|
||||
|
||||
def _reject_private_targets(self, hostname: str) -> None:
|
||||
try:
|
||||
infos = socket.getaddrinfo(hostname, None, proto=socket.IPPROTO_TCP)
|
||||
except socket.gaierror as exc:
|
||||
raise ProviderError(f"DNS lookup failed for {hostname}") from exc
|
||||
for info in infos:
|
||||
ip = ipaddress.ip_address(info[4][0])
|
||||
if ip.is_private or ip.is_loopback or ip.is_link_local or ip.is_multicast or ip.is_reserved or ip.is_unspecified:
|
||||
raise ProviderError(f"Refusing private or non-public target address for {hostname}")
|
||||
136
backend/app/providers/youtube.py
Normal file
136
backend/app/providers/youtube.py
Normal file
@@ -0,0 +1,136 @@
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
|
||||
from app.providers.base import DownloadOptions, DownloadResult, MediaMetadata, ProviderError
|
||||
|
||||
|
||||
class YouTubeProvider:
|
||||
name = "youtube"
|
||||
allowed_hosts = {"youtube.com", "www.youtube.com", "m.youtube.com", "youtu.be"}
|
||||
|
||||
def can_handle(self, url: str) -> bool:
|
||||
host = urlparse(url).hostname or ""
|
||||
return host.lower() in self.allowed_hosts
|
||||
|
||||
async def probe(self, url: str) -> MediaMetadata:
|
||||
if not self.can_handle(url):
|
||||
raise ProviderError("Not a YouTube URL")
|
||||
if shutil.which("yt-dlp") is None:
|
||||
return MediaMetadata(provider=self.name, external_id=self._external_id(url), title="YouTube URL (yt-dlp not installed)")
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"yt-dlp",
|
||||
"--dump-single-json",
|
||||
"--no-playlist",
|
||||
"--no-warnings",
|
||||
url,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=45)
|
||||
if proc.returncode != 0:
|
||||
raise ProviderError((stderr.decode(errors="replace") or "yt-dlp probe failed").strip()[:500])
|
||||
data = json.loads(stdout.decode())
|
||||
return MediaMetadata(
|
||||
provider=self.name,
|
||||
title=data.get("title"),
|
||||
description=data.get("description"),
|
||||
thumbnail=data.get("thumbnail"),
|
||||
duration_seconds=data.get("duration"),
|
||||
external_id=data.get("id") or self._external_id(url),
|
||||
)
|
||||
|
||||
async def download(
|
||||
self,
|
||||
url: str,
|
||||
target_dir: Path,
|
||||
options: DownloadOptions,
|
||||
progress_callback: Callable[[float], None] | None = None,
|
||||
) -> DownloadResult:
|
||||
if shutil.which("yt-dlp") is None:
|
||||
raise ProviderError("yt-dlp is not installed")
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
height_expr = f"bestvideo[height<={options.max_height}]+bestaudio/best[height<={options.max_height}]/best"
|
||||
if options.audio_only:
|
||||
height_expr = "bestaudio/best"
|
||||
output_template = str(target_dir / "%(title).120s [%(id)s].%(ext)s")
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"yt-dlp",
|
||||
"--newline",
|
||||
"--no-playlist",
|
||||
"--no-warnings",
|
||||
"--restrict-filenames",
|
||||
"--write-info-json",
|
||||
"--write-thumbnail",
|
||||
"--format",
|
||||
height_expr,
|
||||
"--output",
|
||||
output_template,
|
||||
url,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
|
||||
captured: list[str] = []
|
||||
|
||||
async def consume(stream: asyncio.StreamReader | None) -> None:
|
||||
if stream is None:
|
||||
return
|
||||
while True:
|
||||
line = await stream.readline()
|
||||
if not line:
|
||||
break
|
||||
text = line.decode(errors="replace").strip()
|
||||
if not text:
|
||||
continue
|
||||
captured.append(text)
|
||||
if len(captured) > 80:
|
||||
del captured[:40]
|
||||
progress = parse_yt_dlp_progress(text)
|
||||
if progress is not None and progress_callback is not None:
|
||||
progress_callback(progress)
|
||||
|
||||
await asyncio.wait_for(asyncio.gather(consume(proc.stdout), consume(proc.stderr), proc.wait()), timeout=60 * 60)
|
||||
if proc.returncode != 0:
|
||||
detail = "\n".join(captured).strip() or "yt-dlp download failed"
|
||||
raise ProviderError(detail[:1000])
|
||||
files = [p for p in target_dir.iterdir() if p.is_file()]
|
||||
media_files = [p for p in files if p.suffix.lower() not in {".json", ".jpg", ".jpeg", ".png", ".webp"}]
|
||||
metadata_files = [p for p in files if p not in media_files]
|
||||
if not media_files:
|
||||
raise ProviderError("yt-dlp finished without producing a media file")
|
||||
if progress_callback is not None:
|
||||
progress_callback(1.0)
|
||||
return DownloadResult(output_files=media_files, metadata_files=metadata_files)
|
||||
|
||||
def _external_id(self, url: str) -> str | None:
|
||||
parsed = urlparse(url)
|
||||
if parsed.hostname == "youtu.be":
|
||||
return parsed.path.strip("/") or None
|
||||
return parse_qs(parsed.query).get("v", [None])[0]
|
||||
|
||||
|
||||
def parse_yt_dlp_progress(line: str) -> float | None:
|
||||
"""Return yt-dlp percent progress as 0.0..1.0, or None for non-progress lines."""
|
||||
if "[download]" not in line:
|
||||
return None
|
||||
match = re.search(r"(\d+(?:\.\d+)?)%", line)
|
||||
if not match:
|
||||
return None
|
||||
return max(0.0, min(float(match.group(1)) / 100.0, 1.0))
|
||||
|
||||
|
||||
async def _main(url: str) -> None:
|
||||
meta = await YouTubeProvider().probe(url)
|
||||
print(meta.model_dump_json(indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 2:
|
||||
raise SystemExit("usage: python -m app.providers.youtube <url>")
|
||||
asyncio.run(_main(sys.argv[1]))
|
||||
115
backend/app/schemas.py
Normal file
115
backend/app/schemas.py
Normal file
@@ -0,0 +1,115 @@
|
||||
from datetime import datetime
|
||||
|
||||
from typing import Any
|
||||
|
||||
from pydantic import AnyHttpUrl, BaseModel, Field
|
||||
|
||||
|
||||
class ProbeRequest(BaseModel):
|
||||
url: AnyHttpUrl
|
||||
|
||||
|
||||
|
||||
|
||||
class AnalyzeRequest(BaseModel):
|
||||
url: AnyHttpUrl
|
||||
|
||||
|
||||
class BrowserCapturedCandidate(BaseModel):
|
||||
url: str
|
||||
title: str | None = None
|
||||
name: str | None = None
|
||||
content_type: str | None = None
|
||||
type: str | None = None
|
||||
quality: str | None = None
|
||||
|
||||
|
||||
class BrowserCaptureRequest(BaseModel):
|
||||
page_url: AnyHttpUrl | None = None
|
||||
candidates: list[BrowserCapturedCandidate] = Field(default_factory=list, max_length=80)
|
||||
|
||||
|
||||
class MediaCandidateResponse(BaseModel):
|
||||
url: str
|
||||
kind: str
|
||||
title: str | None = None
|
||||
source: str
|
||||
content_type: str | None = None
|
||||
content_length: int | None = None
|
||||
file_size: str | None = None
|
||||
quality: str | None = None
|
||||
allowed: bool = True
|
||||
reason: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AnalyzeResponse(BaseModel):
|
||||
page_url: str
|
||||
candidates: list[MediaCandidateResponse]
|
||||
|
||||
|
||||
class InteractiveAnalyzeStartRequest(BaseModel):
|
||||
url: AnyHttpUrl
|
||||
|
||||
|
||||
class InteractiveAnalyzeClickRequest(BaseModel):
|
||||
x: float = Field(ge=0)
|
||||
y: float = Field(ge=0)
|
||||
|
||||
|
||||
class InteractiveAnalyzeKeyRequest(BaseModel):
|
||||
key: str = Field(min_length=1, max_length=40)
|
||||
|
||||
|
||||
class InteractiveAnalyzeResponse(BaseModel):
|
||||
session_id: str
|
||||
page_url: str
|
||||
screenshot: str
|
||||
viewport: dict[str, int]
|
||||
candidates: list[MediaCandidateResponse]
|
||||
ublock_origin: bool = False
|
||||
|
||||
class ProbeResponse(BaseModel):
|
||||
provider: str
|
||||
title: str | None = None
|
||||
description: str | None = None
|
||||
thumbnail: str | None = None
|
||||
duration_seconds: int | None = None
|
||||
external_id: str | None = None
|
||||
allowed: bool = True
|
||||
|
||||
|
||||
class ImportRequest(BaseModel):
|
||||
url: AnyHttpUrl
|
||||
target_profile: str = Field(default="clip", pattern="^(film|serie|clip)$")
|
||||
title: str | None = Field(default=None, max_length=180)
|
||||
year: int | None = Field(default=None, ge=1888, le=2200)
|
||||
series_title: str | None = Field(default=None, max_length=180)
|
||||
season: int | None = Field(default=None, ge=1, le=200)
|
||||
episode: int | None = Field(default=None, ge=1, le=1000)
|
||||
episode_title: str | None = Field(default=None, max_length=180)
|
||||
target_library: str | None = Field(default=None, max_length=80) # legacy API alias
|
||||
max_height: int = Field(default=1080, ge=144, le=2160)
|
||||
audio_only: bool = False
|
||||
|
||||
|
||||
class ImportJobResponse(BaseModel):
|
||||
id: int
|
||||
source_id: int
|
||||
status: str
|
||||
target_library: str
|
||||
target_path: str | None = None
|
||||
progress: float
|
||||
error: str | None = None
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
|
||||
class SourceResponse(BaseModel):
|
||||
id: int
|
||||
kind: str
|
||||
url: str
|
||||
title: str | None = None
|
||||
provider: str | None = None
|
||||
external_id: str | None = None
|
||||
created_at: datetime
|
||||
0
backend/app/services/__init__.py
Normal file
0
backend/app/services/__init__.py
Normal file
122
backend/app/services/auth.py
Normal file
122
backend/app/services/auth.py
Normal file
@@ -0,0 +1,122 @@
|
||||
import base64
|
||||
import hashlib
|
||||
import hmac
|
||||
import secrets
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
|
||||
from argon2 import PasswordHasher
|
||||
from argon2.exceptions import InvalidHashError, VerifyMismatchError
|
||||
from fastapi import HTTPException, Request, Response, status
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.config import Settings
|
||||
|
||||
SESSION_COOKIE = "kino_session"
|
||||
SESSION_TTL_SECONDS = 60 * 60 * 12
|
||||
_ph = PasswordHasher()
|
||||
|
||||
|
||||
class LoginPayload(BaseModel):
|
||||
username: str
|
||||
password: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AuthenticatedUser:
|
||||
username: str
|
||||
|
||||
|
||||
def hash_password(password: str) -> str:
|
||||
return _ph.hash(password)
|
||||
|
||||
|
||||
def verify_password(password: str, password_hash: str) -> bool:
|
||||
try:
|
||||
return _ph.verify(password_hash, password)
|
||||
except (VerifyMismatchError, InvalidHashError):
|
||||
return False
|
||||
|
||||
|
||||
def auth_enabled(settings: Settings) -> bool:
|
||||
return bool(settings.admin_password_hash and settings.session_secret)
|
||||
|
||||
|
||||
def _sign(value: str, secret: str) -> str:
|
||||
return hmac.new(secret.encode("utf-8"), value.encode("utf-8"), hashlib.sha256).hexdigest()
|
||||
|
||||
|
||||
def create_session_cookie(username: str, settings: Settings, now: int | None = None) -> str:
|
||||
if not settings.session_secret:
|
||||
raise RuntimeError("session secret is not configured")
|
||||
issued = int(time.time() if now is None else now)
|
||||
expires = issued + SESSION_TTL_SECONDS
|
||||
payload = f"{username}:{expires}"
|
||||
encoded = base64.urlsafe_b64encode(payload.encode("utf-8")).decode("ascii").rstrip("=")
|
||||
return f"{encoded}.{_sign(encoded, settings.session_secret)}"
|
||||
|
||||
|
||||
def parse_session_cookie(cookie_value: str | None, settings: Settings, now: int | None = None) -> AuthenticatedUser | None:
|
||||
if not cookie_value or not settings.session_secret or "." not in cookie_value:
|
||||
return None
|
||||
encoded, signature = cookie_value.rsplit(".", 1)
|
||||
if not hmac.compare_digest(signature, _sign(encoded, settings.session_secret)):
|
||||
return None
|
||||
try:
|
||||
padding = "=" * (-len(encoded) % 4)
|
||||
payload = base64.urlsafe_b64decode((encoded + padding).encode("ascii")).decode("utf-8")
|
||||
username, expires_raw = payload.rsplit(":", 1)
|
||||
expires = int(expires_raw)
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
return None
|
||||
current = int(time.time() if now is None else now)
|
||||
if expires < current:
|
||||
return None
|
||||
if not secrets.compare_digest(username, settings.admin_username):
|
||||
return None
|
||||
return AuthenticatedUser(username=username)
|
||||
|
||||
|
||||
def require_user(request: Request) -> AuthenticatedUser | None:
|
||||
settings = request.app.state.settings
|
||||
if not auth_enabled(settings):
|
||||
return None
|
||||
user = parse_session_cookie(request.cookies.get(SESSION_COOKIE), settings)
|
||||
if user is None:
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Login required")
|
||||
return user
|
||||
|
||||
|
||||
def should_use_secure_cookie(request: Request) -> bool:
|
||||
forwarded_proto = request.headers.get("x-forwarded-proto", "").split(",", 1)[0].strip().lower()
|
||||
return request.url.scheme == "https" or forwarded_proto == "https"
|
||||
|
||||
|
||||
def login_response(payload: LoginPayload, request: Request, settings: Settings) -> Response:
|
||||
if not auth_enabled(settings):
|
||||
raise HTTPException(status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail="App login is not configured")
|
||||
if not secrets.compare_digest(payload.username, settings.admin_username):
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid username or password")
|
||||
if not verify_password(payload.password, settings.admin_password_hash or ""):
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid username or password")
|
||||
response = Response(status_code=status.HTTP_204_NO_CONTENT)
|
||||
response.set_cookie(
|
||||
SESSION_COOKIE,
|
||||
create_session_cookie(payload.username, settings),
|
||||
httponly=True,
|
||||
secure=should_use_secure_cookie(request),
|
||||
samesite="lax",
|
||||
max_age=SESSION_TTL_SECONDS,
|
||||
)
|
||||
return response
|
||||
|
||||
|
||||
def clear_session_cookie(request: Request | None = None) -> Response:
|
||||
response = Response(status_code=status.HTTP_204_NO_CONTENT)
|
||||
response.delete_cookie(
|
||||
SESSION_COOKIE,
|
||||
httponly=True,
|
||||
secure=should_use_secure_cookie(request) if request is not None else True,
|
||||
samesite="lax",
|
||||
)
|
||||
return response
|
||||
171
backend/app/services/downloader.py
Normal file
171
backend/app/services/downloader.py
Normal file
@@ -0,0 +1,171 @@
|
||||
import asyncio
|
||||
import json
|
||||
import shlex
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
from sqlmodel import Session
|
||||
|
||||
from app.config import Settings
|
||||
from app.models import ImportJob, Source, utcnow
|
||||
from app.providers.base import DownloadOptions, Provider, ProviderError
|
||||
from app.services.jellyfin import refresh_jellyfin
|
||||
from app.services.paths import jellyfin_profile_target, tmp_target
|
||||
|
||||
|
||||
def copy_tree_contents(src: Path, dest: Path) -> list[Path]:
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
copied: list[Path] = []
|
||||
for path in src.iterdir():
|
||||
if path.is_symlink():
|
||||
raise ProviderError(f"Refusing to copy symlink from import output: {path.name}")
|
||||
target = dest / path.name
|
||||
if path.is_dir():
|
||||
for child in path.rglob("*"):
|
||||
if child.is_symlink():
|
||||
raise ProviderError(f"Refusing to copy symlink from import output: {child.relative_to(src)}")
|
||||
if target.exists():
|
||||
shutil.rmtree(target)
|
||||
shutil.copytree(path, target)
|
||||
copied.extend(p for p in target.rglob("*") if p.is_file())
|
||||
else:
|
||||
shutil.copy2(path, target)
|
||||
copied.append(target)
|
||||
return copied
|
||||
|
||||
|
||||
async def sync_to_jellyfin_vm(local_dir: Path, settings: Settings) -> str | None:
|
||||
"""Copy the completed local import directory to Jellyfin's media VM via rsync."""
|
||||
if not settings.rsync_target or not settings.rsync_ssh_key:
|
||||
return None
|
||||
|
||||
local_dir = local_dir.resolve()
|
||||
media_root = settings.media_root.resolve()
|
||||
if media_root != local_dir and media_root not in local_dir.parents:
|
||||
raise ProviderError(f"Refusing to sync path outside media root: {local_dir}")
|
||||
|
||||
relative = local_dir.relative_to(media_root)
|
||||
remote_root = Path(str(settings.rsync_remote_root)).as_posix().rstrip("/") or "/jellyfin"
|
||||
remote_dir = f"{remote_root}/{relative.as_posix()}"
|
||||
ssh_cmd = [
|
||||
"ssh",
|
||||
"-i",
|
||||
str(settings.rsync_ssh_key),
|
||||
"-o",
|
||||
"BatchMode=yes",
|
||||
"-o",
|
||||
"StrictHostKeyChecking=accept-new",
|
||||
]
|
||||
|
||||
mkdir_proc = await asyncio.create_subprocess_exec(
|
||||
*ssh_cmd,
|
||||
settings.rsync_target,
|
||||
f"mkdir -p {shlex.quote(remote_dir)}",
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
_, mkdir_stderr = await mkdir_proc.communicate()
|
||||
if mkdir_proc.returncode != 0:
|
||||
raise ProviderError(f"Jellyfin transfer mkdir failed: {mkdir_stderr.decode(errors='replace')[:500]}")
|
||||
|
||||
rsync_proc = await asyncio.create_subprocess_exec(
|
||||
"rsync",
|
||||
"-a",
|
||||
"--delete",
|
||||
"-s",
|
||||
"-e",
|
||||
" ".join(shlex.quote(part) for part in ssh_cmd),
|
||||
f"{local_dir}/",
|
||||
f"{settings.rsync_target}:{remote_dir}/",
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
_, rsync_stderr = await rsync_proc.communicate()
|
||||
if rsync_proc.returncode != 0:
|
||||
raise ProviderError(f"Jellyfin transfer rsync failed: {rsync_stderr.decode(errors='replace')[:500]}")
|
||||
|
||||
return f"{settings.rsync_target}:{remote_dir}"
|
||||
|
||||
|
||||
def update_job(session: Session, job: ImportJob, **fields) -> None:
|
||||
for key, value in fields.items():
|
||||
setattr(job, key, value)
|
||||
job.updated_at = utcnow()
|
||||
session.add(job)
|
||||
session.commit()
|
||||
session.refresh(job)
|
||||
|
||||
|
||||
async def run_import_job(job_id: int, engine, providers: list[Provider], settings: Settings) -> None:
|
||||
with Session(engine) as session:
|
||||
job = session.get(ImportJob, job_id)
|
||||
if job is None:
|
||||
return
|
||||
source = session.get(Source, job.source_id)
|
||||
if source is None:
|
||||
update_job(session, job, status="failed", error="Source missing")
|
||||
return
|
||||
provider = next((p for p in providers if p.name == source.provider or p.can_handle(source.url)), None)
|
||||
if provider is None:
|
||||
update_job(session, job, status="failed", error="No provider available for source")
|
||||
return
|
||||
try:
|
||||
last_reported_progress = 0.0
|
||||
|
||||
def report_download_progress(provider_progress: float) -> None:
|
||||
nonlocal last_reported_progress
|
||||
mapped = 0.15 + (max(0.0, min(provider_progress, 1.0)) * 0.60)
|
||||
if mapped - last_reported_progress >= 0.01 or mapped >= 0.75:
|
||||
last_reported_progress = mapped
|
||||
update_job(session, job, status="downloading", progress=round(mapped, 3))
|
||||
|
||||
update_job(session, job, status="downloading", progress=0.15)
|
||||
title = source.title or source.external_id or "media"
|
||||
stored_metadata = json.loads(source.metadata_json or "{}")
|
||||
import_request = stored_metadata.get("import_request", {}) if isinstance(stored_metadata, dict) else {}
|
||||
provider_metadata = stored_metadata.get("metadata", stored_metadata) if isinstance(stored_metadata, dict) else {}
|
||||
tmp_dir = tmp_target(settings.tmp_dir, job.id or 0, title)
|
||||
result = await provider.download(
|
||||
source.url,
|
||||
tmp_dir,
|
||||
DownloadOptions(
|
||||
max_height=settings.default_max_height,
|
||||
target_library=job.target_library,
|
||||
max_bytes=settings.max_download_bytes,
|
||||
),
|
||||
progress_callback=report_download_progress,
|
||||
)
|
||||
update_job(session, job, status="postprocessing", progress=0.75)
|
||||
target_dir = jellyfin_profile_target(
|
||||
settings.media_root,
|
||||
job.target_library,
|
||||
import_request.get("title") or title,
|
||||
year=import_request.get("year"),
|
||||
series_title=import_request.get("series_title"),
|
||||
season=import_request.get("season"),
|
||||
episode=import_request.get("episode"),
|
||||
episode_title=import_request.get("episode_title"),
|
||||
channel=provider_metadata.get("uploader") or provider_metadata.get("channel") or source.provider,
|
||||
external_id=source.external_id,
|
||||
)
|
||||
copied = copy_tree_contents(tmp_dir, target_dir)
|
||||
if not copied and not result.output_files:
|
||||
raise ProviderError("No files were produced by the import")
|
||||
remote_target = await sync_to_jellyfin_vm(target_dir, settings)
|
||||
update_job(session, job, target_path=remote_target or str(target_dir), status="refreshing", progress=0.9)
|
||||
refreshed = await refresh_jellyfin(settings)
|
||||
note = None if refreshed else "Import completed; Jellyfin refresh skipped because no API key is configured"
|
||||
update_job(session, job, status="done", progress=1.0, error=note)
|
||||
except Exception as exc: # keep background task failures visible in DB/API
|
||||
update_job(session, job, status="failed", error=str(exc)[:1000])
|
||||
|
||||
|
||||
def schedule_import(job_id: int, engine, providers: list[Provider], settings: Settings) -> None:
|
||||
asyncio.create_task(run_import_job(job_id, engine, providers, settings))
|
||||
|
||||
|
||||
def metadata_json(metadata, *, import_request: dict | None = None) -> str:
|
||||
payload = metadata.model_dump()
|
||||
if import_request is not None:
|
||||
payload = {"metadata": payload, "import_request": import_request}
|
||||
return json.dumps(payload, ensure_ascii=False)
|
||||
202
backend/app/services/interactive_browser.py
Normal file
202
backend/app/services/interactive_browser.py
Normal file
@@ -0,0 +1,202 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import time
|
||||
import uuid
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
from urllib.parse import unquote, urlparse
|
||||
|
||||
from app.config import get_settings
|
||||
from app.services.page_analyzer import (
|
||||
MediaCandidate,
|
||||
_browser_context,
|
||||
_is_ad_or_tracker_url,
|
||||
_looks_like_media_response,
|
||||
_probe_candidates_with_meta,
|
||||
extract_media_urls,
|
||||
)
|
||||
from app.services.url_safety import UnsafeUrlError, assert_public_host, check_url_for_page
|
||||
|
||||
_INTERACTIVE_CAPTURE_LIMIT = 80
|
||||
_DEFAULT_VIEWPORT = {"width": 1280, "height": 720}
|
||||
_SESSION_TTL_SECONDS = 15 * 60
|
||||
|
||||
|
||||
@dataclass
|
||||
class InteractiveBrowserSession:
|
||||
id: str
|
||||
page_url: str
|
||||
context: Any
|
||||
page: Any
|
||||
browser: Any | None = None
|
||||
user_data_dir: Any | None = None
|
||||
ublock_path: str | None = None
|
||||
seen: dict[str, tuple[str | None, str | None]] = field(default_factory=dict)
|
||||
pending_response_tasks: set[asyncio.Task] = field(default_factory=set)
|
||||
created_at: float = field(default_factory=time.monotonic)
|
||||
updated_at: float = field(default_factory=time.monotonic)
|
||||
|
||||
|
||||
class InteractiveBrowserManager:
|
||||
"""Small in-process Playwright session manager for interactive media analysis.
|
||||
|
||||
The browser never receives private/internal network requests: every request is
|
||||
routed through the same public-host guard used by the non-interactive browser
|
||||
analyzer. The UI gets screenshots and sends clicks/keystrokes back, avoiding
|
||||
iframe/CORS limitations while still allowing the user to press Play.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._sessions: dict[str, InteractiveBrowserSession] = {}
|
||||
self._playwright: Any | None = None
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
async def _ensure_playwright(self):
|
||||
if self._playwright is None:
|
||||
from playwright.async_api import async_playwright
|
||||
|
||||
self._playwright = await async_playwright().start()
|
||||
return self._playwright
|
||||
|
||||
async def _close_session(self, session: InteractiveBrowserSession) -> None:
|
||||
for task in list(session.pending_response_tasks):
|
||||
if not task.done():
|
||||
task.cancel()
|
||||
try:
|
||||
await session.context.close()
|
||||
finally:
|
||||
if session.browser is not None:
|
||||
await session.browser.close()
|
||||
if session.user_data_dir is not None:
|
||||
session.user_data_dir.cleanup()
|
||||
|
||||
async def cleanup_expired(self) -> None:
|
||||
now = time.monotonic()
|
||||
expired = [sid for sid, sess in self._sessions.items() if now - sess.updated_at > _SESSION_TTL_SECONDS]
|
||||
for sid in expired:
|
||||
await self.stop(sid)
|
||||
|
||||
async def start(self, url: str) -> InteractiveBrowserSession:
|
||||
await self.cleanup_expired()
|
||||
settings = get_settings()
|
||||
checked_page = await check_url_for_page(url, timeout=settings.request_timeout)
|
||||
playwright = await self._ensure_playwright()
|
||||
context, browser, user_data_dir, ublock_path = await _browser_context(playwright, ignore_https_errors=False)
|
||||
await context.set_extra_http_headers({"DNT": "1"})
|
||||
|
||||
session = InteractiveBrowserSession(
|
||||
id=uuid.uuid4().hex,
|
||||
page_url=checked_page.final_url,
|
||||
context=context,
|
||||
page=await context.new_page(),
|
||||
browser=browser,
|
||||
user_data_dir=user_data_dir,
|
||||
ublock_path=ublock_path,
|
||||
)
|
||||
await session.page.set_viewport_size(_DEFAULT_VIEWPORT)
|
||||
|
||||
async def guard_route(route):
|
||||
request_url = route.request.url
|
||||
parsed = urlparse(request_url)
|
||||
if parsed.scheme not in {"http", "https"}:
|
||||
await route.abort()
|
||||
return
|
||||
if _is_ad_or_tracker_url(request_url):
|
||||
await route.abort()
|
||||
return
|
||||
try:
|
||||
await assert_public_host(request_url)
|
||||
except UnsafeUrlError:
|
||||
await route.abort()
|
||||
return
|
||||
await route.continue_()
|
||||
|
||||
async def on_response(response):
|
||||
response_url = response.url
|
||||
try:
|
||||
headers = await response.all_headers()
|
||||
except Exception:
|
||||
headers = {}
|
||||
content_type = headers.get("content-type")
|
||||
if _looks_like_media_response(response_url, content_type):
|
||||
title = unquote(urlparse(response_url).path.rsplit("/", 1)[-1]) or response_url
|
||||
session.seen.setdefault(response_url, (title, content_type))
|
||||
|
||||
def schedule_response_probe(response):
|
||||
task = asyncio.create_task(on_response(response))
|
||||
session.pending_response_tasks.add(task)
|
||||
task.add_done_callback(session.pending_response_tasks.discard)
|
||||
|
||||
await context.route("**/*", guard_route)
|
||||
context.on("response", schedule_response_probe)
|
||||
try:
|
||||
await session.page.goto(checked_page.final_url, wait_until="domcontentloaded", timeout=int(settings.request_timeout * 1000))
|
||||
try:
|
||||
await session.page.wait_for_load_state("networkidle", timeout=3000)
|
||||
except Exception:
|
||||
pass
|
||||
await self._collect_dom_media(session)
|
||||
except Exception:
|
||||
await self._close_session(session)
|
||||
raise
|
||||
|
||||
self._sessions[session.id] = session
|
||||
return session
|
||||
|
||||
async def _collect_dom_media(self, session: InteractiveBrowserSession) -> None:
|
||||
try:
|
||||
html = await session.page.content()
|
||||
page_url = session.page.url
|
||||
except Exception:
|
||||
return
|
||||
for dom_url in extract_media_urls(html, page_url):
|
||||
session.seen.setdefault(dom_url, (None, None))
|
||||
|
||||
async def _settle(self, session: InteractiveBrowserSession) -> None:
|
||||
session.updated_at = time.monotonic()
|
||||
await asyncio.sleep(0.5)
|
||||
if session.pending_response_tasks:
|
||||
await asyncio.gather(*list(session.pending_response_tasks), return_exceptions=True)
|
||||
await self._collect_dom_media(session)
|
||||
|
||||
def get(self, session_id: str) -> InteractiveBrowserSession:
|
||||
session = self._sessions.get(session_id)
|
||||
if not session:
|
||||
raise KeyError(session_id)
|
||||
session.updated_at = time.monotonic()
|
||||
return session
|
||||
|
||||
async def click(self, session_id: str, x: float, y: float) -> InteractiveBrowserSession:
|
||||
session = self.get(session_id)
|
||||
await session.page.mouse.click(x, y)
|
||||
await self._settle(session)
|
||||
return session
|
||||
|
||||
async def press(self, session_id: str, key: str) -> InteractiveBrowserSession:
|
||||
session = self.get(session_id)
|
||||
await session.page.keyboard.press(key)
|
||||
await self._settle(session)
|
||||
return session
|
||||
|
||||
async def stop(self, session_id: str) -> None:
|
||||
session = self._sessions.pop(session_id, None)
|
||||
if session is not None:
|
||||
await self._close_session(session)
|
||||
|
||||
async def screenshot_data_url(self, session: InteractiveBrowserSession) -> str:
|
||||
png = await session.page.screenshot(type="png", full_page=False)
|
||||
return "data:image/png;base64," + base64.b64encode(png).decode("ascii")
|
||||
|
||||
async def candidates(self, session: InteractiveBrowserSession) -> list[MediaCandidate]:
|
||||
if session.pending_response_tasks:
|
||||
await asyncio.gather(*list(session.pending_response_tasks), return_exceptions=True)
|
||||
await self._collect_dom_media(session)
|
||||
return await _probe_candidates_with_meta(
|
||||
dict(list(session.seen.items())[:_INTERACTIVE_CAPTURE_LIMIT]),
|
||||
source="interactive-browser",
|
||||
)
|
||||
|
||||
|
||||
interactive_browser_manager = InteractiveBrowserManager()
|
||||
23
backend/app/services/jellyfin.py
Normal file
23
backend/app/services/jellyfin.py
Normal file
@@ -0,0 +1,23 @@
|
||||
import httpx
|
||||
|
||||
from app.config import Settings
|
||||
|
||||
|
||||
class JellyfinRefreshSkipped(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
async def refresh_jellyfin(settings: Settings) -> bool:
|
||||
"""Trigger Jellyfin's library refresh when credentials are configured.
|
||||
|
||||
Returns False when no API key is present so local/dev imports can still finish
|
||||
without writing secrets into the project tree.
|
||||
"""
|
||||
if not settings.jellyfin_api_key:
|
||||
return False
|
||||
url = settings.jellyfin_url.rstrip("/") + "/Library/Refresh"
|
||||
headers = {"X-Emby-Token": settings.jellyfin_api_key}
|
||||
async with httpx.AsyncClient(timeout=20) as client:
|
||||
response = await client.post(url, headers=headers)
|
||||
response.raise_for_status()
|
||||
return True
|
||||
489
backend/app/services/page_analyzer.py
Normal file
489
backend/app/services/page_analyzer.py
Normal file
@@ -0,0 +1,489 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import html as html_lib
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
from dataclasses import dataclass, field
|
||||
from html.parser import HTMLParser
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import unquote, urljoin, urlparse
|
||||
|
||||
import httpx
|
||||
try:
|
||||
from yt_dlp import YoutubeDL
|
||||
except ImportError: # optional analyzer backend
|
||||
YoutubeDL = None # type: ignore[assignment]
|
||||
|
||||
from app.config import get_settings
|
||||
from app.services.url_safety import UnsafeUrlError, assert_public_host, check_url_for_download, check_url_for_page
|
||||
|
||||
_MEDIA_EXTENSIONS = (".mp4", ".m4v", ".webm", ".mkv", ".mov", ".m3u8", ".mpd")
|
||||
_MEDIA_CONTENT_TYPES = {
|
||||
"video/mp4": "direct_video",
|
||||
"video/webm": "direct_video",
|
||||
"video/x-matroska": "direct_video",
|
||||
"video/mp2t": "hls_segment",
|
||||
"application/vnd.apple.mpegurl": "hls_manifest",
|
||||
"application/x-mpegurl": "hls_manifest",
|
||||
"application/dash+xml": "dash_manifest",
|
||||
}
|
||||
_QUOTED_URL_RE = re.compile(r"[\"']([^\"']+\.(?:mp4|m4v|webm|mkv|mov|m3u8|mpd)(?:\?[^\"']*)?)[\"']", re.I)
|
||||
_ABSOLUTE_MEDIA_URL_RE = re.compile(
|
||||
r"(?:https?://|https?:\\/\\/|//|\\/\\/)[^\s\"'<>]+?\.(?:mp4|m4v|webm|mkv|mov|m3u8|mpd)(?=$|[?&#\s\"'<>])(?:[?&][^\s\"'<>]*)?",
|
||||
re.I,
|
||||
)
|
||||
_BROWSER_CAPTURE_LIMIT = 80
|
||||
_YTDLP_ALLOWED_HOST_SUFFIXES = (
|
||||
"youtube.com",
|
||||
"youtu.be",
|
||||
"archive.org",
|
||||
"ardmediathek.de",
|
||||
"zdf.de",
|
||||
"arte.tv",
|
||||
)
|
||||
_AD_HOST_FRAGMENTS = (
|
||||
"doubleclick.net",
|
||||
"googlesyndication.com",
|
||||
"googleadservices.com",
|
||||
"adservice.google.",
|
||||
"adsystem.com",
|
||||
"adnxs.com",
|
||||
"adsafeprotected.com",
|
||||
"amazon-adsystem.com",
|
||||
"taboola.com",
|
||||
"outbrain.com",
|
||||
"scorecardresearch.com",
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class MediaCandidate:
|
||||
url: str
|
||||
kind: str
|
||||
title: str | None = None
|
||||
source: str = "html"
|
||||
content_type: str | None = None
|
||||
content_length: int | None = None
|
||||
file_size: str | None = None
|
||||
quality: str | None = None
|
||||
allowed: bool = True
|
||||
reason: str | None = None
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
class _MediaHTMLParser(HTMLParser):
|
||||
def __init__(self, base_url: str) -> None:
|
||||
super().__init__()
|
||||
self.base_url = base_url
|
||||
self.urls: list[str] = []
|
||||
|
||||
def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
|
||||
attr_map = {key.lower(): value for key, value in attrs if value}
|
||||
for attr in ("src", "href", "data-src", "data-url"):
|
||||
value = attr_map.get(attr)
|
||||
if value and _looks_like_media_url(value):
|
||||
self.urls.append(urljoin(self.base_url, value))
|
||||
|
||||
|
||||
def _looks_like_media_url(value: str) -> bool:
|
||||
parsed_path = urlparse(value).path.lower()
|
||||
return parsed_path.endswith(_MEDIA_EXTENSIONS)
|
||||
|
||||
|
||||
def _looks_like_media_response(url: str, content_type: str | None = None) -> bool:
|
||||
clean = (content_type or "").split(";", 1)[0].strip().lower()
|
||||
return clean in _MEDIA_CONTENT_TYPES or clean.startswith("video/") or _looks_like_media_url(url)
|
||||
|
||||
|
||||
def human_size(size: int | None) -> str | None:
|
||||
if size is None:
|
||||
return None
|
||||
value = float(size)
|
||||
for unit in ("B", "KB", "MB", "GB"):
|
||||
if value < 1024 or unit == "GB":
|
||||
if unit == "B":
|
||||
return f"{int(value)} {unit}"
|
||||
return f"{value:.1f} {unit}"
|
||||
value /= 1024
|
||||
|
||||
|
||||
def _is_ad_or_tracker_url(url: str) -> bool:
|
||||
host = (urlparse(url).hostname or "").lower()
|
||||
if any(fragment in host for fragment in _AD_HOST_FRAGMENTS):
|
||||
return True
|
||||
path = urlparse(url).path.lower()
|
||||
return any(marker in path for marker in ("/ads/", "/adserver/", "/prebid", "/vast", "/vpaid"))
|
||||
|
||||
|
||||
def _host_allowed_for_ytdlp(url: str) -> bool:
|
||||
host = (urlparse(url).hostname or "").lower().rstrip(".")
|
||||
return any(host == suffix or host.endswith(f".{suffix}") for suffix in _YTDLP_ALLOWED_HOST_SUFFIXES)
|
||||
|
||||
|
||||
async def _fetch_public_page_body(url: str, *, timeout: float, max_redirects: int = 8) -> tuple[str, str]:
|
||||
"""Fetch a page body while validating every GET redirect hop.
|
||||
|
||||
``check_url_for_page`` validates HEAD redirects, but some origins redirect
|
||||
differently for GET. Keep GET redirects manual so the analyzer cannot be
|
||||
turned into an internal-network fetch proxy by a method-dependent redirect.
|
||||
"""
|
||||
current = url
|
||||
async with httpx.AsyncClient(follow_redirects=False, timeout=timeout) as client:
|
||||
for _ in range(max_redirects + 1):
|
||||
checked = await check_url_for_page(current, timeout=timeout, max_redirects=max_redirects)
|
||||
response = await client.get(checked.final_url)
|
||||
if response.is_redirect:
|
||||
loc = response.headers.get("location")
|
||||
if not loc:
|
||||
raise UnsafeUrlError("Redirect ohne Location-Header")
|
||||
current = str(response.url.join(loc))
|
||||
continue
|
||||
response.raise_for_status()
|
||||
await assert_public_host(str(response.url))
|
||||
return response.text, str(response.url)
|
||||
raise UnsafeUrlError("Zu viele Weiterleitungen")
|
||||
|
||||
|
||||
def _ublock_origin_path() -> Path | None:
|
||||
configured = os.environ.get("KINOPROJEKT_UBLOCK_PATH")
|
||||
candidates = [Path(configured)] if configured else []
|
||||
candidates.extend([Path("/opt/kino-projekt/ublock-origin"), Path("/opt/kino-projekt/ublock")])
|
||||
for candidate in candidates:
|
||||
if not candidate:
|
||||
continue
|
||||
if candidate.exists() and candidate.is_dir() and (candidate / "manifest.json").exists():
|
||||
return candidate
|
||||
nested = candidate / "uBlock0.chromium"
|
||||
if nested.exists() and nested.is_dir() and (nested / "manifest.json").exists():
|
||||
return nested
|
||||
return None
|
||||
|
||||
|
||||
def _has_importable_media(candidates: dict[str, tuple[str | None, str | None]]) -> bool:
|
||||
return any(
|
||||
classify_candidate(url, content_type).kind in {"direct_video", "hls_manifest", "dash_manifest"}
|
||||
for url, (_, content_type) in candidates.items()
|
||||
)
|
||||
|
||||
|
||||
def _normalize_media_url(raw: str, base_url: str) -> str | None:
|
||||
value = html_lib.unescape(raw.strip())
|
||||
if not value:
|
||||
return None
|
||||
value = value.replace("\\/", "/").replace("\\u002F", "/").replace("\\u002f", "/")
|
||||
value = value.rstrip(",;)}]")
|
||||
if value.startswith("//"):
|
||||
value = f"{urlparse(base_url).scheme or 'https'}:{value}"
|
||||
return urljoin(base_url, unquote(value))
|
||||
|
||||
|
||||
def extract_media_urls(html: str, base_url: str) -> list[str]:
|
||||
parser = _MediaHTMLParser(base_url)
|
||||
parser.feed(html)
|
||||
urls = list(parser.urls)
|
||||
for match in _QUOTED_URL_RE.finditer(html):
|
||||
normalized = _normalize_media_url(match.group(1), base_url)
|
||||
if normalized:
|
||||
urls.append(normalized)
|
||||
for match in _ABSOLUTE_MEDIA_URL_RE.finditer(html):
|
||||
normalized = _normalize_media_url(match.group(0), base_url)
|
||||
if normalized:
|
||||
urls.append(normalized)
|
||||
# Stable de-duplication while preserving discovery order.
|
||||
return list(dict.fromkeys(urls))
|
||||
|
||||
|
||||
def classify_candidate(url: str, content_type: str | None = None) -> MediaCandidate:
|
||||
normalized_content_type = (content_type or "").split(";", 1)[0].strip().lower()
|
||||
if normalized_content_type in _MEDIA_CONTENT_TYPES:
|
||||
kind = _MEDIA_CONTENT_TYPES[normalized_content_type]
|
||||
elif normalized_content_type.startswith("video/"):
|
||||
kind = "direct_video"
|
||||
else:
|
||||
path = urlparse(url).path.lower()
|
||||
if path.endswith(".m3u8"):
|
||||
kind = "hls_manifest"
|
||||
elif path.endswith(".mpd"):
|
||||
kind = "dash_manifest"
|
||||
elif path.endswith((".mp4", ".m4v", ".webm", ".mkv", ".mov")):
|
||||
kind = "direct_video"
|
||||
else:
|
||||
kind = "unknown_media"
|
||||
title = unquote(urlparse(url).path.rsplit("/", 1)[-1]) or url
|
||||
return MediaCandidate(url=url, kind=kind, title=title, content_type=content_type)
|
||||
|
||||
|
||||
async def _probe_candidate(url: str, source: str, title: str | None = None, quality: str | None = None) -> MediaCandidate:
|
||||
settings = get_settings()
|
||||
try:
|
||||
checked = await check_url_for_download(url, max_bytes=settings.max_download_bytes, timeout=settings.request_timeout)
|
||||
candidate = classify_candidate(checked.final_url, checked.content_type)
|
||||
candidate.source = source
|
||||
candidate.title = title or candidate.title
|
||||
candidate.quality = quality
|
||||
candidate.content_length = checked.content_length
|
||||
candidate.file_size = human_size(checked.content_length)
|
||||
candidate.metadata = {"redirect_chain": checked.chain}
|
||||
return candidate
|
||||
except UnsafeUrlError as exc:
|
||||
c = classify_candidate(url)
|
||||
c.source = source
|
||||
c.title = title or c.title
|
||||
c.quality = quality
|
||||
c.allowed = False
|
||||
c.reason = str(exc)
|
||||
return c
|
||||
|
||||
|
||||
async def analyze_html_page(url: str) -> list[MediaCandidate]:
|
||||
settings = get_settings()
|
||||
# First validate the page URL itself so the analyzer cannot be used as an internal-network proxy.
|
||||
checked_page = await check_url_for_page(url, timeout=settings.request_timeout)
|
||||
body, final_url = await _fetch_public_page_body(checked_page.final_url, timeout=settings.request_timeout)
|
||||
found = extract_media_urls(body, final_url)
|
||||
return await _probe_candidates(found, source="html")
|
||||
|
||||
|
||||
async def analyze_with_ytdlp(url: str) -> list[MediaCandidate]:
|
||||
checked_page = await check_url_for_page(url, timeout=get_settings().request_timeout)
|
||||
|
||||
if not _host_allowed_for_ytdlp(checked_page.final_url):
|
||||
return []
|
||||
|
||||
if YoutubeDL is None:
|
||||
return []
|
||||
|
||||
def _extract() -> dict[str, Any] | None:
|
||||
opts = {
|
||||
"quiet": True,
|
||||
"skip_download": True,
|
||||
"noplaylist": True,
|
||||
"extract_flat": False,
|
||||
"socket_timeout": get_settings().request_timeout,
|
||||
}
|
||||
with YoutubeDL(opts) as ydl:
|
||||
return ydl.extract_info(checked_page.final_url, download=False)
|
||||
|
||||
try:
|
||||
info = await asyncio.to_thread(_extract)
|
||||
except Exception:
|
||||
return []
|
||||
if not info:
|
||||
return []
|
||||
|
||||
candidates: list[tuple[str, str | None, str | None]] = []
|
||||
title = info.get("title")
|
||||
if info.get("url") and _looks_like_media_url(info["url"]):
|
||||
candidates.append((info["url"], title, None))
|
||||
for fmt in info.get("formats") or []:
|
||||
fmt_url = fmt.get("url")
|
||||
if not fmt_url:
|
||||
continue
|
||||
ext = fmt.get("ext") or ""
|
||||
protocol = fmt.get("protocol") or ""
|
||||
if _looks_like_media_url(fmt_url) or ext in {"mp4", "webm", "m3u8", "mpd"} or "m3u8" in protocol or "dash" in protocol:
|
||||
quality = fmt.get("format_note") or fmt.get("format") or fmt.get("height")
|
||||
candidates.append((fmt_url, title, str(quality) if quality else None))
|
||||
deduped: dict[str, tuple[str | None, str | None]] = {}
|
||||
for candidate_url, candidate_title, quality in candidates:
|
||||
deduped.setdefault(candidate_url, (candidate_title, quality))
|
||||
return await _probe_candidates_with_meta(deduped, source="yt-dlp")
|
||||
|
||||
|
||||
async def _trigger_video_playback(page) -> None:
|
||||
"""Try to start embedded players so lazy video requests become visible."""
|
||||
await page.evaluate(
|
||||
"""
|
||||
async () => {
|
||||
for (const text of ['Akzeptieren', 'Accept', 'I agree', 'OK']) {
|
||||
const buttons = [...document.querySelectorAll('button, [role="button"], input[type="button"], input[type="submit"]')];
|
||||
const button = buttons.find(el => (el.innerText || el.value || '').toLowerCase().includes(text.toLowerCase()));
|
||||
if (button) { try { button.click(); } catch (_) {} }
|
||||
}
|
||||
for (const video of document.querySelectorAll('video')) {
|
||||
try {
|
||||
video.muted = true;
|
||||
video.playsInline = true;
|
||||
video.autoplay = true;
|
||||
video.play().catch(()=>{});
|
||||
} catch (_) {}
|
||||
}
|
||||
const playSelectors = [
|
||||
'[aria-label*="play" i]', '[title*="play" i]', '.play', '.play-button', '.vjs-big-play-button',
|
||||
'button[class*="play" i]', 'button[aria-label*="abspielen" i]'
|
||||
];
|
||||
for (const selector of playSelectors) {
|
||||
for (const el of document.querySelectorAll(selector)) {
|
||||
try { el.click(); } catch (_) {}
|
||||
}
|
||||
}
|
||||
}
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
async def _browser_context(playwright, *, ignore_https_errors: bool = False):
|
||||
base_args = [
|
||||
"--disable-dev-shm-usage",
|
||||
"--disable-gpu",
|
||||
"--no-sandbox",
|
||||
"--disable-setuid-sandbox",
|
||||
"--disable-background-networking",
|
||||
]
|
||||
ublock_path = _ublock_origin_path()
|
||||
if ublock_path:
|
||||
user_data_dir = tempfile.TemporaryDirectory(prefix="kino-browser-")
|
||||
context = await playwright.chromium.launch_persistent_context(
|
||||
user_data_dir.name,
|
||||
headless=True,
|
||||
ignore_https_errors=ignore_https_errors,
|
||||
java_script_enabled=True,
|
||||
args=[
|
||||
*base_args,
|
||||
f"--disable-extensions-except={ublock_path}",
|
||||
f"--load-extension={ublock_path}",
|
||||
],
|
||||
)
|
||||
return context, None, user_data_dir, str(ublock_path)
|
||||
|
||||
browser = await playwright.chromium.launch(headless=True, args=base_args)
|
||||
context = await browser.new_context(ignore_https_errors=ignore_https_errors, java_script_enabled=True)
|
||||
return context, browser, None, None
|
||||
|
||||
|
||||
async def analyze_with_browser(url: str) -> list[MediaCandidate]:
|
||||
"""Headless-Browser-Analyzer (Option B): observe JS-loaded media requests.
|
||||
|
||||
Every browser request is still constrained to public http(s) targets. This keeps the
|
||||
server-side browser from becoming a LAN/internal-network fetch proxy.
|
||||
"""
|
||||
settings = get_settings()
|
||||
checked_page = await check_url_for_page(url, timeout=settings.request_timeout)
|
||||
try:
|
||||
from playwright.async_api import TimeoutError as PlaywrightTimeoutError
|
||||
from playwright.async_api import async_playwright
|
||||
except ImportError:
|
||||
return []
|
||||
|
||||
seen: dict[str, tuple[str | None, str | None]] = {}
|
||||
playback_triggered = False
|
||||
|
||||
async with async_playwright() as p:
|
||||
context, browser, user_data_dir, ublock_path = await _browser_context(p, ignore_https_errors=False)
|
||||
|
||||
async def guard_route(route):
|
||||
request_url = route.request.url
|
||||
parsed = urlparse(request_url)
|
||||
if parsed.scheme not in {"http", "https"}:
|
||||
await route.abort()
|
||||
return
|
||||
if _is_ad_or_tracker_url(request_url):
|
||||
await route.abort()
|
||||
return
|
||||
try:
|
||||
await assert_public_host(request_url)
|
||||
except UnsafeUrlError:
|
||||
await route.abort()
|
||||
return
|
||||
await route.continue_()
|
||||
|
||||
async def on_response(response):
|
||||
response_url = response.url
|
||||
headers = await response.all_headers()
|
||||
content_type = headers.get("content-type")
|
||||
if _looks_like_media_response(response_url, content_type):
|
||||
title = unquote(urlparse(response_url).path.rsplit("/", 1)[-1]) or response_url
|
||||
seen.setdefault(response_url, (title, content_type))
|
||||
|
||||
await context.route("**/*", guard_route)
|
||||
pending_response_tasks: set[asyncio.Task] = set()
|
||||
|
||||
def schedule_response_probe(response):
|
||||
task = asyncio.create_task(on_response(response))
|
||||
pending_response_tasks.add(task)
|
||||
task.add_done_callback(pending_response_tasks.discard)
|
||||
|
||||
context.on("response", schedule_response_probe)
|
||||
page = await context.new_page()
|
||||
try:
|
||||
await page.goto(checked_page.final_url, wait_until="domcontentloaded", timeout=int(settings.request_timeout * 1000))
|
||||
try:
|
||||
await page.wait_for_load_state("networkidle", timeout=5000)
|
||||
except PlaywrightTimeoutError:
|
||||
pass
|
||||
if pending_response_tasks:
|
||||
await asyncio.gather(*pending_response_tasks, return_exceptions=True)
|
||||
if not _has_importable_media(seen):
|
||||
playback_triggered = True
|
||||
await _trigger_video_playback(page)
|
||||
try:
|
||||
await page.wait_for_load_state("networkidle", timeout=5000)
|
||||
except PlaywrightTimeoutError:
|
||||
pass
|
||||
if pending_response_tasks:
|
||||
await asyncio.gather(*pending_response_tasks, return_exceptions=True)
|
||||
html = await page.content()
|
||||
for dom_url in extract_media_urls(html, page.url):
|
||||
seen.setdefault(dom_url, (None, None))
|
||||
finally:
|
||||
await context.close()
|
||||
if browser is not None:
|
||||
await browser.close()
|
||||
if user_data_dir is not None:
|
||||
user_data_dir.cleanup()
|
||||
|
||||
candidates = await _probe_candidates_with_meta(dict(list(seen.items())[:_BROWSER_CAPTURE_LIMIT]), source="browser")
|
||||
for candidate in candidates:
|
||||
candidate.metadata["playback_triggered"] = playback_triggered
|
||||
candidate.metadata["ublock_origin"] = bool(ublock_path)
|
||||
return candidates
|
||||
|
||||
|
||||
async def analyze_browser_capture(captured: list[dict[str, Any]], page_url: str | None = None) -> list[MediaCandidate]:
|
||||
"""Validate and normalize media URLs captured by a user's real browser (Option C)."""
|
||||
base_url = page_url or ""
|
||||
deduped: dict[str, tuple[str | None, str | None]] = {}
|
||||
for item in captured[:_BROWSER_CAPTURE_LIMIT]:
|
||||
raw_url = str(item.get("url") or "").strip()
|
||||
if not raw_url:
|
||||
continue
|
||||
candidate_url = urljoin(base_url, raw_url) if base_url else raw_url
|
||||
if not _looks_like_media_response(candidate_url, item.get("content_type") or item.get("type")):
|
||||
continue
|
||||
title = item.get("title") or item.get("name")
|
||||
quality = item.get("quality")
|
||||
deduped.setdefault(candidate_url, (str(title) if title else None, str(quality) if quality else None))
|
||||
return await _probe_candidates_with_meta(deduped, source="browser-capture")
|
||||
|
||||
|
||||
async def _probe_candidates(urls: list[str], source: str) -> list[MediaCandidate]:
|
||||
return await _probe_candidates_with_meta({url: (None, None) for url in urls}, source=source)
|
||||
|
||||
|
||||
async def _probe_candidates_with_meta(urls: dict[str, tuple[str | None, str | None]], source: str) -> list[MediaCandidate]:
|
||||
candidates = [await _probe_candidate(url, source, title, quality) for url, (title, quality) in list(urls.items())[:50]]
|
||||
if any(candidate.kind in {"hls_manifest", "dash_manifest"} for candidate in candidates):
|
||||
candidates = [candidate for candidate in candidates if candidate.kind != "hls_segment"]
|
||||
# Show usable candidates first, then blocked candidates as diagnostics.
|
||||
candidates.sort(key=lambda c: (not c.allowed, c.kind, c.quality or ""))
|
||||
return candidates
|
||||
|
||||
|
||||
async def analyze_page(url: str) -> list[MediaCandidate]:
|
||||
results = await asyncio.gather(
|
||||
analyze_html_page(url),
|
||||
analyze_with_ytdlp(url),
|
||||
analyze_with_browser(url),
|
||||
return_exceptions=True,
|
||||
)
|
||||
html_candidates, ytdlp_candidates, browser_candidates = [item if isinstance(item, list) else [] for item in results]
|
||||
merged: dict[str, MediaCandidate] = {}
|
||||
# Prefer browser/ytdlp metadata over static HTML when the same URL appears twice.
|
||||
for candidate in [*browser_candidates, *ytdlp_candidates, *html_candidates]:
|
||||
existing = merged.get(candidate.url)
|
||||
if not existing or (not existing.allowed and candidate.allowed):
|
||||
merged[candidate.url] = candidate
|
||||
return list(merged.values())
|
||||
67
backend/app/services/paths.py
Normal file
67
backend/app/services/paths.py
Normal file
@@ -0,0 +1,67 @@
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
_SAFE_CHARS = re.compile(r"[^A-Za-z0-9._ -]+")
|
||||
|
||||
|
||||
def slugify_name(value: str, fallback: str = "media") -> str:
|
||||
cleaned = _SAFE_CHARS.sub("_", value).strip(" ._- ")
|
||||
return cleaned[:120] or fallback
|
||||
|
||||
|
||||
def ensure_under(base: Path, candidate: Path) -> Path:
|
||||
base_resolved = base.expanduser().resolve()
|
||||
path = (base_resolved / candidate).resolve() if not candidate.is_absolute() else candidate.resolve()
|
||||
if base_resolved != path and base_resolved not in path.parents:
|
||||
raise ValueError(f"Path escapes base directory: {candidate}")
|
||||
return path
|
||||
|
||||
|
||||
def tmp_target(tmp_dir: Path, job_id: int, title: str) -> Path:
|
||||
return ensure_under(tmp_dir, Path(str(job_id)) / slugify_name(title))
|
||||
|
||||
|
||||
def media_target(media_root: Path, library: str, title: str) -> Path:
|
||||
return ensure_under(media_root, Path(slugify_name(library, "library")) / slugify_name(title))
|
||||
|
||||
|
||||
def jellyfin_profile_target(
|
||||
media_root: Path,
|
||||
target_profile: str,
|
||||
title: str,
|
||||
*,
|
||||
year: int | None = None,
|
||||
series_title: str | None = None,
|
||||
season: int | None = None,
|
||||
episode: int | None = None,
|
||||
episode_title: str | None = None,
|
||||
channel: str | None = None,
|
||||
external_id: str | None = None,
|
||||
) -> Path:
|
||||
"""Return the Jellyfin-compatible destination directory for a target profile.
|
||||
|
||||
The returned path is always below ``media_root``. The downloader copies the
|
||||
provider output into this directory; provider-selected filenames are kept for
|
||||
now, while the directory structure follows the real Jellyfin layout from the
|
||||
project plan.
|
||||
"""
|
||||
|
||||
profile = target_profile.strip().lower()
|
||||
safe_title = slugify_name(title)
|
||||
if profile == "film":
|
||||
movie_dir = f"{safe_title} ({year})" if year else safe_title
|
||||
return ensure_under(media_root, Path("Filme") / movie_dir)
|
||||
|
||||
if profile == "serie":
|
||||
show = slugify_name(series_title or title, "Serie")
|
||||
season_number = season if season and season > 0 else 1
|
||||
episode_prefix = f"S{season_number:02d}E{episode:02d} - " if episode and episode > 0 else ""
|
||||
episode_dir = slugify_name(f"{episode_prefix}{episode_title or title}", "Episode")
|
||||
return ensure_under(media_root, Path("Serien") / show / f"Staffel{season_number}" / episode_dir)
|
||||
|
||||
if profile == "clip":
|
||||
owner = slugify_name(channel or "Unbekannte Quelle", "Quelle")
|
||||
suffix = f" [{slugify_name(external_id)}]" if external_id else ""
|
||||
return ensure_under(media_root, Path("YouTube-Mediathek") / owner / f"{safe_title}{suffix}")
|
||||
|
||||
raise ValueError(f"Unknown target profile: {target_profile}")
|
||||
170
backend/app/services/url_safety.py
Normal file
170
backend/app/services/url_safety.py
Normal file
@@ -0,0 +1,170 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import ipaddress
|
||||
import socket
|
||||
from dataclasses import dataclass
|
||||
from typing import Iterable
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
BLOCKED_NETWORKS = [
|
||||
ipaddress.ip_network("0.0.0.0/8"),
|
||||
ipaddress.ip_network("10.0.0.0/8"),
|
||||
ipaddress.ip_network("100.64.0.0/10"),
|
||||
ipaddress.ip_network("127.0.0.0/8"),
|
||||
ipaddress.ip_network("169.254.0.0/16"),
|
||||
ipaddress.ip_network("172.16.0.0/12"),
|
||||
ipaddress.ip_network("192.168.0.0/16"),
|
||||
ipaddress.ip_network("224.0.0.0/4"),
|
||||
ipaddress.ip_network("240.0.0.0/4"),
|
||||
ipaddress.ip_network("::1/128"),
|
||||
ipaddress.ip_network("::/128"),
|
||||
ipaddress.ip_network("fc00::/7"),
|
||||
ipaddress.ip_network("fe80::/10"),
|
||||
ipaddress.ip_network("ff00::/8"),
|
||||
]
|
||||
|
||||
ALLOWED_CONTENT_PREFIXES = ("video/", "audio/")
|
||||
ALLOWED_CONTENT_TYPES = {
|
||||
"application/vnd.apple.mpegurl",
|
||||
"application/x-mpegurl",
|
||||
"application/dash+xml",
|
||||
"application/octet-stream", # accepted only with known extension
|
||||
}
|
||||
ALLOWED_EXTENSIONS = (
|
||||
".mp4",
|
||||
".m4v",
|
||||
".mkv",
|
||||
".webm",
|
||||
".mov",
|
||||
".mp3",
|
||||
".m4a",
|
||||
".aac",
|
||||
".ogg",
|
||||
".flac",
|
||||
".m3u8",
|
||||
".mpd",
|
||||
)
|
||||
|
||||
|
||||
class UnsafeUrlError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass
|
||||
class CheckedUrl:
|
||||
final_url: str
|
||||
chain: list[str]
|
||||
content_type: str | None
|
||||
content_length: int | None
|
||||
|
||||
|
||||
def _is_blocked_ip(ip: str) -> bool:
|
||||
addr = ipaddress.ip_address(ip)
|
||||
return any(addr in net for net in BLOCKED_NETWORKS) or addr.is_private or addr.is_loopback or addr.is_link_local
|
||||
|
||||
|
||||
def validate_scheme(url: str) -> None:
|
||||
parsed = urlparse(url)
|
||||
if parsed.scheme not in {"http", "https"}:
|
||||
raise UnsafeUrlError("Nur http/https URLs sind erlaubt")
|
||||
if not parsed.hostname:
|
||||
raise UnsafeUrlError("URL enthält keinen Hostnamen")
|
||||
if parsed.username or parsed.password:
|
||||
raise UnsafeUrlError("Zugangsdaten in URLs sind nicht erlaubt")
|
||||
|
||||
|
||||
async def resolve_host(hostname: str, port: int | None = None) -> list[str]:
|
||||
def _resolve() -> list[str]:
|
||||
infos = socket.getaddrinfo(hostname, port or 443, type=socket.SOCK_STREAM)
|
||||
return sorted({str(info[4][0]) for info in infos})
|
||||
|
||||
return await asyncio.to_thread(_resolve)
|
||||
|
||||
|
||||
async def assert_public_host(url: str) -> None:
|
||||
validate_scheme(url)
|
||||
parsed = urlparse(url)
|
||||
assert parsed.hostname is not None
|
||||
try:
|
||||
ips = await resolve_host(parsed.hostname, parsed.port)
|
||||
except socket.gaierror as exc:
|
||||
raise UnsafeUrlError(f"Hostname kann nicht aufgelöst werden: {parsed.hostname}") from exc
|
||||
if not ips:
|
||||
raise UnsafeUrlError("Hostname hat keine IP-Adressen")
|
||||
blocked = [ip for ip in ips if _is_blocked_ip(ip)]
|
||||
if blocked:
|
||||
raise UnsafeUrlError("URL zeigt auf interne/private IP-Adresse")
|
||||
|
||||
|
||||
def _content_type_allowed(content_type: str | None, url: str) -> bool:
|
||||
if not content_type:
|
||||
return any(urlparse(url).path.lower().endswith(ext) for ext in ALLOWED_EXTENSIONS)
|
||||
clean = content_type.split(";", 1)[0].strip().lower()
|
||||
if clean.startswith(ALLOWED_CONTENT_PREFIXES):
|
||||
return True
|
||||
if clean in ALLOWED_CONTENT_TYPES and any(urlparse(url).path.lower().endswith(ext) for ext in ALLOWED_EXTENSIONS):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
async def check_url_for_page(url: str, *, timeout: float = 15.0, max_redirects: int = 8) -> CheckedUrl:
|
||||
"""Validate URL and redirects for page analysis without requiring media Content-Type."""
|
||||
current = str(url)
|
||||
chain: list[str] = []
|
||||
async with httpx.AsyncClient(follow_redirects=False, timeout=timeout) as client:
|
||||
for _ in range(max_redirects + 1):
|
||||
await assert_public_host(current)
|
||||
chain.append(current)
|
||||
response = await client.head(current)
|
||||
if response.status_code in {405, 403}:
|
||||
response = await client.get(current, headers={"Range": "bytes=0-0"})
|
||||
if response.is_redirect:
|
||||
loc = response.headers.get("location")
|
||||
if not loc:
|
||||
raise UnsafeUrlError("Redirect ohne Location-Header")
|
||||
current = str(response.url.join(loc))
|
||||
continue
|
||||
response.raise_for_status()
|
||||
await assert_public_host(str(response.url))
|
||||
content_type = response.headers.get("content-type")
|
||||
length_header = response.headers.get("content-length")
|
||||
content_length = int(length_header) if length_header and length_header.isdigit() else None
|
||||
return CheckedUrl(str(response.url), chain, content_type, content_length)
|
||||
raise UnsafeUrlError("Zu viele Weiterleitungen")
|
||||
|
||||
|
||||
async def check_url_for_download(url: str, *, max_bytes: int, timeout: float = 15.0, max_redirects: int = 8) -> CheckedUrl:
|
||||
"""Validate URL, every redirect hop, final target, content type, and size.
|
||||
|
||||
This is the no-domain-allowlist SSRF guard for Mediathek/direct links.
|
||||
"""
|
||||
current = str(url)
|
||||
chain: list[str] = []
|
||||
async with httpx.AsyncClient(follow_redirects=False, timeout=timeout) as client:
|
||||
for _ in range(max_redirects + 1):
|
||||
await assert_public_host(current)
|
||||
chain.append(current)
|
||||
response = await client.head(current)
|
||||
if response.status_code in {405, 403}:
|
||||
response = await client.get(current, headers={"Range": "bytes=0-0"})
|
||||
if response.is_redirect:
|
||||
loc = response.headers.get("location")
|
||||
if not loc:
|
||||
raise UnsafeUrlError("Redirect ohne Location-Header")
|
||||
current = str(response.url.join(loc))
|
||||
continue
|
||||
response.raise_for_status()
|
||||
await assert_public_host(str(response.url))
|
||||
content_type = response.headers.get("content-type")
|
||||
length_header = response.headers.get("content-length")
|
||||
content_length = int(length_header) if length_header and length_header.isdigit() else None
|
||||
if content_length is not None and content_length > max_bytes:
|
||||
raise UnsafeUrlError("Datei ist größer als das konfigurierte Limit")
|
||||
if not _content_type_allowed(content_type, str(response.url)):
|
||||
raise UnsafeUrlError("Ziel liefert keinen plausiblen Medien-Content-Type")
|
||||
return CheckedUrl(str(response.url), chain, content_type, content_length)
|
||||
raise UnsafeUrlError("Zu viele Weiterleitungen")
|
||||
336
backend/app/web/app.js
Normal file
336
backend/app/web/app.js
Normal file
@@ -0,0 +1,336 @@
|
||||
const loginSection = document.querySelector('#login-section');
|
||||
const loginForm = document.querySelector('#login-form');
|
||||
const loginMessage = document.querySelector('#login-message');
|
||||
const logoutButton = document.querySelector('#logout');
|
||||
const form = document.querySelector('#probe-form');
|
||||
const result = document.querySelector('#result');
|
||||
const candidatesSection = document.querySelector('#candidates');
|
||||
const analyzeButton = document.querySelector('#analyze-page');
|
||||
const interactiveButton = document.querySelector('#interactive-analyze');
|
||||
const bookmarklet = document.querySelector('#bookmarklet');
|
||||
const copyBookmarklet = document.querySelector('#copy-bookmarklet');
|
||||
const bookmarkletResult = document.querySelector('#bookmarklet-result');
|
||||
|
||||
function escapeHtml(value) {
|
||||
return String(value ?? '').replace(/[&<>'"]/g, (char) => ({'&': '&', '<': '<', '>': '>', "'": ''', '"': '"'}[char]));
|
||||
}
|
||||
|
||||
function setLoginVisible(visible) {
|
||||
loginSection.classList.toggle('hidden', !visible);
|
||||
form.classList.toggle('hidden', visible);
|
||||
logoutButton.classList.toggle('hidden', visible);
|
||||
}
|
||||
|
||||
function renderJob(job) {
|
||||
const percent = Math.round((Number(job.progress) || 0) * 100);
|
||||
if (job.status === 'done') {
|
||||
return `Job #${job.id}: fertig (${percent}%)\nDatei übertragen.`;
|
||||
}
|
||||
const labels = {
|
||||
queued: 'wartet',
|
||||
downloading: 'lädt herunter',
|
||||
postprocessing: 'verarbeitet',
|
||||
refreshing: 'überträgt / aktualisiert Jellyfin',
|
||||
failed: 'fehlgeschlagen',
|
||||
};
|
||||
const lines = [`Job #${job.id}: ${labels[job.status] || job.status} (${percent}%)`];
|
||||
if (job.status === 'failed' && job.error) lines.push(`Fehler: ${job.error}`);
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
function encodeCapturePayload(payload) {
|
||||
return btoa(unescape(encodeURIComponent(JSON.stringify(payload))));
|
||||
}
|
||||
|
||||
function decodeCapturePayload(value) {
|
||||
return JSON.parse(decodeURIComponent(escape(atob(value))));
|
||||
}
|
||||
|
||||
function buildBookmarklet() {
|
||||
const target = `${location.origin}/`;
|
||||
const code = `(()=>{const target='${target}';const mediaExt=/\\.(mp4|m4v|webm|mkv|mov|m3u8|mpd)([?#]|$)/i;const push=(arr,url,title,type)=>{try{if(!url)return;const abs=new URL(url,location.href).href;if(mediaExt.test(abs)||String(type||'').startsWith('video/')||/mpegurl|dash\\+xml/i.test(String(type||'')))arr.push({url:abs,title:title||'',content_type:type||''});}catch(e){}};const c=[];document.querySelectorAll('video,source,a').forEach(el=>push(c,el.currentSrc||el.src||el.href,el.title||el.textContent||document.title,el.type||''));performance.getEntriesByType('resource').forEach(e=>push(c,e.name,document.title,e.initiatorType==='video'?'video/unknown':''));const seen=new Set();const candidates=c.filter(x=>!seen.has(x.url)&&seen.add(x.url)).slice(0,80);if(!candidates.length&&!confirm('Keine Medien-URL im aktuellen Tab gefunden. Trotzdem zur Kino-App wechseln? Tipp: Video erst starten und ein paar Sekunden laufen lassen.'))return;const payload={page_url:location.href,candidates};const encoded=btoa(unescape(encodeURIComponent(JSON.stringify(payload))));const dest=target+'?capture='+encodeURIComponent(encoded);const opened=window.open(dest,'_blank');if(!opened)location.href=dest;})();`;
|
||||
return `javascript:${code}`;
|
||||
}
|
||||
|
||||
function initBookmarklet() {
|
||||
const href = buildBookmarklet();
|
||||
bookmarklet.href = href;
|
||||
copyBookmarklet.addEventListener('click', async () => {
|
||||
await navigator.clipboard.writeText(href);
|
||||
bookmarkletResult.textContent = 'Bookmarklet kopiert. Als neues Lesezeichen speichern und auf der Videoseite ausführen.';
|
||||
});
|
||||
}
|
||||
|
||||
async function pollJob(jobId) {
|
||||
const output = document.querySelector('#job-output');
|
||||
for (;;) {
|
||||
const response = await fetch(`/api/imports/${jobId}`);
|
||||
const job = await response.json();
|
||||
if (response.status === 401) {
|
||||
await refreshAuthStatus();
|
||||
throw new Error('Bitte zuerst einloggen.');
|
||||
}
|
||||
if (!response.ok) throw new Error(job.detail || 'Job-Status konnte nicht gelesen werden');
|
||||
output.textContent = renderJob(job);
|
||||
if (['done', 'failed'].includes(job.status)) return job;
|
||||
await new Promise((resolve) => setTimeout(resolve, 2000));
|
||||
}
|
||||
}
|
||||
|
||||
async function refreshAuthStatus() {
|
||||
const response = await fetch('/api/auth/status');
|
||||
const status = await response.json();
|
||||
setLoginVisible(status.enabled && !status.authenticated);
|
||||
}
|
||||
|
||||
loginForm.addEventListener('submit', async (event) => {
|
||||
event.preventDefault();
|
||||
loginMessage.textContent = 'Melde an…';
|
||||
const username = document.querySelector('#username').value;
|
||||
const password = document.querySelector('#password').value;
|
||||
const response = await fetch('/api/auth/login', {
|
||||
method: 'POST',
|
||||
headers: {'content-type': 'application/json'},
|
||||
body: JSON.stringify({username, password}),
|
||||
});
|
||||
if (!response.ok) {
|
||||
loginMessage.textContent = 'Login fehlgeschlagen.';
|
||||
return;
|
||||
}
|
||||
loginMessage.textContent = '';
|
||||
await refreshAuthStatus();
|
||||
});
|
||||
|
||||
logoutButton.addEventListener('click', async () => {
|
||||
await fetch('/api/auth/logout', {method: 'POST'});
|
||||
await refreshAuthStatus();
|
||||
});
|
||||
|
||||
function currentImportPayload(url) {
|
||||
const payload = {
|
||||
url,
|
||||
target_profile: document.querySelector('input[name="target-profile"]:checked').value,
|
||||
};
|
||||
const textFields = [
|
||||
['title', 'title'],
|
||||
['series-title', 'series_title'],
|
||||
['episode-title', 'episode_title'],
|
||||
];
|
||||
for (const [elementId, key] of textFields) {
|
||||
const value = document.querySelector(`#${elementId}`).value.trim();
|
||||
if (value) payload[key] = value;
|
||||
}
|
||||
const numberFields = [
|
||||
['year', 'year'],
|
||||
['season', 'season'],
|
||||
['episode', 'episode'],
|
||||
];
|
||||
for (const [elementId, key] of numberFields) {
|
||||
const value = document.querySelector(`#${elementId}`).value;
|
||||
if (value) payload[key] = Number(value);
|
||||
}
|
||||
return payload;
|
||||
}
|
||||
|
||||
form.addEventListener('submit', async (event) => {
|
||||
event.preventDefault();
|
||||
const url = document.querySelector('#url').value;
|
||||
result.className = 'card';
|
||||
result.textContent = 'Prüfe Quelle…';
|
||||
try {
|
||||
const response = await fetch('/api/probe', {method: 'POST', headers: {'content-type': 'application/json'}, body: JSON.stringify({url})});
|
||||
const data = await response.json();
|
||||
if (response.status === 401) {
|
||||
await refreshAuthStatus();
|
||||
throw new Error('Bitte zuerst einloggen.');
|
||||
}
|
||||
if (!response.ok) throw new Error(data.detail || 'Probe fehlgeschlagen');
|
||||
result.innerHTML = `
|
||||
<h2>${escapeHtml(data.title || 'Unbenannte Quelle')}</h2>
|
||||
<p>Provider: ${escapeHtml(data.provider)}</p>
|
||||
<p>Dauer: ${escapeHtml(data.duration_seconds ?? 'unbekannt')}</p>
|
||||
<button id="start-import" type="button">Import starten</button>
|
||||
<pre id="job-output"></pre>`;
|
||||
document.querySelector('#start-import').addEventListener('click', async () => {
|
||||
const jobResponse = await fetch('/api/imports', {method: 'POST', headers: {'content-type': 'application/json'}, body: JSON.stringify(currentImportPayload(url))});
|
||||
const job = await jobResponse.json();
|
||||
if (jobResponse.status === 401) {
|
||||
await refreshAuthStatus();
|
||||
throw new Error('Bitte zuerst einloggen.');
|
||||
}
|
||||
if (!jobResponse.ok) throw new Error(job.detail || 'Import konnte nicht gestartet werden');
|
||||
document.querySelector('#job-output').textContent = renderJob(job);
|
||||
await pollJob(job.id);
|
||||
});
|
||||
} catch (err) {
|
||||
result.innerHTML = `<p class="error">${escapeHtml(err.message)}</p>`;
|
||||
}
|
||||
});
|
||||
|
||||
analyzeButton.addEventListener('click', async () => {
|
||||
const url = document.querySelector('#url').value;
|
||||
result.className = 'card';
|
||||
candidatesSection.className = 'card hidden';
|
||||
candidatesSection.innerHTML = '';
|
||||
result.textContent = 'Analysiere Seite per HTML, yt-dlp und Headless-Browser…';
|
||||
try {
|
||||
const response = await fetch('/api/analyze', {method: 'POST', headers: {'content-type': 'application/json'}, body: JSON.stringify({url})});
|
||||
const data = await response.json();
|
||||
if (response.status === 401) {
|
||||
await refreshAuthStatus();
|
||||
throw new Error('Bitte zuerst einloggen.');
|
||||
}
|
||||
if (!response.ok) throw new Error(data.detail || 'Analyse fehlgeschlagen');
|
||||
result.textContent = `${data.candidates.length} Kandidat(en) gefunden.`;
|
||||
renderCandidates(data.candidates || []);
|
||||
} catch (err) {
|
||||
result.innerHTML = `<p class="error">${escapeHtml(err.message)}</p>`;
|
||||
}
|
||||
});
|
||||
|
||||
let interactiveSessionId = null;
|
||||
|
||||
function renderInteractive(data) {
|
||||
interactiveSessionId = data.session_id;
|
||||
const count = (data.candidates || []).length;
|
||||
result.className = 'card interactive-card';
|
||||
result.innerHTML = `
|
||||
<h2>Interaktive Serveranalyse</h2>
|
||||
<p>Screenshot anklicken, um die serverseitige Browser-Session zu steuern. Starte den Player; Medien-Requests erscheinen live unten.</p>
|
||||
<p>${count} Kandidat(en) gefunden.${data.ublock_origin ? ' uBlock Origin ist aktiv.' : ''}</p>
|
||||
<div class="interactive-toolbar">
|
||||
<button id="interactive-space" type="button">Play/Pause (Leertaste)</button>
|
||||
<button id="interactive-refresh" type="button">Kandidaten aktualisieren</button>
|
||||
<button id="interactive-stop" type="button">Session beenden</button>
|
||||
</div>
|
||||
<img id="interactive-screenshot" class="interactive-screenshot" src="${escapeHtml(data.screenshot)}" alt="Interaktive Analyse-Vorschau" />
|
||||
`;
|
||||
const screenshot = document.querySelector('#interactive-screenshot');
|
||||
screenshot.addEventListener('click', async (event) => {
|
||||
const rect = screenshot.getBoundingClientRect();
|
||||
const x = (event.clientX - rect.left) * (data.viewport.width / rect.width);
|
||||
const y = (event.clientY - rect.top) * (data.viewport.height / rect.height);
|
||||
await interactiveAction('click', {x, y});
|
||||
});
|
||||
document.querySelector('#interactive-space').addEventListener('click', () => interactiveAction('key', {key: 'Space'}));
|
||||
document.querySelector('#interactive-refresh').addEventListener('click', () => interactiveAction('key', {key: 'Shift'}));
|
||||
document.querySelector('#interactive-stop').addEventListener('click', stopInteractive);
|
||||
renderCandidates(data.candidates || []);
|
||||
}
|
||||
|
||||
async function interactiveAction(action, payload) {
|
||||
if (!interactiveSessionId) return;
|
||||
result.classList.add('loading');
|
||||
try {
|
||||
const endpoint = action === 'click'
|
||||
? `/api/interactive-analyze/${interactiveSessionId}/click`
|
||||
: `/api/interactive-analyze/${interactiveSessionId}/key`;
|
||||
const response = await fetch(endpoint, {method: 'POST', headers: {'content-type': 'application/json'}, body: JSON.stringify(payload)});
|
||||
const data = await response.json();
|
||||
if (!response.ok) throw new Error(data.detail || 'Interaktive Aktion fehlgeschlagen');
|
||||
renderInteractive(data);
|
||||
} catch (err) {
|
||||
result.innerHTML += `<p class="error">${escapeHtml(err.message)}</p>`;
|
||||
} finally {
|
||||
result.classList.remove('loading');
|
||||
}
|
||||
}
|
||||
|
||||
async function stopInteractive() {
|
||||
if (!interactiveSessionId) return;
|
||||
await fetch(`/api/interactive-analyze/${interactiveSessionId}`, {method: 'DELETE'}).catch(() => {});
|
||||
interactiveSessionId = null;
|
||||
result.className = 'card';
|
||||
result.textContent = 'Interaktive Analyse beendet.';
|
||||
}
|
||||
|
||||
interactiveButton.addEventListener('click', async () => {
|
||||
const url = document.querySelector('#url').value;
|
||||
result.className = 'card';
|
||||
candidatesSection.className = 'card hidden';
|
||||
candidatesSection.innerHTML = '';
|
||||
result.textContent = 'Starte serverseitigen Analyse-Browser…';
|
||||
try {
|
||||
const response = await fetch('/api/interactive-analyze', {method: 'POST', headers: {'content-type': 'application/json'}, body: JSON.stringify({url})});
|
||||
const data = await response.json();
|
||||
if (response.status === 401) {
|
||||
await refreshAuthStatus();
|
||||
throw new Error('Bitte zuerst einloggen.');
|
||||
}
|
||||
if (!response.ok) throw new Error(data.detail || 'Interaktive Analyse konnte nicht gestartet werden');
|
||||
renderInteractive(data);
|
||||
} catch (err) {
|
||||
result.innerHTML = `<p class="error">${escapeHtml(err.message)}</p>`;
|
||||
}
|
||||
});
|
||||
|
||||
function renderCandidates(candidates) {
|
||||
candidatesSection.className = 'card';
|
||||
if (!candidates.length) {
|
||||
candidatesSection.innerHTML = '<p>Keine direkt importierbaren Medienquellen gefunden.</p>';
|
||||
return;
|
||||
}
|
||||
candidatesSection.innerHTML = '<h2>Gefundene Medienquellen</h2>';
|
||||
candidates.forEach((candidate, idx) => {
|
||||
const card = document.createElement('div');
|
||||
card.className = `candidate ${candidate.allowed ? '' : 'blocked'}`;
|
||||
const size = candidate.file_size || (candidate.content_length ? formatBytes(candidate.content_length) : 'Dateigröße unbekannt');
|
||||
const details = [candidate.kind, candidate.source, candidate.quality, `Dateigröße: ${size}`].filter(Boolean).join(' · ');
|
||||
card.innerHTML = `
|
||||
<div>
|
||||
<strong>${escapeHtml(candidate.title || `Kandidat ${idx + 1}`)}</strong>
|
||||
<p>${escapeHtml(details)}</p>
|
||||
<code>${escapeHtml(candidate.url)}</code>
|
||||
${candidate.reason ? `<p class="error">${escapeHtml(candidate.reason)}</p>` : ''}
|
||||
</div>
|
||||
<button ${candidate.allowed ? '' : 'disabled'} data-url="${escapeHtml(candidate.url)}" data-title="${escapeHtml(candidate.title || '')}">Übernehmen</button>
|
||||
`;
|
||||
card.querySelector('button').addEventListener('click', (event) => {
|
||||
document.querySelector('#url').value = event.currentTarget.dataset.url;
|
||||
if (event.currentTarget.dataset.title && !document.querySelector('#title').value) {
|
||||
document.querySelector('#title').value = event.currentTarget.dataset.title;
|
||||
}
|
||||
result.className = 'card';
|
||||
result.textContent = 'Kandidat übernommen. Du kannst jetzt Quelle prüfen oder Import starten.';
|
||||
});
|
||||
candidatesSection.appendChild(card);
|
||||
});
|
||||
}
|
||||
|
||||
function formatBytes(bytes) {
|
||||
const units = ['B', 'KB', 'MB', 'GB'];
|
||||
let value = Number(bytes || 0);
|
||||
for (const unit of units) {
|
||||
if (value < 1024 || unit === 'GB') return unit === 'B' ? `${value} ${unit}` : `${value.toFixed(1)} ${unit}`;
|
||||
value /= 1024;
|
||||
}
|
||||
}
|
||||
|
||||
async function importCapturedFromUrl() {
|
||||
const params = new URLSearchParams(location.search);
|
||||
const encoded = params.get('capture') || (location.hash.startsWith('#capture=') ? location.hash.slice(9) : '');
|
||||
if (!encoded) return;
|
||||
try {
|
||||
const payload = decodeCapturePayload(encoded);
|
||||
result.className = 'card';
|
||||
result.textContent = 'Browser-Capture wird geprüft…';
|
||||
const response = await fetch('/api/browser-capture', {method: 'POST', headers: {'content-type': 'application/json'}, body: JSON.stringify(payload)});
|
||||
const data = await response.json();
|
||||
if (response.status === 401) {
|
||||
await refreshAuthStatus();
|
||||
throw new Error('Bitte zuerst einloggen.');
|
||||
}
|
||||
if (!response.ok) throw new Error(data.detail || 'Browser-Capture konnte nicht geprüft werden');
|
||||
document.querySelector('#url').value = payload.page_url || '';
|
||||
result.textContent = `${data.candidates.length} Kandidat(en) aus Browser-Capture gefunden.`;
|
||||
renderCandidates(data.candidates || []);
|
||||
history.replaceState(null, '', '/');
|
||||
} catch (err) {
|
||||
result.className = 'card';
|
||||
result.innerHTML = `<p class="error">${escapeHtml(err.message)}</p>`;
|
||||
}
|
||||
}
|
||||
|
||||
initBookmarklet();
|
||||
refreshAuthStatus().catch(() => setLoginVisible(false));
|
||||
importCapturedFromUrl();
|
||||
0
backend/app/web/downloads/.gitkeep
Normal file
0
backend/app/web/downloads/.gitkeep
Normal file
71
backend/app/web/index.html
Normal file
71
backend/app/web/index.html
Normal file
@@ -0,0 +1,71 @@
|
||||
<!doctype html>
|
||||
<html lang="de">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>Kino-Projekt</title>
|
||||
<link rel="stylesheet" href="/static/style.css?v=job-status-short-20260707" />
|
||||
</head>
|
||||
<body>
|
||||
<main>
|
||||
<h1>Kino-Projekt</h1>
|
||||
<p>Interner Import für YouTube, legale Mediatheken und direkte erlaubte Medienlinks.</p>
|
||||
<section id="login-section" class="card hidden">
|
||||
<h2>Login</h2>
|
||||
<form id="login-form">
|
||||
<input id="username" autocomplete="username" placeholder="Benutzername" required />
|
||||
<input id="password" type="password" autocomplete="current-password" placeholder="Passwort" required />
|
||||
<button type="submit">Einloggen</button>
|
||||
</form>
|
||||
<p id="login-message" class="error"></p>
|
||||
</section>
|
||||
<button id="logout" type="button" class="hidden">Abmelden</button>
|
||||
<form id="probe-form">
|
||||
<label>
|
||||
Quelle
|
||||
<input id="url" type="url" placeholder="https://…" required />
|
||||
</label>
|
||||
<fieldset>
|
||||
<legend>Zielprofil</legend>
|
||||
<label><input type="radio" name="target-profile" value="clip" checked /> YouTube/Mediathek-Clip</label>
|
||||
<label><input type="radio" name="target-profile" value="film" /> Film</label>
|
||||
<label><input type="radio" name="target-profile" value="serie" /> Serie/Episode</label>
|
||||
</fieldset>
|
||||
<div class="grid">
|
||||
<label>Titel-Korrektur <input id="title" placeholder="optional" /></label>
|
||||
<label>Jahr <input id="year" type="number" min="1888" max="2200" placeholder="optional" /></label>
|
||||
<label>Serientitel <input id="series-title" placeholder="nur Serie" /></label>
|
||||
<label>Staffel <input id="season" type="number" min="1" placeholder="1" /></label>
|
||||
<label>Episode <input id="episode" type="number" min="1" placeholder="optional" /></label>
|
||||
<label>Episodentitel <input id="episode-title" placeholder="optional" /></label>
|
||||
</div>
|
||||
<button type="submit">Quelle prüfen</button>
|
||||
<button id="analyze-page" type="button">Seite analysieren</button>
|
||||
<button id="interactive-analyze" type="button">Interaktive Analyse starten</button>
|
||||
</form>
|
||||
<section id="analyzer-tools" class="card">
|
||||
<h2>Browser-Capture</h2>
|
||||
<p>Wenn die Serveranalyse keine Quelle findet: Für einfache Seiten kannst du weiter das Bookmarklet nutzen. Zuverlässiger ist die Kino-Capture-Browser-Extension, weil sie Medien-Requests des aktiven Tabs beobachtet.</p>
|
||||
<div class="tool-actions">
|
||||
<a id="extension-download" href="/static/downloads/kino-capture-extension.zip" download>Kino-Capture Extension herunterladen</a>
|
||||
<a id="bookmarklet" href="#">Kino-Capture Bookmarklet</a>
|
||||
<button id="copy-bookmarklet" type="button">Bookmarklet kopieren</button>
|
||||
</div>
|
||||
<details>
|
||||
<summary>Extension installieren und nutzen</summary>
|
||||
<ol>
|
||||
<li>ZIP herunterladen und entpacken.</li>
|
||||
<li>Chrome/Edge: <code>chrome://extensions</code> öffnen, Entwicklermodus aktivieren, „Entpackte Erweiterung laden“ wählen.</li>
|
||||
<li>Firefox/LibreWolf: <code>about:debugging#/runtime/this-firefox</code> öffnen und temporäres Add-on über <code>manifest.json</code> laden.</li>
|
||||
<li>Video starten, einige Sekunden laufen lassen, Extension-Icon öffnen und „An Kino-App senden“ klicken.</li>
|
||||
</ol>
|
||||
</details>
|
||||
<p id="bookmarklet-result"></p>
|
||||
</section>
|
||||
<section id="result" class="card hidden"></section>
|
||||
<section id="candidates" class="card hidden"></section>
|
||||
<button id="import" disabled>Import starten</button>
|
||||
</main>
|
||||
<script src="/static/app.js?v=job-status-short-20260707"></script>
|
||||
</body>
|
||||
</html>
|
||||
24
backend/app/web/style.css
Normal file
24
backend/app/web/style.css
Normal file
@@ -0,0 +1,24 @@
|
||||
body { font-family: system-ui, sans-serif; margin: 0; background: #111827; color: #f9fafb; }
|
||||
main { max-width: 860px; margin: 4rem auto; padding: 2rem; }
|
||||
form { display: flex; gap: .75rem; flex-wrap: wrap; }
|
||||
label { display: flex; flex: 1 1 220px; flex-direction: column; gap: .35rem; }
|
||||
fieldset { flex: 1 1 100%; border: 1px solid #374151; border-radius: .75rem; display: flex; gap: 1rem; flex-wrap: wrap; }
|
||||
fieldset label { flex: 0 1 auto; flex-direction: row; align-items: center; }
|
||||
input { flex: 1; padding: .8rem; border-radius: .5rem; border: 1px solid #374151; background: #1f2937; color: #fff; }
|
||||
input[type="radio"] { flex: 0 0 auto; }
|
||||
button { padding: .8rem 1rem; border: 0; border-radius: .5rem; background: #38bdf8; color: #082f49; font-weight: 700; }
|
||||
button:disabled { opacity: .45; cursor: not-allowed; }
|
||||
.grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); gap: .75rem; width: 100%; }
|
||||
.card { margin-top: 1rem; padding: 1rem; border: 1px solid #374151; border-radius: .75rem; background: #1f2937; }
|
||||
.hidden { display: none; }
|
||||
.error { color: #fca5a5; }
|
||||
.candidate { margin-top: .75rem; padding-top: .75rem; border-top: 1px solid #374151; display: grid; gap: .75rem; }
|
||||
.candidate code { display: block; overflow-wrap: anywhere; color: #bae6fd; }
|
||||
.candidate.blocked { opacity: .7; }
|
||||
#bookmarklet, #extension-download { color: #7dd3fc; overflow-wrap: anywhere; }
|
||||
.tool-actions { display: flex; flex-wrap: wrap; align-items: center; gap: .75rem; }
|
||||
details { margin-top: .75rem; }
|
||||
details code { color: #bae6fd; }
|
||||
.interactive-toolbar { display: flex; flex-wrap: wrap; gap: .75rem; margin: .75rem 0; }
|
||||
.interactive-screenshot { width: 100%; max-height: 70vh; object-fit: contain; border: 1px solid #374151; border-radius: .5rem; background: #000; cursor: crosshair; }
|
||||
.loading { opacity: .8; }
|
||||
23
backend/pyproject.toml
Normal file
23
backend/pyproject.toml
Normal file
@@ -0,0 +1,23 @@
|
||||
[project]
|
||||
name = "kino-projekt-backend"
|
||||
version = "0.1.0"
|
||||
description = "Legal-source media import dashboard for Jellyfin"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"fastapi>=0.115",
|
||||
"uvicorn[standard]>=0.30",
|
||||
"pydantic-settings>=2.4",
|
||||
"sqlmodel>=0.0.22",
|
||||
"httpx>=0.27",
|
||||
"argon2-cffi>=23.1",
|
||||
"yt-dlp>=2025.1.1",
|
||||
"playwright>=1.49",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
test = ["pytest>=8.0", "pytest-asyncio>=0.23"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
asyncio_mode = "auto"
|
||||
pythonpath = ["."]
|
||||
testpaths = ["tests"]
|
||||
11
backend/tests/conftest.py
Normal file
11
backend/tests/conftest.py
Normal file
@@ -0,0 +1,11 @@
|
||||
import pytest
|
||||
|
||||
from app.config import Settings
|
||||
from app.main import app
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def reset_app_settings():
|
||||
app.state.settings = Settings(KINOPROJEKT_DB=":memory:")
|
||||
yield
|
||||
app.state.settings = Settings(KINOPROJEKT_DB=":memory:")
|
||||
82
backend/tests/test_analyzer_api.py
Normal file
82
backend/tests/test_analyzer_api.py
Normal file
@@ -0,0 +1,82 @@
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.main import app
|
||||
from app.services.page_analyzer import MediaCandidate
|
||||
|
||||
|
||||
def test_analyze_api_returns_candidates(monkeypatch):
|
||||
async def fake_analyze_page(url):
|
||||
return [
|
||||
MediaCandidate(
|
||||
url="https://cdn.example.org/movie.mp4",
|
||||
kind="direct_video",
|
||||
source="html",
|
||||
content_length=1536,
|
||||
file_size="1.5 KB",
|
||||
)
|
||||
]
|
||||
|
||||
monkeypatch.setattr("app.main.analyze_page", fake_analyze_page)
|
||||
client = TestClient(app)
|
||||
|
||||
response = client.post("/api/analyze", json={"url": "https://example.org/watch"})
|
||||
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["page_url"] == "https://example.org/watch"
|
||||
assert body["candidates"][0]["file_size"] == "1.5 KB"
|
||||
assert body["candidates"][0]["source"] == "html"
|
||||
|
||||
|
||||
def test_browser_capture_api_returns_candidates(monkeypatch):
|
||||
async def fake_analyze_browser_capture(captured, page_url=None):
|
||||
assert page_url == "https://example.org/watch"
|
||||
assert captured[0]["url"] == "/media/movie.mp4"
|
||||
return [MediaCandidate(url="https://example.org/media/movie.mp4", kind="direct_video", source="browser-capture")]
|
||||
|
||||
monkeypatch.setattr("app.main.analyze_browser_capture", fake_analyze_browser_capture)
|
||||
client = TestClient(app)
|
||||
|
||||
response = client.post(
|
||||
"/api/browser-capture",
|
||||
json={"page_url": "https://example.org/watch", "candidates": [{"url": "/media/movie.mp4"}]},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["candidates"][0]["source"] == "browser-capture"
|
||||
|
||||
|
||||
def test_interactive_analyze_api_returns_screenshot_and_candidates(monkeypatch):
|
||||
class FakePage:
|
||||
url = "https://example.org/watch"
|
||||
|
||||
class FakeSession:
|
||||
id = "session-1"
|
||||
page_url = "https://example.org/watch"
|
||||
page = FakePage()
|
||||
ublock_path = None
|
||||
|
||||
async def fake_start(url):
|
||||
assert url == "https://example.org/watch"
|
||||
return FakeSession()
|
||||
|
||||
async def fake_candidates(session):
|
||||
assert session.id == "session-1"
|
||||
return [MediaCandidate(url="https://cdn.example.org/live.m3u8", kind="hls_manifest", source="interactive-browser")]
|
||||
|
||||
async def fake_screenshot(session):
|
||||
return "data:image/png;base64,ZmFrZQ=="
|
||||
|
||||
monkeypatch.setattr("app.main.interactive_browser_manager.start", fake_start)
|
||||
monkeypatch.setattr("app.main.interactive_browser_manager.candidates", fake_candidates)
|
||||
monkeypatch.setattr("app.main.interactive_browser_manager.screenshot_data_url", fake_screenshot)
|
||||
client = TestClient(app)
|
||||
|
||||
response = client.post("/api/interactive-analyze", json={"url": "https://example.org/watch"})
|
||||
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["session_id"] == "session-1"
|
||||
assert body["screenshot"].startswith("data:image/png;base64,")
|
||||
assert body["candidates"][0]["source"] == "interactive-browser"
|
||||
103
backend/tests/test_auth.py
Normal file
103
backend/tests/test_auth.py
Normal file
@@ -0,0 +1,103 @@
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.config import Settings
|
||||
from app.main import app
|
||||
from app.services.auth import hash_password
|
||||
|
||||
|
||||
def configure_auth(password: str = "correct horse battery staple"):
|
||||
app.state.settings = Settings(
|
||||
KINOPROJEKT_ADMIN_USER="admin",
|
||||
KINOPROJEKT_ADMIN_PASSWORD_HASH=hash_password(password),
|
||||
KINOPROJEKT_SESSION_SECRET="test-secret-that-is-long-enough",
|
||||
KINOPROJEKT_DB=":memory:",
|
||||
)
|
||||
return password
|
||||
|
||||
|
||||
def disable_auth():
|
||||
app.state.settings = Settings(KINOPROJEKT_DB=":memory:")
|
||||
|
||||
|
||||
def test_auth_disabled_keeps_mvp_api_accessible():
|
||||
disable_auth()
|
||||
client = TestClient(app)
|
||||
response = client.get("/api/auth/status")
|
||||
assert response.json() == {"enabled": False, "authenticated": True, "username": None}
|
||||
|
||||
|
||||
def test_configured_auth_blocks_api_without_session():
|
||||
configure_auth()
|
||||
client = TestClient(app)
|
||||
response = client.post("/api/probe", json={"url": "http://localhost/video.mp4"})
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
def test_login_sets_secure_httponly_session_and_allows_api():
|
||||
password = configure_auth()
|
||||
client = TestClient(app, base_url="https://testserver")
|
||||
|
||||
login = client.post("/api/auth/login", json={"username": "admin", "password": password})
|
||||
|
||||
assert login.status_code == 204
|
||||
set_cookie = login.headers["set-cookie"]
|
||||
assert "HttpOnly" in set_cookie
|
||||
assert "Secure" in set_cookie
|
||||
assert "SameSite=lax" in set_cookie
|
||||
|
||||
response = client.post("/api/probe", json={"url": "http://localhost/video.mp4"})
|
||||
assert response.status_code == 400
|
||||
assert "private" in response.json()["detail"].lower()
|
||||
|
||||
|
||||
def test_login_omits_secure_cookie_on_plain_http_lan_access():
|
||||
password = configure_auth()
|
||||
client = TestClient(app, base_url="http://testserver")
|
||||
|
||||
login = client.post("/api/auth/login", json={"username": "admin", "password": password})
|
||||
|
||||
assert login.status_code == 204
|
||||
set_cookie = login.headers["set-cookie"]
|
||||
assert "HttpOnly" in set_cookie
|
||||
assert "Secure" not in set_cookie
|
||||
|
||||
|
||||
def test_login_sets_secure_cookie_with_forwarded_https_proto():
|
||||
password = configure_auth()
|
||||
client = TestClient(app, base_url="http://testserver")
|
||||
|
||||
login = client.post(
|
||||
"/api/auth/login",
|
||||
json={"username": "admin", "password": password},
|
||||
headers={"x-forwarded-proto": "https"},
|
||||
)
|
||||
|
||||
assert login.status_code == 204
|
||||
assert "Secure" in login.headers["set-cookie"]
|
||||
|
||||
|
||||
def test_logout_omits_secure_cookie_on_plain_http_lan_access():
|
||||
configure_auth()
|
||||
client = TestClient(app, base_url="http://testserver")
|
||||
|
||||
logout = client.post("/api/auth/logout")
|
||||
|
||||
assert logout.status_code == 204
|
||||
assert "Secure" not in logout.headers["set-cookie"]
|
||||
|
||||
|
||||
def test_logout_sets_secure_cookie_with_forwarded_https_proto():
|
||||
configure_auth()
|
||||
client = TestClient(app, base_url="http://testserver")
|
||||
|
||||
logout = client.post("/api/auth/logout", headers={"x-forwarded-proto": "https"})
|
||||
|
||||
assert logout.status_code == 204
|
||||
assert "Secure" in logout.headers["set-cookie"]
|
||||
|
||||
|
||||
def test_login_rejects_wrong_password():
|
||||
configure_auth()
|
||||
client = TestClient(app, base_url="https://testserver")
|
||||
response = client.post("/api/auth/login", json={"username": "admin", "password": "wrong"})
|
||||
assert response.status_code == 401
|
||||
101
backend/tests/test_downloader.py
Normal file
101
backend/tests/test_downloader.py
Normal file
@@ -0,0 +1,101 @@
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.providers.base import DownloadOptions, MediaMetadata
|
||||
from app.providers.mediathek import MediathekProvider, is_stream_manifest
|
||||
from app.providers.youtube import parse_yt_dlp_progress
|
||||
from app.services.downloader import copy_tree_contents
|
||||
|
||||
|
||||
class FakeStream:
|
||||
def __init__(self, lines: list[bytes]):
|
||||
self.lines = lines
|
||||
|
||||
async def readline(self) -> bytes:
|
||||
if self.lines:
|
||||
return self.lines.pop(0)
|
||||
return b""
|
||||
|
||||
|
||||
class FakeProcess:
|
||||
def __init__(self, output_path: Path):
|
||||
self.returncode = 0
|
||||
self.stdout = FakeStream([b"[download] 100% of 12.00MiB\n"])
|
||||
self.stderr = FakeStream([])
|
||||
self.output_path = output_path
|
||||
|
||||
async def wait(self) -> int:
|
||||
self.output_path.write_bytes(b"real media bytes")
|
||||
return self.returncode
|
||||
|
||||
|
||||
def test_copy_tree_contents_copies_files(tmp_path: Path):
|
||||
src = tmp_path / 'src'
|
||||
src.mkdir()
|
||||
(src / 'movie.mp4').write_bytes(b'data')
|
||||
(src / 'meta.info.json').write_text('{}')
|
||||
dest = tmp_path / 'dest'
|
||||
|
||||
copied = copy_tree_contents(src, dest)
|
||||
|
||||
assert sorted(p.name for p in copied) == ['meta.info.json', 'movie.mp4']
|
||||
assert (dest / 'movie.mp4').read_bytes() == b'data'
|
||||
|
||||
|
||||
def test_copy_tree_contents_rejects_top_level_symlink(tmp_path: Path):
|
||||
src = tmp_path / "src"
|
||||
src.mkdir()
|
||||
(tmp_path / "secret.txt").write_text("secret")
|
||||
(src / "leak.txt").symlink_to(tmp_path / "secret.txt")
|
||||
|
||||
with pytest.raises(Exception, match="symlink"):
|
||||
copy_tree_contents(src, tmp_path / "dest")
|
||||
|
||||
|
||||
def test_copy_tree_contents_rejects_nested_symlink(tmp_path: Path):
|
||||
src = tmp_path / "src"
|
||||
nested = src / "nested"
|
||||
nested.mkdir(parents=True)
|
||||
(tmp_path / "secret.txt").write_text("secret")
|
||||
(nested / "leak.txt").symlink_to(tmp_path / "secret.txt")
|
||||
|
||||
with pytest.raises(Exception, match="symlink"):
|
||||
copy_tree_contents(src, tmp_path / "dest")
|
||||
|
||||
|
||||
def test_parse_yt_dlp_progress_lines():
|
||||
assert parse_yt_dlp_progress('[download] 42.7% of 10.00MiB at 1.00MiB/s ETA 00:05') == pytest.approx(0.427)
|
||||
assert parse_yt_dlp_progress('[download] 100% of 10.00MiB') == 1.0
|
||||
assert parse_yt_dlp_progress('[info] unrelated') is None
|
||||
|
||||
|
||||
def test_stream_manifest_detection():
|
||||
assert is_stream_manifest('https://example.org/master.m3u8')
|
||||
assert is_stream_manifest('https://example.org/manifest.mpd')
|
||||
assert is_stream_manifest('https://example.org/video', 'application/vnd.apple.mpegurl')
|
||||
assert not is_stream_manifest('https://example.org/video.mp4', 'video/mp4')
|
||||
|
||||
|
||||
async def test_m3u8_download_uses_ytdlp_not_manifest_file(monkeypatch, tmp_path: Path):
|
||||
provider = MediathekProvider()
|
||||
manifest_url = 'https://cdn.example.org/master.m3u8'
|
||||
|
||||
async def fake_probe(url: str) -> MediaMetadata:
|
||||
return MediaMetadata(provider='mediathek', title='master.m3u8', external_id=manifest_url)
|
||||
|
||||
async def fake_create_subprocess_exec(*args, **kwargs):
|
||||
assert args[0].endswith('yt-dlp')
|
||||
assert '--proxy' not in args
|
||||
assert '--remux-video' in args
|
||||
output_template = Path(args[args.index('--output') + 1])
|
||||
return FakeProcess(output_template.parent / 'downloaded.mp4')
|
||||
|
||||
monkeypatch.setattr(provider, 'probe', fake_probe)
|
||||
monkeypatch.setattr('app.providers.mediathek.shutil.which', lambda name: '/usr/bin/yt-dlp' if name == 'yt-dlp' else None)
|
||||
monkeypatch.setattr('app.providers.mediathek.asyncio.create_subprocess_exec', fake_create_subprocess_exec)
|
||||
|
||||
result = await provider.download(manifest_url, tmp_path, DownloadOptions())
|
||||
|
||||
assert [p.name for p in result.output_files] == ['downloaded.mp4']
|
||||
assert not (tmp_path / 'master.m3u8').exists()
|
||||
7
backend/tests/test_health.py
Normal file
7
backend/tests/test_health.py
Normal file
@@ -0,0 +1,7 @@
|
||||
from fastapi.testclient import TestClient
|
||||
from app.main import app
|
||||
|
||||
|
||||
def test_health():
|
||||
client = TestClient(app)
|
||||
assert client.get('/health').json() == {'status': 'ok'}
|
||||
45
backend/tests/test_html_import_resolution.py
Normal file
45
backend/tests/test_html_import_resolution.py
Normal file
@@ -0,0 +1,45 @@
|
||||
import pytest
|
||||
|
||||
from app.main import resolve_importable_url
|
||||
from app.providers.base import MediaMetadata, ProviderError
|
||||
from app.services.page_analyzer import MediaCandidate
|
||||
|
||||
|
||||
class FakeProvider:
|
||||
name = "mediathek"
|
||||
|
||||
def can_handle(self, url: str) -> bool:
|
||||
return url.startswith("https://")
|
||||
|
||||
async def probe(self, url: str) -> MediaMetadata:
|
||||
if url == "https://example.org/watch":
|
||||
raise ProviderError("URL is reachable, but content type/extension is not accepted: text/html")
|
||||
if url == "https://cdn.example.org/master.m3u8":
|
||||
return MediaMetadata(provider=self.name, title="master.m3u8", external_id=url)
|
||||
raise ProviderError("unexpected url")
|
||||
|
||||
|
||||
async def test_html_page_import_resolves_to_best_media_candidate(monkeypatch):
|
||||
async def fake_analyze_page(url: str):
|
||||
assert url == "https://example.org/watch"
|
||||
return [
|
||||
MediaCandidate(
|
||||
url="https://cdn.example.org/master.m3u8",
|
||||
kind="hls_manifest",
|
||||
title="HLS stream",
|
||||
source="browser",
|
||||
content_length=4096,
|
||||
allowed=True,
|
||||
)
|
||||
]
|
||||
|
||||
fake_provider = FakeProvider()
|
||||
monkeypatch.setattr("app.main.providers", [fake_provider])
|
||||
monkeypatch.setattr("app.main.analyze_page", fake_analyze_page)
|
||||
|
||||
resolved_url, provider, metadata, candidate = await resolve_importable_url("https://example.org/watch")
|
||||
|
||||
assert resolved_url == "https://cdn.example.org/master.m3u8"
|
||||
assert provider is fake_provider
|
||||
assert metadata.external_id == resolved_url
|
||||
assert candidate.kind == "hls_manifest"
|
||||
46
backend/tests/test_internet_archive_provider.py
Normal file
46
backend/tests/test_internet_archive_provider.py
Normal file
@@ -0,0 +1,46 @@
|
||||
import pytest
|
||||
|
||||
from app.providers.base import ProviderError
|
||||
from app.providers.internet_archive import InternetArchiveProvider
|
||||
|
||||
|
||||
def test_internet_archive_detection():
|
||||
provider = InternetArchiveProvider()
|
||||
assert provider.can_handle("https://archive.org/details/BigBuckBunny_328")
|
||||
assert provider.can_handle("https://archive.org/download/BigBuckBunny_328/video.mp4")
|
||||
assert not provider.can_handle("https://example.com/details/BigBuckBunny_328")
|
||||
|
||||
|
||||
def test_internet_archive_identifier_parsing():
|
||||
provider = InternetArchiveProvider()
|
||||
assert provider._identifier_from_url("https://archive.org/details/BigBuckBunny_328") == "BigBuckBunny_328"
|
||||
assert provider._identifier_from_url("https://archive.org/download/BigBuckBunny_328/video.mp4") == "BigBuckBunny_328"
|
||||
assert provider._identifier_from_url("https://archive.org/metadata/BigBuckBunny_328") == "BigBuckBunny_328"
|
||||
|
||||
|
||||
def test_select_media_file_prefers_original_mp4():
|
||||
provider = InternetArchiveProvider()
|
||||
selected = provider._select_media_file([
|
||||
{"name": "thumb.jpg", "format": "JPEG", "source": "metadata", "size": "10"},
|
||||
{"name": "movie.ogv", "format": "Ogg Video", "source": "derivative", "size": "200000000"},
|
||||
{"name": "movie.mp4", "format": "MPEG4", "source": "original", "size": "100000000"},
|
||||
])
|
||||
assert selected is not None
|
||||
assert selected["name"] == "movie.mp4"
|
||||
|
||||
|
||||
def test_select_media_file_rejects_path_traversal():
|
||||
provider = InternetArchiveProvider()
|
||||
assert provider._select_media_file([{"name": "../evil.mp4", "format": "MPEG4", "source": "original"}]) is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_probe_requires_media_file(monkeypatch):
|
||||
provider = InternetArchiveProvider()
|
||||
|
||||
async def fake_metadata(identifier: str) -> dict:
|
||||
return {"metadata": {"title": "No media"}, "files": [{"name": "notes.txt", "format": "Text"}]}
|
||||
|
||||
monkeypatch.setattr(provider, "_metadata", fake_metadata)
|
||||
with pytest.raises(ProviderError):
|
||||
await provider.probe("https://archive.org/details/no-media")
|
||||
9
backend/tests/test_mediathek_provider.py
Normal file
9
backend/tests/test_mediathek_provider.py
Normal file
@@ -0,0 +1,9 @@
|
||||
import pytest
|
||||
|
||||
from app.providers.base import ProviderError
|
||||
from app.providers.mediathek import MediathekProvider
|
||||
|
||||
|
||||
def test_private_target_rejected():
|
||||
with pytest.raises(ProviderError):
|
||||
MediathekProvider()._reject_private_targets('localhost')
|
||||
19
backend/tests/test_models.py
Normal file
19
backend/tests/test_models.py
Normal file
@@ -0,0 +1,19 @@
|
||||
from sqlmodel import Session, select
|
||||
|
||||
from app.db import get_engine, init_db
|
||||
from app.models import ImportJob, Source
|
||||
|
||||
|
||||
def test_models_roundtrip(tmp_path):
|
||||
engine = get_engine(tmp_path / 'kino.sqlite3')
|
||||
init_db(engine)
|
||||
with Session(engine) as session:
|
||||
src = Source(kind='youtube', url='https://youtu.be/example', title='Example', provider='youtube')
|
||||
session.add(src)
|
||||
session.commit()
|
||||
session.refresh(src)
|
||||
job = ImportJob(source_id=src.id, target_library='movies')
|
||||
session.add(job)
|
||||
session.commit()
|
||||
assert session.exec(select(Source)).one().title == 'Example'
|
||||
assert session.exec(select(ImportJob)).one().status == 'queued'
|
||||
175
backend/tests/test_page_analyzer.py
Normal file
175
backend/tests/test_page_analyzer.py
Normal file
@@ -0,0 +1,175 @@
|
||||
import pytest
|
||||
|
||||
from app.services import page_analyzer
|
||||
from app.services.page_analyzer import (
|
||||
MediaCandidate,
|
||||
analyze_browser_capture,
|
||||
analyze_page,
|
||||
analyze_with_ytdlp,
|
||||
extract_media_urls,
|
||||
human_size,
|
||||
_fetch_public_page_body,
|
||||
)
|
||||
|
||||
|
||||
def test_human_size_labels_file_sizes():
|
||||
assert human_size(None) is None
|
||||
assert human_size(512) == "512 B"
|
||||
assert human_size(1536) == "1.5 KB"
|
||||
assert human_size(5 * 1024 * 1024) == "5.0 MB"
|
||||
|
||||
|
||||
def test_extract_media_urls_finds_encoded_and_unquoted_js_urls():
|
||||
html = """
|
||||
<video data-src="/media/trailer.mp4?token=abc&v=1"></video>
|
||||
<script>
|
||||
const hls = "https:\\/\\/cdn.example.org\\/movie\\/master.m3u8?sig=123";
|
||||
window.player = {dash: https://cdn.example.org/movie/manifest.mpd?x=1};
|
||||
</script>
|
||||
"""
|
||||
|
||||
assert extract_media_urls(html, "https://example.org/watch") == [
|
||||
"https://example.org/media/trailer.mp4?token=abc&v=1",
|
||||
"https://cdn.example.org/movie/master.m3u8?sig=123",
|
||||
"https://cdn.example.org/movie/manifest.mpd?x=1",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_browser_capture_filters_and_normalizes_candidates(monkeypatch):
|
||||
captured_probe_input = {}
|
||||
|
||||
async def fake_probe(urls, source):
|
||||
captured_probe_input["urls"] = urls
|
||||
captured_probe_input["source"] = source
|
||||
return [MediaCandidate(url=url, kind="direct_video", source=source) for url in urls]
|
||||
|
||||
monkeypatch.setattr(page_analyzer, "_probe_candidates_with_meta", fake_probe)
|
||||
|
||||
result = await analyze_browser_capture(
|
||||
[
|
||||
{"url": "/media/movie.mp4", "title": "Movie"},
|
||||
{"url": "https://cdn.example.org/live/master.m3u8", "content_type": "application/vnd.apple.mpegurl"},
|
||||
{"url": "https://cdn.example.org/poster.jpg", "content_type": "image/jpeg"},
|
||||
],
|
||||
page_url="https://example.org/watch/1",
|
||||
)
|
||||
|
||||
assert captured_probe_input["source"] == "browser-capture"
|
||||
assert list(captured_probe_input["urls"]) == [
|
||||
"https://example.org/media/movie.mp4",
|
||||
"https://cdn.example.org/live/master.m3u8",
|
||||
]
|
||||
assert [item.source for item in result] == ["browser-capture", "browser-capture"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_analyze_page_merges_browser_ytdlp_and_html_candidates(monkeypatch):
|
||||
async def fake_html(url):
|
||||
return [MediaCandidate(url="https://cdn.example.org/static.mp4", kind="direct_video", source="html")]
|
||||
|
||||
async def fake_ytdlp(url):
|
||||
return [MediaCandidate(url="https://cdn.example.org/ytdlp.m3u8", kind="hls_manifest", source="yt-dlp")]
|
||||
|
||||
async def fake_browser(url):
|
||||
return [MediaCandidate(url="https://cdn.example.org/browser.mpd", kind="dash_manifest", source="browser")]
|
||||
|
||||
monkeypatch.setattr(page_analyzer, "analyze_html_page", fake_html)
|
||||
monkeypatch.setattr(page_analyzer, "analyze_with_ytdlp", fake_ytdlp)
|
||||
monkeypatch.setattr(page_analyzer, "analyze_with_browser", fake_browser)
|
||||
|
||||
result = await analyze_page("https://example.org/watch")
|
||||
|
||||
assert {candidate.url: candidate.source for candidate in result} == {
|
||||
"https://cdn.example.org/static.mp4": "html",
|
||||
"https://cdn.example.org/ytdlp.m3u8": "yt-dlp",
|
||||
"https://cdn.example.org/browser.mpd": "browser",
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ytdlp_is_skipped_for_generic_untrusted_hosts(monkeypatch):
|
||||
calls = []
|
||||
|
||||
class Checked:
|
||||
final_url = "https://example.org/watch"
|
||||
|
||||
async def fake_check(url, **kwargs):
|
||||
return Checked()
|
||||
|
||||
class FakeYDL:
|
||||
def __init__(self, opts):
|
||||
calls.append(opts)
|
||||
def __enter__(self):
|
||||
return self
|
||||
def __exit__(self, *args):
|
||||
return None
|
||||
def extract_info(self, url, download=False):
|
||||
raise AssertionError("yt-dlp must not run for generic hosts")
|
||||
|
||||
monkeypatch.setattr(page_analyzer, "check_url_for_page", fake_check)
|
||||
monkeypatch.setattr(page_analyzer, "YoutubeDL", FakeYDL)
|
||||
|
||||
assert await analyze_with_ytdlp("https://example.org/watch") == []
|
||||
assert calls == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_public_page_body_revalidates_get_redirect_hops(monkeypatch):
|
||||
checked_urls = []
|
||||
|
||||
class Checked:
|
||||
def __init__(self, final_url):
|
||||
self.final_url = final_url
|
||||
|
||||
async def fake_check(url, **kwargs):
|
||||
checked_urls.append(url)
|
||||
return Checked(url)
|
||||
|
||||
async def fake_assert_public_host(url):
|
||||
checked_urls.append(f"assert:{url}")
|
||||
|
||||
class FakeURL:
|
||||
def __init__(self, url):
|
||||
self._url = url
|
||||
def join(self, loc):
|
||||
return loc
|
||||
def __str__(self):
|
||||
return self._url
|
||||
|
||||
class FakeResponse:
|
||||
def __init__(self, url, redirect_to=None, text=""):
|
||||
self.url = FakeURL(url)
|
||||
self._redirect_to = redirect_to
|
||||
self.headers = {"location": redirect_to} if redirect_to else {}
|
||||
self.text = text
|
||||
@property
|
||||
def is_redirect(self):
|
||||
return self._redirect_to is not None
|
||||
def raise_for_status(self):
|
||||
return None
|
||||
|
||||
class FakeClient:
|
||||
def __init__(self, *args, **kwargs):
|
||||
self.calls = 0
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
async def __aexit__(self, *args):
|
||||
return None
|
||||
async def get(self, url):
|
||||
self.calls += 1
|
||||
if self.calls == 1:
|
||||
return FakeResponse(url, redirect_to="https://cdn.example.org/page")
|
||||
return FakeResponse(url, text="<video src='/movie.mp4'></video>")
|
||||
|
||||
monkeypatch.setattr(page_analyzer, "check_url_for_page", fake_check)
|
||||
monkeypatch.setattr(page_analyzer, "assert_public_host", fake_assert_public_host)
|
||||
monkeypatch.setattr(page_analyzer.httpx, "AsyncClient", FakeClient)
|
||||
|
||||
body, final_url = await _fetch_public_page_body("https://example.org/watch", timeout=1)
|
||||
|
||||
assert body.startswith("<video")
|
||||
assert final_url == "https://cdn.example.org/page"
|
||||
assert "https://example.org/watch" in checked_urls
|
||||
assert "https://cdn.example.org/page" in checked_urls
|
||||
assert "assert:https://cdn.example.org/page" in checked_urls
|
||||
56
backend/tests/test_path_safety.py
Normal file
56
backend/tests/test_path_safety.py
Normal file
@@ -0,0 +1,56 @@
|
||||
from pathlib import Path
|
||||
import pytest
|
||||
|
||||
from app.services.paths import ensure_under, jellyfin_profile_target, media_target, slugify_name, tmp_target
|
||||
|
||||
|
||||
def test_slugify_name_removes_dangerous_chars():
|
||||
assert slugify_name('../Bad:Title?') == 'Bad_Title'
|
||||
|
||||
|
||||
def test_ensure_under_accepts_child(tmp_path: Path):
|
||||
assert ensure_under(tmp_path, Path('a/b')).is_relative_to(tmp_path.resolve())
|
||||
|
||||
|
||||
def test_ensure_under_rejects_escape(tmp_path: Path):
|
||||
with pytest.raises(ValueError):
|
||||
ensure_under(tmp_path, Path('../escape'))
|
||||
|
||||
|
||||
def test_targets_stay_in_configured_roots(tmp_path: Path):
|
||||
assert tmp_target(tmp_path / 'tmp', 7, '../../Film').is_relative_to((tmp_path / 'tmp').resolve())
|
||||
assert media_target(tmp_path / 'media', 'movies', '../../Film').is_relative_to((tmp_path / 'media').resolve())
|
||||
|
||||
|
||||
def test_jellyfin_film_profile_matches_real_library_layout(tmp_path: Path):
|
||||
target = jellyfin_profile_target(tmp_path / 'media', 'film', 'Big Buck Bunny', year=2008)
|
||||
assert target == (tmp_path / 'media' / 'Filme' / 'Big Buck Bunny (2008)').resolve()
|
||||
|
||||
|
||||
def test_jellyfin_series_profile_uses_staffel_folder(tmp_path: Path):
|
||||
target = jellyfin_profile_target(
|
||||
tmp_path / 'media',
|
||||
'serie',
|
||||
'Pilot/Unsafe',
|
||||
series_title='The Show',
|
||||
season=2,
|
||||
episode=3,
|
||||
episode_title='Pilot/Unsafe',
|
||||
)
|
||||
assert target == (tmp_path / 'media' / 'Serien' / 'The Show' / 'Staffel2' / 'S02E03 - Pilot_Unsafe').resolve()
|
||||
|
||||
|
||||
def test_jellyfin_clip_profile_uses_youtube_mediathek(tmp_path: Path):
|
||||
target = jellyfin_profile_target(
|
||||
tmp_path / 'media',
|
||||
'clip',
|
||||
'../../Clip',
|
||||
channel='Archive Org',
|
||||
external_id='abc/123',
|
||||
)
|
||||
assert target == (tmp_path / 'media' / 'YouTube-Mediathek' / 'Archive Org' / 'Clip [abc_123]').resolve()
|
||||
|
||||
|
||||
def test_jellyfin_profile_rejects_unknown_profile(tmp_path: Path):
|
||||
with pytest.raises(ValueError):
|
||||
jellyfin_profile_target(tmp_path / 'media', 'other', 'Title')
|
||||
9
backend/tests/test_probe_api.py
Normal file
9
backend/tests/test_probe_api.py
Normal file
@@ -0,0 +1,9 @@
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.main import app
|
||||
|
||||
|
||||
def test_probe_rejects_private_direct_url():
|
||||
client = TestClient(app)
|
||||
response = client.post('/api/probe', json={'url': 'http://localhost/video.mp4'})
|
||||
assert response.status_code == 400
|
||||
24
backend/tests/test_provider_detection.py
Normal file
24
backend/tests/test_provider_detection.py
Normal file
@@ -0,0 +1,24 @@
|
||||
from app.providers.internet_archive import InternetArchiveProvider
|
||||
from app.providers.mediathek import MediathekProvider
|
||||
from app.providers.youtube import YouTubeProvider
|
||||
|
||||
|
||||
def test_youtube_detection():
|
||||
provider = YouTubeProvider()
|
||||
assert provider.can_handle('https://www.youtube.com/watch?v=abc')
|
||||
assert provider.can_handle('https://youtu.be/abc')
|
||||
assert not provider.can_handle('https://example.com/video.mp4')
|
||||
|
||||
|
||||
def test_mediathek_detection_excludes_youtube_and_internet_archive():
|
||||
provider = MediathekProvider()
|
||||
assert provider.can_handle('https://example.com/video.mp4')
|
||||
assert not provider.can_handle('https://youtube.com/watch?v=abc')
|
||||
assert not provider.can_handle('https://archive.org/details/BigBuckBunny_328')
|
||||
|
||||
|
||||
def test_internet_archive_detection():
|
||||
provider = InternetArchiveProvider()
|
||||
assert provider.can_handle('https://archive.org/details/BigBuckBunny_328')
|
||||
assert provider.can_handle('https://archive.org/download/BigBuckBunny_328/video.mp4')
|
||||
assert not provider.can_handle('https://example.com/video.mp4')
|
||||
7
backend/tests/test_youtube_provider.py
Normal file
7
backend/tests/test_youtube_provider.py
Normal file
@@ -0,0 +1,7 @@
|
||||
from app.providers.youtube import YouTubeProvider
|
||||
|
||||
|
||||
def test_extract_youtube_id():
|
||||
provider = YouTubeProvider()
|
||||
assert provider._external_id('https://youtu.be/abc123') == 'abc123'
|
||||
assert provider._external_id('https://www.youtube.com/watch?v=xyz') == 'xyz'
|
||||
32
browser-extension/kino-capture/README.md
Normal file
32
browser-extension/kino-capture/README.md
Normal file
@@ -0,0 +1,32 @@
|
||||
# Kino Capture Browser-Extension
|
||||
|
||||
Kleine generische Browser-Extension für das Kino-Projekt. Sie sammelt im aktiven Browser-Tab sichtbare Medien-Requests und DOM-/Performance-Kandidaten und übergibt diese an die Kino-App.
|
||||
|
||||
## Installation als entpackte Extension
|
||||
|
||||
### Chromium / Chrome / Edge
|
||||
|
||||
1. Ordner `kino-capture` entpacken.
|
||||
2. `chrome://extensions` öffnen.
|
||||
3. Entwicklermodus aktivieren.
|
||||
4. „Entpackte Erweiterung laden“ klicken.
|
||||
5. Den Ordner `kino-capture` auswählen.
|
||||
|
||||
### Firefox / LibreWolf temporär
|
||||
|
||||
1. `about:debugging#/runtime/this-firefox` öffnen.
|
||||
2. „Temporäres Add-on laden“ klicken.
|
||||
3. `manifest.json` im Ordner `kino-capture` auswählen.
|
||||
|
||||
## Nutzung
|
||||
|
||||
1. Video im Browser starten und einige Sekunden laufen lassen.
|
||||
2. Extension-Icon „Kino Capture“ öffnen.
|
||||
3. Prüfen, dass Kandidaten angezeigt werden.
|
||||
4. Kino-App-URL setzen, z. B. `http://192.168.178.213:8099/`.
|
||||
5. „An Kino-App senden“ klicken.
|
||||
6. In der Kino-App Kandidat übernehmen und Import starten.
|
||||
|
||||
## Grenzen
|
||||
|
||||
Die Extension sammelt nur Medien-URLs, die der Browser normal sichtbar anfragt oder im DOM/Performance-API offenlegt. Sie enthält keine hoster-spezifischen Bypässe, keine CAPTCHA-/DRM-/Token-Umgehung und keine Entschlüsselungslogik.
|
||||
68
browser-extension/kino-capture/content.js
Normal file
68
browser-extension/kino-capture/content.js
Normal file
@@ -0,0 +1,68 @@
|
||||
const MEDIA_EXT_RE = /\.(mp4|m4v|webm|mkv|mov|m3u8|mpd)([?#]|$)/i;
|
||||
const ABSOLUTE_MEDIA_RE = /(?:https?:\/\/|https?:\\\/\\\/|\/\/|\\\/\\\/)[^\s"'<>]+?\.(?:mp4|m4v|webm|mkv|mov|m3u8|mpd)(?=$|[?&#\s"'<>])(?:[?&][^\s"'<>]*)?/gi;
|
||||
|
||||
function ignoreChromeResult(result) {
|
||||
if (result && typeof result.catch === 'function') result.catch(() => {});
|
||||
}
|
||||
|
||||
function normalizeMediaUrl(rawUrl, baseUrl = location.href) {
|
||||
if (!rawUrl || typeof rawUrl !== 'string') return null;
|
||||
try {
|
||||
const cleaned = rawUrl
|
||||
.trim()
|
||||
.replaceAll('\\/', '/')
|
||||
.replaceAll('\\u002F', '/')
|
||||
.replaceAll('\\u002f', '/')
|
||||
.replace(/[;,)}\]]+$/, '');
|
||||
return new URL(cleaned, baseUrl).href;
|
||||
} catch (_) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function pushCandidate(candidates, rawUrl, title = document.title, contentType = '', source = 'dom') {
|
||||
const url = normalizeMediaUrl(rawUrl);
|
||||
if (!url) return;
|
||||
const cleanType = String(contentType || '').split(';', 1)[0].trim();
|
||||
if (MEDIA_EXT_RE.test(url) || cleanType.startsWith('video/') || /mpegurl|dash\+xml/i.test(cleanType)) {
|
||||
candidates.push({url, title: title || document.title, content_type: contentType || '', source});
|
||||
}
|
||||
}
|
||||
|
||||
function collectCandidates() {
|
||||
const candidates = [];
|
||||
document.querySelectorAll('video,source,a').forEach((el) => {
|
||||
pushCandidate(candidates, el.currentSrc || el.src || el.href, el.title || el.textContent || document.title, el.type || '', 'dom');
|
||||
});
|
||||
|
||||
performance.getEntriesByType('resource').forEach((entry) => {
|
||||
pushCandidate(candidates, entry.name, document.title, entry.initiatorType === 'video' ? 'video/unknown' : '', `performance:${entry.initiatorType || 'resource'}`);
|
||||
});
|
||||
|
||||
const html = document.documentElement?.outerHTML || '';
|
||||
for (const match of html.matchAll(ABSOLUTE_MEDIA_RE)) {
|
||||
pushCandidate(candidates, match[0], document.title, '', 'html-js');
|
||||
}
|
||||
|
||||
const seen = new Set();
|
||||
return candidates.filter((candidate) => {
|
||||
if (seen.has(candidate.url)) return false;
|
||||
seen.add(candidate.url);
|
||||
return true;
|
||||
}).slice(0, 120);
|
||||
}
|
||||
|
||||
chrome.runtime.onMessage.addListener((message, _sender, sendResponse) => {
|
||||
if (message?.type === 'collect-media') {
|
||||
sendResponse({candidates: collectCandidates()});
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
});
|
||||
|
||||
// Opportunistisch nach dem Laden melden, damit das Popup sofort Daten hat.
|
||||
setTimeout(() => {
|
||||
for (const candidate of collectCandidates()) {
|
||||
ignoreChromeResult(chrome.runtime.sendMessage({type: 'candidate', candidate}));
|
||||
}
|
||||
}, 1500);
|
||||
30
browser-extension/kino-capture/manifest.json
Normal file
30
browser-extension/kino-capture/manifest.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"manifest_version": 2,
|
||||
"name": "Kino Capture",
|
||||
"version": "0.1.1",
|
||||
"description": "Sammelt sichtbare Medien-Requests im aktiven Tab und übergibt sie an die Kino-Projekt-App.",
|
||||
"permissions": [
|
||||
"activeTab",
|
||||
"storage",
|
||||
"tabs",
|
||||
"webNavigation",
|
||||
"webRequest",
|
||||
"<all_urls>"
|
||||
],
|
||||
"background": {
|
||||
"scripts": ["service_worker.js"],
|
||||
"persistent": false
|
||||
},
|
||||
"content_scripts": [
|
||||
{
|
||||
"matches": ["<all_urls>"],
|
||||
"js": ["content.js"],
|
||||
"run_at": "document_idle",
|
||||
"all_frames": true
|
||||
}
|
||||
],
|
||||
"browser_action": {
|
||||
"default_title": "Kino Capture",
|
||||
"default_popup": "popup.html"
|
||||
}
|
||||
}
|
||||
35
browser-extension/kino-capture/popup.css
Normal file
35
browser-extension/kino-capture/popup.css
Normal file
@@ -0,0 +1,35 @@
|
||||
body {
|
||||
min-width: 380px;
|
||||
max-width: 520px;
|
||||
margin: 0;
|
||||
background: #111827;
|
||||
color: #f9fafb;
|
||||
font: 14px system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
|
||||
}
|
||||
main { padding: 14px; }
|
||||
h1 { margin: 0 0 8px; font-size: 18px; }
|
||||
label { display: grid; gap: 4px; margin: 10px 0; }
|
||||
input {
|
||||
padding: 8px;
|
||||
border-radius: 8px;
|
||||
border: 1px solid #374151;
|
||||
background: #1f2937;
|
||||
color: #fff;
|
||||
}
|
||||
button {
|
||||
padding: 8px 10px;
|
||||
border: 0;
|
||||
border-radius: 8px;
|
||||
background: #38bdf8;
|
||||
color: #082f49;
|
||||
font-weight: 700;
|
||||
cursor: pointer;
|
||||
}
|
||||
button.secondary { background: #374151; color: #f9fafb; }
|
||||
button:disabled { opacity: .5; cursor: not-allowed; }
|
||||
.actions { display: flex; flex-wrap: wrap; gap: 8px; }
|
||||
ol { padding-left: 20px; max-height: 320px; overflow: auto; }
|
||||
li { margin: 8px 0; border-top: 1px solid #374151; padding-top: 8px; }
|
||||
code { display: block; color: #bae6fd; overflow-wrap: anywhere; font-size: 12px; }
|
||||
.meta, .hint { color: #d1d5db; font-size: 12px; }
|
||||
.error { color: #fca5a5; }
|
||||
27
browser-extension/kino-capture/popup.html
Normal file
27
browser-extension/kino-capture/popup.html
Normal file
@@ -0,0 +1,27 @@
|
||||
<!doctype html>
|
||||
<html lang="de">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>Kino Capture</title>
|
||||
<link rel="stylesheet" href="popup.css" />
|
||||
</head>
|
||||
<body>
|
||||
<main>
|
||||
<h1>Kino Capture</h1>
|
||||
<p id="status">Lade Kandidaten…</p>
|
||||
<label>
|
||||
Kino-App URL
|
||||
<input id="app-url" type="url" placeholder="http://192.168.178.213:8099/" />
|
||||
</label>
|
||||
<div class="actions">
|
||||
<button id="refresh" type="button">Neu scannen</button>
|
||||
<button id="send" type="button">An Kino-App senden</button>
|
||||
<button id="clear" type="button" class="secondary">Leeren</button>
|
||||
</div>
|
||||
<ol id="candidates"></ol>
|
||||
<p class="hint">Tipp: Video starten und einige Sekunden laufen lassen, dann dieses Popup öffnen.</p>
|
||||
</main>
|
||||
<script src="popup.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
103
browser-extension/kino-capture/popup.js
Normal file
103
browser-extension/kino-capture/popup.js
Normal file
@@ -0,0 +1,103 @@
|
||||
const DEFAULT_APP_URL = 'http://192.168.178.213:8099/';
|
||||
const statusEl = document.querySelector('#status');
|
||||
const listEl = document.querySelector('#candidates');
|
||||
const appUrlEl = document.querySelector('#app-url');
|
||||
const sendButton = document.querySelector('#send');
|
||||
const refreshButton = document.querySelector('#refresh');
|
||||
const clearButton = document.querySelector('#clear');
|
||||
|
||||
let activeTab = null;
|
||||
let currentPayload = {page_url: '', candidates: []};
|
||||
|
||||
function encodeCapturePayload(payload) {
|
||||
return btoa(unescape(encodeURIComponent(JSON.stringify(payload))));
|
||||
}
|
||||
|
||||
function normalizeAppUrl(value) {
|
||||
try {
|
||||
const url = new URL(value || DEFAULT_APP_URL);
|
||||
if (!url.pathname.endsWith('/')) url.pathname = `${url.pathname}/`;
|
||||
url.search = '';
|
||||
url.hash = '';
|
||||
return url.href;
|
||||
} catch (_) {
|
||||
return DEFAULT_APP_URL;
|
||||
}
|
||||
}
|
||||
|
||||
function render(payload) {
|
||||
currentPayload = payload || {page_url: activeTab?.url || '', candidates: []};
|
||||
const candidates = currentPayload.candidates || [];
|
||||
statusEl.textContent = `${candidates.length} Kandidat(en) im aktiven Tab gefunden.`;
|
||||
statusEl.className = candidates.length ? '' : 'error';
|
||||
sendButton.disabled = candidates.length === 0;
|
||||
listEl.innerHTML = '';
|
||||
for (const candidate of candidates.slice(0, 30)) {
|
||||
const li = document.createElement('li');
|
||||
const type = candidate.content_type || candidate.source || 'media';
|
||||
li.innerHTML = `<div class="meta"></div><code></code>`;
|
||||
li.querySelector('.meta').textContent = type;
|
||||
li.querySelector('code').textContent = candidate.url;
|
||||
listEl.appendChild(li);
|
||||
}
|
||||
}
|
||||
|
||||
async function getActiveTab() {
|
||||
const [tab] = await chrome.tabs.query({active: true, currentWindow: true});
|
||||
return tab;
|
||||
}
|
||||
|
||||
async function refresh() {
|
||||
activeTab = await getActiveTab();
|
||||
if (!activeTab?.id) {
|
||||
statusEl.textContent = 'Kein aktiver Tab gefunden.';
|
||||
statusEl.className = 'error';
|
||||
return;
|
||||
}
|
||||
const response = await chrome.runtime.sendMessage({type: 'get-candidates', tabId: activeTab.id});
|
||||
if (!response?.ok) {
|
||||
statusEl.textContent = response?.error || 'Kandidaten konnten nicht gelesen werden.';
|
||||
statusEl.className = 'error';
|
||||
return;
|
||||
}
|
||||
render(response);
|
||||
}
|
||||
|
||||
async function saveAppUrl() {
|
||||
const appUrl = normalizeAppUrl(appUrlEl.value);
|
||||
appUrlEl.value = appUrl;
|
||||
await chrome.storage.local.set({kinoAppUrl: appUrl});
|
||||
return appUrl;
|
||||
}
|
||||
|
||||
async function sendToKinoApp() {
|
||||
const appUrl = await saveAppUrl();
|
||||
const encoded = encodeCapturePayload({
|
||||
page_url: currentPayload.page_url || activeTab?.url || '',
|
||||
candidates: currentPayload.candidates || [],
|
||||
});
|
||||
const destination = `${appUrl}?capture=${encodeURIComponent(encoded)}`;
|
||||
await chrome.tabs.create({url: destination});
|
||||
}
|
||||
|
||||
async function clearCandidates() {
|
||||
if (!activeTab?.id) activeTab = await getActiveTab();
|
||||
if (activeTab?.id) await chrome.runtime.sendMessage({type: 'clear-candidates', tabId: activeTab.id});
|
||||
render({page_url: activeTab?.url || '', candidates: []});
|
||||
}
|
||||
|
||||
async function init() {
|
||||
const stored = await chrome.storage.local.get({kinoAppUrl: DEFAULT_APP_URL});
|
||||
appUrlEl.value = normalizeAppUrl(stored.kinoAppUrl);
|
||||
await refresh();
|
||||
}
|
||||
|
||||
appUrlEl.addEventListener('change', saveAppUrl);
|
||||
refreshButton.addEventListener('click', refresh);
|
||||
sendButton.addEventListener('click', sendToKinoApp);
|
||||
clearButton.addEventListener('click', clearCandidates);
|
||||
|
||||
init().catch((error) => {
|
||||
statusEl.textContent = error?.message || String(error);
|
||||
statusEl.className = 'error';
|
||||
});
|
||||
138
browser-extension/kino-capture/service_worker.js
Normal file
138
browser-extension/kino-capture/service_worker.js
Normal file
@@ -0,0 +1,138 @@
|
||||
const MEDIA_EXT_RE = /\.(mp4|m4v|webm|mkv|mov|m3u8|mpd)([?#]|$)/i;
|
||||
const MEDIA_CONTENT_RE = /^(video\/|application\/(vnd\.apple\.mpegurl|x-mpegurl|dash\+xml)|audio\/mpegurl)/i;
|
||||
const MAX_CANDIDATES_PER_TAB = 120;
|
||||
|
||||
const tabCandidates = new Map();
|
||||
const tabPageUrls = new Map();
|
||||
|
||||
const actionApi = chrome.action || chrome.browserAction;
|
||||
|
||||
function ignoreChromeResult(result) {
|
||||
if (result && typeof result.catch === 'function') result.catch(() => {});
|
||||
}
|
||||
|
||||
function setBadgeTextSafe(tabId, text) {
|
||||
if (actionApi?.setBadgeText) ignoreChromeResult(actionApi.setBadgeText({tabId, text}));
|
||||
}
|
||||
|
||||
function setBadgeBackgroundColorSafe(tabId, color) {
|
||||
if (actionApi?.setBadgeBackgroundColor) ignoreChromeResult(actionApi.setBadgeBackgroundColor({tabId, color}));
|
||||
}
|
||||
|
||||
function candidateKey(candidate) {
|
||||
return candidate.url;
|
||||
}
|
||||
|
||||
function normalizeUrl(rawUrl) {
|
||||
if (!rawUrl || typeof rawUrl !== 'string') return null;
|
||||
try {
|
||||
return new URL(rawUrl).href;
|
||||
} catch (_) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function looksLikeMedia(url, contentType = '') {
|
||||
return MEDIA_EXT_RE.test(url) || MEDIA_CONTENT_RE.test(String(contentType).split(';', 1)[0].trim());
|
||||
}
|
||||
|
||||
function addCandidate(tabId, candidate) {
|
||||
if (tabId < 0 || !candidate?.url) return;
|
||||
const url = normalizeUrl(candidate.url);
|
||||
if (!url) return;
|
||||
const contentType = candidate.content_type || candidate.type || '';
|
||||
if (!looksLikeMedia(url, contentType)) return;
|
||||
|
||||
const existing = tabCandidates.get(tabId) || [];
|
||||
if (existing.some((item) => candidateKey(item) === url)) return;
|
||||
existing.unshift({
|
||||
url,
|
||||
title: candidate.title || candidate.name || '',
|
||||
content_type: contentType,
|
||||
quality: candidate.quality || '',
|
||||
source: candidate.source || 'webRequest',
|
||||
ts: Date.now(),
|
||||
});
|
||||
tabCandidates.set(tabId, existing.slice(0, MAX_CANDIDATES_PER_TAB));
|
||||
setBadgeTextSafe(tabId, String(Math.min(existing.length, 99)));
|
||||
setBadgeBackgroundColorSafe(tabId, '#38bdf8');
|
||||
}
|
||||
|
||||
chrome.webRequest.onBeforeRequest.addListener(
|
||||
(details) => {
|
||||
if (details.tabId < 0 || details.type === 'image') return;
|
||||
if (looksLikeMedia(details.url)) {
|
||||
addCandidate(details.tabId, {url: details.url, source: `request:${details.type}`});
|
||||
}
|
||||
},
|
||||
{urls: ['<all_urls>']}
|
||||
);
|
||||
|
||||
chrome.webRequest.onHeadersReceived.addListener(
|
||||
(details) => {
|
||||
if (details.tabId < 0) return;
|
||||
const header = (details.responseHeaders || []).find((item) => item.name.toLowerCase() === 'content-type');
|
||||
const contentType = header?.value || '';
|
||||
if (looksLikeMedia(details.url, contentType)) {
|
||||
addCandidate(details.tabId, {url: details.url, content_type: contentType, source: 'response'});
|
||||
}
|
||||
},
|
||||
{urls: ['<all_urls>']},
|
||||
['responseHeaders']
|
||||
);
|
||||
|
||||
chrome.webNavigation?.onCommitted?.addListener?.((details) => {
|
||||
if (details.frameId === 0) {
|
||||
tabPageUrls.set(details.tabId, details.url);
|
||||
tabCandidates.delete(details.tabId);
|
||||
setBadgeTextSafe(details.tabId, '');
|
||||
}
|
||||
});
|
||||
|
||||
chrome.tabs.onRemoved.addListener((tabId) => {
|
||||
tabCandidates.delete(tabId);
|
||||
tabPageUrls.delete(tabId);
|
||||
});
|
||||
|
||||
async function collectDomCandidates(tabId) {
|
||||
try {
|
||||
const responses = await chrome.tabs.sendMessage(tabId, {type: 'collect-media'});
|
||||
return Array.isArray(responses) ? responses : responses?.candidates || [];
|
||||
} catch (_) {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
chrome.runtime.onMessage.addListener((message, sender, sendResponse) => {
|
||||
(async () => {
|
||||
if (message?.type === 'candidate' && sender.tab?.id != null) {
|
||||
addCandidate(sender.tab.id, message.candidate);
|
||||
sendResponse({ok: true});
|
||||
return;
|
||||
}
|
||||
|
||||
if (message?.type === 'get-candidates') {
|
||||
const tabId = message.tabId;
|
||||
const domCandidates = await collectDomCandidates(tabId);
|
||||
for (const candidate of domCandidates) addCandidate(tabId, {...candidate, source: candidate.source || 'dom'});
|
||||
const tab = await chrome.tabs.get(tabId).catch(() => null);
|
||||
sendResponse({
|
||||
ok: true,
|
||||
page_url: tab?.url || tabPageUrls.get(tabId) || '',
|
||||
candidates: tabCandidates.get(tabId) || [],
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (message?.type === 'clear-candidates') {
|
||||
const tabId = message.tabId;
|
||||
tabCandidates.delete(tabId);
|
||||
setBadgeTextSafe(tabId, '');
|
||||
sendResponse({ok: true});
|
||||
return;
|
||||
}
|
||||
|
||||
sendResponse({ok: false, error: 'unknown message'});
|
||||
})();
|
||||
return true;
|
||||
});
|
||||
11
docker-compose.yml
Normal file
11
docker-compose.yml
Normal file
@@ -0,0 +1,11 @@
|
||||
services:
|
||||
kino-projekt:
|
||||
build: .
|
||||
ports:
|
||||
- "127.0.0.1:8099:8080"
|
||||
volumes:
|
||||
- ./data:/data
|
||||
- /jellyfin:/jellyfin
|
||||
env_file:
|
||||
- .env
|
||||
restart: unless-stopped
|
||||
19
docs/deployment.md
Normal file
19
docs/deployment.md
Normal file
@@ -0,0 +1,19 @@
|
||||
# Deployment
|
||||
|
||||
1. `.env` auf dem Zielhost aus `.env.example` erstellen; `JELLYFIN_API_KEY`, `KINOPROJEKT_ADMIN_PASSWORD_HASH` und `KINOPROJEKT_SESSION_SECRET` nur dort oder in Vaultwarden speichern.
|
||||
2. Medien- und Datenverzeichnisse als Volumes mounten.
|
||||
3. Dienst nur Mesh-only veröffentlichen und den App-Login aktivieren.
|
||||
4. Nach Start prüfen:
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:8080/health
|
||||
curl -X POST http://127.0.0.1:8080/api/probe \
|
||||
-H 'content-type: application/json' \
|
||||
--data '{"url":"http://localhost/video.mp4"}'
|
||||
# Erwartet: HTTP 400, weil private/loopback Ziele abgelehnt werden.
|
||||
```
|
||||
|
||||
5. Login prüfen: Ohne Session antworten geschützte API-Routen mit `401`; nach `POST /api/auth/login` setzt die App ein `HttpOnly; Secure; SameSite=Lax` Session-Cookie.
|
||||
6. Zielprofile prüfen: Ein Import mit `target_profile=film|serie|clip` muss unter `/jellyfin/Filme`, `/jellyfin/Serien/.../Staffel<N>` oder `/jellyfin/YouTube-Mediathek` landen. Der Ordner `/jellyfin/YouTube-Mediathek` sollte auf dem Zielhost `jellyfinuser:media` gehören und setgid wie die vorhandenen Bibliotheken nutzen.
|
||||
7. Für echte Importe muss `yt-dlp` im Container/Host verfügbar sein; das Dockerfile installiert es.
|
||||
8. Jellyfin-Refresh wird über `POST <JELLYFIN_URL>/Library/Refresh` mit `X-Emby-Token` ausgelöst, wenn `JELLYFIN_API_KEY` gesetzt ist.
|
||||
11
docs/provider-policy.md
Normal file
11
docs/provider-policy.md
Normal file
@@ -0,0 +1,11 @@
|
||||
# Provider Policy
|
||||
|
||||
Erlaubt sind nur öffentlich und legal erreichbare Quellen ohne Umgehung technischer oder rechtlicher Zugriffsbeschränkungen.
|
||||
|
||||
## Implementierte Provider
|
||||
|
||||
- **YouTube:** Metadaten/Downloads über `yt-dlp`, ohne Cookies, Login-Flows oder Umgehungsflags; Default-Qualität maximal 1080p.
|
||||
- **Internet Archive:** akzeptiert `archive.org/details/<identifier>`, `archive.org/download/<identifier>/...` und `archive.org/metadata/<identifier>`; Metadaten kommen aus der öffentlichen Archive-Metadata-API, heruntergeladen wird nur ein plausibler öffentlicher Medien-Track.
|
||||
- **Mediathek/Direktlink:** direkte URLs werden vor Verwendung gegen SSRF-Risiken geprüft: Schema, DNS, Redirects, private IP-Ranges und Content-Type/Dateiendung.
|
||||
|
||||
Nicht erlaubt sind Piracy-Scraper, DRM-/Paywall-/Login-/Geoblocking-Umgehung, Cookie-Importe und freie Server-Schreibpfade.
|
||||
Reference in New Issue
Block a user