chore: archive current Kino runtime and deployment #5
@@ -2,6 +2,8 @@
|
||||
|
||||
Docker-hosted web GUI for searching and downloading movies/series, with Jellyfin integration.
|
||||
|
||||
> **Current runtime snapshot (2026-08-21):** The latest VM207/CT201 FastAPI implementation, browser extension, tests, Docker stack, token monitor, and migration report are under [`runtime-current/`](runtime-current/README.md). The older root frontend/backend/worker layout below remains as project history and has not been overwritten.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
|
||||
27
runtime-current/.gitignore
vendored
Normal file
27
runtime-current/.gitignore
vendored
Normal file
@@ -0,0 +1,27 @@
|
||||
# Secrets and local runtime config
|
||||
*.env
|
||||
!.env.example
|
||||
kino/kino-projekt.env
|
||||
token-monitor/token-monitor.env
|
||||
|
||||
# Runtime data
|
||||
**/data/
|
||||
**/export/
|
||||
*.sqlite
|
||||
*.sqlite3
|
||||
*.sqlite3-wal
|
||||
*.sqlite3-shm
|
||||
*.db
|
||||
*.db-wal
|
||||
*.db-shm
|
||||
|
||||
# Credentials and browser/runtime caches
|
||||
**/.ssh/
|
||||
**/known_hosts
|
||||
**/.cache/
|
||||
**/.venv/
|
||||
**/venv/
|
||||
**/__pycache__/
|
||||
**/*.pyc
|
||||
**/*.egg-info/
|
||||
**/.pytest_cache/
|
||||
63
runtime-current/MIGRATION-2026-08-21.md
Normal file
63
runtime-current/MIGRATION-2026-08-21.md
Normal file
@@ -0,0 +1,63 @@
|
||||
# Migration Kino-Projekt: VM207 → Docker-CT201
|
||||
|
||||
**Datum:** 21.08.2026
|
||||
**Status:** erfolgreich migriert; VM207 gestoppt, nicht gelöscht
|
||||
|
||||
## Zielzustand
|
||||
|
||||
| Komponente | Neuer Ort | Port/Status |
|
||||
|---|---|---|
|
||||
| Kino-Projekt | Docker-CT201 `docker`, `192.168.178.224` | `8099`, Container `kino-projekt`, healthy |
|
||||
| Token-Monitor | Docker-CT201 | `8098`, Container `token-monitor`, healthy |
|
||||
| Öffentlicher Kino-Tunnel | CT201 systemd `kino-vps-tunnel.service` | aktiv, Remote-Port `8892` |
|
||||
| Jellyfin-Ziel | CT100, `192.168.178.222` | rsync als `kino-transfer` weiterhin aktiv |
|
||||
| Alte VM | VM207 `kino-projekt`, `192.168.178.213` | gestoppt, Disk für Rollback erhalten |
|
||||
|
||||
## Durchgeführt
|
||||
|
||||
- Live-Dienste, Daten, Umgebungsdateien, SSH-Transferkeys, Reverse-Tunnel und aktive Jobs auf VM207 inventarisiert.
|
||||
- Keine laufenden Importe vor dem Cutover festgestellt; zwei vorhandene Jobs waren bereits `done`.
|
||||
- Quellcode, SQLite-Datenbank, Export-Staging, Browser-Erweiterung, Transferkey, Token-Monitor-Daten und Tunnelkey gesichert übertragen.
|
||||
- Rollbackfähigen Docker-Stack unter `/srv/kino-stack/` auf CT201 aufgebaut.
|
||||
- Kino-Image auf Python 3.12 mit ffmpeg, rsync, OpenSSH, yt-dlp und Playwright/Chromium gebaut.
|
||||
- Build-Test: **45 Tests bestanden**, eine unkritische Starlette/httpx-Deprecation-Warnung.
|
||||
- Container laufen im Host-Netz, da Docker-Portweiterleitung auf CT201 von den bestehenden Meshguard-/Docker-Regeln für neue Ports nicht zuverlässig erreichbar war.
|
||||
- Jellyfin-SSH-Regel um `kino-transfer@192.168.178.224` ergänzt; alter VM207-Zugang für Rollback beibehalten.
|
||||
- Reverse-Tunnel von VM207 auf CT201 übertragen; NPM-Container erreicht `headscale-tunnel-sshd:8892/health` mit `{"status":"ok"}`.
|
||||
- Alte Dienste auf VM207 gestoppt und deaktiviert; finalen Datenstand danach erneut synchronisiert.
|
||||
- VM207 kontrolliert heruntergefahren.
|
||||
|
||||
## Verifikation
|
||||
|
||||
- `http://192.168.178.224:8099/health` → HTTP 200 `{"status":"ok"}`
|
||||
- `http://192.168.178.224:8099/` → HTTP 200
|
||||
- `http://192.168.178.224:8099/api/auth/status` → bisheriges Verhalten erhalten (`enabled=false`)
|
||||
- `http://192.168.178.224:8098/` → HTTP 200
|
||||
- Playwright/Chromium-Start im Container → `PLAYWRIGHT_OK`
|
||||
- Analyse einer öffentlichen Internet-Archive-Seite → erlaubter direkter MP4-Kandidat erkannt
|
||||
- SSH-/rsync-Schreibtest aus dem Kino-Container nach `/jellyfin` → erfolgreich
|
||||
- Realer Testimport einer öffentlichen MP4-Datei → Job `done`, Datei per rsync auf Jellyfin angekommen; Testjob und Testmedium anschließend entfernt
|
||||
- Container-Restart → beide Container erneut `healthy`
|
||||
- Tunnel-Restart → aktiv; NPM-interner Upstream-Healthcheck weiterhin OK
|
||||
- Öffentlicher Schutz unverändert: HTTP → HTTPS 301, HTTPS ohne Zugangsdaten → 401
|
||||
- Nach Abschaltung von VM207: alte Ports `192.168.178.213:8099/8080` nicht mehr erreichbar; neue Ports weiterhin HTTP 200
|
||||
|
||||
## Daten und Backups
|
||||
|
||||
- CT201 Stack: `/srv/kino-stack/`
|
||||
- CT201 Vorbereitungsbackup: `/root/kino-stack-backups/pre-migration-20260821-172103`
|
||||
- VM207 Rollbackbackup: `/root/kino-migration-backup-20260821-154102`
|
||||
- PVE-Konfigurationssnapshot: `/root/decommissioned-vm-configs/vm207-kino-projekt-20260821-175059.conf`
|
||||
- VM207 wurde **nicht gelöscht**; ein Start ist als schneller Infrastruktur-Rollback weiterhin möglich.
|
||||
|
||||
## Rollback
|
||||
|
||||
1. Auf CT201 `kino-vps-tunnel.service` stoppen und `/srv/kino-stack` herunterfahren.
|
||||
2. VM207 starten.
|
||||
3. Auf VM207 `kino-projekt.service`, `token-monitor.service` und `kino-vps-tunnel.service` wieder aktivieren/starten.
|
||||
4. Health, rsync und NPM-Tunnel erneut prüfen.
|
||||
|
||||
## Hinweise
|
||||
|
||||
- Die vorhandene Token-Monitor-Datenbank wurde unverändert übernommen. Ihr Hermes-History-Snapshot war bereits vor der Migration nicht aktuell; die Provider-Liveabfragen und Alarmzustände laufen weiter. Eine erneuerte automatische Synchronisation der Hermes-`state.db` ist ein separates Folgeprojekt.
|
||||
- Für den Kino-Dienst ist die neue LAN-Adresse `http://192.168.178.224:8099`. Der öffentliche Hostname `https://kino.dasposchi.de` bleibt unverändert.
|
||||
50
runtime-current/README.md
Normal file
50
runtime-current/README.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# Current Kino/Token-Monitor Runtime Export
|
||||
|
||||
This directory contains the latest **sanitized source and deployment definition** copied from Docker CT201 on 2026-08-21.
|
||||
|
||||
The two runtime containers were removed afterward at the user's request. This is therefore a reproducible source/configuration export, not proof that the services are currently running.
|
||||
|
||||
## Contents
|
||||
|
||||
- `kino/app-src/` – latest Kino-Projekt backend, web UI, browser extension, tests and Dockerfile
|
||||
- `token-monitor/` – token monitor source and Dockerfile
|
||||
- `compose.yml` – last validated CT201 Docker Compose topology
|
||||
- `deployment/kino-vps-tunnel.service.example` – reverse-tunnel unit without private keys
|
||||
- `MIGRATION-2026-08-21.md` – VM207 → CT201 migration and verification report
|
||||
|
||||
## Intentionally excluded
|
||||
|
||||
Runtime and secret-bearing data must not be committed:
|
||||
|
||||
- `kino/kino-projekt.env`
|
||||
- `token-monitor/token-monitor.env`
|
||||
- SQLite databases and WAL/SHM files
|
||||
- imported/exported media
|
||||
- SSH private keys and `known_hosts`
|
||||
- browser caches and virtual environments
|
||||
- `/var/lib/token-monitor/hermes-state.db`
|
||||
|
||||
Use the two `.env.example` templates and inject real values from Vaultwarden or host-managed environment files.
|
||||
|
||||
## Last verified build
|
||||
|
||||
```text
|
||||
45 tests passed
|
||||
Playwright Chromium launch: OK
|
||||
Kino health: HTTP 200
|
||||
Token monitor: HTTP 200
|
||||
Jellyfin rsync: OK
|
||||
Real public-media import: done, test media removed
|
||||
```
|
||||
|
||||
## Rebuild
|
||||
|
||||
```bash
|
||||
cp kino/kino-projekt.env.example kino/kino-projekt.env
|
||||
cp token-monitor/token-monitor.env.example token-monitor/token-monitor.env
|
||||
# Fill secrets outside Git.
|
||||
docker compose build
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
The Compose file uses host networking with Kino on `8099` and Token Monitor on `8098`, matching the last validated CT201 deployment.
|
||||
21
runtime-current/REMOVAL-2026-08-21.md
Normal file
21
runtime-current/REMOVAL-2026-08-21.md
Normal file
@@ -0,0 +1,21 @@
|
||||
# Container Removal – 2026-08-21
|
||||
|
||||
At the user's request, the Docker containers `kino-projekt` and `token-monitor` were stopped and removed from CT201 after the verified VM207 migration.
|
||||
|
||||
Also disabled/stopped:
|
||||
|
||||
- `kino-vps-tunnel.service`
|
||||
|
||||
Preserved on CT201 (not committed to Git):
|
||||
|
||||
- `/srv/kino-stack/kino/data/`
|
||||
- `/srv/kino-stack/kino/export/`
|
||||
- `/srv/kino-stack/kino/ssh/`
|
||||
- `/srv/kino-stack/kino/kino-projekt.env`
|
||||
- `/srv/kino-stack/token-monitor/data/`
|
||||
- `/srv/kino-stack/token-monitor/token-monitor.env`
|
||||
- Docker images and `/srv/kino-stack/` deployment files
|
||||
|
||||
VM207 remains stopped and not deleted, so it is still an infrastructure-level rollback option.
|
||||
|
||||
This Git export contains source code, tests, sanitized deployment files and `.env.example` templates only. It deliberately excludes credentials, SSH keys, imported media, SQLite state and Hermes history data.
|
||||
57
runtime-current/compose.yml
Normal file
57
runtime-current/compose.yml
Normal file
@@ -0,0 +1,57 @@
|
||||
services:
|
||||
kino-projekt:
|
||||
build:
|
||||
context: ./kino/app-src
|
||||
dockerfile: Dockerfile
|
||||
container_name: kino-projekt
|
||||
init: true
|
||||
restart: unless-stopped
|
||||
env_file:
|
||||
- ./kino/kino-projekt.env
|
||||
environment:
|
||||
PLAYWRIGHT_BROWSERS_PATH: /ms-playwright
|
||||
KINOPROJEKT_UBLOCK_PATH: /opt/kino-projekt/ublock-origin
|
||||
network_mode: host
|
||||
command: ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8099"]
|
||||
volumes:
|
||||
- ./kino/data:/opt/kino-projekt/data
|
||||
- ./kino/export:/opt/kino-projekt/export
|
||||
- ./kino/ssh:/opt/kino-projekt/.ssh:ro
|
||||
- ./kino/ublock-origin:/opt/kino-projekt/ublock-origin:ro
|
||||
shm_size: "1gb"
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
cap_drop:
|
||||
- ALL
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8099/health', timeout=3).read()"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
|
||||
token-monitor:
|
||||
build:
|
||||
context: ./token-monitor
|
||||
dockerfile: Dockerfile
|
||||
container_name: token-monitor
|
||||
init: true
|
||||
restart: unless-stopped
|
||||
env_file:
|
||||
- ./token-monitor/token-monitor.env
|
||||
environment:
|
||||
DATA_DIR: /data
|
||||
network_mode: host
|
||||
command: ["gunicorn", "--bind", "0.0.0.0:8098", "--workers", "1", "--threads", "4", "--timeout", "30", "app:app"]
|
||||
volumes:
|
||||
- ./token-monitor/data:/data
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
cap_drop:
|
||||
- ALL
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8098/', timeout=3).read()"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 15s
|
||||
16
runtime-current/deployment/kino-vps-tunnel.service.example
Normal file
16
runtime-current/deployment/kino-vps-tunnel.service.example
Normal file
@@ -0,0 +1,16 @@
|
||||
[Unit]
|
||||
Description=Reverse SSH tunnel for containerized Kino-Projekt via public VPS/NPM
|
||||
After=network-online.target docker.service
|
||||
Wants=network-online.target
|
||||
Requires=docker.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do curl -fsS http://127.0.0.1:8099/health >/dev/null && exit 0; sleep 2; done; exit 1'
|
||||
ExecStart=/usr/bin/ssh -N -T -p 2222 -i /root/.ssh/kino_reverse_tunnel_ed25519 -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o StrictHostKeyChecking=yes -R 0.0.0.0:8892:127.0.0.1:8099 root@194.59.204.106
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
12
runtime-current/kino/app-src/.dockerignore
Normal file
12
runtime-current/kino/app-src/.dockerignore
Normal file
@@ -0,0 +1,12 @@
|
||||
.venv/
|
||||
**/.venv/
|
||||
__pycache__/
|
||||
**/__pycache__/
|
||||
.pytest_cache/
|
||||
**/.pytest_cache/
|
||||
*.pyc
|
||||
*.pyo
|
||||
backend/*.egg-info/
|
||||
backend/**/*.egg-info/
|
||||
plugin.video.xstream/
|
||||
script.module.xstreamscraper/
|
||||
13
runtime-current/kino/app-src/.env.example
Normal file
13
runtime-current/kino/app-src/.env.example
Normal file
@@ -0,0 +1,13 @@
|
||||
KINOPROJEKT_DB=/data/kino.sqlite3
|
||||
KINOPROJEKT_MEDIA_ROOT=/jellyfin
|
||||
KINOPROJEKT_TMP=/data/tmp
|
||||
JELLYFIN_URL=http://jellyfin:8096
|
||||
# JELLYFIN_API_KEY belongs in the protected target-host .env or Vaultwarden, never in Nextcloud notes.
|
||||
JELLYFIN_API_KEY=
|
||||
DEFAULT_MAX_HEIGHT=1080
|
||||
MAX_DOWNLOAD_BYTES=15000000000
|
||||
KINOPROJEKT_ADMIN_USER=admin
|
||||
# Generate with: python -c "from app.services.auth import hash_password; import getpass; print(hash_password(getpass.getpass()))"
|
||||
KINOPROJEKT_ADMIN_PASSWORD_HASH=
|
||||
# Generate with: python -c "import secrets; print(secrets.token_urlsafe(48))"
|
||||
KINOPROJEKT_SESSION_SECRET=
|
||||
26
runtime-current/kino/app-src/Dockerfile
Normal file
26
runtime-current/kino/app-src/Dockerfile
Normal file
@@ -0,0 +1,26 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PLAYWRIGHT_BROWSERS_PATH=/ms-playwright \
|
||||
HOME=/opt/kino-projekt
|
||||
|
||||
WORKDIR /app/backend
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
ca-certificates curl ffmpeg openssh-client rsync \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY backend/ ./
|
||||
RUN pip install --no-cache-dir .[test] yt-dlp \
|
||||
&& python -m playwright install --with-deps chromium \
|
||||
&& pytest -q
|
||||
|
||||
RUN groupadd --gid 10001 kino \
|
||||
&& useradd --uid 10001 --gid 10001 --create-home --home-dir /opt/kino-projekt --shell /usr/sbin/nologin kino \
|
||||
&& mkdir -p /opt/kino-projekt/data /opt/kino-projekt/export /opt/kino-projekt/.ssh /opt/kino-projekt/.cache \
|
||||
&& chown -R kino:kino /opt/kino-projekt /app /ms-playwright
|
||||
|
||||
USER kino
|
||||
EXPOSE 8080
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080"]
|
||||
51
runtime-current/kino/app-src/README.md
Normal file
51
runtime-current/kino/app-src/README.md
Normal file
@@ -0,0 +1,51 @@
|
||||
# Kino-Projekt
|
||||
|
||||
Interne WebUI zum Prüfen und Importieren von Inhalten aus legalen Quellen in Jellyfin.
|
||||
|
||||
## Sicherheitsgrenzen
|
||||
|
||||
Dieses Projekt implementiert bewusst **keine** Scraper für Piracy-Seiten, keine DRM-/Paywall-/Login-Umgehung und keinen Cookie-Import. Unterstützt werden YouTube-Metadaten via `yt-dlp`, öffentliche Internet-Archive-Items sowie konservativ geprüfte öffentliche Medien-/Mediathek-URLs.
|
||||
|
||||
## Entwicklung
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
python -m venv .venv
|
||||
. .venv/bin/activate
|
||||
pip install -e '.[test]'
|
||||
pytest -q
|
||||
uvicorn app.main:app --host 0.0.0.0 --port 8080
|
||||
```
|
||||
|
||||
Healthcheck: `curl http://localhost:8080/health`
|
||||
|
||||
## API-MVP
|
||||
|
||||
- `POST /api/probe` prüft eine URL und liefert Metadaten, ohne einen Download zu starten.
|
||||
- `POST /api/imports` legt Source + Import-Job in SQLite an und startet den Import im Hintergrund.
|
||||
- `GET /api/imports` und `GET /api/imports/{id}` zeigen Status/Fortschritt/Fehler an.
|
||||
- `GET /api/sources` listet bekannte Quellen.
|
||||
|
||||
## Zielprofile
|
||||
|
||||
Die WebUI bietet drei Jellyfin-Zielprofile an:
|
||||
|
||||
- `film` → `/jellyfin/Filme/<Titel> (<Jahr>)/`
|
||||
- `serie` → `/jellyfin/Serien/<Serientitel>/Staffel<N>/<SxxExx - Episodentitel>/`
|
||||
- `clip` → `/jellyfin/YouTube-Mediathek/<Kanal oder Quelle>/<Titel> [<id>]/`
|
||||
|
||||
Optionale Titel-/Jahr-/Serienfelder dienen nur zur Zielpfad-Korrektur; Pfade werden serverseitig normalisiert und bleiben unter `KINOPROJEKT_MEDIA_ROOT`.
|
||||
|
||||
## App-Login
|
||||
|
||||
Wenn `KINOPROJEKT_ADMIN_PASSWORD_HASH` und `KINOPROJEKT_SESSION_SECRET` gesetzt sind, schützt ein lokaler App-Login die API. Das Passwort wird nur als Argon2-Hash gespeichert; die Session liegt in einem `HttpOnly`, `Secure`, `SameSite=Lax` Cookie.
|
||||
|
||||
Hash/Secret für die geschützte Zielhost-`.env` erzeugen:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
python -c "from app.services.auth import hash_password; import getpass; print(hash_password(getpass.getpass()))"
|
||||
python -c "import secrets; print(secrets.token_urlsafe(48))"
|
||||
```
|
||||
|
||||
Jellyfin wird nur aktualisiert, wenn `JELLYFIN_API_KEY` in einer geschützten Zielhost-Umgebung gesetzt ist; Secrets gehören nicht in Nextcloud.
|
||||
54
runtime-current/kino/app-src/backend/app/config.py
Normal file
54
runtime-current/kino/app-src/backend/app/config.py
Normal file
@@ -0,0 +1,54 @@
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from urllib.parse import unquote, urlparse
|
||||
|
||||
from pydantic import Field, model_validator
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
def sqlite_url_to_path(value: str | None) -> Path | None:
|
||||
if not value:
|
||||
return None
|
||||
parsed = urlparse(value)
|
||||
if parsed.scheme != "sqlite":
|
||||
return None
|
||||
if parsed.netloc and parsed.netloc != "localhost":
|
||||
return None
|
||||
return Path(unquote(parsed.path))
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
db_path: Path = Field(default=Path("/data/kino.sqlite3"), alias="KINOPROJEKT_DB")
|
||||
legacy_db_url: str | None = Field(default=None, alias="KINOPROJEKT_DB_URL")
|
||||
media_root: Path = Field(default=Path("/jellyfin"), alias="KINOPROJEKT_MEDIA_ROOT")
|
||||
tmp_dir: Path = Field(default=Path("/data/tmp"), alias="KINOPROJEKT_TMP")
|
||||
jellyfin_url: str = Field(default="http://jellyfin:8096", alias="JELLYFIN_URL")
|
||||
jellyfin_api_key: str | None = Field(default=None, alias="JELLYFIN_API_KEY")
|
||||
default_max_height: int = Field(default=1080, alias="DEFAULT_MAX_HEIGHT")
|
||||
max_download_bytes: int = Field(default=15_000_000_000, alias="MAX_DOWNLOAD_BYTES")
|
||||
request_timeout: float = Field(default=15.0, alias="KINOPROJEKT_REQUEST_TIMEOUT")
|
||||
admin_username: str = Field(default="admin", alias="KINOPROJEKT_ADMIN_USER")
|
||||
admin_password_hash: str | None = Field(default=None, alias="KINOPROJEKT_ADMIN_PASSWORD_HASH")
|
||||
session_secret: str | None = Field(default=None, alias="KINOPROJEKT_SESSION_SECRET")
|
||||
rsync_target: str | None = Field(default=None, alias="KINOPROJEKT_RSYNC_TARGET")
|
||||
rsync_ssh_key: Path | None = Field(default=None, alias="KINOPROJEKT_RSYNC_SSH_KEY")
|
||||
rsync_remote_root: Path = Field(default=Path("/jellyfin"), alias="KINOPROJEKT_RSYNC_REMOTE_ROOT")
|
||||
import_concurrency: int = Field(default=3, ge=1, le=8, alias="KINOPROJEKT_IMPORT_CONCURRENCY")
|
||||
|
||||
@model_validator(mode="after")
|
||||
def apply_legacy_db_url(self) -> "Settings":
|
||||
# The first target-host deployment used KINOPROJEKT_DB_URL=sqlite:////...
|
||||
# Keep it working unless the newer KINOPROJEKT_DB path is explicitly set.
|
||||
fields_set = getattr(self, "__pydantic_fields_set__", set())
|
||||
if "db_path" not in fields_set:
|
||||
legacy_path = sqlite_url_to_path(self.legacy_db_url)
|
||||
if legacy_path is not None:
|
||||
self.db_path = legacy_path
|
||||
return self
|
||||
|
||||
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
|
||||
|
||||
|
||||
@lru_cache
|
||||
def get_settings() -> Settings:
|
||||
return Settings()
|
||||
22
runtime-current/kino/app-src/backend/app/db.py
Normal file
22
runtime-current/kino/app-src/backend/app/db.py
Normal file
@@ -0,0 +1,22 @@
|
||||
from pathlib import Path
|
||||
|
||||
from sqlmodel import SQLModel, Session, create_engine
|
||||
|
||||
from app.config import get_settings
|
||||
|
||||
|
||||
def get_engine(db_path: Path | None = None):
|
||||
settings = get_settings()
|
||||
path = db_path or settings.db_path
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
return create_engine(f"sqlite:///{path}", connect_args={"check_same_thread": False, "timeout": 30})
|
||||
|
||||
|
||||
def init_db(engine=None) -> None:
|
||||
SQLModel.metadata.create_all(engine or get_engine())
|
||||
|
||||
|
||||
def get_session():
|
||||
engine = get_engine()
|
||||
with Session(engine) as session:
|
||||
yield session
|
||||
431
runtime-current/kino/app-src/backend/app/main.py
Normal file
431
runtime-current/kino/app-src/backend/app/main.py
Normal file
@@ -0,0 +1,431 @@
|
||||
from contextlib import asynccontextmanager
|
||||
import json
|
||||
|
||||
from fastapi import Depends, FastAPI, HTTPException, Request
|
||||
from fastapi.responses import FileResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from sqlmodel import Session, select
|
||||
|
||||
from app.config import get_settings
|
||||
from app.db import get_engine, get_session, init_db
|
||||
from app.models import ImportJob, Source
|
||||
from app.providers.base import ProviderError
|
||||
from app.providers.internet_archive import InternetArchiveProvider
|
||||
from app.providers.mediathek import MediathekProvider
|
||||
from app.providers.youtube import YouTubeProvider
|
||||
from app.schemas import (
|
||||
AnalyzeRequest,
|
||||
AnalyzeResponse,
|
||||
BrowserCaptureRequest,
|
||||
ImportJobResponse,
|
||||
ImportRequest,
|
||||
InteractiveAnalyzeClickRequest,
|
||||
InteractiveAnalyzeKeyRequest,
|
||||
InteractiveAnalyzeResponse,
|
||||
InteractiveAnalyzeStartRequest,
|
||||
MediaCandidateResponse,
|
||||
ProbeRequest,
|
||||
ProbeResponse,
|
||||
SourceResponse,
|
||||
)
|
||||
from app.services.downloader import metadata_json, run_import_job
|
||||
from app.services.import_queue import TERMINAL_IMPORT_STATUSES, import_queue_manager
|
||||
from app.services.interactive_browser import interactive_browser_manager
|
||||
from app.services.page_analyzer import analyze_browser_capture, analyze_page
|
||||
from app.services.url_safety import UnsafeUrlError
|
||||
from app.services.auth import LoginPayload, auth_enabled, clear_session_cookie, login_response, require_user
|
||||
|
||||
providers = [YouTubeProvider(), InternetArchiveProvider(), MediathekProvider()]
|
||||
engine = get_engine()
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
app.state.settings = get_settings()
|
||||
init_db(engine)
|
||||
await import_queue_manager.start(
|
||||
engine=engine,
|
||||
providers=providers,
|
||||
settings=app.state.settings,
|
||||
runner=run_import_job,
|
||||
concurrency=app.state.settings.import_concurrency,
|
||||
)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
await import_queue_manager.stop()
|
||||
|
||||
|
||||
app = FastAPI(title="Kino-Projekt", version="0.2.0", lifespan=lifespan)
|
||||
app.state.settings = get_settings()
|
||||
|
||||
|
||||
def pick_provider(url: str):
|
||||
return next((p for p in providers if p.can_handle(url)), None)
|
||||
|
||||
|
||||
IMPORTABLE_CANDIDATE_KINDS = {"direct_video", "hls_manifest", "dash_manifest"}
|
||||
|
||||
|
||||
def candidate_import_score(candidate) -> tuple[int, int]:
|
||||
kind_priority = {"direct_video": 3, "hls_manifest": 2, "dash_manifest": 2}.get(candidate.kind, 0)
|
||||
return kind_priority, int(candidate.content_length or 0)
|
||||
|
||||
|
||||
async def resolve_importable_url(url: str):
|
||||
"""Return a provider/metadata for an importable media URL.
|
||||
|
||||
Users often paste the player/page URL first. If that page is reachable as
|
||||
text/html, the generic provider cannot import it directly; analyze it and
|
||||
switch to the best discovered media candidate before creating the job.
|
||||
"""
|
||||
provider = pick_provider(url)
|
||||
if provider is None:
|
||||
raise HTTPException(status_code=400, detail="No legal provider accepted this URL")
|
||||
try:
|
||||
metadata = await provider.probe(url)
|
||||
return url, provider, metadata, None
|
||||
except ProviderError as exc:
|
||||
original_error = exc
|
||||
if "text/html" not in str(exc).lower():
|
||||
raise
|
||||
|
||||
candidates = await analyze_page(url)
|
||||
allowed = [
|
||||
candidate
|
||||
for candidate in candidates
|
||||
if candidate.allowed and candidate.kind in IMPORTABLE_CANDIDATE_KINDS
|
||||
]
|
||||
if not allowed:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=(
|
||||
"HTML page is reachable, but no importable media URL was found automatically. "
|
||||
"Use 'Seite analysieren' or the Browser-Capture bookmarklet after playing the video, "
|
||||
"then import one of the discovered media candidates."
|
||||
),
|
||||
) from original_error
|
||||
candidate = max(allowed, key=candidate_import_score)
|
||||
provider = pick_provider(candidate.url)
|
||||
if provider is None:
|
||||
raise HTTPException(status_code=400, detail="No legal provider accepted the discovered media URL")
|
||||
metadata = await provider.probe(candidate.url)
|
||||
return candidate.url, provider, metadata, candidate
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
def health() -> dict[str, str]:
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@app.get("/api/auth/status")
|
||||
def auth_status(request: Request) -> dict[str, bool | str | None]:
|
||||
settings = request.app.state.settings
|
||||
user = None
|
||||
if auth_enabled(settings):
|
||||
try:
|
||||
current = require_user(request)
|
||||
user = current.username if current else None
|
||||
except HTTPException:
|
||||
user = None
|
||||
return {"enabled": auth_enabled(settings), "authenticated": user is not None or not auth_enabled(settings), "username": user}
|
||||
|
||||
|
||||
@app.post("/api/auth/login", status_code=204)
|
||||
def login(payload: LoginPayload, request: Request):
|
||||
return login_response(payload, request, request.app.state.settings)
|
||||
|
||||
|
||||
@app.post("/api/auth/logout", status_code=204)
|
||||
def logout(request: Request):
|
||||
return clear_session_cookie(request)
|
||||
|
||||
|
||||
@app.post("/api/probe", response_model=ProbeResponse)
|
||||
async def probe(payload: ProbeRequest, _: object = Depends(require_user)) -> ProbeResponse:
|
||||
url = str(payload.url)
|
||||
provider = pick_provider(url)
|
||||
if provider is None:
|
||||
raise HTTPException(status_code=400, detail="No legal provider accepted this URL")
|
||||
try:
|
||||
resolved_url, provider, metadata, candidate = await resolve_importable_url(url)
|
||||
except ProviderError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
response = metadata.model_dump()
|
||||
if candidate is not None:
|
||||
response["title"] = candidate.title or metadata.title
|
||||
response["external_id"] = resolved_url
|
||||
response["description"] = "Automatisch aus der HTML-Seite erkannte Medienquelle"
|
||||
return ProbeResponse(**response, allowed=True)
|
||||
|
||||
|
||||
|
||||
|
||||
@app.post("/api/analyze", response_model=AnalyzeResponse)
|
||||
async def analyze(payload: AnalyzeRequest, _: object = Depends(require_user)) -> AnalyzeResponse:
|
||||
try:
|
||||
candidates = await analyze_page(str(payload.url))
|
||||
except UnsafeUrlError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
return AnalyzeResponse(
|
||||
page_url=str(payload.url),
|
||||
candidates=[MediaCandidateResponse(**candidate.__dict__) for candidate in candidates],
|
||||
)
|
||||
|
||||
|
||||
@app.post("/api/browser-capture", response_model=AnalyzeResponse)
|
||||
async def browser_capture(payload: BrowserCaptureRequest, _: object = Depends(require_user)) -> AnalyzeResponse:
|
||||
try:
|
||||
candidates = await analyze_browser_capture(
|
||||
[item.model_dump() for item in payload.candidates],
|
||||
str(payload.page_url) if payload.page_url else None,
|
||||
)
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
return AnalyzeResponse(
|
||||
page_url=str(payload.page_url) if payload.page_url else "browser-capture",
|
||||
candidates=[MediaCandidateResponse(**candidate.__dict__) for candidate in candidates],
|
||||
)
|
||||
|
||||
|
||||
async def interactive_response(session) -> InteractiveAnalyzeResponse:
|
||||
candidates = await interactive_browser_manager.candidates(session)
|
||||
return InteractiveAnalyzeResponse(
|
||||
session_id=session.id,
|
||||
page_url=session.page.url or session.page_url,
|
||||
screenshot=await interactive_browser_manager.screenshot_data_url(session),
|
||||
viewport={"width": 1280, "height": 720},
|
||||
candidates=[MediaCandidateResponse(**candidate.__dict__) for candidate in candidates],
|
||||
ublock_origin=bool(session.ublock_path),
|
||||
)
|
||||
|
||||
|
||||
@app.post("/api/interactive-analyze", response_model=InteractiveAnalyzeResponse)
|
||||
async def start_interactive_analyze(payload: InteractiveAnalyzeStartRequest, _: object = Depends(require_user)) -> InteractiveAnalyzeResponse:
|
||||
try:
|
||||
session = await interactive_browser_manager.start(str(payload.url))
|
||||
return await interactive_response(session)
|
||||
except UnsafeUrlError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@app.post("/api/interactive-analyze/{session_id}/click", response_model=InteractiveAnalyzeResponse)
|
||||
async def click_interactive_analyze(session_id: str, payload: InteractiveAnalyzeClickRequest, _: object = Depends(require_user)) -> InteractiveAnalyzeResponse:
|
||||
try:
|
||||
session = await interactive_browser_manager.click(session_id, payload.x, payload.y)
|
||||
return await interactive_response(session)
|
||||
except KeyError as exc:
|
||||
raise HTTPException(status_code=404, detail="Interactive analysis session not found") from exc
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@app.post("/api/interactive-analyze/{session_id}/key", response_model=InteractiveAnalyzeResponse)
|
||||
async def key_interactive_analyze(session_id: str, payload: InteractiveAnalyzeKeyRequest, _: object = Depends(require_user)) -> InteractiveAnalyzeResponse:
|
||||
try:
|
||||
session = await interactive_browser_manager.press(session_id, payload.key)
|
||||
return await interactive_response(session)
|
||||
except KeyError as exc:
|
||||
raise HTTPException(status_code=404, detail="Interactive analysis session not found") from exc
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@app.delete("/api/interactive-analyze/{session_id}", status_code=204)
|
||||
async def stop_interactive_analyze(session_id: str, _: object = Depends(require_user)) -> None:
|
||||
await interactive_browser_manager.stop(session_id)
|
||||
|
||||
|
||||
@app.post("/api/imports", response_model=ImportJobResponse, status_code=202)
|
||||
async def create_import(
|
||||
request: ImportRequest,
|
||||
_: object = Depends(require_user),
|
||||
session: Session = Depends(get_session),
|
||||
) -> ImportJobResponse:
|
||||
job = await create_import_job(request, session)
|
||||
return import_job_response(job, session)
|
||||
|
||||
|
||||
def import_job_response(job: ImportJob, session: Session) -> ImportJobResponse:
|
||||
source = session.get(Source, job.source_id)
|
||||
return ImportJobResponse(
|
||||
id=job.id or 0,
|
||||
source_id=job.source_id,
|
||||
title=source.title if source is not None else None,
|
||||
status=job.status,
|
||||
target_library=job.target_library,
|
||||
target_path=job.target_path,
|
||||
progress=job.progress,
|
||||
error=job.error,
|
||||
created_at=job.created_at,
|
||||
updated_at=job.updated_at,
|
||||
)
|
||||
|
||||
|
||||
async def create_import_job(request: ImportRequest, session: Session) -> ImportJob:
|
||||
url = str(request.url)
|
||||
provider = pick_provider(url)
|
||||
if provider is None:
|
||||
raise HTTPException(status_code=400, detail="No legal provider accepted this URL")
|
||||
try:
|
||||
resolved_url, provider, metadata, candidate = await resolve_importable_url(url)
|
||||
except ProviderError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
|
||||
source = Source(
|
||||
kind=provider.name,
|
||||
provider=provider.name,
|
||||
url=resolved_url,
|
||||
title=request.title or (candidate.title if candidate is not None else None) or metadata.title,
|
||||
external_id=metadata.external_id,
|
||||
metadata_json=metadata_json(
|
||||
metadata,
|
||||
import_request={
|
||||
"source_page_url": url if resolved_url != url else None,
|
||||
"discovered_media_url": resolved_url if resolved_url != url else None,
|
||||
"discovered_candidate_kind": candidate.kind if candidate is not None else None,
|
||||
"target_profile": request.target_profile,
|
||||
"title": request.title,
|
||||
"year": request.year,
|
||||
"series_title": request.series_title,
|
||||
"season": request.season,
|
||||
"episode": request.episode,
|
||||
"episode_title": request.episode_title,
|
||||
},
|
||||
),
|
||||
)
|
||||
session.add(source)
|
||||
session.commit()
|
||||
session.refresh(source)
|
||||
|
||||
job = ImportJob(source_id=source.id, target_library=request.target_profile, status="queued")
|
||||
session.add(job)
|
||||
session.commit()
|
||||
session.refresh(job)
|
||||
if job.id is not None:
|
||||
await import_queue_manager.enqueue(job.id)
|
||||
return job
|
||||
|
||||
|
||||
@app.get("/api/imports", response_model=list[ImportJobResponse])
|
||||
def list_imports(_: object = Depends(require_user), session: Session = Depends(get_session)) -> list[ImportJobResponse]:
|
||||
jobs = list(session.exec(select(ImportJob).order_by(ImportJob.created_at.desc())).all())
|
||||
return [import_job_response(job, session) for job in jobs]
|
||||
|
||||
|
||||
@app.get("/api/imports/{job_id}", response_model=ImportJobResponse)
|
||||
def get_import(job_id: int, _: object = Depends(require_user), session: Session = Depends(get_session)) -> ImportJobResponse:
|
||||
job = session.get(ImportJob, job_id)
|
||||
if job is None:
|
||||
raise HTTPException(status_code=404, detail="Import job not found")
|
||||
return import_job_response(job, session)
|
||||
|
||||
|
||||
@app.delete("/api/imports/{job_id}", status_code=204)
|
||||
def delete_import(job_id: int, _: object = Depends(require_user), session: Session = Depends(get_session)) -> None:
|
||||
job = session.get(ImportJob, job_id)
|
||||
if job is None:
|
||||
raise HTTPException(status_code=404, detail="Import job not found")
|
||||
if job.status not in TERMINAL_IMPORT_STATUSES:
|
||||
raise HTTPException(status_code=409, detail="Only completed, failed or cancelled imports can be deleted from the overview")
|
||||
# This deletes only the import record from the overview. Jellyfin media is not removed here.
|
||||
session.delete(job)
|
||||
session.commit()
|
||||
|
||||
|
||||
@app.post("/api/imports/{job_id}/retry", response_model=ImportJobResponse, status_code=202)
|
||||
async def retry_import(job_id: int, _: object = Depends(require_user), session: Session = Depends(get_session)) -> ImportJobResponse:
|
||||
old_job = session.get(ImportJob, job_id)
|
||||
if old_job is None:
|
||||
raise HTTPException(status_code=404, detail="Import job not found")
|
||||
if old_job.status not in TERMINAL_IMPORT_STATUSES:
|
||||
raise HTTPException(status_code=409, detail="Only completed, failed or cancelled imports can be retried")
|
||||
new_job = ImportJob(source_id=old_job.source_id, target_library=old_job.target_library, status="queued")
|
||||
session.add(new_job)
|
||||
session.commit()
|
||||
session.refresh(new_job)
|
||||
if new_job.id is not None:
|
||||
await import_queue_manager.enqueue(new_job.id)
|
||||
return import_job_response(new_job, session)
|
||||
|
||||
|
||||
@app.post("/api/imports/{job_id}/retry-alternative", response_model=ImportJobResponse, status_code=202)
|
||||
async def retry_import_alternative(job_id: int, _: object = Depends(require_user), session: Session = Depends(get_session)) -> ImportJobResponse:
|
||||
old_job = session.get(ImportJob, job_id)
|
||||
if old_job is None:
|
||||
raise HTTPException(status_code=404, detail="Import job not found")
|
||||
if old_job.status not in TERMINAL_IMPORT_STATUSES:
|
||||
raise HTTPException(status_code=409, detail="Only completed, failed or cancelled imports can use alternative retry")
|
||||
source = session.get(Source, old_job.source_id)
|
||||
if source is None:
|
||||
raise HTTPException(status_code=404, detail="Import source missing")
|
||||
try:
|
||||
stored = json.loads(source.metadata_json or "{}")
|
||||
except Exception:
|
||||
stored = {}
|
||||
import_request = stored.get("import_request", {}) if isinstance(stored, dict) else {}
|
||||
retry_url = import_request.get("source_page_url") or source.url
|
||||
try:
|
||||
resolved_url, provider, metadata, candidate = await resolve_importable_url(retry_url)
|
||||
except ProviderError as exc:
|
||||
raise HTTPException(status_code=400, detail=f"Alternative Analyse fehlgeschlagen: {exc}") from exc
|
||||
new_source = Source(
|
||||
kind=provider.name,
|
||||
provider=provider.name,
|
||||
url=resolved_url,
|
||||
title=source.title or (candidate.title if candidate is not None else None) or metadata.title,
|
||||
external_id=metadata.external_id,
|
||||
metadata_json=metadata_json(
|
||||
metadata,
|
||||
import_request={
|
||||
**import_request,
|
||||
"source_page_url": retry_url if resolved_url != retry_url else import_request.get("source_page_url"),
|
||||
"discovered_media_url": resolved_url if resolved_url != retry_url else None,
|
||||
"discovered_candidate_kind": candidate.kind if candidate is not None else None,
|
||||
"alternative_retry_of": job_id,
|
||||
},
|
||||
),
|
||||
)
|
||||
session.add(new_source)
|
||||
session.commit()
|
||||
session.refresh(new_source)
|
||||
new_job = ImportJob(source_id=new_source.id, target_library=old_job.target_library, status="queued")
|
||||
session.add(new_job)
|
||||
session.commit()
|
||||
session.refresh(new_job)
|
||||
if new_job.id is not None:
|
||||
await import_queue_manager.enqueue(new_job.id)
|
||||
return import_job_response(new_job, session)
|
||||
|
||||
|
||||
@app.post("/api/imports/{job_id}/cancel", response_model=ImportJobResponse)
|
||||
def cancel_import(job_id: int, _: object = Depends(require_user), session: Session = Depends(get_session)) -> ImportJobResponse:
|
||||
job = session.get(ImportJob, job_id)
|
||||
if job is None:
|
||||
raise HTTPException(status_code=404, detail="Import job not found")
|
||||
if job.status in TERMINAL_IMPORT_STATUSES:
|
||||
raise HTTPException(status_code=409, detail="Import is already finished")
|
||||
job.status = "cancelled" if job.status == "queued" else "cancelling"
|
||||
job.error = "Abbruch angefordert"
|
||||
job.updated_at = utcnow()
|
||||
session.add(job)
|
||||
session.commit()
|
||||
session.refresh(job)
|
||||
return import_job_response(job, session)
|
||||
|
||||
|
||||
@app.get("/api/sources", response_model=list[SourceResponse])
|
||||
def list_sources(_: object = Depends(require_user), session: Session = Depends(get_session)) -> list[Source]:
|
||||
return list(session.exec(select(Source).order_by(Source.created_at.desc())).all())
|
||||
|
||||
|
||||
app.mount("/static", StaticFiles(directory="app/web"), name="static")
|
||||
|
||||
|
||||
@app.get("/", include_in_schema=False)
|
||||
def index() -> FileResponse:
|
||||
return FileResponse("app/web/index.html", headers={"Cache-Control": "no-store"})
|
||||
30
runtime-current/kino/app-src/backend/app/models.py
Normal file
30
runtime-current/kino/app-src/backend/app/models.py
Normal file
@@ -0,0 +1,30 @@
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlmodel import Field, SQLModel
|
||||
|
||||
|
||||
def utcnow() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
class Source(SQLModel, table=True):
|
||||
id: int | None = Field(default=None, primary_key=True)
|
||||
kind: str
|
||||
url: str
|
||||
title: str | None = None
|
||||
provider: str | None = None
|
||||
external_id: str | None = None
|
||||
metadata_json: str = "{}"
|
||||
created_at: datetime = Field(default_factory=utcnow)
|
||||
|
||||
|
||||
class ImportJob(SQLModel, table=True):
|
||||
id: int | None = Field(default=None, primary_key=True)
|
||||
source_id: int = Field(foreign_key="source.id")
|
||||
status: str = "queued"
|
||||
target_library: str
|
||||
target_path: str | None = None
|
||||
progress: float = 0.0
|
||||
error: str | None = None
|
||||
created_at: datetime = Field(default_factory=utcnow)
|
||||
updated_at: datetime = Field(default_factory=utcnow)
|
||||
46
runtime-current/kino/app-src/backend/app/providers/base.py
Normal file
46
runtime-current/kino/app-src/backend/app/providers/base.py
Normal file
@@ -0,0 +1,46 @@
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from typing import Protocol
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class ProviderError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class MediaMetadata(BaseModel):
|
||||
provider: str
|
||||
title: str | None = None
|
||||
description: str | None = None
|
||||
thumbnail: str | None = None
|
||||
duration_seconds: int | None = None
|
||||
external_id: str | None = None
|
||||
|
||||
|
||||
class DownloadOptions(BaseModel):
|
||||
max_height: int = 1080
|
||||
target_library: str = "movies"
|
||||
audio_only: bool = False
|
||||
max_bytes: int = 15_000_000_000
|
||||
|
||||
|
||||
class DownloadResult(BaseModel):
|
||||
output_files: list[Path]
|
||||
metadata_files: list[Path] = []
|
||||
|
||||
|
||||
class Provider(Protocol):
|
||||
name: str
|
||||
|
||||
def can_handle(self, url: str) -> bool: ...
|
||||
|
||||
async def probe(self, url: str) -> MediaMetadata: ...
|
||||
|
||||
async def download(
|
||||
self,
|
||||
url: str,
|
||||
target_dir: Path,
|
||||
options: DownloadOptions,
|
||||
progress_callback: Callable[[float], None] | None = None,
|
||||
) -> DownloadResult: ...
|
||||
@@ -0,0 +1,123 @@
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from urllib.parse import quote, urlparse
|
||||
|
||||
import httpx
|
||||
|
||||
from app.providers.base import DownloadOptions, DownloadResult, MediaMetadata, ProviderError
|
||||
from app.providers.mediathek import MEDIA_EXTENSIONS, MediathekProvider
|
||||
from app.services.url_safety import BROWSER_LIKE_HEADERS
|
||||
|
||||
ARCHIVE_HOSTS = {"archive.org", "www.archive.org"}
|
||||
METADATA_BASE = "https://archive.org/metadata"
|
||||
DOWNLOAD_BASE = "https://archive.org/download"
|
||||
|
||||
|
||||
class InternetArchiveProvider:
|
||||
name = "internet_archive"
|
||||
|
||||
def can_handle(self, url: str) -> bool:
|
||||
parsed = urlparse(url)
|
||||
return parsed.scheme in {"http", "https"} and (parsed.hostname or "").lower() in ARCHIVE_HOSTS and self._identifier_from_url(url) is not None
|
||||
|
||||
async def probe(self, url: str) -> MediaMetadata:
|
||||
identifier = self._identifier_from_url(url)
|
||||
if not identifier:
|
||||
raise ProviderError("Internet Archive URL must contain an item identifier")
|
||||
metadata = await self._metadata(identifier)
|
||||
meta = metadata.get("metadata") or {}
|
||||
files = metadata.get("files") or []
|
||||
selected = self._select_media_file(files)
|
||||
if selected is None:
|
||||
raise ProviderError("Internet Archive item has no accepted public media file")
|
||||
title = self._string(meta.get("title")) or identifier
|
||||
description = self._string(meta.get("description"))
|
||||
return MediaMetadata(
|
||||
provider=self.name,
|
||||
title=title,
|
||||
description=description,
|
||||
external_id=identifier,
|
||||
)
|
||||
|
||||
async def download(
|
||||
self,
|
||||
url: str,
|
||||
target_dir: Path,
|
||||
options: DownloadOptions,
|
||||
progress_callback: Callable[[float], None] | None = None,
|
||||
) -> DownloadResult:
|
||||
identifier = self._identifier_from_url(url)
|
||||
if not identifier:
|
||||
raise ProviderError("Internet Archive URL must contain an item identifier")
|
||||
metadata = await self._metadata(identifier)
|
||||
selected = self._select_media_file(metadata.get("files") or [])
|
||||
if selected is None:
|
||||
raise ProviderError("Internet Archive item has no accepted public media file")
|
||||
file_name = selected["name"]
|
||||
safe_url = f"{DOWNLOAD_BASE}/{quote(identifier, safe='')}/{quote(file_name, safe='/')}"
|
||||
return await MediathekProvider().download(safe_url, target_dir, options, progress_callback=progress_callback)
|
||||
|
||||
async def _metadata(self, identifier: str) -> dict:
|
||||
url = f"{METADATA_BASE}/{quote(identifier, safe='')}"
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(15, read=20), follow_redirects=True) as client:
|
||||
response = await client.get(url, headers=BROWSER_LIKE_HEADERS)
|
||||
if response.status_code == 404:
|
||||
raise ProviderError(f"Internet Archive item not found: {identifier}")
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
if not isinstance(data, dict):
|
||||
raise ProviderError("Internet Archive metadata response was not an object")
|
||||
return data
|
||||
|
||||
def _identifier_from_url(self, url: str) -> str | None:
|
||||
parsed = urlparse(url)
|
||||
parts = [part for part in parsed.path.split("/") if part]
|
||||
if not parts:
|
||||
return None
|
||||
if parts[0] in {"details", "download", "metadata"} and len(parts) >= 2:
|
||||
return parts[1]
|
||||
return parts[0]
|
||||
|
||||
def _select_media_file(self, files: list[dict]) -> dict | None:
|
||||
candidates: list[tuple[int, dict]] = []
|
||||
for item in files:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
name = item.get("name")
|
||||
if not isinstance(name, str) or not name or name.endswith("/") or ".." in Path(name).parts:
|
||||
continue
|
||||
lower = name.lower()
|
||||
ext = Path(lower).suffix
|
||||
fmt = str(item.get("format") or "").lower()
|
||||
source = str(item.get("source") or "").lower()
|
||||
if ext not in MEDIA_EXTENSIONS and "mpeg4" not in fmt and "h.264" not in fmt:
|
||||
continue
|
||||
if source in {"metadata", "derivative"} and ext not in {".mp4", ".m4v", ".mp3", ".m4a", ".ogg", ".webm"}:
|
||||
continue
|
||||
size = self._int_or_zero(item.get("size"))
|
||||
# Prefer original/public mp4-ish media, then larger files as likely higher quality.
|
||||
score = 0
|
||||
if source == "original":
|
||||
score += 1000
|
||||
if ext in {".mp4", ".m4v", ".webm"}:
|
||||
score += 500
|
||||
if size:
|
||||
score += min(size // 1_000_000, 400)
|
||||
candidates.append((score, item))
|
||||
if not candidates:
|
||||
return None
|
||||
candidates.sort(key=lambda pair: pair[0], reverse=True)
|
||||
return candidates[0][1]
|
||||
|
||||
def _int_or_zero(self, value: object) -> int:
|
||||
try:
|
||||
return int(value) # type: ignore[arg-type]
|
||||
except (TypeError, ValueError):
|
||||
return 0
|
||||
|
||||
def _string(self, value: object) -> str | None:
|
||||
if isinstance(value, str):
|
||||
return value.strip() or None
|
||||
if isinstance(value, list) and value and isinstance(value[0], str):
|
||||
return value[0].strip() or None
|
||||
return None
|
||||
204
runtime-current/kino/app-src/backend/app/providers/mediathek.py
Normal file
204
runtime-current/kino/app-src/backend/app/providers/mediathek.py
Normal file
@@ -0,0 +1,204 @@
|
||||
import asyncio
|
||||
import ipaddress
|
||||
import shutil
|
||||
import socket
|
||||
import sys
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path, PurePosixPath
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
|
||||
from app.providers.base import DownloadOptions, DownloadResult, MediaMetadata, ProviderError
|
||||
from app.providers.youtube import parse_yt_dlp_progress
|
||||
from app.services.url_safety import BROWSER_LIKE_MEDIA_HEADERS, CHROME_WINDOWS_11_USER_AGENT
|
||||
|
||||
MEDIA_CONTENT_TYPES = (
|
||||
"video/",
|
||||
"audio/",
|
||||
"application/vnd.apple.mpegurl",
|
||||
"application/x-mpegurl",
|
||||
"application/dash+xml",
|
||||
)
|
||||
MEDIA_EXTENSIONS = {
|
||||
".mp4",
|
||||
".mkv",
|
||||
".webm",
|
||||
".mov",
|
||||
".m4v",
|
||||
".mp3",
|
||||
".m4a",
|
||||
".aac",
|
||||
".ogg",
|
||||
".flac",
|
||||
".m3u8",
|
||||
".mpd",
|
||||
}
|
||||
MANIFEST_CONTENT_TYPES = {"application/vnd.apple.mpegurl", "application/x-mpegurl", "application/dash+xml"}
|
||||
MANIFEST_EXTENSIONS = {".m3u8", ".mpd"}
|
||||
|
||||
|
||||
def is_stream_manifest(url: str, content_type: str | None = None) -> bool:
|
||||
clean_content_type = (content_type or "").split(";", 1)[0].lower()
|
||||
ext = PurePosixPath(urlparse(url).path).suffix.lower()
|
||||
return clean_content_type in MANIFEST_CONTENT_TYPES or ext in MANIFEST_EXTENSIONS
|
||||
|
||||
|
||||
def ytdlp_command() -> list[str]:
|
||||
binary = shutil.which("yt-dlp")
|
||||
if binary:
|
||||
return [binary]
|
||||
return [sys.executable, "-m", "yt_dlp"]
|
||||
|
||||
|
||||
class MediathekProvider:
|
||||
name = "mediathek"
|
||||
|
||||
def can_handle(self, url: str) -> bool:
|
||||
parsed = urlparse(url)
|
||||
host = parsed.hostname or ""
|
||||
excluded_hosts = {"youtube.com", "www.youtube.com", "m.youtube.com", "youtu.be", "archive.org", "www.archive.org"}
|
||||
return parsed.scheme in {"http", "https"} and host not in excluded_hosts
|
||||
|
||||
async def probe(self, url: str) -> MediaMetadata:
|
||||
parsed = urlparse(url)
|
||||
if parsed.scheme not in {"http", "https"} or not parsed.hostname:
|
||||
raise ProviderError("Only http/https URLs are allowed")
|
||||
self._reject_private_targets(parsed.hostname)
|
||||
async with httpx.AsyncClient(follow_redirects=True, timeout=httpx.Timeout(10, read=10), max_redirects=5) as client:
|
||||
response = await client.head(url, headers=BROWSER_LIKE_MEDIA_HEADERS)
|
||||
if response.status_code in {405, 403}:
|
||||
response = await client.get(url, headers={**BROWSER_LIKE_MEDIA_HEADERS, "Range": "bytes=0-0"})
|
||||
for hop in [*response.history, response]:
|
||||
host = hop.url.host
|
||||
if host:
|
||||
self._reject_private_targets(host)
|
||||
content_type = response.headers.get("content-type", "").split(";", 1)[0].lower()
|
||||
ext = PurePosixPath(urlparse(str(response.url)).path).suffix.lower()
|
||||
if not (content_type.startswith(MEDIA_CONTENT_TYPES) or ext in MEDIA_EXTENSIONS):
|
||||
raise ProviderError(f"URL is reachable, but content type/extension is not accepted: {content_type or ext or 'unknown'}")
|
||||
title = PurePosixPath(urlparse(str(response.url)).path).name or str(response.url.host)
|
||||
return MediaMetadata(provider=self.name, title=title, external_id=str(response.url))
|
||||
|
||||
async def download(
|
||||
self,
|
||||
url: str,
|
||||
target_dir: Path,
|
||||
options: DownloadOptions,
|
||||
progress_callback: Callable[[float], None] | None = None,
|
||||
) -> DownloadResult:
|
||||
metadata = await self.probe(url)
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
source_url = metadata.external_id or url
|
||||
if is_stream_manifest(source_url):
|
||||
return await self._download_with_ytdlp(source_url, target_dir, options, progress_callback)
|
||||
|
||||
name = PurePosixPath(urlparse(source_url).path).name or "download.bin"
|
||||
output_path = target_dir / name
|
||||
written = 0
|
||||
async with httpx.AsyncClient(follow_redirects=True, timeout=httpx.Timeout(20, read=60), max_redirects=5) as client:
|
||||
async with client.stream("GET", source_url, headers=BROWSER_LIKE_MEDIA_HEADERS) as response:
|
||||
response.raise_for_status()
|
||||
for hop in [*response.history, response]:
|
||||
host = hop.url.host
|
||||
if host:
|
||||
self._reject_private_targets(host)
|
||||
total = int(response.headers.get("content-length") or 0)
|
||||
with output_path.open("wb") as fh:
|
||||
async for chunk in response.aiter_bytes():
|
||||
written += len(chunk)
|
||||
if written > options.max_bytes:
|
||||
raise ProviderError("Download exceeds configured maximum size")
|
||||
fh.write(chunk)
|
||||
if total > 0 and progress_callback is not None:
|
||||
progress_callback(min(written / total, 1.0))
|
||||
if progress_callback is not None:
|
||||
progress_callback(1.0)
|
||||
return DownloadResult(output_files=[output_path])
|
||||
|
||||
async def _download_with_ytdlp(
|
||||
self,
|
||||
url: str,
|
||||
target_dir: Path,
|
||||
options: DownloadOptions,
|
||||
progress_callback: Callable[[float], None] | None = None,
|
||||
) -> DownloadResult:
|
||||
"""Download HLS/DASH manifests as real media, not the tiny manifest file.
|
||||
|
||||
A .m3u8/.mpd is only a playlist/manifest. Saving it directly produces a
|
||||
few-KB text file in Jellyfin instead of the actual media stream. Let
|
||||
yt-dlp drive ffmpeg so segments are downloaded and remuxed into MP4.
|
||||
"""
|
||||
command = ytdlp_command()
|
||||
output_template = str(target_dir / "%(title,playlist_index,id).120s.%(ext)s")
|
||||
height_expr = f"bestvideo[height<={options.max_height}]+bestaudio/best[height<={options.max_height}]/best"
|
||||
if options.audio_only:
|
||||
height_expr = "bestaudio/best"
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*command,
|
||||
"--newline",
|
||||
"--no-playlist",
|
||||
"--no-warnings",
|
||||
"--restrict-filenames",
|
||||
"--format",
|
||||
height_expr,
|
||||
"--remux-video",
|
||||
"mp4",
|
||||
"--max-filesize",
|
||||
str(options.max_bytes),
|
||||
"--user-agent",
|
||||
CHROME_WINDOWS_11_USER_AGENT,
|
||||
"--add-header",
|
||||
f"Accept-Language: {BROWSER_LIKE_MEDIA_HEADERS['Accept-Language']}",
|
||||
"--output",
|
||||
output_template,
|
||||
url,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
captured: list[str] = []
|
||||
|
||||
async def consume(stream: asyncio.StreamReader | None) -> None:
|
||||
if stream is None:
|
||||
return
|
||||
while True:
|
||||
line = await stream.readline()
|
||||
if not line:
|
||||
break
|
||||
text = line.decode(errors="replace").strip()
|
||||
if not text:
|
||||
continue
|
||||
captured.append(text)
|
||||
if len(captured) > 80:
|
||||
del captured[:40]
|
||||
progress = parse_yt_dlp_progress(text)
|
||||
if progress is not None and progress_callback is not None:
|
||||
progress_callback(progress)
|
||||
|
||||
try:
|
||||
await asyncio.wait_for(asyncio.gather(consume(proc.stdout), consume(proc.stderr), proc.wait()), timeout=60 * 60)
|
||||
except BaseException:
|
||||
if proc.returncode is None:
|
||||
proc.kill()
|
||||
await proc.wait()
|
||||
raise
|
||||
if proc.returncode != 0:
|
||||
detail = "\n".join(captured).strip() or "yt-dlp manifest download failed"
|
||||
raise ProviderError(detail[:1000])
|
||||
files = [p for p in target_dir.iterdir() if p.is_file()]
|
||||
media_files = [p for p in files if p.suffix.lower() not in {".json", ".jpg", ".jpeg", ".png", ".webp", ".m3u8", ".mpd"}]
|
||||
if not media_files:
|
||||
raise ProviderError("yt-dlp finished without producing a media file")
|
||||
if progress_callback is not None:
|
||||
progress_callback(1.0)
|
||||
return DownloadResult(output_files=media_files)
|
||||
|
||||
def _reject_private_targets(self, hostname: str) -> None:
|
||||
try:
|
||||
infos = socket.getaddrinfo(hostname, None, proto=socket.IPPROTO_TCP)
|
||||
except socket.gaierror as exc:
|
||||
raise ProviderError(f"DNS lookup failed for {hostname}") from exc
|
||||
for info in infos:
|
||||
ip = ipaddress.ip_address(info[4][0])
|
||||
if ip.is_private or ip.is_loopback or ip.is_link_local or ip.is_multicast or ip.is_reserved or ip.is_unspecified:
|
||||
raise ProviderError(f"Refusing private or non-public target address for {hostname}")
|
||||
151
runtime-current/kino/app-src/backend/app/providers/youtube.py
Normal file
151
runtime-current/kino/app-src/backend/app/providers/youtube.py
Normal file
@@ -0,0 +1,151 @@
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
|
||||
from app.providers.base import DownloadOptions, DownloadResult, MediaMetadata, ProviderError
|
||||
from app.services.url_safety import CHROME_WINDOWS_11_USER_AGENT, BROWSER_LIKE_HEADERS
|
||||
|
||||
|
||||
class YouTubeProvider:
|
||||
name = "youtube"
|
||||
allowed_hosts = {"youtube.com", "www.youtube.com", "m.youtube.com", "youtu.be"}
|
||||
|
||||
def can_handle(self, url: str) -> bool:
|
||||
host = urlparse(url).hostname or ""
|
||||
return host.lower() in self.allowed_hosts
|
||||
|
||||
async def probe(self, url: str) -> MediaMetadata:
|
||||
if not self.can_handle(url):
|
||||
raise ProviderError("Not a YouTube URL")
|
||||
if shutil.which("yt-dlp") is None:
|
||||
return MediaMetadata(provider=self.name, external_id=self._external_id(url), title="YouTube URL (yt-dlp not installed)")
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"yt-dlp",
|
||||
"--dump-single-json",
|
||||
"--no-playlist",
|
||||
"--no-warnings",
|
||||
"--user-agent",
|
||||
CHROME_WINDOWS_11_USER_AGENT,
|
||||
"--add-header",
|
||||
f"Accept-Language: {BROWSER_LIKE_HEADERS['Accept-Language']}",
|
||||
url,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=45)
|
||||
if proc.returncode != 0:
|
||||
raise ProviderError((stderr.decode(errors="replace") or "yt-dlp probe failed").strip()[:500])
|
||||
data = json.loads(stdout.decode())
|
||||
return MediaMetadata(
|
||||
provider=self.name,
|
||||
title=data.get("title"),
|
||||
description=data.get("description"),
|
||||
thumbnail=data.get("thumbnail"),
|
||||
duration_seconds=data.get("duration"),
|
||||
external_id=data.get("id") or self._external_id(url),
|
||||
)
|
||||
|
||||
async def download(
|
||||
self,
|
||||
url: str,
|
||||
target_dir: Path,
|
||||
options: DownloadOptions,
|
||||
progress_callback: Callable[[float], None] | None = None,
|
||||
) -> DownloadResult:
|
||||
if shutil.which("yt-dlp") is None:
|
||||
raise ProviderError("yt-dlp is not installed")
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
height_expr = f"bestvideo[height<={options.max_height}]+bestaudio/best[height<={options.max_height}]/best"
|
||||
if options.audio_only:
|
||||
height_expr = "bestaudio/best"
|
||||
output_template = str(target_dir / "%(title).120s [%(id)s].%(ext)s")
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"yt-dlp",
|
||||
"--newline",
|
||||
"--no-playlist",
|
||||
"--no-warnings",
|
||||
"--restrict-filenames",
|
||||
"--write-info-json",
|
||||
"--write-thumbnail",
|
||||
"--user-agent",
|
||||
CHROME_WINDOWS_11_USER_AGENT,
|
||||
"--add-header",
|
||||
f"Accept-Language: {BROWSER_LIKE_HEADERS['Accept-Language']}",
|
||||
"--format",
|
||||
height_expr,
|
||||
"--output",
|
||||
output_template,
|
||||
url,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
|
||||
captured: list[str] = []
|
||||
|
||||
async def consume(stream: asyncio.StreamReader | None) -> None:
|
||||
if stream is None:
|
||||
return
|
||||
while True:
|
||||
line = await stream.readline()
|
||||
if not line:
|
||||
break
|
||||
text = line.decode(errors="replace").strip()
|
||||
if not text:
|
||||
continue
|
||||
captured.append(text)
|
||||
if len(captured) > 80:
|
||||
del captured[:40]
|
||||
progress = parse_yt_dlp_progress(text)
|
||||
if progress is not None and progress_callback is not None:
|
||||
progress_callback(progress)
|
||||
|
||||
try:
|
||||
await asyncio.wait_for(asyncio.gather(consume(proc.stdout), consume(proc.stderr), proc.wait()), timeout=60 * 60)
|
||||
except BaseException:
|
||||
if proc.returncode is None:
|
||||
proc.kill()
|
||||
await proc.wait()
|
||||
raise
|
||||
if proc.returncode != 0:
|
||||
detail = "\n".join(captured).strip() or "yt-dlp download failed"
|
||||
raise ProviderError(detail[:1000])
|
||||
files = [p for p in target_dir.iterdir() if p.is_file()]
|
||||
media_files = [p for p in files if p.suffix.lower() not in {".json", ".jpg", ".jpeg", ".png", ".webp"}]
|
||||
metadata_files = [p for p in files if p not in media_files]
|
||||
if not media_files:
|
||||
raise ProviderError("yt-dlp finished without producing a media file")
|
||||
if progress_callback is not None:
|
||||
progress_callback(1.0)
|
||||
return DownloadResult(output_files=media_files, metadata_files=metadata_files)
|
||||
|
||||
def _external_id(self, url: str) -> str | None:
|
||||
parsed = urlparse(url)
|
||||
if parsed.hostname == "youtu.be":
|
||||
return parsed.path.strip("/") or None
|
||||
return parse_qs(parsed.query).get("v", [None])[0]
|
||||
|
||||
|
||||
def parse_yt_dlp_progress(line: str) -> float | None:
|
||||
"""Return yt-dlp percent progress as 0.0..1.0, or None for non-progress lines."""
|
||||
if "[download]" not in line:
|
||||
return None
|
||||
match = re.search(r"(\d+(?:\.\d+)?)%", line)
|
||||
if not match:
|
||||
return None
|
||||
return max(0.0, min(float(match.group(1)) / 100.0, 1.0))
|
||||
|
||||
|
||||
async def _main(url: str) -> None:
|
||||
meta = await YouTubeProvider().probe(url)
|
||||
print(meta.model_dump_json(indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 2:
|
||||
raise SystemExit("usage: python -m app.providers.youtube <url>")
|
||||
asyncio.run(_main(sys.argv[1]))
|
||||
116
runtime-current/kino/app-src/backend/app/schemas.py
Normal file
116
runtime-current/kino/app-src/backend/app/schemas.py
Normal file
@@ -0,0 +1,116 @@
|
||||
from datetime import datetime
|
||||
|
||||
from typing import Any
|
||||
|
||||
from pydantic import AnyHttpUrl, BaseModel, Field
|
||||
|
||||
|
||||
class ProbeRequest(BaseModel):
|
||||
url: AnyHttpUrl
|
||||
|
||||
|
||||
|
||||
|
||||
class AnalyzeRequest(BaseModel):
|
||||
url: AnyHttpUrl
|
||||
|
||||
|
||||
class BrowserCapturedCandidate(BaseModel):
|
||||
url: str
|
||||
title: str | None = None
|
||||
name: str | None = None
|
||||
content_type: str | None = None
|
||||
type: str | None = None
|
||||
quality: str | None = None
|
||||
|
||||
|
||||
class BrowserCaptureRequest(BaseModel):
|
||||
page_url: AnyHttpUrl | None = None
|
||||
candidates: list[BrowserCapturedCandidate] = Field(default_factory=list, max_length=80)
|
||||
|
||||
|
||||
class MediaCandidateResponse(BaseModel):
|
||||
url: str
|
||||
kind: str
|
||||
title: str | None = None
|
||||
source: str
|
||||
content_type: str | None = None
|
||||
content_length: int | None = None
|
||||
file_size: str | None = None
|
||||
quality: str | None = None
|
||||
allowed: bool = True
|
||||
reason: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class AnalyzeResponse(BaseModel):
|
||||
page_url: str
|
||||
candidates: list[MediaCandidateResponse]
|
||||
|
||||
|
||||
class InteractiveAnalyzeStartRequest(BaseModel):
|
||||
url: AnyHttpUrl
|
||||
|
||||
|
||||
class InteractiveAnalyzeClickRequest(BaseModel):
|
||||
x: float = Field(ge=0)
|
||||
y: float = Field(ge=0)
|
||||
|
||||
|
||||
class InteractiveAnalyzeKeyRequest(BaseModel):
|
||||
key: str = Field(min_length=1, max_length=40)
|
||||
|
||||
|
||||
class InteractiveAnalyzeResponse(BaseModel):
|
||||
session_id: str
|
||||
page_url: str
|
||||
screenshot: str
|
||||
viewport: dict[str, int]
|
||||
candidates: list[MediaCandidateResponse]
|
||||
ublock_origin: bool = False
|
||||
|
||||
class ProbeResponse(BaseModel):
|
||||
provider: str
|
||||
title: str | None = None
|
||||
description: str | None = None
|
||||
thumbnail: str | None = None
|
||||
duration_seconds: int | None = None
|
||||
external_id: str | None = None
|
||||
allowed: bool = True
|
||||
|
||||
|
||||
class ImportRequest(BaseModel):
|
||||
url: AnyHttpUrl
|
||||
target_profile: str = Field(default="clip", pattern="^(film|serie|clip)$")
|
||||
title: str | None = Field(default=None, max_length=180)
|
||||
year: int | None = Field(default=None, ge=1888, le=2200)
|
||||
series_title: str | None = Field(default=None, max_length=180)
|
||||
season: int | None = Field(default=None, ge=1, le=200)
|
||||
episode: int | None = Field(default=None, ge=1, le=1000)
|
||||
episode_title: str | None = Field(default=None, max_length=180)
|
||||
target_library: str | None = Field(default=None, max_length=80) # legacy API alias
|
||||
max_height: int = Field(default=1080, ge=144, le=2160)
|
||||
audio_only: bool = False
|
||||
|
||||
|
||||
class ImportJobResponse(BaseModel):
|
||||
id: int
|
||||
source_id: int
|
||||
title: str | None = None
|
||||
status: str
|
||||
target_library: str
|
||||
target_path: str | None = None
|
||||
progress: float
|
||||
error: str | None = None
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
|
||||
class SourceResponse(BaseModel):
|
||||
id: int
|
||||
kind: str
|
||||
url: str
|
||||
title: str | None = None
|
||||
provider: str | None = None
|
||||
external_id: str | None = None
|
||||
created_at: datetime
|
||||
122
runtime-current/kino/app-src/backend/app/services/auth.py
Normal file
122
runtime-current/kino/app-src/backend/app/services/auth.py
Normal file
@@ -0,0 +1,122 @@
|
||||
import base64
|
||||
import hashlib
|
||||
import hmac
|
||||
import secrets
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
|
||||
from argon2 import PasswordHasher
|
||||
from argon2.exceptions import InvalidHashError, VerifyMismatchError
|
||||
from fastapi import HTTPException, Request, Response, status
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.config import Settings
|
||||
|
||||
SESSION_COOKIE = "kino_session"
|
||||
SESSION_TTL_SECONDS = 60 * 60 * 12
|
||||
_ph = PasswordHasher()
|
||||
|
||||
|
||||
class LoginPayload(BaseModel):
|
||||
username: str
|
||||
password: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AuthenticatedUser:
|
||||
username: str
|
||||
|
||||
|
||||
def hash_password(password: str) -> str:
|
||||
return _ph.hash(password)
|
||||
|
||||
|
||||
def verify_password(password: str, password_hash: str) -> bool:
|
||||
try:
|
||||
return _ph.verify(password_hash, password)
|
||||
except (VerifyMismatchError, InvalidHashError):
|
||||
return False
|
||||
|
||||
|
||||
def auth_enabled(settings: Settings) -> bool:
|
||||
return bool(settings.admin_password_hash and settings.session_secret)
|
||||
|
||||
|
||||
def _sign(value: str, secret: str) -> str:
|
||||
return hmac.new(secret.encode("utf-8"), value.encode("utf-8"), hashlib.sha256).hexdigest()
|
||||
|
||||
|
||||
def create_session_cookie(username: str, settings: Settings, now: int | None = None) -> str:
|
||||
if not settings.session_secret:
|
||||
raise RuntimeError("session secret is not configured")
|
||||
issued = int(time.time() if now is None else now)
|
||||
expires = issued + SESSION_TTL_SECONDS
|
||||
payload = f"{username}:{expires}"
|
||||
encoded = base64.urlsafe_b64encode(payload.encode("utf-8")).decode("ascii").rstrip("=")
|
||||
return f"{encoded}.{_sign(encoded, settings.session_secret)}"
|
||||
|
||||
|
||||
def parse_session_cookie(cookie_value: str | None, settings: Settings, now: int | None = None) -> AuthenticatedUser | None:
|
||||
if not cookie_value or not settings.session_secret or "." not in cookie_value:
|
||||
return None
|
||||
encoded, signature = cookie_value.rsplit(".", 1)
|
||||
if not hmac.compare_digest(signature, _sign(encoded, settings.session_secret)):
|
||||
return None
|
||||
try:
|
||||
padding = "=" * (-len(encoded) % 4)
|
||||
payload = base64.urlsafe_b64decode((encoded + padding).encode("ascii")).decode("utf-8")
|
||||
username, expires_raw = payload.rsplit(":", 1)
|
||||
expires = int(expires_raw)
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
return None
|
||||
current = int(time.time() if now is None else now)
|
||||
if expires < current:
|
||||
return None
|
||||
if not secrets.compare_digest(username, settings.admin_username):
|
||||
return None
|
||||
return AuthenticatedUser(username=username)
|
||||
|
||||
|
||||
def require_user(request: Request) -> AuthenticatedUser | None:
|
||||
settings = request.app.state.settings
|
||||
if not auth_enabled(settings):
|
||||
return None
|
||||
user = parse_session_cookie(request.cookies.get(SESSION_COOKIE), settings)
|
||||
if user is None:
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Login required")
|
||||
return user
|
||||
|
||||
|
||||
def should_use_secure_cookie(request: Request) -> bool:
|
||||
forwarded_proto = request.headers.get("x-forwarded-proto", "").split(",", 1)[0].strip().lower()
|
||||
return request.url.scheme == "https" or forwarded_proto == "https"
|
||||
|
||||
|
||||
def login_response(payload: LoginPayload, request: Request, settings: Settings) -> Response:
|
||||
if not auth_enabled(settings):
|
||||
raise HTTPException(status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail="App login is not configured")
|
||||
if not secrets.compare_digest(payload.username, settings.admin_username):
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid username or password")
|
||||
if not verify_password(payload.password, settings.admin_password_hash or ""):
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid username or password")
|
||||
response = Response(status_code=status.HTTP_204_NO_CONTENT)
|
||||
response.set_cookie(
|
||||
SESSION_COOKIE,
|
||||
create_session_cookie(payload.username, settings),
|
||||
httponly=True,
|
||||
secure=should_use_secure_cookie(request),
|
||||
samesite="lax",
|
||||
max_age=SESSION_TTL_SECONDS,
|
||||
)
|
||||
return response
|
||||
|
||||
|
||||
def clear_session_cookie(request: Request | None = None) -> Response:
|
||||
response = Response(status_code=status.HTTP_204_NO_CONTENT)
|
||||
response.delete_cookie(
|
||||
SESSION_COOKIE,
|
||||
httponly=True,
|
||||
secure=should_use_secure_cookie(request) if request is not None else True,
|
||||
samesite="lax",
|
||||
)
|
||||
return response
|
||||
293
runtime-current/kino/app-src/backend/app/services/downloader.py
Normal file
293
runtime-current/kino/app-src/backend/app/services/downloader.py
Normal file
@@ -0,0 +1,293 @@
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import shlex
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
from sqlmodel import Session
|
||||
|
||||
from app.config import Settings
|
||||
from app.models import ImportJob, Source, utcnow
|
||||
from app.providers.base import DownloadOptions, Provider, ProviderError
|
||||
from app.services.jellyfin import refresh_jellyfin
|
||||
from app.services.paths import jellyfin_profile_target, tmp_target
|
||||
|
||||
|
||||
class ImportCancelled(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def file_md5(path: Path, chunk_size: int = 1024 * 1024) -> str:
|
||||
digest = hashlib.md5() # nosec B324 - integrity check, not security/auth
|
||||
with path.open("rb") as fh:
|
||||
while chunk := fh.read(chunk_size):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def local_md5_manifest(base_dir: Path) -> dict[str, str]:
|
||||
base_dir = base_dir.resolve()
|
||||
manifest: dict[str, str] = {}
|
||||
if not base_dir.exists():
|
||||
return manifest
|
||||
for path in sorted(p for p in base_dir.rglob("*") if p.is_file()):
|
||||
if path.is_symlink():
|
||||
raise ProviderError(f"Refusing to hash symlink: {path.relative_to(base_dir)}")
|
||||
manifest[path.relative_to(base_dir).as_posix()] = file_md5(path)
|
||||
return manifest
|
||||
|
||||
|
||||
def job_title(source: Source | None) -> str:
|
||||
if source is None:
|
||||
return "media"
|
||||
return source.title or source.external_id or "media"
|
||||
|
||||
|
||||
def ensure_not_cancelled(session: Session, job: ImportJob) -> None:
|
||||
session.refresh(job)
|
||||
if job.status in {"cancelling", "cancelled"}:
|
||||
raise ImportCancelled("Import abgebrochen")
|
||||
|
||||
|
||||
async def remote_md5_manifest(remote_dir: str, settings: Settings) -> dict[str, str]:
|
||||
if not settings.rsync_target or not settings.rsync_ssh_key:
|
||||
return {}
|
||||
ssh_cmd = [
|
||||
"ssh",
|
||||
"-i",
|
||||
str(settings.rsync_ssh_key),
|
||||
"-o",
|
||||
"BatchMode=yes",
|
||||
"-o",
|
||||
"StrictHostKeyChecking=accept-new",
|
||||
settings.rsync_target,
|
||||
"cd " + shlex.quote(remote_dir) + " && find . -type f -print0 | sort -z | xargs -0 md5sum",
|
||||
]
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*ssh_cmd,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
stdout, stderr = await proc.communicate()
|
||||
if proc.returncode != 0:
|
||||
raise ProviderError(f"Jellyfin md5 verification failed: {stderr.decode(errors='replace')[:500]}")
|
||||
manifest: dict[str, str] = {}
|
||||
for line in stdout.decode(errors="replace").splitlines():
|
||||
if not line.strip():
|
||||
continue
|
||||
checksum, _, rel = line.partition(" ")
|
||||
if not rel:
|
||||
checksum, _, rel = line.partition(" ")
|
||||
manifest[rel.removeprefix("./")] = checksum.strip()
|
||||
return manifest
|
||||
|
||||
|
||||
def remote_dir_from_target(target_path: str | None, settings: Settings) -> str | None:
|
||||
if not target_path or not settings.rsync_target:
|
||||
return None
|
||||
prefix = f"{settings.rsync_target}:"
|
||||
if target_path.startswith(prefix):
|
||||
return target_path[len(prefix):]
|
||||
return None
|
||||
|
||||
|
||||
async def verify_jellyfin_transfer(local_dir: Path, target_path: str | None, settings: Settings) -> bool:
|
||||
local_manifest = local_md5_manifest(local_dir)
|
||||
if not local_manifest:
|
||||
raise ProviderError("No local files available for md5 verification")
|
||||
remote_dir = remote_dir_from_target(target_path, settings)
|
||||
if remote_dir is None:
|
||||
# No remote rsync target: the local media root is the final target.
|
||||
return local_md5_manifest(local_dir) == local_manifest
|
||||
remote_manifest = await remote_md5_manifest(remote_dir, settings)
|
||||
if remote_manifest != local_manifest:
|
||||
missing = sorted(set(local_manifest) - set(remote_manifest))[:5]
|
||||
changed = sorted(k for k in local_manifest.keys() & remote_manifest.keys() if local_manifest[k] != remote_manifest[k])[:5]
|
||||
raise ProviderError(f"Jellyfin md5 verification mismatch; missing={missing}, changed={changed}")
|
||||
return True
|
||||
|
||||
|
||||
async def cleanup_verified_download_data(tmp_dir: Path, target_dir: Path, target_path: str | None, settings: Settings) -> None:
|
||||
if tmp_dir.exists():
|
||||
shutil.rmtree(tmp_dir)
|
||||
# When rsync copied to a remote Jellyfin host, local media_root is staging and can be removed after md5 proof.
|
||||
if remote_dir_from_target(target_path, settings) is not None and target_dir.exists():
|
||||
media_root = settings.media_root.resolve()
|
||||
resolved = target_dir.resolve()
|
||||
if resolved != media_root and media_root in resolved.parents:
|
||||
shutil.rmtree(resolved)
|
||||
|
||||
|
||||
|
||||
def copy_tree_contents(src: Path, dest: Path) -> list[Path]:
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
copied: list[Path] = []
|
||||
for path in src.iterdir():
|
||||
if path.is_symlink():
|
||||
raise ProviderError(f"Refusing to copy symlink from import output: {path.name}")
|
||||
target = dest / path.name
|
||||
if path.is_dir():
|
||||
for child in path.rglob("*"):
|
||||
if child.is_symlink():
|
||||
raise ProviderError(f"Refusing to copy symlink from import output: {child.relative_to(src)}")
|
||||
if target.exists():
|
||||
shutil.rmtree(target)
|
||||
shutil.copytree(path, target)
|
||||
copied.extend(p for p in target.rglob("*") if p.is_file())
|
||||
else:
|
||||
shutil.copy2(path, target)
|
||||
copied.append(target)
|
||||
return copied
|
||||
|
||||
|
||||
async def sync_to_jellyfin_vm(local_dir: Path, settings: Settings) -> str | None:
|
||||
"""Copy the completed local import directory to Jellyfin's media VM via rsync."""
|
||||
if not settings.rsync_target or not settings.rsync_ssh_key:
|
||||
return None
|
||||
|
||||
local_dir = local_dir.resolve()
|
||||
media_root = settings.media_root.resolve()
|
||||
if media_root != local_dir and media_root not in local_dir.parents:
|
||||
raise ProviderError(f"Refusing to sync path outside media root: {local_dir}")
|
||||
|
||||
relative = local_dir.relative_to(media_root)
|
||||
remote_root = Path(str(settings.rsync_remote_root)).as_posix().rstrip("/") or "/jellyfin"
|
||||
remote_dir = f"{remote_root}/{relative.as_posix()}"
|
||||
ssh_cmd = [
|
||||
"ssh",
|
||||
"-i",
|
||||
str(settings.rsync_ssh_key),
|
||||
"-o",
|
||||
"BatchMode=yes",
|
||||
"-o",
|
||||
"StrictHostKeyChecking=accept-new",
|
||||
]
|
||||
|
||||
mkdir_proc = await asyncio.create_subprocess_exec(
|
||||
*ssh_cmd,
|
||||
settings.rsync_target,
|
||||
f"mkdir -p {shlex.quote(remote_dir)}",
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
_, mkdir_stderr = await mkdir_proc.communicate()
|
||||
if mkdir_proc.returncode != 0:
|
||||
raise ProviderError(f"Jellyfin transfer mkdir failed: {mkdir_stderr.decode(errors='replace')[:500]}")
|
||||
|
||||
rsync_proc = await asyncio.create_subprocess_exec(
|
||||
"rsync",
|
||||
"-a",
|
||||
"--delete",
|
||||
"-s",
|
||||
"-e",
|
||||
" ".join(shlex.quote(part) for part in ssh_cmd),
|
||||
f"{local_dir}/",
|
||||
f"{settings.rsync_target}:{remote_dir}/",
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
_, rsync_stderr = await rsync_proc.communicate()
|
||||
if rsync_proc.returncode != 0:
|
||||
raise ProviderError(f"Jellyfin transfer rsync failed: {rsync_stderr.decode(errors='replace')[:500]}")
|
||||
|
||||
return f"{settings.rsync_target}:{remote_dir}"
|
||||
|
||||
|
||||
def update_job(session: Session, job: ImportJob, **fields) -> None:
|
||||
for key, value in fields.items():
|
||||
setattr(job, key, value)
|
||||
job.updated_at = utcnow()
|
||||
session.add(job)
|
||||
session.commit()
|
||||
session.refresh(job)
|
||||
|
||||
|
||||
async def run_import_job(job_id: int, engine, providers: list[Provider], settings: Settings) -> None:
|
||||
tmp_dir: Path | None = None
|
||||
target_dir: Path | None = None
|
||||
with Session(engine) as session:
|
||||
job = session.get(ImportJob, job_id)
|
||||
if job is None:
|
||||
return
|
||||
source = session.get(Source, job.source_id)
|
||||
if source is None:
|
||||
update_job(session, job, status="failed", error="Source missing")
|
||||
return
|
||||
provider = next((p for p in providers if p.name == source.provider or p.can_handle(source.url)), None)
|
||||
if provider is None:
|
||||
update_job(session, job, status="failed", error="No provider available for source")
|
||||
return
|
||||
try:
|
||||
ensure_not_cancelled(session, job)
|
||||
last_reported_progress = 0.0
|
||||
|
||||
def report_download_progress(provider_progress: float) -> None:
|
||||
nonlocal last_reported_progress
|
||||
ensure_not_cancelled(session, job)
|
||||
mapped = 0.15 + (max(0.0, min(provider_progress, 1.0)) * 0.60)
|
||||
if mapped - last_reported_progress >= 0.01 or mapped >= 0.75:
|
||||
last_reported_progress = mapped
|
||||
update_job(session, job, status="downloading", progress=round(mapped, 3))
|
||||
|
||||
update_job(session, job, status="downloading", progress=0.15, error=None)
|
||||
title = job_title(source)
|
||||
stored_metadata = json.loads(source.metadata_json or "{}")
|
||||
import_request = stored_metadata.get("import_request", {}) if isinstance(stored_metadata, dict) else {}
|
||||
provider_metadata = stored_metadata.get("metadata", stored_metadata) if isinstance(stored_metadata, dict) else {}
|
||||
tmp_dir = tmp_target(settings.tmp_dir, job.id or 0, title)
|
||||
result = await provider.download(
|
||||
source.url,
|
||||
tmp_dir,
|
||||
DownloadOptions(
|
||||
max_height=settings.default_max_height,
|
||||
target_library=job.target_library,
|
||||
max_bytes=settings.max_download_bytes,
|
||||
),
|
||||
progress_callback=report_download_progress,
|
||||
)
|
||||
ensure_not_cancelled(session, job)
|
||||
update_job(session, job, status="postprocessing", progress=0.75)
|
||||
target_dir = jellyfin_profile_target(
|
||||
settings.media_root,
|
||||
job.target_library,
|
||||
import_request.get("title") or title,
|
||||
year=import_request.get("year"),
|
||||
series_title=import_request.get("series_title"),
|
||||
season=import_request.get("season"),
|
||||
episode=import_request.get("episode"),
|
||||
episode_title=import_request.get("episode_title"),
|
||||
channel=provider_metadata.get("uploader") or provider_metadata.get("channel") or source.provider,
|
||||
external_id=source.external_id,
|
||||
)
|
||||
update_job(session, job, status="copying", progress=0.82)
|
||||
copied = copy_tree_contents(tmp_dir, target_dir)
|
||||
if not copied and not result.output_files:
|
||||
raise ProviderError("No files were produced by the import")
|
||||
ensure_not_cancelled(session, job)
|
||||
remote_target = await sync_to_jellyfin_vm(target_dir, settings)
|
||||
final_target = remote_target or str(target_dir)
|
||||
update_job(session, job, target_path=final_target, status="refreshing", progress=0.9)
|
||||
verify_ok = await verify_jellyfin_transfer(target_dir, final_target, settings)
|
||||
if verify_ok:
|
||||
await cleanup_verified_download_data(tmp_dir, target_dir, final_target, settings)
|
||||
refreshed = await refresh_jellyfin(settings)
|
||||
note = None if refreshed else "Import completed; Jellyfin refresh skipped because no API key is configured"
|
||||
update_job(session, job, status="done", progress=1.0, error=note)
|
||||
except ImportCancelled:
|
||||
if tmp_dir and tmp_dir.exists():
|
||||
shutil.rmtree(tmp_dir)
|
||||
update_job(session, job, status="cancelled", error="Import abgebrochen; temporäre Download-Dateien gelöscht")
|
||||
except Exception as exc: # keep background task failures visible in DB/API
|
||||
update_job(session, job, status="failed", error=str(exc)[:1000])
|
||||
|
||||
|
||||
def schedule_import(job_id: int, engine, providers: list[Provider], settings: Settings) -> None:
|
||||
asyncio.create_task(run_import_job(job_id, engine, providers, settings))
|
||||
|
||||
|
||||
def metadata_json(metadata, *, import_request: dict | None = None) -> str:
|
||||
payload = metadata.model_dump()
|
||||
if import_request is not None:
|
||||
payload = {"metadata": payload, "import_request": import_request}
|
||||
return json.dumps(payload, ensure_ascii=False)
|
||||
@@ -0,0 +1,94 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
from sqlmodel import Session, select
|
||||
|
||||
from app.models import ImportJob, utcnow
|
||||
|
||||
ACTIVE_IMPORT_STATUSES = {"queued", "downloading", "postprocessing", "copying", "refreshing"}
|
||||
TERMINAL_IMPORT_STATUSES = {"done", "failed"}
|
||||
|
||||
|
||||
class ImportQueueManager:
|
||||
"""In-process background worker pool for import jobs.
|
||||
|
||||
Jobs are persisted in SQLite, so the UI can poll progress and a service restart
|
||||
requeues any jobs that were active when the process stopped. The heavy work is
|
||||
still executed inside the app process, but decoupled from the request that
|
||||
created the job and bounded by a configurable concurrency limit.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._queue: asyncio.Queue[int] = asyncio.Queue()
|
||||
self._workers: list[asyncio.Task] = []
|
||||
self._known: set[int] = set()
|
||||
self._stopping = False
|
||||
|
||||
async def start(
|
||||
self,
|
||||
*,
|
||||
engine,
|
||||
providers,
|
||||
settings,
|
||||
runner: Callable[[int, object, list, object], Awaitable[None]],
|
||||
concurrency: int,
|
||||
) -> None:
|
||||
self._stopping = False
|
||||
self._engine = engine
|
||||
self._providers = providers
|
||||
self._settings = settings
|
||||
self._runner = runner
|
||||
self._concurrency = max(1, int(concurrency or 1))
|
||||
await self.requeue_active_jobs()
|
||||
self._workers = [asyncio.create_task(self._worker(idx)) for idx in range(self._concurrency)]
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._stopping = True
|
||||
for worker in self._workers:
|
||||
worker.cancel()
|
||||
if self._workers:
|
||||
await asyncio.gather(*self._workers, return_exceptions=True)
|
||||
self._workers = []
|
||||
self._known.clear()
|
||||
|
||||
async def enqueue(self, job_id: int) -> None:
|
||||
if self._stopping or job_id in self._known:
|
||||
return
|
||||
self._known.add(job_id)
|
||||
await self._queue.put(job_id)
|
||||
|
||||
async def requeue_active_jobs(self) -> int:
|
||||
count = 0
|
||||
with Session(self._engine) as session:
|
||||
jobs = list(
|
||||
session.exec(
|
||||
select(ImportJob).where(ImportJob.status.in_(ACTIVE_IMPORT_STATUSES)).order_by(ImportJob.created_at)
|
||||
).all()
|
||||
)
|
||||
for job in jobs:
|
||||
# Jobs interrupted during a restart become queued again; completed
|
||||
# progress is intentionally reset because the temp dir may be gone.
|
||||
job.status = "queued"
|
||||
job.progress = 0.0
|
||||
job.updated_at = utcnow()
|
||||
session.add(job)
|
||||
session.commit()
|
||||
for job in jobs:
|
||||
if job.id is not None:
|
||||
await self.enqueue(job.id)
|
||||
count += 1
|
||||
return count
|
||||
|
||||
async def _worker(self, idx: int) -> None:
|
||||
while True:
|
||||
job_id = await self._queue.get()
|
||||
try:
|
||||
await self._runner(job_id, self._engine, self._providers, self._settings)
|
||||
finally:
|
||||
self._known.discard(job_id)
|
||||
self._queue.task_done()
|
||||
|
||||
|
||||
import_queue_manager = ImportQueueManager()
|
||||
@@ -0,0 +1,202 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import time
|
||||
import uuid
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
from urllib.parse import unquote, urlparse
|
||||
|
||||
from app.config import get_settings
|
||||
from app.services.page_analyzer import (
|
||||
MediaCandidate,
|
||||
_browser_context,
|
||||
_is_ad_or_tracker_url,
|
||||
_looks_like_media_response,
|
||||
_probe_candidates_with_meta,
|
||||
extract_media_urls,
|
||||
)
|
||||
from app.services.url_safety import BROWSER_LIKE_HEADERS, UnsafeUrlError, assert_public_host, check_url_for_page
|
||||
|
||||
_INTERACTIVE_CAPTURE_LIMIT = 80
|
||||
_DEFAULT_VIEWPORT = {"width": 1280, "height": 720}
|
||||
_SESSION_TTL_SECONDS = 15 * 60
|
||||
|
||||
|
||||
@dataclass
|
||||
class InteractiveBrowserSession:
|
||||
id: str
|
||||
page_url: str
|
||||
context: Any
|
||||
page: Any
|
||||
browser: Any | None = None
|
||||
user_data_dir: Any | None = None
|
||||
ublock_path: str | None = None
|
||||
seen: dict[str, tuple[str | None, str | None]] = field(default_factory=dict)
|
||||
pending_response_tasks: set[asyncio.Task] = field(default_factory=set)
|
||||
created_at: float = field(default_factory=time.monotonic)
|
||||
updated_at: float = field(default_factory=time.monotonic)
|
||||
|
||||
|
||||
class InteractiveBrowserManager:
|
||||
"""Small in-process Playwright session manager for interactive media analysis.
|
||||
|
||||
The browser never receives private/internal network requests: every request is
|
||||
routed through the same public-host guard used by the non-interactive browser
|
||||
analyzer. The UI gets screenshots and sends clicks/keystrokes back, avoiding
|
||||
iframe/CORS limitations while still allowing the user to press Play.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._sessions: dict[str, InteractiveBrowserSession] = {}
|
||||
self._playwright: Any | None = None
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
async def _ensure_playwright(self):
|
||||
if self._playwright is None:
|
||||
from playwright.async_api import async_playwright
|
||||
|
||||
self._playwright = await async_playwright().start()
|
||||
return self._playwright
|
||||
|
||||
async def _close_session(self, session: InteractiveBrowserSession) -> None:
|
||||
for task in list(session.pending_response_tasks):
|
||||
if not task.done():
|
||||
task.cancel()
|
||||
try:
|
||||
await session.context.close()
|
||||
finally:
|
||||
if session.browser is not None:
|
||||
await session.browser.close()
|
||||
if session.user_data_dir is not None:
|
||||
session.user_data_dir.cleanup()
|
||||
|
||||
async def cleanup_expired(self) -> None:
|
||||
now = time.monotonic()
|
||||
expired = [sid for sid, sess in self._sessions.items() if now - sess.updated_at > _SESSION_TTL_SECONDS]
|
||||
for sid in expired:
|
||||
await self.stop(sid)
|
||||
|
||||
async def start(self, url: str) -> InteractiveBrowserSession:
|
||||
await self.cleanup_expired()
|
||||
settings = get_settings()
|
||||
checked_page = await check_url_for_page(url, timeout=settings.request_timeout)
|
||||
playwright = await self._ensure_playwright()
|
||||
context, browser, user_data_dir, ublock_path = await _browser_context(playwright, ignore_https_errors=False)
|
||||
await context.set_extra_http_headers({**BROWSER_LIKE_HEADERS, "DNT": "1"})
|
||||
|
||||
session = InteractiveBrowserSession(
|
||||
id=uuid.uuid4().hex,
|
||||
page_url=checked_page.final_url,
|
||||
context=context,
|
||||
page=await context.new_page(),
|
||||
browser=browser,
|
||||
user_data_dir=user_data_dir,
|
||||
ublock_path=ublock_path,
|
||||
)
|
||||
await session.page.set_viewport_size(_DEFAULT_VIEWPORT)
|
||||
|
||||
async def guard_route(route):
|
||||
request_url = route.request.url
|
||||
parsed = urlparse(request_url)
|
||||
if parsed.scheme not in {"http", "https"}:
|
||||
await route.abort()
|
||||
return
|
||||
if _is_ad_or_tracker_url(request_url):
|
||||
await route.abort()
|
||||
return
|
||||
try:
|
||||
await assert_public_host(request_url)
|
||||
except UnsafeUrlError:
|
||||
await route.abort()
|
||||
return
|
||||
await route.continue_()
|
||||
|
||||
async def on_response(response):
|
||||
response_url = response.url
|
||||
try:
|
||||
headers = await response.all_headers()
|
||||
except Exception:
|
||||
headers = {}
|
||||
content_type = headers.get("content-type")
|
||||
if _looks_like_media_response(response_url, content_type):
|
||||
title = unquote(urlparse(response_url).path.rsplit("/", 1)[-1]) or response_url
|
||||
session.seen.setdefault(response_url, (title, content_type))
|
||||
|
||||
def schedule_response_probe(response):
|
||||
task = asyncio.create_task(on_response(response))
|
||||
session.pending_response_tasks.add(task)
|
||||
task.add_done_callback(session.pending_response_tasks.discard)
|
||||
|
||||
await context.route("**/*", guard_route)
|
||||
context.on("response", schedule_response_probe)
|
||||
try:
|
||||
await session.page.goto(checked_page.final_url, wait_until="domcontentloaded", timeout=int(settings.request_timeout * 1000))
|
||||
try:
|
||||
await session.page.wait_for_load_state("networkidle", timeout=3000)
|
||||
except Exception:
|
||||
pass
|
||||
await self._collect_dom_media(session)
|
||||
except Exception:
|
||||
await self._close_session(session)
|
||||
raise
|
||||
|
||||
self._sessions[session.id] = session
|
||||
return session
|
||||
|
||||
async def _collect_dom_media(self, session: InteractiveBrowserSession) -> None:
|
||||
try:
|
||||
html = await session.page.content()
|
||||
page_url = session.page.url
|
||||
except Exception:
|
||||
return
|
||||
for dom_url in extract_media_urls(html, page_url):
|
||||
session.seen.setdefault(dom_url, (None, None))
|
||||
|
||||
async def _settle(self, session: InteractiveBrowserSession) -> None:
|
||||
session.updated_at = time.monotonic()
|
||||
await asyncio.sleep(0.5)
|
||||
if session.pending_response_tasks:
|
||||
await asyncio.gather(*list(session.pending_response_tasks), return_exceptions=True)
|
||||
await self._collect_dom_media(session)
|
||||
|
||||
def get(self, session_id: str) -> InteractiveBrowserSession:
|
||||
session = self._sessions.get(session_id)
|
||||
if not session:
|
||||
raise KeyError(session_id)
|
||||
session.updated_at = time.monotonic()
|
||||
return session
|
||||
|
||||
async def click(self, session_id: str, x: float, y: float) -> InteractiveBrowserSession:
|
||||
session = self.get(session_id)
|
||||
await session.page.mouse.click(x, y)
|
||||
await self._settle(session)
|
||||
return session
|
||||
|
||||
async def press(self, session_id: str, key: str) -> InteractiveBrowserSession:
|
||||
session = self.get(session_id)
|
||||
await session.page.keyboard.press(key)
|
||||
await self._settle(session)
|
||||
return session
|
||||
|
||||
async def stop(self, session_id: str) -> None:
|
||||
session = self._sessions.pop(session_id, None)
|
||||
if session is not None:
|
||||
await self._close_session(session)
|
||||
|
||||
async def screenshot_data_url(self, session: InteractiveBrowserSession) -> str:
|
||||
png = await session.page.screenshot(type="png", full_page=False)
|
||||
return "data:image/png;base64," + base64.b64encode(png).decode("ascii")
|
||||
|
||||
async def candidates(self, session: InteractiveBrowserSession) -> list[MediaCandidate]:
|
||||
if session.pending_response_tasks:
|
||||
await asyncio.gather(*list(session.pending_response_tasks), return_exceptions=True)
|
||||
await self._collect_dom_media(session)
|
||||
return await _probe_candidates_with_meta(
|
||||
dict(list(session.seen.items())[:_INTERACTIVE_CAPTURE_LIMIT]),
|
||||
source="interactive-browser",
|
||||
)
|
||||
|
||||
|
||||
interactive_browser_manager = InteractiveBrowserManager()
|
||||
@@ -0,0 +1,23 @@
|
||||
import httpx
|
||||
|
||||
from app.config import Settings
|
||||
|
||||
|
||||
class JellyfinRefreshSkipped(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
async def refresh_jellyfin(settings: Settings) -> bool:
|
||||
"""Trigger Jellyfin's library refresh when credentials are configured.
|
||||
|
||||
Returns False when no API key is present so local/dev imports can still finish
|
||||
without writing secrets into the project tree.
|
||||
"""
|
||||
if not settings.jellyfin_api_key:
|
||||
return False
|
||||
url = settings.jellyfin_url.rstrip("/") + "/Library/Refresh"
|
||||
headers = {"X-Emby-Token": settings.jellyfin_api_key}
|
||||
async with httpx.AsyncClient(timeout=20) as client:
|
||||
response = await client.post(url, headers=headers)
|
||||
response.raise_for_status()
|
||||
return True
|
||||
@@ -0,0 +1,503 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import html as html_lib
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
from dataclasses import dataclass, field
|
||||
from html.parser import HTMLParser
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import unquote, urljoin, urlparse
|
||||
|
||||
import httpx
|
||||
try:
|
||||
from yt_dlp import YoutubeDL
|
||||
except ImportError: # optional analyzer backend
|
||||
YoutubeDL = None # type: ignore[assignment]
|
||||
|
||||
from app.config import get_settings
|
||||
from app.services.url_safety import BROWSER_LIKE_HEADERS, BROWSER_LIKE_MEDIA_HEADERS, CHROME_WINDOWS_11_USER_AGENT, UnsafeUrlError, assert_public_host, check_url_for_download, check_url_for_page
|
||||
|
||||
_MEDIA_EXTENSIONS = (".mp4", ".m4v", ".webm", ".mkv", ".mov", ".m3u8", ".mpd")
|
||||
_MEDIA_CONTENT_TYPES = {
|
||||
"video/mp4": "direct_video",
|
||||
"video/webm": "direct_video",
|
||||
"video/x-matroska": "direct_video",
|
||||
"video/mp2t": "hls_segment",
|
||||
"application/vnd.apple.mpegurl": "hls_manifest",
|
||||
"application/x-mpegurl": "hls_manifest",
|
||||
"application/dash+xml": "dash_manifest",
|
||||
}
|
||||
_QUOTED_URL_RE = re.compile(r"[\"']([^\"']+\.(?:mp4|m4v|webm|mkv|mov|m3u8|mpd)(?:\?[^\"']*)?)[\"']", re.I)
|
||||
_ABSOLUTE_MEDIA_URL_RE = re.compile(
|
||||
r"(?:https?://|https?:\\/\\/|//|\\/\\/)[^\s\"'<>]+?\.(?:mp4|m4v|webm|mkv|mov|m3u8|mpd)(?=$|[?&#\s\"'<>])(?:[?&][^\s\"'<>]*)?",
|
||||
re.I,
|
||||
)
|
||||
_BROWSER_CAPTURE_LIMIT = 80
|
||||
_YTDLP_ALLOWED_HOST_SUFFIXES = (
|
||||
"youtube.com",
|
||||
"youtu.be",
|
||||
"archive.org",
|
||||
"ardmediathek.de",
|
||||
"zdf.de",
|
||||
"arte.tv",
|
||||
)
|
||||
_AD_HOST_FRAGMENTS = (
|
||||
"doubleclick.net",
|
||||
"googlesyndication.com",
|
||||
"googleadservices.com",
|
||||
"adservice.google.",
|
||||
"adsystem.com",
|
||||
"adnxs.com",
|
||||
"adsafeprotected.com",
|
||||
"amazon-adsystem.com",
|
||||
"taboola.com",
|
||||
"outbrain.com",
|
||||
"scorecardresearch.com",
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class MediaCandidate:
|
||||
url: str
|
||||
kind: str
|
||||
title: str | None = None
|
||||
source: str = "html"
|
||||
content_type: str | None = None
|
||||
content_length: int | None = None
|
||||
file_size: str | None = None
|
||||
quality: str | None = None
|
||||
allowed: bool = True
|
||||
reason: str | None = None
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
class _MediaHTMLParser(HTMLParser):
|
||||
def __init__(self, base_url: str) -> None:
|
||||
super().__init__()
|
||||
self.base_url = base_url
|
||||
self.urls: list[str] = []
|
||||
|
||||
def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
|
||||
attr_map = {key.lower(): value for key, value in attrs if value}
|
||||
for attr in ("src", "href", "data-src", "data-url"):
|
||||
value = attr_map.get(attr)
|
||||
if value and _looks_like_media_url(value):
|
||||
self.urls.append(urljoin(self.base_url, value))
|
||||
|
||||
|
||||
def _looks_like_media_url(value: str) -> bool:
|
||||
parsed_path = urlparse(value).path.lower()
|
||||
return parsed_path.endswith(_MEDIA_EXTENSIONS)
|
||||
|
||||
|
||||
def _looks_like_media_response(url: str, content_type: str | None = None) -> bool:
|
||||
clean = (content_type or "").split(";", 1)[0].strip().lower()
|
||||
return clean in _MEDIA_CONTENT_TYPES or clean.startswith("video/") or _looks_like_media_url(url)
|
||||
|
||||
|
||||
def human_size(size: int | None) -> str | None:
|
||||
if size is None:
|
||||
return None
|
||||
value = float(size)
|
||||
for unit in ("B", "KB", "MB", "GB"):
|
||||
if value < 1024 or unit == "GB":
|
||||
if unit == "B":
|
||||
return f"{int(value)} {unit}"
|
||||
return f"{value:.1f} {unit}"
|
||||
value /= 1024
|
||||
|
||||
|
||||
def _is_ad_or_tracker_url(url: str) -> bool:
|
||||
host = (urlparse(url).hostname or "").lower()
|
||||
if any(fragment in host for fragment in _AD_HOST_FRAGMENTS):
|
||||
return True
|
||||
path = urlparse(url).path.lower()
|
||||
return any(marker in path for marker in ("/ads/", "/adserver/", "/prebid", "/vast", "/vpaid"))
|
||||
|
||||
|
||||
def _host_allowed_for_ytdlp(url: str) -> bool:
|
||||
host = (urlparse(url).hostname or "").lower().rstrip(".")
|
||||
return any(host == suffix or host.endswith(f".{suffix}") for suffix in _YTDLP_ALLOWED_HOST_SUFFIXES)
|
||||
|
||||
|
||||
async def _fetch_public_page_body(url: str, *, timeout: float, max_redirects: int = 8) -> tuple[str, str]:
|
||||
"""Fetch a page body while validating every GET redirect hop.
|
||||
|
||||
``check_url_for_page`` validates HEAD redirects, but some origins redirect
|
||||
differently for GET. Keep GET redirects manual so the analyzer cannot be
|
||||
turned into an internal-network fetch proxy by a method-dependent redirect.
|
||||
"""
|
||||
current = url
|
||||
async with httpx.AsyncClient(follow_redirects=False, timeout=timeout) as client:
|
||||
for _ in range(max_redirects + 1):
|
||||
checked = await check_url_for_page(current, timeout=timeout, max_redirects=max_redirects)
|
||||
response = await client.get(checked.final_url, headers=BROWSER_LIKE_HEADERS)
|
||||
if response.is_redirect:
|
||||
loc = response.headers.get("location")
|
||||
if not loc:
|
||||
raise UnsafeUrlError("Redirect ohne Location-Header")
|
||||
current = str(response.url.join(loc))
|
||||
continue
|
||||
response.raise_for_status()
|
||||
await assert_public_host(str(response.url))
|
||||
return response.text, str(response.url)
|
||||
raise UnsafeUrlError("Zu viele Weiterleitungen")
|
||||
|
||||
|
||||
def _ublock_origin_path() -> Path | None:
|
||||
configured = os.environ.get("KINOPROJEKT_UBLOCK_PATH")
|
||||
candidates = [Path(configured)] if configured else []
|
||||
candidates.extend([Path("/opt/kino-projekt/ublock-origin"), Path("/opt/kino-projekt/ublock")])
|
||||
for candidate in candidates:
|
||||
if not candidate:
|
||||
continue
|
||||
if candidate.exists() and candidate.is_dir() and (candidate / "manifest.json").exists():
|
||||
return candidate
|
||||
nested = candidate / "uBlock0.chromium"
|
||||
if nested.exists() and nested.is_dir() and (nested / "manifest.json").exists():
|
||||
return nested
|
||||
return None
|
||||
|
||||
|
||||
def _has_importable_media(candidates: dict[str, tuple[str | None, str | None]]) -> bool:
|
||||
return any(
|
||||
classify_candidate(url, content_type).kind in {"direct_video", "hls_manifest", "dash_manifest"}
|
||||
for url, (_, content_type) in candidates.items()
|
||||
)
|
||||
|
||||
|
||||
def _normalize_media_url(raw: str, base_url: str) -> str | None:
|
||||
value = html_lib.unescape(raw.strip())
|
||||
if not value:
|
||||
return None
|
||||
value = value.replace("\\/", "/").replace("\\u002F", "/").replace("\\u002f", "/")
|
||||
value = value.rstrip(",;)}]")
|
||||
if value.startswith("//"):
|
||||
value = f"{urlparse(base_url).scheme or 'https'}:{value}"
|
||||
return urljoin(base_url, unquote(value))
|
||||
|
||||
|
||||
def extract_media_urls(html: str, base_url: str) -> list[str]:
|
||||
parser = _MediaHTMLParser(base_url)
|
||||
parser.feed(html)
|
||||
urls = list(parser.urls)
|
||||
for match in _QUOTED_URL_RE.finditer(html):
|
||||
normalized = _normalize_media_url(match.group(1), base_url)
|
||||
if normalized:
|
||||
urls.append(normalized)
|
||||
for match in _ABSOLUTE_MEDIA_URL_RE.finditer(html):
|
||||
normalized = _normalize_media_url(match.group(0), base_url)
|
||||
if normalized:
|
||||
urls.append(normalized)
|
||||
# Stable de-duplication while preserving discovery order.
|
||||
return list(dict.fromkeys(urls))
|
||||
|
||||
|
||||
def classify_candidate(url: str, content_type: str | None = None) -> MediaCandidate:
|
||||
normalized_content_type = (content_type or "").split(";", 1)[0].strip().lower()
|
||||
if normalized_content_type in _MEDIA_CONTENT_TYPES:
|
||||
kind = _MEDIA_CONTENT_TYPES[normalized_content_type]
|
||||
elif normalized_content_type.startswith("video/"):
|
||||
kind = "direct_video"
|
||||
else:
|
||||
path = urlparse(url).path.lower()
|
||||
if path.endswith(".m3u8"):
|
||||
kind = "hls_manifest"
|
||||
elif path.endswith(".mpd"):
|
||||
kind = "dash_manifest"
|
||||
elif path.endswith((".mp4", ".m4v", ".webm", ".mkv", ".mov")):
|
||||
kind = "direct_video"
|
||||
else:
|
||||
kind = "unknown_media"
|
||||
title = unquote(urlparse(url).path.rsplit("/", 1)[-1]) or url
|
||||
return MediaCandidate(url=url, kind=kind, title=title, content_type=content_type)
|
||||
|
||||
|
||||
async def _probe_candidate(url: str, source: str, title: str | None = None, quality: str | None = None) -> MediaCandidate:
|
||||
settings = get_settings()
|
||||
try:
|
||||
checked = await check_url_for_download(url, max_bytes=settings.max_download_bytes, timeout=settings.request_timeout)
|
||||
candidate = classify_candidate(checked.final_url, checked.content_type)
|
||||
candidate.source = source
|
||||
candidate.title = title or candidate.title
|
||||
candidate.quality = quality
|
||||
candidate.content_length = checked.content_length
|
||||
candidate.file_size = human_size(checked.content_length)
|
||||
candidate.metadata = {"redirect_chain": checked.chain}
|
||||
return candidate
|
||||
except UnsafeUrlError as exc:
|
||||
c = classify_candidate(url)
|
||||
c.source = source
|
||||
c.title = title or c.title
|
||||
c.quality = quality
|
||||
c.allowed = False
|
||||
c.reason = str(exc)
|
||||
return c
|
||||
|
||||
|
||||
async def analyze_html_page(url: str) -> list[MediaCandidate]:
|
||||
settings = get_settings()
|
||||
# First validate the page URL itself so the analyzer cannot be used as an internal-network proxy.
|
||||
checked_page = await check_url_for_page(url, timeout=settings.request_timeout)
|
||||
body, final_url = await _fetch_public_page_body(checked_page.final_url, timeout=settings.request_timeout)
|
||||
found = extract_media_urls(body, final_url)
|
||||
return await _probe_candidates(found, source="html")
|
||||
|
||||
|
||||
async def analyze_with_ytdlp(url: str) -> list[MediaCandidate]:
|
||||
checked_page = await check_url_for_page(url, timeout=get_settings().request_timeout)
|
||||
|
||||
if not _host_allowed_for_ytdlp(checked_page.final_url):
|
||||
return []
|
||||
|
||||
if YoutubeDL is None:
|
||||
return []
|
||||
|
||||
def _extract() -> dict[str, Any] | None:
|
||||
opts = {
|
||||
"quiet": True,
|
||||
"skip_download": True,
|
||||
"noplaylist": True,
|
||||
"extract_flat": False,
|
||||
"socket_timeout": get_settings().request_timeout,
|
||||
}
|
||||
with YoutubeDL(opts) as ydl:
|
||||
return ydl.extract_info(checked_page.final_url, download=False)
|
||||
|
||||
try:
|
||||
info = await asyncio.to_thread(_extract)
|
||||
except Exception:
|
||||
return []
|
||||
if not info:
|
||||
return []
|
||||
|
||||
candidates: list[tuple[str, str | None, str | None]] = []
|
||||
title = info.get("title")
|
||||
if info.get("url") and _looks_like_media_url(info["url"]):
|
||||
candidates.append((info["url"], title, None))
|
||||
for fmt in info.get("formats") or []:
|
||||
fmt_url = fmt.get("url")
|
||||
if not fmt_url:
|
||||
continue
|
||||
ext = fmt.get("ext") or ""
|
||||
protocol = fmt.get("protocol") or ""
|
||||
if _looks_like_media_url(fmt_url) or ext in {"mp4", "webm", "m3u8", "mpd"} or "m3u8" in protocol or "dash" in protocol:
|
||||
quality = fmt.get("format_note") or fmt.get("format") or fmt.get("height")
|
||||
candidates.append((fmt_url, title, str(quality) if quality else None))
|
||||
deduped: dict[str, tuple[str | None, str | None]] = {}
|
||||
for candidate_url, candidate_title, quality in candidates:
|
||||
deduped.setdefault(candidate_url, (candidate_title, quality))
|
||||
return await _probe_candidates_with_meta(deduped, source="yt-dlp")
|
||||
|
||||
|
||||
async def _trigger_video_playback(page) -> None:
|
||||
"""Try to start embedded players so lazy video requests become visible."""
|
||||
await page.evaluate(
|
||||
"""
|
||||
async () => {
|
||||
for (const text of ['Akzeptieren', 'Accept', 'I agree', 'OK']) {
|
||||
const buttons = [...document.querySelectorAll('button, [role="button"], input[type="button"], input[type="submit"]')];
|
||||
const button = buttons.find(el => (el.innerText || el.value || '').toLowerCase().includes(text.toLowerCase()));
|
||||
if (button) { try { button.click(); } catch (_) {} }
|
||||
}
|
||||
for (const video of document.querySelectorAll('video')) {
|
||||
try {
|
||||
video.muted = true;
|
||||
video.playsInline = true;
|
||||
video.autoplay = true;
|
||||
video.play().catch(()=>{});
|
||||
} catch (_) {}
|
||||
}
|
||||
const playSelectors = [
|
||||
'[aria-label*="play" i]', '[title*="play" i]', '.play', '.play-button', '.vjs-big-play-button',
|
||||
'button[class*="play" i]', 'button[aria-label*="abspielen" i]'
|
||||
];
|
||||
for (const selector of playSelectors) {
|
||||
for (const el of document.querySelectorAll(selector)) {
|
||||
try { el.click(); } catch (_) {}
|
||||
}
|
||||
}
|
||||
}
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
async def _browser_context(playwright, *, ignore_https_errors: bool = False):
|
||||
base_args = [
|
||||
"--disable-dev-shm-usage",
|
||||
"--disable-gpu",
|
||||
"--no-sandbox",
|
||||
"--disable-setuid-sandbox",
|
||||
"--disable-background-networking",
|
||||
"--window-size=1365,768",
|
||||
]
|
||||
context_options = {
|
||||
"ignore_https_errors": ignore_https_errors,
|
||||
"java_script_enabled": True,
|
||||
"user_agent": CHROME_WINDOWS_11_USER_AGENT,
|
||||
"locale": "de-DE",
|
||||
"timezone_id": "Europe/Berlin",
|
||||
"viewport": {"width": 1365, "height": 768},
|
||||
"device_scale_factor": 1,
|
||||
"is_mobile": False,
|
||||
"has_touch": False,
|
||||
"extra_http_headers": {
|
||||
key: value for key, value in BROWSER_LIKE_HEADERS.items() if key.lower() != "user-agent"
|
||||
},
|
||||
}
|
||||
ublock_path = _ublock_origin_path()
|
||||
if ublock_path:
|
||||
user_data_dir = tempfile.TemporaryDirectory(prefix="kino-browser-")
|
||||
context = await playwright.chromium.launch_persistent_context(
|
||||
user_data_dir.name,
|
||||
headless=True,
|
||||
args=[
|
||||
*base_args,
|
||||
f"--disable-extensions-except={ublock_path}",
|
||||
f"--load-extension={ublock_path}",
|
||||
],
|
||||
**context_options,
|
||||
)
|
||||
return context, None, user_data_dir, str(ublock_path)
|
||||
|
||||
browser = await playwright.chromium.launch(headless=True, args=base_args)
|
||||
context = await browser.new_context(**context_options)
|
||||
return context, browser, None, None
|
||||
|
||||
|
||||
async def analyze_with_browser(url: str) -> list[MediaCandidate]:
|
||||
"""Headless-Browser-Analyzer (Option B): observe JS-loaded media requests.
|
||||
|
||||
Every browser request is still constrained to public http(s) targets. This keeps the
|
||||
server-side browser from becoming a LAN/internal-network fetch proxy.
|
||||
"""
|
||||
settings = get_settings()
|
||||
checked_page = await check_url_for_page(url, timeout=settings.request_timeout)
|
||||
try:
|
||||
from playwright.async_api import TimeoutError as PlaywrightTimeoutError
|
||||
from playwright.async_api import async_playwright
|
||||
except ImportError:
|
||||
return []
|
||||
|
||||
seen: dict[str, tuple[str | None, str | None]] = {}
|
||||
playback_triggered = False
|
||||
|
||||
async with async_playwright() as p:
|
||||
context, browser, user_data_dir, ublock_path = await _browser_context(p, ignore_https_errors=False)
|
||||
|
||||
async def guard_route(route):
|
||||
request_url = route.request.url
|
||||
parsed = urlparse(request_url)
|
||||
if parsed.scheme not in {"http", "https"}:
|
||||
await route.abort()
|
||||
return
|
||||
if _is_ad_or_tracker_url(request_url):
|
||||
await route.abort()
|
||||
return
|
||||
try:
|
||||
await assert_public_host(request_url)
|
||||
except UnsafeUrlError:
|
||||
await route.abort()
|
||||
return
|
||||
await route.continue_()
|
||||
|
||||
async def on_response(response):
|
||||
response_url = response.url
|
||||
headers = await response.all_headers()
|
||||
content_type = headers.get("content-type")
|
||||
if _looks_like_media_response(response_url, content_type):
|
||||
title = unquote(urlparse(response_url).path.rsplit("/", 1)[-1]) or response_url
|
||||
seen.setdefault(response_url, (title, content_type))
|
||||
|
||||
await context.route("**/*", guard_route)
|
||||
pending_response_tasks: set[asyncio.Task] = set()
|
||||
|
||||
def schedule_response_probe(response):
|
||||
task = asyncio.create_task(on_response(response))
|
||||
pending_response_tasks.add(task)
|
||||
task.add_done_callback(pending_response_tasks.discard)
|
||||
|
||||
context.on("response", schedule_response_probe)
|
||||
page = await context.new_page()
|
||||
try:
|
||||
await page.goto(checked_page.final_url, wait_until="domcontentloaded", timeout=int(settings.request_timeout * 1000))
|
||||
try:
|
||||
await page.wait_for_load_state("networkidle", timeout=5000)
|
||||
except PlaywrightTimeoutError:
|
||||
pass
|
||||
if pending_response_tasks:
|
||||
await asyncio.gather(*pending_response_tasks, return_exceptions=True)
|
||||
if not _has_importable_media(seen):
|
||||
playback_triggered = True
|
||||
await _trigger_video_playback(page)
|
||||
try:
|
||||
await page.wait_for_load_state("networkidle", timeout=5000)
|
||||
except PlaywrightTimeoutError:
|
||||
pass
|
||||
if pending_response_tasks:
|
||||
await asyncio.gather(*pending_response_tasks, return_exceptions=True)
|
||||
html = await page.content()
|
||||
for dom_url in extract_media_urls(html, page.url):
|
||||
seen.setdefault(dom_url, (None, None))
|
||||
finally:
|
||||
await context.close()
|
||||
if browser is not None:
|
||||
await browser.close()
|
||||
if user_data_dir is not None:
|
||||
user_data_dir.cleanup()
|
||||
|
||||
candidates = await _probe_candidates_with_meta(dict(list(seen.items())[:_BROWSER_CAPTURE_LIMIT]), source="browser")
|
||||
for candidate in candidates:
|
||||
candidate.metadata["playback_triggered"] = playback_triggered
|
||||
candidate.metadata["ublock_origin"] = bool(ublock_path)
|
||||
return candidates
|
||||
|
||||
|
||||
async def analyze_browser_capture(captured: list[dict[str, Any]], page_url: str | None = None) -> list[MediaCandidate]:
|
||||
"""Validate and normalize media URLs captured by a user's real browser (Option C)."""
|
||||
base_url = page_url or ""
|
||||
deduped: dict[str, tuple[str | None, str | None]] = {}
|
||||
for item in captured[:_BROWSER_CAPTURE_LIMIT]:
|
||||
raw_url = str(item.get("url") or "").strip()
|
||||
if not raw_url:
|
||||
continue
|
||||
candidate_url = urljoin(base_url, raw_url) if base_url else raw_url
|
||||
if not _looks_like_media_response(candidate_url, item.get("content_type") or item.get("type")):
|
||||
continue
|
||||
title = item.get("title") or item.get("name")
|
||||
quality = item.get("quality")
|
||||
deduped.setdefault(candidate_url, (str(title) if title else None, str(quality) if quality else None))
|
||||
return await _probe_candidates_with_meta(deduped, source="browser-capture")
|
||||
|
||||
|
||||
async def _probe_candidates(urls: list[str], source: str) -> list[MediaCandidate]:
|
||||
return await _probe_candidates_with_meta({url: (None, None) for url in urls}, source=source)
|
||||
|
||||
|
||||
async def _probe_candidates_with_meta(urls: dict[str, tuple[str | None, str | None]], source: str) -> list[MediaCandidate]:
|
||||
candidates = [await _probe_candidate(url, source, title, quality) for url, (title, quality) in list(urls.items())[:50]]
|
||||
if any(candidate.kind in {"hls_manifest", "dash_manifest"} for candidate in candidates):
|
||||
candidates = [candidate for candidate in candidates if candidate.kind != "hls_segment"]
|
||||
# Show usable candidates first, then blocked candidates as diagnostics.
|
||||
candidates.sort(key=lambda c: (not c.allowed, c.kind, c.quality or ""))
|
||||
return candidates
|
||||
|
||||
|
||||
async def analyze_page(url: str) -> list[MediaCandidate]:
|
||||
results = await asyncio.gather(
|
||||
analyze_html_page(url),
|
||||
analyze_with_ytdlp(url),
|
||||
analyze_with_browser(url),
|
||||
return_exceptions=True,
|
||||
)
|
||||
html_candidates, ytdlp_candidates, browser_candidates = [item if isinstance(item, list) else [] for item in results]
|
||||
merged: dict[str, MediaCandidate] = {}
|
||||
# Prefer browser/ytdlp metadata over static HTML when the same URL appears twice.
|
||||
for candidate in [*browser_candidates, *ytdlp_candidates, *html_candidates]:
|
||||
existing = merged.get(candidate.url)
|
||||
if not existing or (not existing.allowed and candidate.allowed):
|
||||
merged[candidate.url] = candidate
|
||||
return list(merged.values())
|
||||
67
runtime-current/kino/app-src/backend/app/services/paths.py
Normal file
67
runtime-current/kino/app-src/backend/app/services/paths.py
Normal file
@@ -0,0 +1,67 @@
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
_SAFE_CHARS = re.compile(r"[^A-Za-z0-9._ -]+")
|
||||
|
||||
|
||||
def slugify_name(value: str, fallback: str = "media") -> str:
|
||||
cleaned = _SAFE_CHARS.sub("_", value).strip(" ._- ")
|
||||
return cleaned[:120] or fallback
|
||||
|
||||
|
||||
def ensure_under(base: Path, candidate: Path) -> Path:
|
||||
base_resolved = base.expanduser().resolve()
|
||||
path = (base_resolved / candidate).resolve() if not candidate.is_absolute() else candidate.resolve()
|
||||
if base_resolved != path and base_resolved not in path.parents:
|
||||
raise ValueError(f"Path escapes base directory: {candidate}")
|
||||
return path
|
||||
|
||||
|
||||
def tmp_target(tmp_dir: Path, job_id: int, title: str) -> Path:
|
||||
return ensure_under(tmp_dir, Path(str(job_id)) / slugify_name(title))
|
||||
|
||||
|
||||
def media_target(media_root: Path, library: str, title: str) -> Path:
|
||||
return ensure_under(media_root, Path(slugify_name(library, "library")) / slugify_name(title))
|
||||
|
||||
|
||||
def jellyfin_profile_target(
|
||||
media_root: Path,
|
||||
target_profile: str,
|
||||
title: str,
|
||||
*,
|
||||
year: int | None = None,
|
||||
series_title: str | None = None,
|
||||
season: int | None = None,
|
||||
episode: int | None = None,
|
||||
episode_title: str | None = None,
|
||||
channel: str | None = None,
|
||||
external_id: str | None = None,
|
||||
) -> Path:
|
||||
"""Return the Jellyfin-compatible destination directory for a target profile.
|
||||
|
||||
The returned path is always below ``media_root``. The downloader copies the
|
||||
provider output into this directory; provider-selected filenames are kept for
|
||||
now, while the directory structure follows the real Jellyfin layout from the
|
||||
project plan.
|
||||
"""
|
||||
|
||||
profile = target_profile.strip().lower()
|
||||
safe_title = slugify_name(title)
|
||||
if profile == "film":
|
||||
movie_dir = f"{safe_title} ({year})" if year else safe_title
|
||||
return ensure_under(media_root, Path("Filme") / movie_dir)
|
||||
|
||||
if profile == "serie":
|
||||
show = slugify_name(series_title or title, "Serie")
|
||||
season_number = season if season and season > 0 else 1
|
||||
episode_prefix = f"S{season_number:02d}E{episode:02d} - " if episode and episode > 0 else ""
|
||||
episode_dir = slugify_name(f"{episode_prefix}{episode_title or title}", "Episode")
|
||||
return ensure_under(media_root, Path("Serien") / show / f"Staffel{season_number}" / episode_dir)
|
||||
|
||||
if profile == "clip":
|
||||
owner = slugify_name(channel or "Unbekannte Quelle", "Quelle")
|
||||
suffix = f" [{slugify_name(external_id)}]" if external_id else ""
|
||||
return ensure_under(media_root, Path("YouTube-Mediathek") / owner / f"{safe_title}{suffix}")
|
||||
|
||||
raise ValueError(f"Unknown target profile: {target_profile}")
|
||||
209
runtime-current/kino/app-src/backend/app/services/url_safety.py
Normal file
209
runtime-current/kino/app-src/backend/app/services/url_safety.py
Normal file
@@ -0,0 +1,209 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import ipaddress
|
||||
import socket
|
||||
from dataclasses import dataclass
|
||||
from typing import Iterable
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
BLOCKED_NETWORKS = [
|
||||
ipaddress.ip_network("0.0.0.0/8"),
|
||||
ipaddress.ip_network("10.0.0.0/8"),
|
||||
ipaddress.ip_network("100.64.0.0/10"),
|
||||
ipaddress.ip_network("127.0.0.0/8"),
|
||||
ipaddress.ip_network("169.254.0.0/16"),
|
||||
ipaddress.ip_network("172.16.0.0/12"),
|
||||
ipaddress.ip_network("192.168.0.0/16"),
|
||||
ipaddress.ip_network("224.0.0.0/4"),
|
||||
ipaddress.ip_network("240.0.0.0/4"),
|
||||
ipaddress.ip_network("::1/128"),
|
||||
ipaddress.ip_network("::/128"),
|
||||
ipaddress.ip_network("fc00::/7"),
|
||||
ipaddress.ip_network("fe80::/10"),
|
||||
ipaddress.ip_network("ff00::/8"),
|
||||
]
|
||||
|
||||
ALLOWED_CONTENT_PREFIXES = ("video/", "audio/")
|
||||
ALLOWED_CONTENT_TYPES = {
|
||||
"application/vnd.apple.mpegurl",
|
||||
"application/x-mpegurl",
|
||||
"application/dash+xml",
|
||||
"application/octet-stream", # accepted only with known extension
|
||||
}
|
||||
ALLOWED_EXTENSIONS = (
|
||||
".mp4",
|
||||
".m4v",
|
||||
".mkv",
|
||||
".webm",
|
||||
".mov",
|
||||
".mp3",
|
||||
".m4a",
|
||||
".aac",
|
||||
".ogg",
|
||||
".flac",
|
||||
".m3u8",
|
||||
".mpd",
|
||||
)
|
||||
|
||||
CHROME_WINDOWS_11_USER_AGENT = (
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
|
||||
"AppleWebKit/537.36 (KHTML, like Gecko) "
|
||||
"Chrome/126.0.0.0 Safari/537.36"
|
||||
)
|
||||
|
||||
# Chrome on Windows 11 intentionally still reports Windows NT 10.0 in the
|
||||
# classic User-Agent for compatibility. The client-hint headers below are the
|
||||
# browser-like Windows/Chrome signals many upstreams check before serving pages
|
||||
# or media manifests. Keep this central so safety checks, analyzer fetches and
|
||||
# provider downloads use the same fingerprint.
|
||||
BROWSER_LIKE_HEADERS = {
|
||||
"User-Agent": CHROME_WINDOWS_11_USER_AGENT,
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,video/*;q=0.8,*/*;q=0.7",
|
||||
"Accept-Language": "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7",
|
||||
"Sec-CH-UA": '"Google Chrome";v="126", "Chromium";v="126", "Not-A.Brand";v="99"',
|
||||
"Sec-CH-UA-Mobile": "?0",
|
||||
"Sec-CH-UA-Platform": '"Windows"',
|
||||
"Upgrade-Insecure-Requests": "1",
|
||||
"Sec-Fetch-Dest": "document",
|
||||
"Sec-Fetch-Mode": "navigate",
|
||||
"Sec-Fetch-Site": "none",
|
||||
"Sec-Fetch-User": "?1",
|
||||
}
|
||||
|
||||
BROWSER_LIKE_MEDIA_HEADERS = {
|
||||
**BROWSER_LIKE_HEADERS,
|
||||
"Accept": "video/webm,video/mp4,video/*;q=0.9,application/vnd.apple.mpegurl;q=0.9,application/dash+xml;q=0.9,*/*;q=0.8",
|
||||
"Sec-Fetch-Dest": "video",
|
||||
"Sec-Fetch-Mode": "no-cors",
|
||||
"Sec-Fetch-Site": "cross-site",
|
||||
}
|
||||
|
||||
|
||||
class UnsafeUrlError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass
|
||||
class CheckedUrl:
|
||||
final_url: str
|
||||
chain: list[str]
|
||||
content_type: str | None
|
||||
content_length: int | None
|
||||
|
||||
|
||||
def _is_blocked_ip(ip: str) -> bool:
|
||||
addr = ipaddress.ip_address(ip)
|
||||
return any(addr in net for net in BLOCKED_NETWORKS) or addr.is_private or addr.is_loopback or addr.is_link_local
|
||||
|
||||
|
||||
def validate_scheme(url: str) -> None:
|
||||
parsed = urlparse(url)
|
||||
if parsed.scheme not in {"http", "https"}:
|
||||
raise UnsafeUrlError("Nur http/https URLs sind erlaubt")
|
||||
if not parsed.hostname:
|
||||
raise UnsafeUrlError("URL enthält keinen Hostnamen")
|
||||
if parsed.username or parsed.password:
|
||||
raise UnsafeUrlError("Zugangsdaten in URLs sind nicht erlaubt")
|
||||
|
||||
|
||||
async def resolve_host(hostname: str, port: int | None = None) -> list[str]:
|
||||
def _resolve() -> list[str]:
|
||||
infos = socket.getaddrinfo(hostname, port or 443, type=socket.SOCK_STREAM)
|
||||
return sorted({str(info[4][0]) for info in infos})
|
||||
|
||||
return await asyncio.to_thread(_resolve)
|
||||
|
||||
|
||||
async def assert_public_host(url: str) -> None:
|
||||
validate_scheme(url)
|
||||
parsed = urlparse(url)
|
||||
assert parsed.hostname is not None
|
||||
try:
|
||||
ips = await resolve_host(parsed.hostname, parsed.port)
|
||||
except socket.gaierror as exc:
|
||||
raise UnsafeUrlError(f"Hostname kann nicht aufgelöst werden: {parsed.hostname}") from exc
|
||||
if not ips:
|
||||
raise UnsafeUrlError("Hostname hat keine IP-Adressen")
|
||||
blocked = [ip for ip in ips if _is_blocked_ip(ip)]
|
||||
if blocked:
|
||||
raise UnsafeUrlError("URL zeigt auf interne/private IP-Adresse")
|
||||
|
||||
|
||||
def _content_type_allowed(content_type: str | None, url: str) -> bool:
|
||||
if not content_type:
|
||||
return any(urlparse(url).path.lower().endswith(ext) for ext in ALLOWED_EXTENSIONS)
|
||||
clean = content_type.split(";", 1)[0].strip().lower()
|
||||
if clean.startswith(ALLOWED_CONTENT_PREFIXES):
|
||||
return True
|
||||
if clean in ALLOWED_CONTENT_TYPES and any(urlparse(url).path.lower().endswith(ext) for ext in ALLOWED_EXTENSIONS):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
async def check_url_for_page(url: str, *, timeout: float = 15.0, max_redirects: int = 8) -> CheckedUrl:
|
||||
"""Validate URL and redirects for page analysis without requiring media Content-Type."""
|
||||
current = str(url)
|
||||
chain: list[str] = []
|
||||
async with httpx.AsyncClient(follow_redirects=False, timeout=timeout) as client:
|
||||
for _ in range(max_redirects + 1):
|
||||
await assert_public_host(current)
|
||||
chain.append(current)
|
||||
try:
|
||||
response = await client.head(current, headers=BROWSER_LIKE_HEADERS)
|
||||
except httpx.TimeoutException:
|
||||
response = await client.get(current, headers={**BROWSER_LIKE_MEDIA_HEADERS, "Range": "bytes=0-0"})
|
||||
if response.status_code in {405, 403}:
|
||||
response = await client.get(current, headers={**BROWSER_LIKE_MEDIA_HEADERS, "Range": "bytes=0-0"})
|
||||
if response.is_redirect:
|
||||
loc = response.headers.get("location")
|
||||
if not loc:
|
||||
raise UnsafeUrlError("Redirect ohne Location-Header")
|
||||
current = str(response.url.join(loc))
|
||||
continue
|
||||
response.raise_for_status()
|
||||
await assert_public_host(str(response.url))
|
||||
content_type = response.headers.get("content-type")
|
||||
length_header = response.headers.get("content-length")
|
||||
content_length = int(length_header) if length_header and length_header.isdigit() else None
|
||||
return CheckedUrl(str(response.url), chain, content_type, content_length)
|
||||
raise UnsafeUrlError("Zu viele Weiterleitungen")
|
||||
|
||||
|
||||
async def check_url_for_download(url: str, *, max_bytes: int, timeout: float = 15.0, max_redirects: int = 8) -> CheckedUrl:
|
||||
"""Validate URL, every redirect hop, final target, content type, and size.
|
||||
|
||||
This is the no-domain-allowlist SSRF guard for Mediathek/direct links.
|
||||
"""
|
||||
current = str(url)
|
||||
chain: list[str] = []
|
||||
async with httpx.AsyncClient(follow_redirects=False, timeout=timeout) as client:
|
||||
for _ in range(max_redirects + 1):
|
||||
await assert_public_host(current)
|
||||
chain.append(current)
|
||||
try:
|
||||
response = await client.head(current, headers=BROWSER_LIKE_HEADERS)
|
||||
except httpx.TimeoutException:
|
||||
response = await client.get(current, headers={**BROWSER_LIKE_MEDIA_HEADERS, "Range": "bytes=0-0"})
|
||||
if response.status_code in {405, 403}:
|
||||
response = await client.get(current, headers={**BROWSER_LIKE_MEDIA_HEADERS, "Range": "bytes=0-0"})
|
||||
if response.is_redirect:
|
||||
loc = response.headers.get("location")
|
||||
if not loc:
|
||||
raise UnsafeUrlError("Redirect ohne Location-Header")
|
||||
current = str(response.url.join(loc))
|
||||
continue
|
||||
response.raise_for_status()
|
||||
await assert_public_host(str(response.url))
|
||||
content_type = response.headers.get("content-type")
|
||||
length_header = response.headers.get("content-length")
|
||||
content_length = int(length_header) if length_header and length_header.isdigit() else None
|
||||
if content_length is not None and content_length > max_bytes:
|
||||
raise UnsafeUrlError("Datei ist größer als das konfigurierte Limit")
|
||||
if not _content_type_allowed(content_type, str(response.url)):
|
||||
raise UnsafeUrlError("Ziel liefert keinen plausiblen Medien-Content-Type")
|
||||
return CheckedUrl(str(response.url), chain, content_type, content_length)
|
||||
raise UnsafeUrlError("Zu viele Weiterleitungen")
|
||||
477
runtime-current/kino/app-src/backend/app/web/app.js
Normal file
477
runtime-current/kino/app-src/backend/app/web/app.js
Normal file
@@ -0,0 +1,477 @@
|
||||
const loginSection = document.querySelector('#login-section');
|
||||
const loginForm = document.querySelector('#login-form');
|
||||
const loginMessage = document.querySelector('#login-message');
|
||||
const logoutButton = document.querySelector('#logout');
|
||||
const form = document.querySelector('#probe-form');
|
||||
const result = document.querySelector('#result');
|
||||
const candidatesSection = document.querySelector('#candidates');
|
||||
const importsSection = document.querySelector('#imports');
|
||||
const analyzeButton = document.querySelector('#analyze-page');
|
||||
const interactiveButton = document.querySelector('#interactive-analyze');
|
||||
function escapeHtml(value) {
|
||||
return String(value ?? '').replace(/[&<>'"]/g, (char) => ({'&': '&', '<': '<', '>': '>', "'": ''', '"': '"'}[char]));
|
||||
}
|
||||
|
||||
function setLoginVisible(visible) {
|
||||
loginSection.classList.toggle('hidden', !visible);
|
||||
form.classList.toggle('hidden', visible);
|
||||
logoutButton.classList.toggle('hidden', visible);
|
||||
}
|
||||
|
||||
function renderJob(job) {
|
||||
const percent = Math.round((Number(job.progress) || 0) * 100);
|
||||
if (job.status === 'done') {
|
||||
return `Job #${job.id}: fertig (${percent}%)\nDatei übertragen.`;
|
||||
}
|
||||
const labels = {
|
||||
queued: 'wartet',
|
||||
downloading: 'Download',
|
||||
postprocessing: 'Verarbeitung',
|
||||
copying: 'Kopiervorgang',
|
||||
refreshing: 'Kopiervorgang / Jellyfin aktualisieren',
|
||||
failed: 'Fehlgeschlagen',
|
||||
cancelling: 'Abbruch läuft',
|
||||
cancelled: 'Abgebrochen',
|
||||
};
|
||||
const lines = [`Job #${job.id}: ${labels[job.status] || job.status} (${percent}%)`];
|
||||
if (job.status === 'failed' && job.error) lines.push(`Fehler: ${job.error}`);
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
function decodeCapturePayload(value) {
|
||||
return JSON.parse(decodeURIComponent(escape(atob(value))));
|
||||
}
|
||||
|
||||
const terminalStatuses = new Set(['done', 'failed', 'cancelled']);
|
||||
const activeStatuses = new Set(['queued', 'downloading', 'postprocessing', 'copying', 'refreshing', 'cancelling']);
|
||||
const hideDoneAfterMs = 5 * 60 * 1000;
|
||||
|
||||
function statusLabel(status) {
|
||||
return {
|
||||
queued: 'Wartet',
|
||||
downloading: 'Download',
|
||||
postprocessing: 'Verarbeitung',
|
||||
copying: 'Kopiervorgang',
|
||||
refreshing: 'Kopiervorgang',
|
||||
done: 'Fertig',
|
||||
failed: 'Fehlgeschlagen',
|
||||
cancelling: 'Abbruch läuft',
|
||||
cancelled: 'Abgebrochen',
|
||||
}[status] || status;
|
||||
}
|
||||
|
||||
function renderImports(jobs) {
|
||||
importsSection.className = 'card';
|
||||
if (!jobs.length) {
|
||||
importsSection.innerHTML = '<h2>Importe</h2><p class="muted">Noch keine Importe.</p>';
|
||||
return;
|
||||
}
|
||||
importsSection.innerHTML = '<h2>Importe</h2>';
|
||||
const visibleJobs = jobs.filter((job) => {
|
||||
if (job.status !== 'done') return true;
|
||||
const updated = Date.parse(job.updated_at);
|
||||
return Number.isNaN(updated) || Date.now() - updated < hideDoneAfterMs;
|
||||
});
|
||||
if (!visibleJobs.length) {
|
||||
importsSection.innerHTML += '<p class="muted">Keine sichtbaren Importe. Fertige Importe werden nach fünf Minuten automatisch ausgeblendet.</p>';
|
||||
return;
|
||||
}
|
||||
visibleJobs.forEach((job) => {
|
||||
const percent = Math.round((Number(job.progress) || 0) * 100);
|
||||
const card = document.createElement('div');
|
||||
card.className = `import-job status-${escapeHtml(job.status)}`;
|
||||
card.innerHTML = `
|
||||
<div class="import-header">
|
||||
<strong>#${escapeHtml(job.id)} · ${escapeHtml(job.title || 'Unbenannter Import')} · ${escapeHtml(statusLabel(job.status))}</strong>
|
||||
<span class="muted">${percent}% · ${escapeHtml(job.target_library || '')}</span>
|
||||
</div>
|
||||
<div class="progress" aria-label="Fortschritt ${percent}%"><span style="width:${Math.max(0, Math.min(percent, 100))}%"></span></div>
|
||||
${job.error ? `<p class="error">${escapeHtml(job.error)}</p>` : ''}
|
||||
${job.target_path && job.status === 'done' ? '<p class="muted">Datei per MD5 auf Jellyfin verifiziert. Lokale Download-Daten wurden bereinigt.</p>' : ''}
|
||||
<div class="import-actions"></div>
|
||||
`;
|
||||
const actions = card.querySelector('.import-actions');
|
||||
if (activeStatuses.has(job.status) && job.status !== 'cancelling') {
|
||||
const cancel = document.createElement('button');
|
||||
cancel.type = 'button';
|
||||
cancel.textContent = 'Abbrechen';
|
||||
cancel.addEventListener('click', () => importAction(job.id, 'cancel'));
|
||||
actions.appendChild(cancel);
|
||||
}
|
||||
if (terminalStatuses.has(job.status)) {
|
||||
const retry = document.createElement('button');
|
||||
retry.type = 'button';
|
||||
retry.textContent = 'Wiederholen';
|
||||
retry.addEventListener('click', () => importAction(job.id, 'retry'));
|
||||
actions.appendChild(retry);
|
||||
if (job.status === 'failed') {
|
||||
const alternative = document.createElement('button');
|
||||
alternative.type = 'button';
|
||||
alternative.textContent = 'Alternative Fehlerbehebung';
|
||||
alternative.addEventListener('click', () => importAction(job.id, 'retry-alternative'));
|
||||
actions.appendChild(alternative);
|
||||
}
|
||||
const remove = document.createElement('button');
|
||||
remove.type = 'button';
|
||||
remove.textContent = 'Aus Liste löschen';
|
||||
remove.addEventListener('click', () => importAction(job.id, 'delete'));
|
||||
actions.appendChild(remove);
|
||||
}
|
||||
importsSection.appendChild(card);
|
||||
});
|
||||
}
|
||||
|
||||
async function refreshImports() {
|
||||
const response = await fetch('/api/imports');
|
||||
if (response.status === 401) {
|
||||
await refreshAuthStatus();
|
||||
return;
|
||||
}
|
||||
const jobs = await response.json();
|
||||
if (!response.ok) throw new Error(jobs.detail || 'Importstatus konnte nicht gelesen werden');
|
||||
renderImports(jobs);
|
||||
}
|
||||
|
||||
async function importAction(jobId, action) {
|
||||
try {
|
||||
let endpoint = `/api/imports/${jobId}`;
|
||||
const options = {method: 'DELETE'};
|
||||
if (action !== 'delete') {
|
||||
endpoint += `/${action}`;
|
||||
options.method = 'POST';
|
||||
}
|
||||
const response = await fetch(endpoint, options);
|
||||
if (!response.ok) {
|
||||
const data = await response.json().catch(() => ({}));
|
||||
throw new Error(data.detail || 'Import-Aktion fehlgeschlagen');
|
||||
}
|
||||
await refreshImports();
|
||||
} catch (err) {
|
||||
result.className = 'card';
|
||||
result.innerHTML = `<p class="error">${escapeHtml(err.message)}</p>`;
|
||||
}
|
||||
}
|
||||
|
||||
function startImportPolling() {
|
||||
refreshImports().catch(() => {});
|
||||
setInterval(() => refreshImports().catch(() => {}), 2000);
|
||||
}
|
||||
|
||||
async function pollJob(jobId) {
|
||||
const output = document.querySelector('#job-output');
|
||||
for (;;) {
|
||||
const response = await fetch(`/api/imports/${jobId}`);
|
||||
const job = await response.json();
|
||||
if (response.status === 401) {
|
||||
await refreshAuthStatus();
|
||||
throw new Error('Bitte zuerst einloggen.');
|
||||
}
|
||||
if (!response.ok) throw new Error(job.detail || 'Job-Status konnte nicht gelesen werden');
|
||||
output.textContent = renderJob(job);
|
||||
if (['done', 'failed'].includes(job.status)) return job;
|
||||
await new Promise((resolve) => setTimeout(resolve, 2000));
|
||||
}
|
||||
}
|
||||
|
||||
async function refreshAuthStatus() {
|
||||
const response = await fetch('/api/auth/status');
|
||||
const status = await response.json();
|
||||
setLoginVisible(status.enabled && !status.authenticated);
|
||||
}
|
||||
|
||||
loginForm.addEventListener('submit', async (event) => {
|
||||
event.preventDefault();
|
||||
loginMessage.textContent = 'Melde an…';
|
||||
const username = document.querySelector('#username').value;
|
||||
const password = document.querySelector('#password').value;
|
||||
const response = await fetch('/api/auth/login', {
|
||||
method: 'POST',
|
||||
headers: {'content-type': 'application/json'},
|
||||
body: JSON.stringify({username, password}),
|
||||
});
|
||||
if (!response.ok) {
|
||||
loginMessage.textContent = 'Login fehlgeschlagen.';
|
||||
return;
|
||||
}
|
||||
loginMessage.textContent = '';
|
||||
await refreshAuthStatus();
|
||||
});
|
||||
|
||||
logoutButton.addEventListener('click', async () => {
|
||||
await fetch('/api/auth/logout', {method: 'POST'});
|
||||
await refreshAuthStatus();
|
||||
});
|
||||
|
||||
function currentImportPayload(url) {
|
||||
const payload = {
|
||||
url,
|
||||
target_profile: document.querySelector('input[name="target-profile"]:checked').value,
|
||||
};
|
||||
const textFields = [
|
||||
['title', 'title'],
|
||||
['series-title', 'series_title'],
|
||||
['episode-title', 'episode_title'],
|
||||
];
|
||||
for (const [elementId, key] of textFields) {
|
||||
const value = document.querySelector(`#${elementId}`).value.trim();
|
||||
if (value) payload[key] = value;
|
||||
}
|
||||
const numberFields = [
|
||||
['year', 'year'],
|
||||
['season', 'season'],
|
||||
['episode', 'episode'],
|
||||
];
|
||||
for (const [elementId, key] of numberFields) {
|
||||
const value = document.querySelector(`#${elementId}`).value;
|
||||
if (value) payload[key] = Number(value);
|
||||
}
|
||||
return payload;
|
||||
}
|
||||
|
||||
function isImportableCandidate(candidate) {
|
||||
return candidate && candidate.allowed && ['direct_video', 'hls_manifest', 'dash_manifest'].includes(candidate.kind);
|
||||
}
|
||||
|
||||
function candidateScore(candidate) {
|
||||
const kindPriority = {direct_video: 3, hls_manifest: 2, dash_manifest: 2}[candidate.kind] || 0;
|
||||
return (kindPriority * 1_000_000_000_000) + (Number(candidate.content_length) || 0);
|
||||
}
|
||||
|
||||
function bestImportableCandidate(candidates) {
|
||||
return (candidates || []).filter(isImportableCandidate).sort((a, b) => candidateScore(b) - candidateScore(a))[0] || null;
|
||||
}
|
||||
|
||||
async function startImportForUrl(url, title) {
|
||||
result.className = 'card';
|
||||
result.innerHTML = `<h2>Download gestartet</h2><p>${escapeHtml(title || url)}</p><pre id="job-output">Starte Import…</pre>`;
|
||||
const payload = currentImportPayload(url);
|
||||
if (title && !payload.title) payload.title = title;
|
||||
const jobResponse = await fetch('/api/imports', {method: 'POST', headers: {'content-type': 'application/json'}, body: JSON.stringify(payload)});
|
||||
const job = await jobResponse.json();
|
||||
if (jobResponse.status === 401) {
|
||||
await refreshAuthStatus();
|
||||
throw new Error('Bitte zuerst einloggen.');
|
||||
}
|
||||
if (!jobResponse.ok) throw new Error(job.detail || 'Import konnte nicht gestartet werden');
|
||||
document.querySelector('#job-output').textContent = renderJob(job);
|
||||
await refreshImports();
|
||||
}
|
||||
|
||||
form.addEventListener('submit', async (event) => {
|
||||
event.preventDefault();
|
||||
const url = document.querySelector('#url').value;
|
||||
result.className = 'card';
|
||||
result.textContent = 'Prüfe Quelle…';
|
||||
try {
|
||||
const response = await fetch('/api/probe', {method: 'POST', headers: {'content-type': 'application/json'}, body: JSON.stringify({url})});
|
||||
const data = await response.json();
|
||||
if (response.status === 401) {
|
||||
await refreshAuthStatus();
|
||||
throw new Error('Bitte zuerst einloggen.');
|
||||
}
|
||||
if (!response.ok) throw new Error(data.detail || 'Probe fehlgeschlagen');
|
||||
result.innerHTML = `
|
||||
<h2>${escapeHtml(data.title || 'Unbenannte Quelle')}</h2>
|
||||
<p>Provider: ${escapeHtml(data.provider)}</p>
|
||||
<p>Dauer: ${escapeHtml(data.duration_seconds ?? 'unbekannt')}</p>
|
||||
<button id="start-import" type="button">Import starten</button>
|
||||
<pre id="job-output"></pre>`;
|
||||
document.querySelector('#start-import').addEventListener('click', async () => {
|
||||
await startImportForUrl(url, data.title);
|
||||
});
|
||||
} catch (err) {
|
||||
result.innerHTML = `<p class="error">${escapeHtml(err.message)}</p>`;
|
||||
}
|
||||
});
|
||||
|
||||
analyzeButton.addEventListener('click', async () => {
|
||||
const url = document.querySelector('#url').value;
|
||||
result.className = 'card';
|
||||
candidatesSection.className = 'card hidden';
|
||||
candidatesSection.innerHTML = '';
|
||||
result.textContent = 'Starte serverseitige Analyse mit Vorschau…';
|
||||
try {
|
||||
const response = await fetch('/api/interactive-analyze', {method: 'POST', headers: {'content-type': 'application/json'}, body: JSON.stringify({url})});
|
||||
const data = await response.json();
|
||||
if (response.status === 401) {
|
||||
await refreshAuthStatus();
|
||||
throw new Error('Bitte zuerst einloggen.');
|
||||
}
|
||||
if (!response.ok) throw new Error(data.detail || 'Analyse konnte nicht gestartet werden');
|
||||
renderInteractive(data);
|
||||
} catch (err) {
|
||||
result.innerHTML = `<p class="error">${escapeHtml(err.message)}</p>`;
|
||||
}
|
||||
});
|
||||
|
||||
let interactiveSessionId = null;
|
||||
|
||||
function renderInteractive(data) {
|
||||
interactiveSessionId = data.session_id;
|
||||
const count = (data.candidates || []).length;
|
||||
result.className = 'card interactive-card';
|
||||
result.innerHTML = `
|
||||
<h2>Interaktive Serveranalyse</h2>
|
||||
<p>Die Seite läuft in einem kurzlebigen Server-Browser. Klicke in die Vorschau oder nutze Play/Pause; Medien-Requests werden während der Wiedergabe erneut ausgewertet.</p>
|
||||
<p>${count} Kandidat(en) gefunden.${data.ublock_origin ? ' uBlock Origin ist aktiv.' : ''}</p>
|
||||
<div class="interactive-toolbar">
|
||||
<button id="interactive-space" type="button">Play/Pause (Leertaste)</button>
|
||||
<button id="interactive-refresh" type="button">Kandidaten aktualisieren</button>
|
||||
<button id="interactive-stop" type="button">Session beenden</button>
|
||||
</div>
|
||||
<img id="interactive-screenshot" class="interactive-screenshot" src="${escapeHtml(data.screenshot)}" alt="Interaktive Analyse-Vorschau" />
|
||||
`;
|
||||
const screenshot = document.querySelector('#interactive-screenshot');
|
||||
screenshot.addEventListener('click', async (event) => {
|
||||
const rect = screenshot.getBoundingClientRect();
|
||||
const x = (event.clientX - rect.left) * (data.viewport.width / rect.width);
|
||||
const y = (event.clientY - rect.top) * (data.viewport.height / rect.height);
|
||||
await interactiveAction('click', {x, y});
|
||||
});
|
||||
document.querySelector('#interactive-space').addEventListener('click', () => interactiveAction('key', {key: 'Space'}));
|
||||
document.querySelector('#interactive-refresh').addEventListener('click', () => interactiveAction('key', {key: 'Shift'}));
|
||||
document.querySelector('#interactive-stop').addEventListener('click', stopInteractive);
|
||||
renderCandidates(data.candidates || [], {showBestDownload: true});
|
||||
}
|
||||
|
||||
async function interactiveAction(action, payload) {
|
||||
if (!interactiveSessionId) return;
|
||||
result.classList.add('loading');
|
||||
try {
|
||||
const endpoint = action === 'click'
|
||||
? `/api/interactive-analyze/${interactiveSessionId}/click`
|
||||
: `/api/interactive-analyze/${interactiveSessionId}/key`;
|
||||
const response = await fetch(endpoint, {method: 'POST', headers: {'content-type': 'application/json'}, body: JSON.stringify(payload)});
|
||||
const data = await response.json();
|
||||
if (!response.ok) throw new Error(data.detail || 'Interaktive Aktion fehlgeschlagen');
|
||||
renderInteractive(data);
|
||||
} catch (err) {
|
||||
result.innerHTML += `<p class="error">${escapeHtml(err.message)}</p>`;
|
||||
} finally {
|
||||
result.classList.remove('loading');
|
||||
}
|
||||
}
|
||||
|
||||
async function stopInteractive() {
|
||||
if (!interactiveSessionId) return;
|
||||
await fetch(`/api/interactive-analyze/${interactiveSessionId}`, {method: 'DELETE'}).catch(() => {});
|
||||
interactiveSessionId = null;
|
||||
result.className = 'card';
|
||||
result.textContent = 'Interaktive Analyse beendet.';
|
||||
}
|
||||
|
||||
interactiveButton.addEventListener('click', async () => {
|
||||
const url = document.querySelector('#url').value;
|
||||
result.className = 'card';
|
||||
candidatesSection.className = 'card hidden';
|
||||
candidatesSection.innerHTML = '';
|
||||
result.textContent = 'Starte serverseitigen Analyse-Browser…';
|
||||
try {
|
||||
const response = await fetch('/api/interactive-analyze', {method: 'POST', headers: {'content-type': 'application/json'}, body: JSON.stringify({url})});
|
||||
const data = await response.json();
|
||||
if (response.status === 401) {
|
||||
await refreshAuthStatus();
|
||||
throw new Error('Bitte zuerst einloggen.');
|
||||
}
|
||||
if (!response.ok) throw new Error(data.detail || 'Interaktive Analyse konnte nicht gestartet werden');
|
||||
renderInteractive(data);
|
||||
} catch (err) {
|
||||
result.innerHTML = `<p class="error">${escapeHtml(err.message)}</p>`;
|
||||
}
|
||||
});
|
||||
|
||||
function renderCandidates(candidates, options = {}) {
|
||||
candidatesSection.className = 'card';
|
||||
const importable = (candidates || []).filter(isImportableCandidate);
|
||||
const best = bestImportableCandidate(candidates);
|
||||
if (!candidates.length) {
|
||||
candidatesSection.innerHTML = '<h2>Gefundene Medienquellen</h2><p>Noch keine direkt importierbare Medienquelle gefunden. Starte das Video in der Vorschau und drücke dann „Kandidaten aktualisieren“ oder Play/Pause.</p>';
|
||||
return;
|
||||
}
|
||||
candidatesSection.innerHTML = '<h2>Gefundene Medienquellen</h2>';
|
||||
if (options.showBestDownload && best) {
|
||||
const bestBox = document.createElement('div');
|
||||
bestBox.className = 'candidate best-candidate';
|
||||
bestBox.innerHTML = `
|
||||
<div>
|
||||
<strong>Bester Kandidat: ${escapeHtml(best.title || best.url)}</strong>
|
||||
<p>${escapeHtml([best.kind, best.source, best.quality, best.file_size || (best.content_length ? formatBytes(best.content_length) : '')].filter(Boolean).join(' · '))}</p>
|
||||
</div>
|
||||
<button id="download-best" type="button">Besten Kandidaten herunterladen</button>
|
||||
`;
|
||||
bestBox.querySelector('#download-best').addEventListener('click', async () => {
|
||||
try {
|
||||
await startImportForUrl(best.url, best.title);
|
||||
} catch (err) {
|
||||
result.innerHTML = `<p class="error">${escapeHtml(err.message)}</p>`;
|
||||
}
|
||||
});
|
||||
candidatesSection.appendChild(bestBox);
|
||||
} else if (options.showBestDownload && !importable.length) {
|
||||
const hint = document.createElement('p');
|
||||
hint.textContent = 'Es wurden nur Diagnose-/Blockier-Kandidaten gefunden. Bitte Video in der Vorschau starten und erneut aktualisieren.';
|
||||
candidatesSection.appendChild(hint);
|
||||
}
|
||||
candidates.forEach((candidate, idx) => {
|
||||
const card = document.createElement('div');
|
||||
card.className = `candidate ${candidate.allowed ? '' : 'blocked'}`;
|
||||
const size = candidate.file_size || (candidate.content_length ? formatBytes(candidate.content_length) : 'Dateigröße unbekannt');
|
||||
const details = [candidate.kind, candidate.source, candidate.quality, `Dateigröße: ${size}`].filter(Boolean).join(' · ');
|
||||
card.innerHTML = `
|
||||
<div>
|
||||
<strong>${escapeHtml(candidate.title || `Kandidat ${idx + 1}`)}</strong>
|
||||
<p>${escapeHtml(details)}</p>
|
||||
<code>${escapeHtml(candidate.url)}</code>
|
||||
${candidate.reason ? `<p class="error">${escapeHtml(candidate.reason)}</p>` : ''}
|
||||
</div>
|
||||
<button ${candidate.allowed ? '' : 'disabled'} data-url="${escapeHtml(candidate.url)}" data-title="${escapeHtml(candidate.title || '')}">Übernehmen</button>
|
||||
`;
|
||||
card.querySelector('button').addEventListener('click', (event) => {
|
||||
document.querySelector('#url').value = event.currentTarget.dataset.url;
|
||||
if (event.currentTarget.dataset.title && !document.querySelector('#title').value) {
|
||||
document.querySelector('#title').value = event.currentTarget.dataset.title;
|
||||
}
|
||||
result.className = 'card';
|
||||
result.textContent = 'Kandidat übernommen. Du kannst jetzt Quelle prüfen oder Import starten.';
|
||||
});
|
||||
candidatesSection.appendChild(card);
|
||||
});
|
||||
}
|
||||
|
||||
function formatBytes(bytes) {
|
||||
const units = ['B', 'KB', 'MB', 'GB'];
|
||||
let value = Number(bytes || 0);
|
||||
for (const unit of units) {
|
||||
if (value < 1024 || unit === 'GB') return unit === 'B' ? `${value} ${unit}` : `${value.toFixed(1)} ${unit}`;
|
||||
value /= 1024;
|
||||
}
|
||||
}
|
||||
|
||||
async function importCapturedFromUrl() {
|
||||
const params = new URLSearchParams(location.search);
|
||||
const encoded = params.get('capture') || (location.hash.startsWith('#capture=') ? location.hash.slice(9) : '');
|
||||
if (!encoded) return;
|
||||
try {
|
||||
const payload = decodeCapturePayload(encoded);
|
||||
result.className = 'card';
|
||||
result.textContent = 'Browser-Capture wird geprüft…';
|
||||
const response = await fetch('/api/browser-capture', {method: 'POST', headers: {'content-type': 'application/json'}, body: JSON.stringify(payload)});
|
||||
const data = await response.json();
|
||||
if (response.status === 401) {
|
||||
await refreshAuthStatus();
|
||||
throw new Error('Bitte zuerst einloggen.');
|
||||
}
|
||||
if (!response.ok) throw new Error(data.detail || 'Browser-Capture konnte nicht geprüft werden');
|
||||
document.querySelector('#url').value = payload.page_url || '';
|
||||
result.textContent = `${data.candidates.length} Kandidat(en) aus Browser-Capture gefunden.`;
|
||||
renderCandidates(data.candidates || []);
|
||||
history.replaceState(null, '', '/');
|
||||
} catch (err) {
|
||||
result.className = 'card';
|
||||
result.innerHTML = `<p class="error">${escapeHtml(err.message)}</p>`;
|
||||
}
|
||||
}
|
||||
|
||||
refreshAuthStatus().catch(() => setLoginVisible(false));
|
||||
startImportPolling();
|
||||
importCapturedFromUrl();
|
||||
Binary file not shown.
60
runtime-current/kino/app-src/backend/app/web/index.html
Normal file
60
runtime-current/kino/app-src/backend/app/web/index.html
Normal file
@@ -0,0 +1,60 @@
|
||||
<!doctype html>
|
||||
<html lang="de">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>Kino-Projekt</title>
|
||||
<link rel="stylesheet" href="/static/style.css?v=import-cancel-md5-20260714" />
|
||||
</head>
|
||||
<body>
|
||||
<main>
|
||||
<h1>Kino-Projekt</h1>
|
||||
<p>Interner Import für YouTube, legale Mediatheken und direkte erlaubte Medienlinks.</p>
|
||||
<section id="login-section" class="card hidden">
|
||||
<h2>Login</h2>
|
||||
<form id="login-form">
|
||||
<input id="username" autocomplete="username" placeholder="Benutzername" required />
|
||||
<input id="password" type="password" autocomplete="current-password" placeholder="Passwort" required />
|
||||
<button type="submit">Einloggen</button>
|
||||
</form>
|
||||
<p id="login-message" class="error"></p>
|
||||
</section>
|
||||
<button id="logout" type="button" class="hidden">Abmelden</button>
|
||||
<form id="probe-form">
|
||||
<label>
|
||||
Quelle
|
||||
<input id="url" type="url" placeholder="https://…" required />
|
||||
</label>
|
||||
<fieldset>
|
||||
<legend>Zielprofil</legend>
|
||||
<label><input type="radio" name="target-profile" value="clip" checked /> YouTube/Mediathek-Clip</label>
|
||||
<label><input type="radio" name="target-profile" value="film" /> Film</label>
|
||||
<label><input type="radio" name="target-profile" value="serie" /> Serie/Episode</label>
|
||||
</fieldset>
|
||||
<div class="grid">
|
||||
<label>Titel-Korrektur <input id="title" placeholder="optional" /></label>
|
||||
<label>Jahr <input id="year" type="number" min="1888" max="2200" placeholder="optional" /></label>
|
||||
<label>Serientitel <input id="series-title" placeholder="nur Serie" /></label>
|
||||
<label>Staffel <input id="season" type="number" min="1" placeholder="1" /></label>
|
||||
<label>Episode <input id="episode" type="number" min="1" placeholder="optional" /></label>
|
||||
<label>Episodentitel <input id="episode-title" placeholder="optional" /></label>
|
||||
</div>
|
||||
<button type="submit">Quelle prüfen</button>
|
||||
<button id="analyze-page" type="button">Analysieren</button>
|
||||
<button id="interactive-analyze" type="button" class="hidden">Interaktive Analyse starten</button>
|
||||
</form>
|
||||
<section id="analyzer-tools" class="card">
|
||||
<h2>Ablauf</h2>
|
||||
<p>Link einfügen und „Analysieren“ klicken: Die Seite wird serverseitig als Vorschau geöffnet. Findet die Analyse einen importierbaren Kandidaten, erscheint direkt der Button „Besten Kandidaten herunterladen“. Falls noch nichts gefunden wurde, starte das Video in der Vorschau und aktualisiere die Kandidaten während der Wiedergabe.</p>
|
||||
</section>
|
||||
<section id="result" class="card hidden"></section>
|
||||
<section id="candidates" class="card hidden"></section>
|
||||
<section id="imports" class="card">
|
||||
<h2>Importe</h2>
|
||||
<p>Lade Importstatus…</p>
|
||||
</section>
|
||||
<button id="import" disabled>Import starten</button>
|
||||
</main>
|
||||
<script src="/static/app.js?v=import-cancel-md5-20260714"></script>
|
||||
</body>
|
||||
</html>
|
||||
37
runtime-current/kino/app-src/backend/app/web/style.css
Normal file
37
runtime-current/kino/app-src/backend/app/web/style.css
Normal file
@@ -0,0 +1,37 @@
|
||||
body { font-family: system-ui, sans-serif; margin: 0; background: #111827; color: #f9fafb; }
|
||||
main { max-width: 860px; margin: 4rem auto; padding: 2rem; }
|
||||
form { display: flex; gap: .75rem; flex-wrap: wrap; }
|
||||
label { display: flex; flex: 1 1 220px; flex-direction: column; gap: .35rem; }
|
||||
fieldset { flex: 1 1 100%; border: 1px solid #374151; border-radius: .75rem; display: flex; gap: 1rem; flex-wrap: wrap; }
|
||||
fieldset label { flex: 0 1 auto; flex-direction: row; align-items: center; }
|
||||
input { flex: 1; padding: .8rem; border-radius: .5rem; border: 1px solid #374151; background: #1f2937; color: #fff; }
|
||||
input[type="radio"] { flex: 0 0 auto; }
|
||||
button { padding: .8rem 1rem; border: 0; border-radius: .5rem; background: #38bdf8; color: #082f49; font-weight: 700; }
|
||||
button:disabled { opacity: .45; cursor: not-allowed; }
|
||||
.grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); gap: .75rem; width: 100%; }
|
||||
.card { margin-top: 1rem; padding: 1rem; border: 1px solid #374151; border-radius: .75rem; background: #1f2937; }
|
||||
.hidden { display: none; }
|
||||
.error { color: #fca5a5; }
|
||||
.candidate { margin-top: .75rem; padding-top: .75rem; border-top: 1px solid #374151; display: grid; gap: .75rem; }
|
||||
.candidate code { display: block; overflow-wrap: anywhere; color: #bae6fd; }
|
||||
.candidate.blocked { opacity: .7; }
|
||||
.best-candidate { border: 1px solid #38bdf8; border-radius: .75rem; padding: 1rem; background: #0f2f3f; }
|
||||
.import-job { border-top: 1px solid #374151; padding: .9rem 0; display: grid; gap: .5rem; }
|
||||
.import-job:first-of-type { border-top: 0; }
|
||||
.import-header { display: flex; justify-content: space-between; gap: .75rem; flex-wrap: wrap; }
|
||||
.import-actions { display: flex; gap: .5rem; flex-wrap: wrap; }
|
||||
.import-actions button { padding: .55rem .75rem; }
|
||||
.progress { height: .7rem; border-radius: 999px; background: #111827; overflow: hidden; border: 1px solid #374151; }
|
||||
.progress > span { display: block; height: 100%; background: #38bdf8; }
|
||||
.status-done .progress > span { background: #22c55e; }
|
||||
.status-failed .progress > span { background: #ef4444; }
|
||||
.muted { color: #9ca3af; }
|
||||
.tool-actions { display: flex; flex-wrap: wrap; align-items: center; gap: .75rem; }
|
||||
details { margin-top: .75rem; }
|
||||
details code { color: #bae6fd; }
|
||||
.interactive-toolbar { display: flex; flex-wrap: wrap; gap: .75rem; margin: .75rem 0; }
|
||||
.interactive-screenshot { width: 100%; max-height: 70vh; object-fit: contain; border: 1px solid #374151; border-radius: .5rem; background: #000; cursor: crosshair; }
|
||||
.loading { opacity: .8; }
|
||||
|
||||
.status-cancelling, .status-cancelled { border-color: #f59e0b; }
|
||||
.import-actions button { margin-right: .5rem; margin-top: .35rem; }
|
||||
23
runtime-current/kino/app-src/backend/pyproject.toml
Normal file
23
runtime-current/kino/app-src/backend/pyproject.toml
Normal file
@@ -0,0 +1,23 @@
|
||||
[project]
|
||||
name = "kino-projekt-backend"
|
||||
version = "0.1.0"
|
||||
description = "Legal-source media import dashboard for Jellyfin"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"fastapi>=0.115",
|
||||
"uvicorn[standard]>=0.30",
|
||||
"pydantic-settings>=2.4",
|
||||
"sqlmodel>=0.0.22",
|
||||
"httpx>=0.27",
|
||||
"argon2-cffi>=23.1",
|
||||
"yt-dlp>=2025.1.1",
|
||||
"playwright>=1.49",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
test = ["pytest>=8.0", "pytest-asyncio>=0.23"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
asyncio_mode = "auto"
|
||||
pythonpath = ["."]
|
||||
testpaths = ["tests"]
|
||||
11
runtime-current/kino/app-src/backend/tests/conftest.py
Normal file
11
runtime-current/kino/app-src/backend/tests/conftest.py
Normal file
@@ -0,0 +1,11 @@
|
||||
import pytest
|
||||
|
||||
from app.config import Settings
|
||||
from app.main import app
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def reset_app_settings():
|
||||
app.state.settings = Settings(KINOPROJEKT_DB=":memory:")
|
||||
yield
|
||||
app.state.settings = Settings(KINOPROJEKT_DB=":memory:")
|
||||
@@ -0,0 +1,82 @@
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.main import app
|
||||
from app.services.page_analyzer import MediaCandidate
|
||||
|
||||
|
||||
def test_analyze_api_returns_candidates(monkeypatch):
|
||||
async def fake_analyze_page(url):
|
||||
return [
|
||||
MediaCandidate(
|
||||
url="https://cdn.example.org/movie.mp4",
|
||||
kind="direct_video",
|
||||
source="html",
|
||||
content_length=1536,
|
||||
file_size="1.5 KB",
|
||||
)
|
||||
]
|
||||
|
||||
monkeypatch.setattr("app.main.analyze_page", fake_analyze_page)
|
||||
client = TestClient(app)
|
||||
|
||||
response = client.post("/api/analyze", json={"url": "https://example.org/watch"})
|
||||
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["page_url"] == "https://example.org/watch"
|
||||
assert body["candidates"][0]["file_size"] == "1.5 KB"
|
||||
assert body["candidates"][0]["source"] == "html"
|
||||
|
||||
|
||||
def test_browser_capture_api_returns_candidates(monkeypatch):
|
||||
async def fake_analyze_browser_capture(captured, page_url=None):
|
||||
assert page_url == "https://example.org/watch"
|
||||
assert captured[0]["url"] == "/media/movie.mp4"
|
||||
return [MediaCandidate(url="https://example.org/media/movie.mp4", kind="direct_video", source="browser-capture")]
|
||||
|
||||
monkeypatch.setattr("app.main.analyze_browser_capture", fake_analyze_browser_capture)
|
||||
client = TestClient(app)
|
||||
|
||||
response = client.post(
|
||||
"/api/browser-capture",
|
||||
json={"page_url": "https://example.org/watch", "candidates": [{"url": "/media/movie.mp4"}]},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["candidates"][0]["source"] == "browser-capture"
|
||||
|
||||
|
||||
def test_interactive_analyze_api_returns_screenshot_and_candidates(monkeypatch):
|
||||
class FakePage:
|
||||
url = "https://example.org/watch"
|
||||
|
||||
class FakeSession:
|
||||
id = "session-1"
|
||||
page_url = "https://example.org/watch"
|
||||
page = FakePage()
|
||||
ublock_path = None
|
||||
|
||||
async def fake_start(url):
|
||||
assert url == "https://example.org/watch"
|
||||
return FakeSession()
|
||||
|
||||
async def fake_candidates(session):
|
||||
assert session.id == "session-1"
|
||||
return [MediaCandidate(url="https://cdn.example.org/live.m3u8", kind="hls_manifest", source="interactive-browser")]
|
||||
|
||||
async def fake_screenshot(session):
|
||||
return "data:image/png;base64,ZmFrZQ=="
|
||||
|
||||
monkeypatch.setattr("app.main.interactive_browser_manager.start", fake_start)
|
||||
monkeypatch.setattr("app.main.interactive_browser_manager.candidates", fake_candidates)
|
||||
monkeypatch.setattr("app.main.interactive_browser_manager.screenshot_data_url", fake_screenshot)
|
||||
client = TestClient(app)
|
||||
|
||||
response = client.post("/api/interactive-analyze", json={"url": "https://example.org/watch"})
|
||||
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["session_id"] == "session-1"
|
||||
assert body["screenshot"].startswith("data:image/png;base64,")
|
||||
assert body["candidates"][0]["source"] == "interactive-browser"
|
||||
103
runtime-current/kino/app-src/backend/tests/test_auth.py
Normal file
103
runtime-current/kino/app-src/backend/tests/test_auth.py
Normal file
@@ -0,0 +1,103 @@
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.config import Settings
|
||||
from app.main import app
|
||||
from app.services.auth import hash_password
|
||||
|
||||
|
||||
def configure_auth(password: str = "correct horse battery staple"):
|
||||
app.state.settings = Settings(
|
||||
KINOPROJEKT_ADMIN_USER="admin",
|
||||
KINOPROJEKT_ADMIN_PASSWORD_HASH=hash_password(password),
|
||||
KINOPROJEKT_SESSION_SECRET="test-secret-that-is-long-enough",
|
||||
KINOPROJEKT_DB=":memory:",
|
||||
)
|
||||
return password
|
||||
|
||||
|
||||
def disable_auth():
|
||||
app.state.settings = Settings(KINOPROJEKT_DB=":memory:")
|
||||
|
||||
|
||||
def test_auth_disabled_keeps_mvp_api_accessible():
|
||||
disable_auth()
|
||||
client = TestClient(app)
|
||||
response = client.get("/api/auth/status")
|
||||
assert response.json() == {"enabled": False, "authenticated": True, "username": None}
|
||||
|
||||
|
||||
def test_configured_auth_blocks_api_without_session():
|
||||
configure_auth()
|
||||
client = TestClient(app)
|
||||
response = client.post("/api/probe", json={"url": "http://localhost/video.mp4"})
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
def test_login_sets_secure_httponly_session_and_allows_api():
|
||||
password = configure_auth()
|
||||
client = TestClient(app, base_url="https://testserver")
|
||||
|
||||
login = client.post("/api/auth/login", json={"username": "admin", "password": password})
|
||||
|
||||
assert login.status_code == 204
|
||||
set_cookie = login.headers["set-cookie"]
|
||||
assert "HttpOnly" in set_cookie
|
||||
assert "Secure" in set_cookie
|
||||
assert "SameSite=lax" in set_cookie
|
||||
|
||||
response = client.post("/api/probe", json={"url": "http://localhost/video.mp4"})
|
||||
assert response.status_code == 400
|
||||
assert "private" in response.json()["detail"].lower()
|
||||
|
||||
|
||||
def test_login_omits_secure_cookie_on_plain_http_lan_access():
|
||||
password = configure_auth()
|
||||
client = TestClient(app, base_url="http://testserver")
|
||||
|
||||
login = client.post("/api/auth/login", json={"username": "admin", "password": password})
|
||||
|
||||
assert login.status_code == 204
|
||||
set_cookie = login.headers["set-cookie"]
|
||||
assert "HttpOnly" in set_cookie
|
||||
assert "Secure" not in set_cookie
|
||||
|
||||
|
||||
def test_login_sets_secure_cookie_with_forwarded_https_proto():
|
||||
password = configure_auth()
|
||||
client = TestClient(app, base_url="http://testserver")
|
||||
|
||||
login = client.post(
|
||||
"/api/auth/login",
|
||||
json={"username": "admin", "password": password},
|
||||
headers={"x-forwarded-proto": "https"},
|
||||
)
|
||||
|
||||
assert login.status_code == 204
|
||||
assert "Secure" in login.headers["set-cookie"]
|
||||
|
||||
|
||||
def test_logout_omits_secure_cookie_on_plain_http_lan_access():
|
||||
configure_auth()
|
||||
client = TestClient(app, base_url="http://testserver")
|
||||
|
||||
logout = client.post("/api/auth/logout")
|
||||
|
||||
assert logout.status_code == 204
|
||||
assert "Secure" not in logout.headers["set-cookie"]
|
||||
|
||||
|
||||
def test_logout_sets_secure_cookie_with_forwarded_https_proto():
|
||||
configure_auth()
|
||||
client = TestClient(app, base_url="http://testserver")
|
||||
|
||||
logout = client.post("/api/auth/logout", headers={"x-forwarded-proto": "https"})
|
||||
|
||||
assert logout.status_code == 204
|
||||
assert "Secure" in logout.headers["set-cookie"]
|
||||
|
||||
|
||||
def test_login_rejects_wrong_password():
|
||||
configure_auth()
|
||||
client = TestClient(app, base_url="https://testserver")
|
||||
response = client.post("/api/auth/login", json={"username": "admin", "password": "wrong"})
|
||||
assert response.status_code == 401
|
||||
101
runtime-current/kino/app-src/backend/tests/test_downloader.py
Normal file
101
runtime-current/kino/app-src/backend/tests/test_downloader.py
Normal file
@@ -0,0 +1,101 @@
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.providers.base import DownloadOptions, MediaMetadata
|
||||
from app.providers.mediathek import MediathekProvider, is_stream_manifest
|
||||
from app.providers.youtube import parse_yt_dlp_progress
|
||||
from app.services.downloader import copy_tree_contents
|
||||
|
||||
|
||||
class FakeStream:
|
||||
def __init__(self, lines: list[bytes]):
|
||||
self.lines = lines
|
||||
|
||||
async def readline(self) -> bytes:
|
||||
if self.lines:
|
||||
return self.lines.pop(0)
|
||||
return b""
|
||||
|
||||
|
||||
class FakeProcess:
|
||||
def __init__(self, output_path: Path):
|
||||
self.returncode = 0
|
||||
self.stdout = FakeStream([b"[download] 100% of 12.00MiB\n"])
|
||||
self.stderr = FakeStream([])
|
||||
self.output_path = output_path
|
||||
|
||||
async def wait(self) -> int:
|
||||
self.output_path.write_bytes(b"real media bytes")
|
||||
return self.returncode
|
||||
|
||||
|
||||
def test_copy_tree_contents_copies_files(tmp_path: Path):
|
||||
src = tmp_path / 'src'
|
||||
src.mkdir()
|
||||
(src / 'movie.mp4').write_bytes(b'data')
|
||||
(src / 'meta.info.json').write_text('{}')
|
||||
dest = tmp_path / 'dest'
|
||||
|
||||
copied = copy_tree_contents(src, dest)
|
||||
|
||||
assert sorted(p.name for p in copied) == ['meta.info.json', 'movie.mp4']
|
||||
assert (dest / 'movie.mp4').read_bytes() == b'data'
|
||||
|
||||
|
||||
def test_copy_tree_contents_rejects_top_level_symlink(tmp_path: Path):
|
||||
src = tmp_path / "src"
|
||||
src.mkdir()
|
||||
(tmp_path / "secret.txt").write_text("secret")
|
||||
(src / "leak.txt").symlink_to(tmp_path / "secret.txt")
|
||||
|
||||
with pytest.raises(Exception, match="symlink"):
|
||||
copy_tree_contents(src, tmp_path / "dest")
|
||||
|
||||
|
||||
def test_copy_tree_contents_rejects_nested_symlink(tmp_path: Path):
|
||||
src = tmp_path / "src"
|
||||
nested = src / "nested"
|
||||
nested.mkdir(parents=True)
|
||||
(tmp_path / "secret.txt").write_text("secret")
|
||||
(nested / "leak.txt").symlink_to(tmp_path / "secret.txt")
|
||||
|
||||
with pytest.raises(Exception, match="symlink"):
|
||||
copy_tree_contents(src, tmp_path / "dest")
|
||||
|
||||
|
||||
def test_parse_yt_dlp_progress_lines():
|
||||
assert parse_yt_dlp_progress('[download] 42.7% of 10.00MiB at 1.00MiB/s ETA 00:05') == pytest.approx(0.427)
|
||||
assert parse_yt_dlp_progress('[download] 100% of 10.00MiB') == 1.0
|
||||
assert parse_yt_dlp_progress('[info] unrelated') is None
|
||||
|
||||
|
||||
def test_stream_manifest_detection():
|
||||
assert is_stream_manifest('https://example.org/master.m3u8')
|
||||
assert is_stream_manifest('https://example.org/manifest.mpd')
|
||||
assert is_stream_manifest('https://example.org/video', 'application/vnd.apple.mpegurl')
|
||||
assert not is_stream_manifest('https://example.org/video.mp4', 'video/mp4')
|
||||
|
||||
|
||||
async def test_m3u8_download_uses_ytdlp_not_manifest_file(monkeypatch, tmp_path: Path):
|
||||
provider = MediathekProvider()
|
||||
manifest_url = 'https://cdn.example.org/master.m3u8'
|
||||
|
||||
async def fake_probe(url: str) -> MediaMetadata:
|
||||
return MediaMetadata(provider='mediathek', title='master.m3u8', external_id=manifest_url)
|
||||
|
||||
async def fake_create_subprocess_exec(*args, **kwargs):
|
||||
assert args[0].endswith('yt-dlp')
|
||||
assert '--proxy' not in args
|
||||
assert '--remux-video' in args
|
||||
output_template = Path(args[args.index('--output') + 1])
|
||||
return FakeProcess(output_template.parent / 'downloaded.mp4')
|
||||
|
||||
monkeypatch.setattr(provider, 'probe', fake_probe)
|
||||
monkeypatch.setattr('app.providers.mediathek.shutil.which', lambda name: '/usr/bin/yt-dlp' if name == 'yt-dlp' else None)
|
||||
monkeypatch.setattr('app.providers.mediathek.asyncio.create_subprocess_exec', fake_create_subprocess_exec)
|
||||
|
||||
result = await provider.download(manifest_url, tmp_path, DownloadOptions())
|
||||
|
||||
assert [p.name for p in result.output_files] == ['downloaded.mp4']
|
||||
assert not (tmp_path / 'master.m3u8').exists()
|
||||
@@ -0,0 +1,7 @@
|
||||
from fastapi.testclient import TestClient
|
||||
from app.main import app
|
||||
|
||||
|
||||
def test_health():
|
||||
client = TestClient(app)
|
||||
assert client.get('/health').json() == {'status': 'ok'}
|
||||
@@ -0,0 +1,45 @@
|
||||
import pytest
|
||||
|
||||
from app.main import resolve_importable_url
|
||||
from app.providers.base import MediaMetadata, ProviderError
|
||||
from app.services.page_analyzer import MediaCandidate
|
||||
|
||||
|
||||
class FakeProvider:
|
||||
name = "mediathek"
|
||||
|
||||
def can_handle(self, url: str) -> bool:
|
||||
return url.startswith("https://")
|
||||
|
||||
async def probe(self, url: str) -> MediaMetadata:
|
||||
if url == "https://example.org/watch":
|
||||
raise ProviderError("URL is reachable, but content type/extension is not accepted: text/html")
|
||||
if url == "https://cdn.example.org/master.m3u8":
|
||||
return MediaMetadata(provider=self.name, title="master.m3u8", external_id=url)
|
||||
raise ProviderError("unexpected url")
|
||||
|
||||
|
||||
async def test_html_page_import_resolves_to_best_media_candidate(monkeypatch):
|
||||
async def fake_analyze_page(url: str):
|
||||
assert url == "https://example.org/watch"
|
||||
return [
|
||||
MediaCandidate(
|
||||
url="https://cdn.example.org/master.m3u8",
|
||||
kind="hls_manifest",
|
||||
title="HLS stream",
|
||||
source="browser",
|
||||
content_length=4096,
|
||||
allowed=True,
|
||||
)
|
||||
]
|
||||
|
||||
fake_provider = FakeProvider()
|
||||
monkeypatch.setattr("app.main.providers", [fake_provider])
|
||||
monkeypatch.setattr("app.main.analyze_page", fake_analyze_page)
|
||||
|
||||
resolved_url, provider, metadata, candidate = await resolve_importable_url("https://example.org/watch")
|
||||
|
||||
assert resolved_url == "https://cdn.example.org/master.m3u8"
|
||||
assert provider is fake_provider
|
||||
assert metadata.external_id == resolved_url
|
||||
assert candidate.kind == "hls_manifest"
|
||||
@@ -0,0 +1,46 @@
|
||||
import pytest
|
||||
|
||||
from app.providers.base import ProviderError
|
||||
from app.providers.internet_archive import InternetArchiveProvider
|
||||
|
||||
|
||||
def test_internet_archive_detection():
|
||||
provider = InternetArchiveProvider()
|
||||
assert provider.can_handle("https://archive.org/details/BigBuckBunny_328")
|
||||
assert provider.can_handle("https://archive.org/download/BigBuckBunny_328/video.mp4")
|
||||
assert not provider.can_handle("https://example.com/details/BigBuckBunny_328")
|
||||
|
||||
|
||||
def test_internet_archive_identifier_parsing():
|
||||
provider = InternetArchiveProvider()
|
||||
assert provider._identifier_from_url("https://archive.org/details/BigBuckBunny_328") == "BigBuckBunny_328"
|
||||
assert provider._identifier_from_url("https://archive.org/download/BigBuckBunny_328/video.mp4") == "BigBuckBunny_328"
|
||||
assert provider._identifier_from_url("https://archive.org/metadata/BigBuckBunny_328") == "BigBuckBunny_328"
|
||||
|
||||
|
||||
def test_select_media_file_prefers_original_mp4():
|
||||
provider = InternetArchiveProvider()
|
||||
selected = provider._select_media_file([
|
||||
{"name": "thumb.jpg", "format": "JPEG", "source": "metadata", "size": "10"},
|
||||
{"name": "movie.ogv", "format": "Ogg Video", "source": "derivative", "size": "200000000"},
|
||||
{"name": "movie.mp4", "format": "MPEG4", "source": "original", "size": "100000000"},
|
||||
])
|
||||
assert selected is not None
|
||||
assert selected["name"] == "movie.mp4"
|
||||
|
||||
|
||||
def test_select_media_file_rejects_path_traversal():
|
||||
provider = InternetArchiveProvider()
|
||||
assert provider._select_media_file([{"name": "../evil.mp4", "format": "MPEG4", "source": "original"}]) is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_probe_requires_media_file(monkeypatch):
|
||||
provider = InternetArchiveProvider()
|
||||
|
||||
async def fake_metadata(identifier: str) -> dict:
|
||||
return {"metadata": {"title": "No media"}, "files": [{"name": "notes.txt", "format": "Text"}]}
|
||||
|
||||
monkeypatch.setattr(provider, "_metadata", fake_metadata)
|
||||
with pytest.raises(ProviderError):
|
||||
await provider.probe("https://archive.org/details/no-media")
|
||||
@@ -0,0 +1,9 @@
|
||||
import pytest
|
||||
|
||||
from app.providers.base import ProviderError
|
||||
from app.providers.mediathek import MediathekProvider
|
||||
|
||||
|
||||
def test_private_target_rejected():
|
||||
with pytest.raises(ProviderError):
|
||||
MediathekProvider()._reject_private_targets('localhost')
|
||||
19
runtime-current/kino/app-src/backend/tests/test_models.py
Normal file
19
runtime-current/kino/app-src/backend/tests/test_models.py
Normal file
@@ -0,0 +1,19 @@
|
||||
from sqlmodel import Session, select
|
||||
|
||||
from app.db import get_engine, init_db
|
||||
from app.models import ImportJob, Source
|
||||
|
||||
|
||||
def test_models_roundtrip(tmp_path):
|
||||
engine = get_engine(tmp_path / 'kino.sqlite3')
|
||||
init_db(engine)
|
||||
with Session(engine) as session:
|
||||
src = Source(kind='youtube', url='https://youtu.be/example', title='Example', provider='youtube')
|
||||
session.add(src)
|
||||
session.commit()
|
||||
session.refresh(src)
|
||||
job = ImportJob(source_id=src.id, target_library='movies')
|
||||
session.add(job)
|
||||
session.commit()
|
||||
assert session.exec(select(Source)).one().title == 'Example'
|
||||
assert session.exec(select(ImportJob)).one().status == 'queued'
|
||||
175
runtime-current/kino/app-src/backend/tests/test_page_analyzer.py
Normal file
175
runtime-current/kino/app-src/backend/tests/test_page_analyzer.py
Normal file
@@ -0,0 +1,175 @@
|
||||
import pytest
|
||||
|
||||
from app.services import page_analyzer
|
||||
from app.services.page_analyzer import (
|
||||
MediaCandidate,
|
||||
analyze_browser_capture,
|
||||
analyze_page,
|
||||
analyze_with_ytdlp,
|
||||
extract_media_urls,
|
||||
human_size,
|
||||
_fetch_public_page_body,
|
||||
)
|
||||
|
||||
|
||||
def test_human_size_labels_file_sizes():
|
||||
assert human_size(None) is None
|
||||
assert human_size(512) == "512 B"
|
||||
assert human_size(1536) == "1.5 KB"
|
||||
assert human_size(5 * 1024 * 1024) == "5.0 MB"
|
||||
|
||||
|
||||
def test_extract_media_urls_finds_encoded_and_unquoted_js_urls():
|
||||
html = """
|
||||
<video data-src="/media/trailer.mp4?token=abc&v=1"></video>
|
||||
<script>
|
||||
const hls = "https:\\/\\/cdn.example.org\\/movie\\/master.m3u8?sig=123";
|
||||
window.player = {dash: https://cdn.example.org/movie/manifest.mpd?x=1};
|
||||
</script>
|
||||
"""
|
||||
|
||||
assert extract_media_urls(html, "https://example.org/watch") == [
|
||||
"https://example.org/media/trailer.mp4?token=abc&v=1",
|
||||
"https://cdn.example.org/movie/master.m3u8?sig=123",
|
||||
"https://cdn.example.org/movie/manifest.mpd?x=1",
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_browser_capture_filters_and_normalizes_candidates(monkeypatch):
|
||||
captured_probe_input = {}
|
||||
|
||||
async def fake_probe(urls, source):
|
||||
captured_probe_input["urls"] = urls
|
||||
captured_probe_input["source"] = source
|
||||
return [MediaCandidate(url=url, kind="direct_video", source=source) for url in urls]
|
||||
|
||||
monkeypatch.setattr(page_analyzer, "_probe_candidates_with_meta", fake_probe)
|
||||
|
||||
result = await analyze_browser_capture(
|
||||
[
|
||||
{"url": "/media/movie.mp4", "title": "Movie"},
|
||||
{"url": "https://cdn.example.org/live/master.m3u8", "content_type": "application/vnd.apple.mpegurl"},
|
||||
{"url": "https://cdn.example.org/poster.jpg", "content_type": "image/jpeg"},
|
||||
],
|
||||
page_url="https://example.org/watch/1",
|
||||
)
|
||||
|
||||
assert captured_probe_input["source"] == "browser-capture"
|
||||
assert list(captured_probe_input["urls"]) == [
|
||||
"https://example.org/media/movie.mp4",
|
||||
"https://cdn.example.org/live/master.m3u8",
|
||||
]
|
||||
assert [item.source for item in result] == ["browser-capture", "browser-capture"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_analyze_page_merges_browser_ytdlp_and_html_candidates(monkeypatch):
|
||||
async def fake_html(url):
|
||||
return [MediaCandidate(url="https://cdn.example.org/static.mp4", kind="direct_video", source="html")]
|
||||
|
||||
async def fake_ytdlp(url):
|
||||
return [MediaCandidate(url="https://cdn.example.org/ytdlp.m3u8", kind="hls_manifest", source="yt-dlp")]
|
||||
|
||||
async def fake_browser(url):
|
||||
return [MediaCandidate(url="https://cdn.example.org/browser.mpd", kind="dash_manifest", source="browser")]
|
||||
|
||||
monkeypatch.setattr(page_analyzer, "analyze_html_page", fake_html)
|
||||
monkeypatch.setattr(page_analyzer, "analyze_with_ytdlp", fake_ytdlp)
|
||||
monkeypatch.setattr(page_analyzer, "analyze_with_browser", fake_browser)
|
||||
|
||||
result = await analyze_page("https://example.org/watch")
|
||||
|
||||
assert {candidate.url: candidate.source for candidate in result} == {
|
||||
"https://cdn.example.org/static.mp4": "html",
|
||||
"https://cdn.example.org/ytdlp.m3u8": "yt-dlp",
|
||||
"https://cdn.example.org/browser.mpd": "browser",
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_ytdlp_is_skipped_for_generic_untrusted_hosts(monkeypatch):
|
||||
calls = []
|
||||
|
||||
class Checked:
|
||||
final_url = "https://example.org/watch"
|
||||
|
||||
async def fake_check(url, **kwargs):
|
||||
return Checked()
|
||||
|
||||
class FakeYDL:
|
||||
def __init__(self, opts):
|
||||
calls.append(opts)
|
||||
def __enter__(self):
|
||||
return self
|
||||
def __exit__(self, *args):
|
||||
return None
|
||||
def extract_info(self, url, download=False):
|
||||
raise AssertionError("yt-dlp must not run for generic hosts")
|
||||
|
||||
monkeypatch.setattr(page_analyzer, "check_url_for_page", fake_check)
|
||||
monkeypatch.setattr(page_analyzer, "YoutubeDL", FakeYDL)
|
||||
|
||||
assert await analyze_with_ytdlp("https://example.org/watch") == []
|
||||
assert calls == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_public_page_body_revalidates_get_redirect_hops(monkeypatch):
|
||||
checked_urls = []
|
||||
|
||||
class Checked:
|
||||
def __init__(self, final_url):
|
||||
self.final_url = final_url
|
||||
|
||||
async def fake_check(url, **kwargs):
|
||||
checked_urls.append(url)
|
||||
return Checked(url)
|
||||
|
||||
async def fake_assert_public_host(url):
|
||||
checked_urls.append(f"assert:{url}")
|
||||
|
||||
class FakeURL:
|
||||
def __init__(self, url):
|
||||
self._url = url
|
||||
def join(self, loc):
|
||||
return loc
|
||||
def __str__(self):
|
||||
return self._url
|
||||
|
||||
class FakeResponse:
|
||||
def __init__(self, url, redirect_to=None, text=""):
|
||||
self.url = FakeURL(url)
|
||||
self._redirect_to = redirect_to
|
||||
self.headers = {"location": redirect_to} if redirect_to else {}
|
||||
self.text = text
|
||||
@property
|
||||
def is_redirect(self):
|
||||
return self._redirect_to is not None
|
||||
def raise_for_status(self):
|
||||
return None
|
||||
|
||||
class FakeClient:
|
||||
def __init__(self, *args, **kwargs):
|
||||
self.calls = 0
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
async def __aexit__(self, *args):
|
||||
return None
|
||||
async def get(self, url, **kwargs):
|
||||
self.calls += 1
|
||||
if self.calls == 1:
|
||||
return FakeResponse(url, redirect_to="https://cdn.example.org/page")
|
||||
return FakeResponse(url, text="<video src='/movie.mp4'></video>")
|
||||
|
||||
monkeypatch.setattr(page_analyzer, "check_url_for_page", fake_check)
|
||||
monkeypatch.setattr(page_analyzer, "assert_public_host", fake_assert_public_host)
|
||||
monkeypatch.setattr(page_analyzer.httpx, "AsyncClient", FakeClient)
|
||||
|
||||
body, final_url = await _fetch_public_page_body("https://example.org/watch", timeout=1)
|
||||
|
||||
assert body.startswith("<video")
|
||||
assert final_url == "https://cdn.example.org/page"
|
||||
assert "https://example.org/watch" in checked_urls
|
||||
assert "https://cdn.example.org/page" in checked_urls
|
||||
assert "assert:https://cdn.example.org/page" in checked_urls
|
||||
@@ -0,0 +1,56 @@
|
||||
from pathlib import Path
|
||||
import pytest
|
||||
|
||||
from app.services.paths import ensure_under, jellyfin_profile_target, media_target, slugify_name, tmp_target
|
||||
|
||||
|
||||
def test_slugify_name_removes_dangerous_chars():
|
||||
assert slugify_name('../Bad:Title?') == 'Bad_Title'
|
||||
|
||||
|
||||
def test_ensure_under_accepts_child(tmp_path: Path):
|
||||
assert ensure_under(tmp_path, Path('a/b')).is_relative_to(tmp_path.resolve())
|
||||
|
||||
|
||||
def test_ensure_under_rejects_escape(tmp_path: Path):
|
||||
with pytest.raises(ValueError):
|
||||
ensure_under(tmp_path, Path('../escape'))
|
||||
|
||||
|
||||
def test_targets_stay_in_configured_roots(tmp_path: Path):
|
||||
assert tmp_target(tmp_path / 'tmp', 7, '../../Film').is_relative_to((tmp_path / 'tmp').resolve())
|
||||
assert media_target(tmp_path / 'media', 'movies', '../../Film').is_relative_to((tmp_path / 'media').resolve())
|
||||
|
||||
|
||||
def test_jellyfin_film_profile_matches_real_library_layout(tmp_path: Path):
|
||||
target = jellyfin_profile_target(tmp_path / 'media', 'film', 'Big Buck Bunny', year=2008)
|
||||
assert target == (tmp_path / 'media' / 'Filme' / 'Big Buck Bunny (2008)').resolve()
|
||||
|
||||
|
||||
def test_jellyfin_series_profile_uses_staffel_folder(tmp_path: Path):
|
||||
target = jellyfin_profile_target(
|
||||
tmp_path / 'media',
|
||||
'serie',
|
||||
'Pilot/Unsafe',
|
||||
series_title='The Show',
|
||||
season=2,
|
||||
episode=3,
|
||||
episode_title='Pilot/Unsafe',
|
||||
)
|
||||
assert target == (tmp_path / 'media' / 'Serien' / 'The Show' / 'Staffel2' / 'S02E03 - Pilot_Unsafe').resolve()
|
||||
|
||||
|
||||
def test_jellyfin_clip_profile_uses_youtube_mediathek(tmp_path: Path):
|
||||
target = jellyfin_profile_target(
|
||||
tmp_path / 'media',
|
||||
'clip',
|
||||
'../../Clip',
|
||||
channel='Archive Org',
|
||||
external_id='abc/123',
|
||||
)
|
||||
assert target == (tmp_path / 'media' / 'YouTube-Mediathek' / 'Archive Org' / 'Clip [abc_123]').resolve()
|
||||
|
||||
|
||||
def test_jellyfin_profile_rejects_unknown_profile(tmp_path: Path):
|
||||
with pytest.raises(ValueError):
|
||||
jellyfin_profile_target(tmp_path / 'media', 'other', 'Title')
|
||||
@@ -0,0 +1,9 @@
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app.main import app
|
||||
|
||||
|
||||
def test_probe_rejects_private_direct_url():
|
||||
client = TestClient(app)
|
||||
response = client.post('/api/probe', json={'url': 'http://localhost/video.mp4'})
|
||||
assert response.status_code == 400
|
||||
@@ -0,0 +1,24 @@
|
||||
from app.providers.internet_archive import InternetArchiveProvider
|
||||
from app.providers.mediathek import MediathekProvider
|
||||
from app.providers.youtube import YouTubeProvider
|
||||
|
||||
|
||||
def test_youtube_detection():
|
||||
provider = YouTubeProvider()
|
||||
assert provider.can_handle('https://www.youtube.com/watch?v=abc')
|
||||
assert provider.can_handle('https://youtu.be/abc')
|
||||
assert not provider.can_handle('https://example.com/video.mp4')
|
||||
|
||||
|
||||
def test_mediathek_detection_excludes_youtube_and_internet_archive():
|
||||
provider = MediathekProvider()
|
||||
assert provider.can_handle('https://example.com/video.mp4')
|
||||
assert not provider.can_handle('https://youtube.com/watch?v=abc')
|
||||
assert not provider.can_handle('https://archive.org/details/BigBuckBunny_328')
|
||||
|
||||
|
||||
def test_internet_archive_detection():
|
||||
provider = InternetArchiveProvider()
|
||||
assert provider.can_handle('https://archive.org/details/BigBuckBunny_328')
|
||||
assert provider.can_handle('https://archive.org/download/BigBuckBunny_328/video.mp4')
|
||||
assert not provider.can_handle('https://example.com/video.mp4')
|
||||
@@ -0,0 +1,7 @@
|
||||
from app.providers.youtube import YouTubeProvider
|
||||
|
||||
|
||||
def test_extract_youtube_id():
|
||||
provider = YouTubeProvider()
|
||||
assert provider._external_id('https://youtu.be/abc123') == 'abc123'
|
||||
assert provider._external_id('https://www.youtube.com/watch?v=xyz') == 'xyz'
|
||||
@@ -0,0 +1,32 @@
|
||||
# Kino Capture Browser-Extension
|
||||
|
||||
Kleine generische Browser-Extension für das Kino-Projekt. Sie sammelt im aktiven Browser-Tab sichtbare Medien-Requests und DOM-/Performance-Kandidaten und übergibt diese an die Kino-App.
|
||||
|
||||
## Installation als entpackte Extension
|
||||
|
||||
### Chromium / Chrome / Edge
|
||||
|
||||
1. Ordner `kino-capture` entpacken.
|
||||
2. `chrome://extensions` öffnen.
|
||||
3. Entwicklermodus aktivieren.
|
||||
4. „Entpackte Erweiterung laden“ klicken.
|
||||
5. Den Ordner `kino-capture` auswählen.
|
||||
|
||||
### Firefox / LibreWolf temporär
|
||||
|
||||
1. `about:debugging#/runtime/this-firefox` öffnen.
|
||||
2. „Temporäres Add-on laden“ klicken.
|
||||
3. `manifest.json` im Ordner `kino-capture` auswählen.
|
||||
|
||||
## Nutzung
|
||||
|
||||
1. Video im Browser starten und einige Sekunden laufen lassen.
|
||||
2. Extension-Icon „Kino Capture“ öffnen.
|
||||
3. Prüfen, dass Kandidaten angezeigt werden.
|
||||
4. Kino-App-URL setzen, z. B. `http://192.168.178.213:8099/`.
|
||||
5. „An Kino-App senden“ klicken.
|
||||
6. In der Kino-App Kandidat übernehmen und Import starten.
|
||||
|
||||
## Grenzen
|
||||
|
||||
Die Extension sammelt nur Medien-URLs, die der Browser normal sichtbar anfragt oder im DOM/Performance-API offenlegt. Sie enthält keine hoster-spezifischen Bypässe, keine CAPTCHA-/DRM-/Token-Umgehung und keine Entschlüsselungslogik.
|
||||
@@ -0,0 +1,68 @@
|
||||
const MEDIA_EXT_RE = /\.(mp4|m4v|webm|mkv|mov|m3u8|mpd)([?#]|$)/i;
|
||||
const ABSOLUTE_MEDIA_RE = /(?:https?:\/\/|https?:\\\/\\\/|\/\/|\\\/\\\/)[^\s"'<>]+?\.(?:mp4|m4v|webm|mkv|mov|m3u8|mpd)(?=$|[?&#\s"'<>])(?:[?&][^\s"'<>]*)?/gi;
|
||||
|
||||
function ignoreChromeResult(result) {
|
||||
if (result && typeof result.catch === 'function') result.catch(() => {});
|
||||
}
|
||||
|
||||
function normalizeMediaUrl(rawUrl, baseUrl = location.href) {
|
||||
if (!rawUrl || typeof rawUrl !== 'string') return null;
|
||||
try {
|
||||
const cleaned = rawUrl
|
||||
.trim()
|
||||
.replaceAll('\\/', '/')
|
||||
.replaceAll('\\u002F', '/')
|
||||
.replaceAll('\\u002f', '/')
|
||||
.replace(/[;,)}\]]+$/, '');
|
||||
return new URL(cleaned, baseUrl).href;
|
||||
} catch (_) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function pushCandidate(candidates, rawUrl, title = document.title, contentType = '', source = 'dom') {
|
||||
const url = normalizeMediaUrl(rawUrl);
|
||||
if (!url) return;
|
||||
const cleanType = String(contentType || '').split(';', 1)[0].trim();
|
||||
if (MEDIA_EXT_RE.test(url) || cleanType.startsWith('video/') || /mpegurl|dash\+xml/i.test(cleanType)) {
|
||||
candidates.push({url, title: title || document.title, content_type: contentType || '', source});
|
||||
}
|
||||
}
|
||||
|
||||
function collectCandidates() {
|
||||
const candidates = [];
|
||||
document.querySelectorAll('video,source,a').forEach((el) => {
|
||||
pushCandidate(candidates, el.currentSrc || el.src || el.href, el.title || el.textContent || document.title, el.type || '', 'dom');
|
||||
});
|
||||
|
||||
performance.getEntriesByType('resource').forEach((entry) => {
|
||||
pushCandidate(candidates, entry.name, document.title, entry.initiatorType === 'video' ? 'video/unknown' : '', `performance:${entry.initiatorType || 'resource'}`);
|
||||
});
|
||||
|
||||
const html = document.documentElement?.outerHTML || '';
|
||||
for (const match of html.matchAll(ABSOLUTE_MEDIA_RE)) {
|
||||
pushCandidate(candidates, match[0], document.title, '', 'html-js');
|
||||
}
|
||||
|
||||
const seen = new Set();
|
||||
return candidates.filter((candidate) => {
|
||||
if (seen.has(candidate.url)) return false;
|
||||
seen.add(candidate.url);
|
||||
return true;
|
||||
}).slice(0, 120);
|
||||
}
|
||||
|
||||
chrome.runtime.onMessage.addListener((message, _sender, sendResponse) => {
|
||||
if (message?.type === 'collect-media') {
|
||||
sendResponse({candidates: collectCandidates()});
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
});
|
||||
|
||||
// Opportunistisch nach dem Laden melden, damit das Popup sofort Daten hat.
|
||||
setTimeout(() => {
|
||||
for (const candidate of collectCandidates()) {
|
||||
ignoreChromeResult(chrome.runtime.sendMessage({type: 'candidate', candidate}));
|
||||
}
|
||||
}, 1500);
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"manifest_version": 2,
|
||||
"name": "Kino Capture",
|
||||
"version": "0.1.1",
|
||||
"description": "Sammelt sichtbare Medien-Requests im aktiven Tab und übergibt sie an die Kino-Projekt-App.",
|
||||
"permissions": [
|
||||
"activeTab",
|
||||
"storage",
|
||||
"tabs",
|
||||
"webNavigation",
|
||||
"webRequest",
|
||||
"<all_urls>"
|
||||
],
|
||||
"background": {
|
||||
"scripts": ["service_worker.js"],
|
||||
"persistent": false
|
||||
},
|
||||
"content_scripts": [
|
||||
{
|
||||
"matches": ["<all_urls>"],
|
||||
"js": ["content.js"],
|
||||
"run_at": "document_idle",
|
||||
"all_frames": true
|
||||
}
|
||||
],
|
||||
"browser_action": {
|
||||
"default_title": "Kino Capture",
|
||||
"default_popup": "popup.html"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
body {
|
||||
min-width: 380px;
|
||||
max-width: 520px;
|
||||
margin: 0;
|
||||
background: #111827;
|
||||
color: #f9fafb;
|
||||
font: 14px system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
|
||||
}
|
||||
main { padding: 14px; }
|
||||
h1 { margin: 0 0 8px; font-size: 18px; }
|
||||
label { display: grid; gap: 4px; margin: 10px 0; }
|
||||
input {
|
||||
padding: 8px;
|
||||
border-radius: 8px;
|
||||
border: 1px solid #374151;
|
||||
background: #1f2937;
|
||||
color: #fff;
|
||||
}
|
||||
button {
|
||||
padding: 8px 10px;
|
||||
border: 0;
|
||||
border-radius: 8px;
|
||||
background: #38bdf8;
|
||||
color: #082f49;
|
||||
font-weight: 700;
|
||||
cursor: pointer;
|
||||
}
|
||||
button.secondary { background: #374151; color: #f9fafb; }
|
||||
button:disabled { opacity: .5; cursor: not-allowed; }
|
||||
.actions { display: flex; flex-wrap: wrap; gap: 8px; }
|
||||
ol { padding-left: 20px; max-height: 320px; overflow: auto; }
|
||||
li { margin: 8px 0; border-top: 1px solid #374151; padding-top: 8px; }
|
||||
code { display: block; color: #bae6fd; overflow-wrap: anywhere; font-size: 12px; }
|
||||
.meta, .hint { color: #d1d5db; font-size: 12px; }
|
||||
.error { color: #fca5a5; }
|
||||
@@ -0,0 +1,27 @@
|
||||
<!doctype html>
|
||||
<html lang="de">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>Kino Capture</title>
|
||||
<link rel="stylesheet" href="popup.css" />
|
||||
</head>
|
||||
<body>
|
||||
<main>
|
||||
<h1>Kino Capture</h1>
|
||||
<p id="status">Lade Kandidaten…</p>
|
||||
<label>
|
||||
Kino-App URL
|
||||
<input id="app-url" type="url" placeholder="http://192.168.178.213:8099/" />
|
||||
</label>
|
||||
<div class="actions">
|
||||
<button id="refresh" type="button">Neu scannen</button>
|
||||
<button id="send" type="button">An Kino-App senden</button>
|
||||
<button id="clear" type="button" class="secondary">Leeren</button>
|
||||
</div>
|
||||
<ol id="candidates"></ol>
|
||||
<p class="hint">Tipp: Video starten und einige Sekunden laufen lassen, dann dieses Popup öffnen.</p>
|
||||
</main>
|
||||
<script src="popup.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,103 @@
|
||||
const DEFAULT_APP_URL = 'http://192.168.178.213:8099/';
|
||||
const statusEl = document.querySelector('#status');
|
||||
const listEl = document.querySelector('#candidates');
|
||||
const appUrlEl = document.querySelector('#app-url');
|
||||
const sendButton = document.querySelector('#send');
|
||||
const refreshButton = document.querySelector('#refresh');
|
||||
const clearButton = document.querySelector('#clear');
|
||||
|
||||
let activeTab = null;
|
||||
let currentPayload = {page_url: '', candidates: []};
|
||||
|
||||
function encodeCapturePayload(payload) {
|
||||
return btoa(unescape(encodeURIComponent(JSON.stringify(payload))));
|
||||
}
|
||||
|
||||
function normalizeAppUrl(value) {
|
||||
try {
|
||||
const url = new URL(value || DEFAULT_APP_URL);
|
||||
if (!url.pathname.endsWith('/')) url.pathname = `${url.pathname}/`;
|
||||
url.search = '';
|
||||
url.hash = '';
|
||||
return url.href;
|
||||
} catch (_) {
|
||||
return DEFAULT_APP_URL;
|
||||
}
|
||||
}
|
||||
|
||||
function render(payload) {
|
||||
currentPayload = payload || {page_url: activeTab?.url || '', candidates: []};
|
||||
const candidates = currentPayload.candidates || [];
|
||||
statusEl.textContent = `${candidates.length} Kandidat(en) im aktiven Tab gefunden.`;
|
||||
statusEl.className = candidates.length ? '' : 'error';
|
||||
sendButton.disabled = candidates.length === 0;
|
||||
listEl.innerHTML = '';
|
||||
for (const candidate of candidates.slice(0, 30)) {
|
||||
const li = document.createElement('li');
|
||||
const type = candidate.content_type || candidate.source || 'media';
|
||||
li.innerHTML = `<div class="meta"></div><code></code>`;
|
||||
li.querySelector('.meta').textContent = type;
|
||||
li.querySelector('code').textContent = candidate.url;
|
||||
listEl.appendChild(li);
|
||||
}
|
||||
}
|
||||
|
||||
async function getActiveTab() {
|
||||
const [tab] = await chrome.tabs.query({active: true, currentWindow: true});
|
||||
return tab;
|
||||
}
|
||||
|
||||
async function refresh() {
|
||||
activeTab = await getActiveTab();
|
||||
if (!activeTab?.id) {
|
||||
statusEl.textContent = 'Kein aktiver Tab gefunden.';
|
||||
statusEl.className = 'error';
|
||||
return;
|
||||
}
|
||||
const response = await chrome.runtime.sendMessage({type: 'get-candidates', tabId: activeTab.id});
|
||||
if (!response?.ok) {
|
||||
statusEl.textContent = response?.error || 'Kandidaten konnten nicht gelesen werden.';
|
||||
statusEl.className = 'error';
|
||||
return;
|
||||
}
|
||||
render(response);
|
||||
}
|
||||
|
||||
async function saveAppUrl() {
|
||||
const appUrl = normalizeAppUrl(appUrlEl.value);
|
||||
appUrlEl.value = appUrl;
|
||||
await chrome.storage.local.set({kinoAppUrl: appUrl});
|
||||
return appUrl;
|
||||
}
|
||||
|
||||
async function sendToKinoApp() {
|
||||
const appUrl = await saveAppUrl();
|
||||
const encoded = encodeCapturePayload({
|
||||
page_url: currentPayload.page_url || activeTab?.url || '',
|
||||
candidates: currentPayload.candidates || [],
|
||||
});
|
||||
const destination = `${appUrl}?capture=${encodeURIComponent(encoded)}`;
|
||||
await chrome.tabs.create({url: destination});
|
||||
}
|
||||
|
||||
async function clearCandidates() {
|
||||
if (!activeTab?.id) activeTab = await getActiveTab();
|
||||
if (activeTab?.id) await chrome.runtime.sendMessage({type: 'clear-candidates', tabId: activeTab.id});
|
||||
render({page_url: activeTab?.url || '', candidates: []});
|
||||
}
|
||||
|
||||
async function init() {
|
||||
const stored = await chrome.storage.local.get({kinoAppUrl: DEFAULT_APP_URL});
|
||||
appUrlEl.value = normalizeAppUrl(stored.kinoAppUrl);
|
||||
await refresh();
|
||||
}
|
||||
|
||||
appUrlEl.addEventListener('change', saveAppUrl);
|
||||
refreshButton.addEventListener('click', refresh);
|
||||
sendButton.addEventListener('click', sendToKinoApp);
|
||||
clearButton.addEventListener('click', clearCandidates);
|
||||
|
||||
init().catch((error) => {
|
||||
statusEl.textContent = error?.message || String(error);
|
||||
statusEl.className = 'error';
|
||||
});
|
||||
@@ -0,0 +1,138 @@
|
||||
const MEDIA_EXT_RE = /\.(mp4|m4v|webm|mkv|mov|m3u8|mpd)([?#]|$)/i;
|
||||
const MEDIA_CONTENT_RE = /^(video\/|application\/(vnd\.apple\.mpegurl|x-mpegurl|dash\+xml)|audio\/mpegurl)/i;
|
||||
const MAX_CANDIDATES_PER_TAB = 120;
|
||||
|
||||
const tabCandidates = new Map();
|
||||
const tabPageUrls = new Map();
|
||||
|
||||
const actionApi = chrome.action || chrome.browserAction;
|
||||
|
||||
function ignoreChromeResult(result) {
|
||||
if (result && typeof result.catch === 'function') result.catch(() => {});
|
||||
}
|
||||
|
||||
function setBadgeTextSafe(tabId, text) {
|
||||
if (actionApi?.setBadgeText) ignoreChromeResult(actionApi.setBadgeText({tabId, text}));
|
||||
}
|
||||
|
||||
function setBadgeBackgroundColorSafe(tabId, color) {
|
||||
if (actionApi?.setBadgeBackgroundColor) ignoreChromeResult(actionApi.setBadgeBackgroundColor({tabId, color}));
|
||||
}
|
||||
|
||||
function candidateKey(candidate) {
|
||||
return candidate.url;
|
||||
}
|
||||
|
||||
function normalizeUrl(rawUrl) {
|
||||
if (!rawUrl || typeof rawUrl !== 'string') return null;
|
||||
try {
|
||||
return new URL(rawUrl).href;
|
||||
} catch (_) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function looksLikeMedia(url, contentType = '') {
|
||||
return MEDIA_EXT_RE.test(url) || MEDIA_CONTENT_RE.test(String(contentType).split(';', 1)[0].trim());
|
||||
}
|
||||
|
||||
function addCandidate(tabId, candidate) {
|
||||
if (tabId < 0 || !candidate?.url) return;
|
||||
const url = normalizeUrl(candidate.url);
|
||||
if (!url) return;
|
||||
const contentType = candidate.content_type || candidate.type || '';
|
||||
if (!looksLikeMedia(url, contentType)) return;
|
||||
|
||||
const existing = tabCandidates.get(tabId) || [];
|
||||
if (existing.some((item) => candidateKey(item) === url)) return;
|
||||
existing.unshift({
|
||||
url,
|
||||
title: candidate.title || candidate.name || '',
|
||||
content_type: contentType,
|
||||
quality: candidate.quality || '',
|
||||
source: candidate.source || 'webRequest',
|
||||
ts: Date.now(),
|
||||
});
|
||||
tabCandidates.set(tabId, existing.slice(0, MAX_CANDIDATES_PER_TAB));
|
||||
setBadgeTextSafe(tabId, String(Math.min(existing.length, 99)));
|
||||
setBadgeBackgroundColorSafe(tabId, '#38bdf8');
|
||||
}
|
||||
|
||||
chrome.webRequest.onBeforeRequest.addListener(
|
||||
(details) => {
|
||||
if (details.tabId < 0 || details.type === 'image') return;
|
||||
if (looksLikeMedia(details.url)) {
|
||||
addCandidate(details.tabId, {url: details.url, source: `request:${details.type}`});
|
||||
}
|
||||
},
|
||||
{urls: ['<all_urls>']}
|
||||
);
|
||||
|
||||
chrome.webRequest.onHeadersReceived.addListener(
|
||||
(details) => {
|
||||
if (details.tabId < 0) return;
|
||||
const header = (details.responseHeaders || []).find((item) => item.name.toLowerCase() === 'content-type');
|
||||
const contentType = header?.value || '';
|
||||
if (looksLikeMedia(details.url, contentType)) {
|
||||
addCandidate(details.tabId, {url: details.url, content_type: contentType, source: 'response'});
|
||||
}
|
||||
},
|
||||
{urls: ['<all_urls>']},
|
||||
['responseHeaders']
|
||||
);
|
||||
|
||||
chrome.webNavigation?.onCommitted?.addListener?.((details) => {
|
||||
if (details.frameId === 0) {
|
||||
tabPageUrls.set(details.tabId, details.url);
|
||||
tabCandidates.delete(details.tabId);
|
||||
setBadgeTextSafe(details.tabId, '');
|
||||
}
|
||||
});
|
||||
|
||||
chrome.tabs.onRemoved.addListener((tabId) => {
|
||||
tabCandidates.delete(tabId);
|
||||
tabPageUrls.delete(tabId);
|
||||
});
|
||||
|
||||
async function collectDomCandidates(tabId) {
|
||||
try {
|
||||
const responses = await chrome.tabs.sendMessage(tabId, {type: 'collect-media'});
|
||||
return Array.isArray(responses) ? responses : responses?.candidates || [];
|
||||
} catch (_) {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
chrome.runtime.onMessage.addListener((message, sender, sendResponse) => {
|
||||
(async () => {
|
||||
if (message?.type === 'candidate' && sender.tab?.id != null) {
|
||||
addCandidate(sender.tab.id, message.candidate);
|
||||
sendResponse({ok: true});
|
||||
return;
|
||||
}
|
||||
|
||||
if (message?.type === 'get-candidates') {
|
||||
const tabId = message.tabId;
|
||||
const domCandidates = await collectDomCandidates(tabId);
|
||||
for (const candidate of domCandidates) addCandidate(tabId, {...candidate, source: candidate.source || 'dom'});
|
||||
const tab = await chrome.tabs.get(tabId).catch(() => null);
|
||||
sendResponse({
|
||||
ok: true,
|
||||
page_url: tab?.url || tabPageUrls.get(tabId) || '',
|
||||
candidates: tabCandidates.get(tabId) || [],
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (message?.type === 'clear-candidates') {
|
||||
const tabId = message.tabId;
|
||||
tabCandidates.delete(tabId);
|
||||
setBadgeTextSafe(tabId, '');
|
||||
sendResponse({ok: true});
|
||||
return;
|
||||
}
|
||||
|
||||
sendResponse({ok: false, error: 'unknown message'});
|
||||
})();
|
||||
return true;
|
||||
});
|
||||
11
runtime-current/kino/app-src/docker-compose.yml
Normal file
11
runtime-current/kino/app-src/docker-compose.yml
Normal file
@@ -0,0 +1,11 @@
|
||||
services:
|
||||
kino-projekt:
|
||||
build: .
|
||||
ports:
|
||||
- "127.0.0.1:8099:8080"
|
||||
volumes:
|
||||
- ./data:/data
|
||||
- /jellyfin:/jellyfin
|
||||
env_file:
|
||||
- .env
|
||||
restart: unless-stopped
|
||||
19
runtime-current/kino/app-src/docs/deployment.md
Normal file
19
runtime-current/kino/app-src/docs/deployment.md
Normal file
@@ -0,0 +1,19 @@
|
||||
# Deployment
|
||||
|
||||
1. `.env` auf dem Zielhost aus `.env.example` erstellen; `JELLYFIN_API_KEY`, `KINOPROJEKT_ADMIN_PASSWORD_HASH` und `KINOPROJEKT_SESSION_SECRET` nur dort oder in Vaultwarden speichern.
|
||||
2. Medien- und Datenverzeichnisse als Volumes mounten.
|
||||
3. Dienst nur Mesh-only veröffentlichen und den App-Login aktivieren.
|
||||
4. Nach Start prüfen:
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:8080/health
|
||||
curl -X POST http://127.0.0.1:8080/api/probe \
|
||||
-H 'content-type: application/json' \
|
||||
--data '{"url":"http://localhost/video.mp4"}'
|
||||
# Erwartet: HTTP 400, weil private/loopback Ziele abgelehnt werden.
|
||||
```
|
||||
|
||||
5. Login prüfen: Ohne Session antworten geschützte API-Routen mit `401`; nach `POST /api/auth/login` setzt die App ein `HttpOnly; Secure; SameSite=Lax` Session-Cookie.
|
||||
6. Zielprofile prüfen: Ein Import mit `target_profile=film|serie|clip` muss unter `/jellyfin/Filme`, `/jellyfin/Serien/.../Staffel<N>` oder `/jellyfin/YouTube-Mediathek` landen. Der Ordner `/jellyfin/YouTube-Mediathek` sollte auf dem Zielhost `jellyfinuser:media` gehören und setgid wie die vorhandenen Bibliotheken nutzen.
|
||||
7. Für echte Importe muss `yt-dlp` im Container/Host verfügbar sein; das Dockerfile installiert es.
|
||||
8. Jellyfin-Refresh wird über `POST <JELLYFIN_URL>/Library/Refresh` mit `X-Emby-Token` ausgelöst, wenn `JELLYFIN_API_KEY` gesetzt ist.
|
||||
11
runtime-current/kino/app-src/docs/provider-policy.md
Normal file
11
runtime-current/kino/app-src/docs/provider-policy.md
Normal file
@@ -0,0 +1,11 @@
|
||||
# Provider Policy
|
||||
|
||||
Erlaubt sind nur öffentlich und legal erreichbare Quellen ohne Umgehung technischer oder rechtlicher Zugriffsbeschränkungen.
|
||||
|
||||
## Implementierte Provider
|
||||
|
||||
- **YouTube:** Metadaten/Downloads über `yt-dlp`, ohne Cookies, Login-Flows oder Umgehungsflags; Default-Qualität maximal 1080p.
|
||||
- **Internet Archive:** akzeptiert `archive.org/details/<identifier>`, `archive.org/download/<identifier>/...` und `archive.org/metadata/<identifier>`; Metadaten kommen aus der öffentlichen Archive-Metadata-API, heruntergeladen wird nur ein plausibler öffentlicher Medien-Track.
|
||||
- **Mediathek/Direktlink:** direkte URLs werden vor Verwendung gegen SSRF-Risiken geprüft: Schema, DNS, Redirects, private IP-Ranges und Content-Type/Dateiendung.
|
||||
|
||||
Nicht erlaubt sind Piracy-Scraper, DRM-/Paywall-/Login-/Geoblocking-Umgehung, Cookie-Importe und freie Server-Schreibpfade.
|
||||
13
runtime-current/kino/kino-projekt.env.example
Normal file
13
runtime-current/kino/kino-projekt.env.example
Normal file
@@ -0,0 +1,13 @@
|
||||
DEFAULT_MAX_HEIGHT=1080
|
||||
MAX_DOWNLOAD_BYTES=5368709120
|
||||
JELLYFIN_URL=http://192.168.178.222:8096
|
||||
JELLYFIN_API_KEY=CHANGE_ME
|
||||
KINOPROJEKT_DB_URL=sqlite:////opt/kino-projekt/data/kino.sqlite3
|
||||
KINOPROJEKT_MEDIA_ROOT=/opt/kino-projekt/export
|
||||
KINOPROJEKT_TMP=/opt/kino-projekt/data/tmp
|
||||
KINOPROJEKT_RSYNC_TARGET=kino-transfer@192.168.178.222
|
||||
KINOPROJEKT_RSYNC_SSH_KEY=/opt/kino-projekt/.ssh/jellyfin_transfer_ed25519
|
||||
KINOPROJEKT_RSYNC_REMOTE_ROOT=/jellyfin
|
||||
KINOPROJEKT_ADMIN_USER=admin
|
||||
KINOPROJEKT_ADMIN_PASSWORD_HASH=
|
||||
KINOPROJEKT_SESSION_SECRET=CHANGE_ME
|
||||
11
runtime-current/token-monitor/Dockerfile
Normal file
11
runtime-current/token-monitor/Dockerfile
Normal file
@@ -0,0 +1,11 @@
|
||||
FROM python:3.12-slim
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 DATA_DIR=/data
|
||||
WORKDIR /app
|
||||
RUN pip install --no-cache-dir flask requests gunicorn
|
||||
COPY app.py /app/app.py
|
||||
RUN useradd --uid 10002 --create-home --shell /usr/sbin/nologin monitor \
|
||||
&& mkdir -p /data \
|
||||
&& chown -R monitor:monitor /app /data
|
||||
USER monitor
|
||||
EXPOSE 8080
|
||||
CMD ["gunicorn", "--bind", "0.0.0.0:8080", "--workers", "1", "--threads", "4", "--timeout", "30", "app:app"]
|
||||
253
runtime-current/token-monitor/app.py
Normal file
253
runtime-current/token-monitor/app.py
Normal file
@@ -0,0 +1,253 @@
|
||||
#!/usr/bin/env python3
|
||||
import os, sqlite3, time, json, datetime, pathlib, threading
|
||||
from collections import defaultdict
|
||||
from flask import Flask, jsonify, render_template_string
|
||||
import requests
|
||||
|
||||
DATA_DIR=pathlib.Path(os.getenv('DATA_DIR','/var/lib/token-monitor'))
|
||||
DB=DATA_DIR/'hermes-state.db'
|
||||
ALERT_STATE=DATA_DIR/'alert-state.json'
|
||||
# Legacy Hermes-internal limits are kept for backwards-compatible API fields only.
|
||||
# Provider stock/remaining budget is calculated in provider_inventory() below.
|
||||
DAILY_LIMIT=int(os.getenv('DAILY_TOKEN_LIMIT','0') or 0)
|
||||
MONTHLY_LIMIT=int(os.getenv('MONTHLY_TOKEN_LIMIT','0') or 0)
|
||||
DISCORD_BOT_TOKEN=os.getenv('DISCORD_BOT_TOKEN','')
|
||||
DISCORD_CHANNEL_ID=os.getenv('DISCORD_CHANNEL_ID','')
|
||||
REFRESH_SECONDS=int(os.getenv('REFRESH_SECONDS','300'))
|
||||
app=Flask(__name__)
|
||||
|
||||
PROVIDER_LABELS={
|
||||
'anthropic':'Anthropic', 'nvidia':'NVIDIA', 'openrouter':'OpenRouter',
|
||||
'ollama':'Ollama Cloud', 'openai':'OpenAI', 'openai-codex':'OpenAI/Codex'
|
||||
}
|
||||
|
||||
|
||||
def _connect():
|
||||
if not DB.exists(): return None
|
||||
return sqlite3.connect(f'file:{DB}?mode=ro', uri=True)
|
||||
|
||||
|
||||
def _num_env(name):
|
||||
raw=os.getenv(name,'').strip()
|
||||
if not raw: return None
|
||||
try: return float(raw.replace('_',''))
|
||||
except ValueError: return None
|
||||
|
||||
|
||||
def _fmt_amount(value, unit):
|
||||
if value is None: return None
|
||||
if unit == 'usd':
|
||||
return f'${value:,.2f}'
|
||||
if unit == 'credits':
|
||||
return f'{value:,.2f} Credits'
|
||||
return f'{int(value):,} Tokens'.replace(',', '.')
|
||||
|
||||
|
||||
def provider_for_model(model):
|
||||
m=(model or '').lower()
|
||||
if m.startswith('anthropic/') or m.startswith('claude-') or '/claude-' in m:
|
||||
return 'anthropic'
|
||||
if m.startswith('nvidia/') or 'nemotron' in m:
|
||||
return 'nvidia'
|
||||
if m.startswith('openai/') or m.startswith('gpt-') or '/gpt-' in m:
|
||||
return 'openai-codex'
|
||||
if m.startswith('openrouter/'):
|
||||
return 'openrouter'
|
||||
if m.startswith('ollama/'):
|
||||
return 'ollama'
|
||||
return 'unknown'
|
||||
|
||||
|
||||
def usage_by_provider(rows):
|
||||
daily=defaultdict(int); monthly=defaultdict(int); total=defaultdict(int)
|
||||
now=datetime.datetime.now()
|
||||
month_start=now.replace(day=1,hour=0,minute=0,second=0,microsecond=0).timestamp()
|
||||
day_start=now.replace(hour=0,minute=0,second=0,microsecond=0).timestamp()
|
||||
for r in rows:
|
||||
prov=provider_for_model(r['model'])
|
||||
tok=int(r['tok'] or 0); ts=float(r['timestamp'] or 0)
|
||||
total[prov]+=tok
|
||||
if ts>=day_start: daily[prov]+=tok
|
||||
if ts>=month_start: monthly[prov]+=tok
|
||||
return {'daily':dict(daily), 'monthly':dict(monthly), 'total':dict(total)}
|
||||
|
||||
|
||||
def provider_inventory(usage=None):
|
||||
"""Return provider-reported or explicitly configured remaining stock.
|
||||
|
||||
We intentionally do not treat Hermes message history as the source of truth for
|
||||
available quota. It is only used as a subtraction term when a provider-specific
|
||||
stock limit was explicitly configured via environment variables.
|
||||
"""
|
||||
usage=usage or {'daily':{}, 'monthly':{}, 'total':{}}
|
||||
providers=[]
|
||||
candidates=[]
|
||||
if os.getenv('ANTHROPIC_API_KEY') or os.getenv('ANTHROPIC_TOKEN'):
|
||||
candidates.append('anthropic')
|
||||
if os.getenv('NVIDIA_API_KEY'):
|
||||
candidates.append('nvidia')
|
||||
if os.getenv('OPENROUTER_API_KEY'):
|
||||
candidates.append('openrouter')
|
||||
if os.getenv('OLLAMA_API_KEY'):
|
||||
candidates.append('ollama')
|
||||
for extra in os.getenv('TOKEN_MONITOR_PROVIDERS','').split(','):
|
||||
extra=extra.strip().lower()
|
||||
if extra and extra not in candidates: candidates.append(extra)
|
||||
|
||||
for name in candidates:
|
||||
unit=os.getenv(f'{name.upper().replace("-","_")}_STOCK_UNIT','tokens').lower()
|
||||
entry={
|
||||
'provider': name,
|
||||
'label': PROVIDER_LABELS.get(name,name),
|
||||
'ok': None,
|
||||
'status': None,
|
||||
'source': 'unavailable',
|
||||
'unit': unit,
|
||||
'daily_limit': _num_env(f'{name.upper().replace("-","_")}_DAILY_TOKEN_LIMIT'),
|
||||
'total_limit': _num_env(f'{name.upper().replace("-","_")}_TOTAL_TOKEN_LIMIT'),
|
||||
'daily_used': usage.get('daily',{}).get(name,0),
|
||||
'total_used': usage.get('total',{}).get(name,0),
|
||||
'daily_remaining': None,
|
||||
'total_remaining': None,
|
||||
'note': None,
|
||||
}
|
||||
try:
|
||||
if name=='openrouter':
|
||||
r=requests.get('https://openrouter.ai/api/v1/credits',headers={'Authorization':'Bearer '+os.getenv('OPENROUTER_API_KEY','')},timeout=8)
|
||||
entry.update({'ok':r.status_code<400,'status':r.status_code})
|
||||
if r.ok:
|
||||
data=r.json().get('data',{})
|
||||
credits=float(data.get('total_credits') or 0)
|
||||
used=float(data.get('total_usage') or 0)
|
||||
entry.update({
|
||||
'source':'provider_api', 'unit':'usd',
|
||||
'total_limit': credits,
|
||||
'total_used': used,
|
||||
'total_remaining': max(0.0, credits-used),
|
||||
'note':'OpenRouter meldet Gesamt-Credits und Gesamtverbrauch; kein Tagesbestand verfügbar.'
|
||||
})
|
||||
elif name=='anthropic':
|
||||
r=requests.get('https://api.anthropic.com/v1/models',headers={'x-api-key':os.getenv('ANTHROPIC_API_KEY') or os.getenv('ANTHROPIC_TOKEN',''),'anthropic-version':'2023-06-01'},timeout=8)
|
||||
entry.update({'ok':r.status_code<400,'status':r.status_code,'note':'Anthropic stellt für normale API-Keys keinen Restbestand/Quota-Endpunkt bereit.'})
|
||||
elif name=='nvidia':
|
||||
r=requests.get('https://integrate.api.nvidia.com/v1/models',headers={'Authorization':'Bearer '+os.getenv('NVIDIA_API_KEY','')},timeout=8)
|
||||
entry.update({'ok':r.status_code<400,'status':r.status_code,'note':'NVIDIA meldet per Models-API nur Erreichbarkeit, keinen Token-Restbestand.'})
|
||||
elif name=='ollama':
|
||||
r=requests.get('https://ollama.com/v1/models',headers={'Authorization':'Bearer '+os.getenv('OLLAMA_API_KEY','')},timeout=8)
|
||||
entry.update({'ok':r.status_code<400,'status':r.status_code,'note':'Ollama Cloud meldet hier nur API-Erreichbarkeit, keinen Restbestand.'})
|
||||
except Exception as e:
|
||||
entry.update({'ok':False,'error':str(e)[:160]})
|
||||
|
||||
if entry['daily_limit'] is not None and entry['daily_remaining'] is None:
|
||||
entry['daily_remaining']=max(0, entry['daily_limit']-entry['daily_used'])
|
||||
entry['source']='configured_limit_minus_hermes_usage'
|
||||
if entry['total_limit'] is not None and entry['total_remaining'] is None:
|
||||
entry['total_remaining']=max(0, entry['total_limit']-entry['total_used'])
|
||||
entry['source']='configured_limit_minus_hermes_usage'
|
||||
entry['daily_remaining_display']=_fmt_amount(entry['daily_remaining'], entry['unit'])
|
||||
entry['total_remaining_display']=_fmt_amount(entry['total_remaining'], entry['unit'])
|
||||
entry['daily_limit_display']=_fmt_amount(entry['daily_limit'], entry['unit'])
|
||||
entry['total_limit_display']=_fmt_amount(entry['total_limit'], entry['unit'])
|
||||
providers.append(entry)
|
||||
return providers
|
||||
|
||||
|
||||
def metrics():
|
||||
con=_connect()
|
||||
now=datetime.datetime.now()
|
||||
month_start=now.replace(day=1,hour=0,minute=0,second=0,microsecond=0).timestamp()
|
||||
day_start=now.replace(hour=0,minute=0,second=0,microsecond=0).timestamp()
|
||||
out={'updated_at': time.time(), 'daily_limit':DAILY_LIMIT,'monthly_limit':MONTHLY_LIMIT,
|
||||
'total_tokens':0,'daily_tokens':0,'monthly_tokens':0,'by_model':[], 'by_day':[], 'by_source':[],
|
||||
'recent_sessions':[], 'provider_inventory': [], 'provider_health': []}
|
||||
rows=[]
|
||||
if not con:
|
||||
out['error']='Hermes state DB noch nicht synchronisiert.'
|
||||
out['provider_inventory']=provider_inventory()
|
||||
out['provider_health']=out['provider_inventory']
|
||||
return out
|
||||
con.row_factory=sqlite3.Row
|
||||
cur=con.cursor()
|
||||
cur.execute('''select coalesce(s.model,"unknown") model, coalesce(s.source,"unknown") source, m.timestamp,
|
||||
case when m.token_count is not null then m.token_count
|
||||
else max(1, cast(length(coalesce(m.content,''))/4 as integer)) end tok
|
||||
from messages m join sessions s on s.id=m.session_id''')
|
||||
rows=cur.fetchall()
|
||||
by_model=defaultdict(int); by_source=defaultdict(int); by_day=defaultdict(int)
|
||||
for r in rows:
|
||||
tok=int(r['tok'] or 0); ts=float(r['timestamp'] or 0)
|
||||
out['total_tokens']+=tok
|
||||
by_model[r['model']]+=tok; by_source[r['source']]+=tok
|
||||
by_day[datetime.datetime.fromtimestamp(ts).strftime('%Y-%m-%d')]+=tok
|
||||
if ts>=day_start: out['daily_tokens']+=tok
|
||||
if ts>=month_start: out['monthly_tokens']+=tok
|
||||
usage=usage_by_provider(rows)
|
||||
out['provider_inventory']=provider_inventory(usage)
|
||||
out['provider_health']=out['provider_inventory'] # backwards-compatible name for old UI/API users
|
||||
out['by_provider_usage']=[{'provider':k,'tokens':v} for k,v in sorted(usage['total'].items(), key=lambda x:-x[1])]
|
||||
out['by_model']=[{'model':k,'tokens':v} for k,v in sorted(by_model.items(), key=lambda x:-x[1])]
|
||||
out['by_source']=[{'source':k,'tokens':v} for k,v in sorted(by_source.items(), key=lambda x:-x[1])]
|
||||
out['by_day']=[{'day':k,'tokens':v} for k,v in sorted(by_day.items())[-60:]]
|
||||
cur.execute('select id, source, model, started_at, ended_at, message_count from sessions order by started_at desc limit 20')
|
||||
out['recent_sessions']=[dict(r) for r in cur.fetchall()]
|
||||
return out
|
||||
|
||||
|
||||
def send_discord(msg):
|
||||
if not (DISCORD_BOT_TOKEN and DISCORD_CHANNEL_ID): return False
|
||||
r=requests.post(f'https://discord.com/api/v10/channels/{DISCORD_CHANNEL_ID}/messages',headers={'Authorization':'Bot '+DISCORD_BOT_TOKEN,'Content-Type':'application/json'},json={'content':msg},timeout=10)
|
||||
return r.status_code<300
|
||||
|
||||
|
||||
def alert_loop():
|
||||
DATA_DIR.mkdir(parents=True, exist_ok=True)
|
||||
while True:
|
||||
try:
|
||||
m=metrics(); state={}
|
||||
if ALERT_STATE.exists(): state=json.loads(ALERT_STATE.read_text())
|
||||
today=datetime.date.today().isoformat(); month=datetime.date.today().strftime('%Y-%m')
|
||||
for p in m.get('provider_inventory',[]):
|
||||
for scope,period in [('daily',today),('total',month)]:
|
||||
rem=p.get(f'{scope}_remaining'); limit=p.get(f'{scope}_limit')
|
||||
if rem is None or not limit or limit <= 0: continue
|
||||
ratio=rem/limit
|
||||
for threshold,label in [(0.2,'unter 20%'),(0.0,'aufgebraucht')]:
|
||||
if ratio<=threshold:
|
||||
sid=f'provider-stock:{p["provider"]}:{scope}:{period}:{threshold}'
|
||||
if not state.get(sid):
|
||||
send_discord(f'⚠️ Token-Monitor: {p.get("label",p["provider"])} {scope}-Bestand {label}: {p.get(f"{scope}_remaining_display") or rem} verfügbar')
|
||||
state[sid]=time.time()
|
||||
ALERT_STATE.write_text(json.dumps(state,indent=2))
|
||||
except Exception as e:
|
||||
print('alert_loop error',e,flush=True)
|
||||
time.sleep(REFRESH_SECONDS)
|
||||
|
||||
|
||||
@app.route('/api/metrics')
|
||||
def api_metrics(): return jsonify(metrics())
|
||||
|
||||
|
||||
@app.route('/')
|
||||
def index():
|
||||
return render_template_string('''<!doctype html><html lang="de"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Model Token Monitor</title><script src="https://cdn.jsdelivr.net/npm/chart.js"></script><style>body{font-family:system-ui;background:#0b1020;color:#e5eefc;margin:0}main{max-width:1200px;margin:auto;padding:24px}.cards{display:grid;grid-template-columns:repeat(auto-fit,minmax(240px,1fr));gap:16px}.card{background:#111a33;border:1px solid #26375f;border-radius:16px;padding:16px}.muted{color:#9fb0d0}.big{font-size:1.8rem;font-weight:700;margin:.2rem 0}.pill{display:inline-block;border-radius:999px;padding:2px 8px;background:#25375f;margin-left:6px}canvas{background:#111a33;border-radius:16px;padding:12px;margin-top:16px}.ok{color:#56d364}.bad{color:#ff7b72}.unknown{color:#d29922}table{width:100%;border-collapse:collapse}td,th{padding:8px;border-bottom:1px solid #26375f;text-align:left}code{color:#7ee787}</style></head><body><main><h1>Model Token Monitor</h1><p>Bestand der Modell-Anbieter: wie viel heute bzw. insgesamt noch verfügbar ist. Hermes-Verbrauch bleibt darunter nur als Historie sichtbar.</p><div class="cards" id="cards"></div><h2>Anbieter-Bestand</h2><div class="cards" id="providers"></div><h2>Hermes-Nutzungshistorie</h2><canvas id="day"></canvas><canvas id="model"></canvas><h2>Letzte Sessions</h2><div id="sessions"></div></main><script>
|
||||
let charts=[];
|
||||
function fmt(n){return (n||0).toLocaleString('de-DE')}
|
||||
function val(v,fallback='nicht verfügbar'){return (v===null||v===undefined||v==='')?fallback:v}
|
||||
function statusClass(p){return p.ok===true?'ok':(p.ok===false?'bad':'unknown')}
|
||||
async function load(){let m=await (await fetch('/api/metrics')).json();
|
||||
let providers=m.provider_inventory||[];
|
||||
let availableDaily=providers.filter(p=>p.daily_remaining_display).length;
|
||||
let availableTotal=providers.filter(p=>p.total_remaining_display).length;
|
||||
cards.innerHTML=`<div class=card><b>Anbieter geprüft</b><div class=big>${providers.length}</div></div><div class=card><b>Tagesbestand bekannt</b><div class=big>${availableDaily}/${providers.length}</div></div><div class=card><b>Gesamtbestand bekannt</b><div class=big>${availableTotal}/${providers.length}</div></div><div class=card><b>Update</b><div class=big>${new Date(m.updated_at*1000).toLocaleString('de-DE')}</div></div>`;
|
||||
providers.innerHTML=providers.map(p=>`<div class=card><b>${p.label||p.provider}</b><span class="pill ${statusClass(p)}">${p.ok===true?'API OK':p.ok===false?'API FEHLER':'Bestand unbekannt'}</span><p class=muted>Quelle: ${p.source||'unbekannt'} ${p.status?`· HTTP ${p.status}`:''}</p><div>Täglich verfügbar</div><div class=big>${val(p.daily_remaining_display)}</div>${p.daily_limit_display?`<p class=muted>Limit ${p.daily_limit_display}, Hermes heute genutzt ${fmt(p.daily_used)}</p>`:''}<div>Insgesamt verfügbar</div><div class=big>${val(p.total_remaining_display)}</div>${p.total_limit_display?`<p class=muted>Limit/Credits ${p.total_limit_display}, genutzt ${p.total_used}</p>`:''}${p.note?`<p class=muted>${p.note}</p>`:''}${p.error?`<code>${p.error}</code>`:''}</div>`).join('')||'<p>Keine Provider-Credentials oder TOKEN_MONITOR_PROVIDERS konfiguriert.</p>';
|
||||
charts.forEach(c=>c.destroy()); charts=[];
|
||||
charts.push(new Chart(day,{type:'line',data:{labels:m.by_day.map(x=>x.day),datasets:[{label:'Hermes Tokens/Tag (Historie)',data:m.by_day.map(x=>x.tokens),borderColor:'#00d4ff'}]}}));
|
||||
charts.push(new Chart(model,{type:'bar',data:{labels:m.by_model.slice(0,12).map(x=>x.model),datasets:[{label:'Hermes Tokens nach Modell',data:m.by_model.slice(0,12).map(x=>x.tokens),backgroundColor:'#00d4ff'}]},options:{indexAxis:'y'}}));
|
||||
sessions.innerHTML='<table><tr><th>Zeit</th><th>Quelle</th><th>Modell</th><th>Msgs</th></tr>'+m.recent_sessions.map(s=>`<tr><td>${new Date(s.started_at*1000).toLocaleString('de-DE')}</td><td>${s.source}</td><td>${s.model}</td><td>${s.message_count}</td></tr>`).join('')+'</table>';
|
||||
} load(); setInterval(load,300000);
|
||||
</script></body></html>''')
|
||||
|
||||
|
||||
if __name__=='__main__':
|
||||
threading.Thread(target=alert_loop,daemon=True).start()
|
||||
app.run(host='0.0.0.0',port=8080)
|
||||
11
runtime-current/token-monitor/token-monitor.env.example
Normal file
11
runtime-current/token-monitor/token-monitor.env.example
Normal file
@@ -0,0 +1,11 @@
|
||||
DATA_DIR=/data
|
||||
DAILY_TOKEN_LIMIT=0
|
||||
MONTHLY_TOKEN_LIMIT=0
|
||||
REFRESH_SECONDS=300
|
||||
TOKEN_MONITOR_PROVIDERS=nvidia,ollama
|
||||
NVIDIA_API_KEY=CHANGE_ME
|
||||
OLLAMA_API_KEY=CHANGE_ME
|
||||
DISCORD_BOT_TOKEN=CHANGE_ME
|
||||
DISCORD_CHANNEL_ID=CHANGE_ME
|
||||
PVE_HOST=192.168.178.30
|
||||
HERMES_CTID=109
|
||||
Reference in New Issue
Block a user