40 Commits

Author SHA1 Message Date
Joachim Friberg 72e62bb2ff Updated appstore 2026-04-22 20:15:14 +02:00
Joachim Friberg 236ed8ef97 Changed category on llama-server and open-webui 2026-04-21 21:03:18 +02:00
Joachim Friberg 0d2dbf0139 Updated appstore 2026-04-21 18:01:52 +02:00
jfriberg a94117c696 Merge pull request 'Add llama-server and open-webui apps for local LLM inference' (#4) from llama-server+open-webui/initial/local-llm-inference into main
Reviewed-on: phirna/zima-apps#4
2026-04-21 17:56:28 +02:00
Joachim Friberg 5f20bd3193 Merge branch 'main' into llama-server+open-webui/initial/local-llm-inference 2026-04-21 17:56:12 +02:00
jfriberg 1b35702b1b Add Snacks app: automated video library encoder with hardware acceleration (#6)
Co-authored-by: Joachim Friberg <joachim.friberg@ip-solutions.se>
Reviewed-on: phirna/zima-apps#6
2026-04-20 20:09:08 +02:00
jfriberg d226ee0b1e Merge pull request 'Add Gitea bot (mimir) configuration for tea-CLI operations' (#5) from agents/update/gitea-bot-mimir into main
Reviewed-on: phirna/zima-apps#5
2026-04-20 15:42:38 +02:00
Joachim Friberg 42a5d231b8 Add Gitea bot (mimir) configuration for tea-CLI operations
- Add AGENTS.md section 11 documenting mimir bot user for Gitea
- Store token via tea logins system with repository:write and user:read scopes
- Document common tea commands for branch/PR creation and issue management
- Enable agents to create branches, commits and PRs via tea-CLI
2026-04-20 13:24:57 +02:00
Joachim Friberg 100213de2d Added publishing info to agents.md 2026-04-20 10:19:43 +02:00
Joachim Friberg eb340eb664 Rebuild appstore zip with fixed llama.cpp image 2026-04-19 22:32:01 +02:00
Joachim Friberg 5cdf645865 Fix llama.cpp image reference: use ghcr.io/ggml-org/llama.cpp:server-b8840 with digest 2026-04-19 22:30:54 +02:00
Joachim Friberg ed75613a08 Build appstore zip with llama-server and open-webui 2026-04-19 22:26:10 +02:00
Joachim Friberg 0aabfc8a72 Add llama-server and open-webui apps for local LLM inference
- llama-server: llama.cpp REST API server, 8G memory, port 8080
- open-webui: Chat UI connecting to llama-server, 2G memory, port 3000
- Both include x-casaos metadata for ZimaOS app store
- README with model download instructions and API examples
2026-04-19 22:25:22 +02:00
Joachim Friberg 231aba08b0 Updated script 2026-04-19 22:23:57 +02:00
Joachim Friberg d6650108e8 Remove timemachine app and refresh appstore zip 2026-04-01 21:05:14 +02:00
Joachim Friberg 6e1720891f Added timemachine - second try 2026-04-01 17:25:21 +02:00
Joachim Friberg e239867ce3 Added timemachine 2026-04-01 14:55:14 +02:00
Joachim Friberg 6c8c570239 Updated appstore 2026-04-01 14:03:43 +02:00
Joachim Friberg bb6d36ad7a chore: apply agent/template updates and refresh appstore zip 2026-03-24 11:32:01 +01:00
Joachim Friberg 2f1babc681 fix(ark-suac): align ports, service naming, and permissions bootstrap 2026-03-24 11:21:57 +01:00
Joachim Friberg 0e7c188845 fix: remove template app from zip and label ark correctly 2026-03-24 09:34:33 +01:00
Joachim Friberg 38676de168 chore: switch image digests to tags 2026-03-24 08:25:24 +01:00
Joachim Friberg 9c5ea400fb chore: align ark-suac app and refresh appstore zip 2026-03-23 21:23:16 +01:00
Joachim Friberg 9c4265b429 Added Ark
Initial commit for adding AsA as a gameserver
2026-03-23 15:20:25 +01:00
Joachim Friberg 2346d5a096 Add LAN-only status UI for caddy-autogen 2026-03-23 12:47:30 +01:00
Joachim Friberg 5b15a0aedd Updated appstore 2026-03-20 13:17:09 +01:00
Joachim Friberg 4b43e80f06 Updated metadata
Changed author and developer to Joachim Friberg
2026-03-20 13:15:56 +01:00
Joachim Friberg 97396afe88 Updated appstore 2026-03-20 11:21:44 +01:00
Joachim Friberg 3d9a599f71 Use object-based category-list schema 2026-03-20 11:18:11 +01:00
Joachim Friberg 237be77014 Document script flags and image verification behavior 2026-03-19 22:47:49 +01:00
Joachim Friberg afea257f86 Updated appstore 2026-03-19 22:40:58 +01:00
Joachim Friberg 55dc745a5e Updated image tags 2026-03-19 22:40:24 +01:00
Joachim Friberg 20323f2ecd Add online image verification policy and CI-enforced zip checks 2026-03-19 22:39:31 +01:00
Joachim Friberg b1f74d4495 Updated appstore 2026-03-19 22:33:36 +01:00
Joachim Friberg 56dfa7a653 Enforce container name limit and update appstore push workflow 2026-03-19 22:16:12 +01:00
Joachim Friberg 6baad76e0d Updated appstore 2026-03-19 21:55:41 +01:00
Joachim Friberg f5580b3be3 Remove compose env interpolation and set fixed TZ defaults 2026-03-19 20:58:24 +01:00
Joachim Friberg 18bf5b2736 Add joafri image metadata in compose and multi-arch build/push script 2026-03-19 14:32:24 +01:00
Joachim Friberg 84ee052d71 Updated appstore 2026-03-18 21:58:27 +01:00
jfriberg e9834a1997 Merge pull request 'Add steam headless apps with security docs and verification guide' (#2) from docker-ip-addr-manager/initial/fastapi-vue-ip-manager into main
Reviewed-on: phirna/zima-apps#2
2026-03-18 21:55:48 +01:00
75 changed files with 5967 additions and 155 deletions
+7
View File
@@ -0,0 +1,7 @@
*.agent_wip
handoff.md
artifacts/unraid-images-live.txt
artifacts/unraid-images-with-names.tsv
artifacts/test-zima-inventory.yaml
artifacts/test-zima-map.yaml
artifacts/unraid-to-public-stores-map.yaml
+3
View File
@@ -0,0 +1,3 @@
[submodule "Apps/ark-survival-ascended-linux-container-image"]
path = Apps/ark-survival-ascended-linux-container-image
url = https://github.com/mschnitzer/ark-survival-ascended-linux-container-image.git
+110
View File
@@ -0,0 +1,110 @@
# Plan: Add Snacks app to zima-apps
## Context
`apps.md` lists "Snacks" (https://github.com/derekshreds/snacks) as a pending app. It is an automated video library encoder with hardware acceleration (NVENC, QSV, VAAPI, AMF).
This plan has two parts:
1. Create the Snacks app definition
2. Update `apps.md` with agent instructions for future additions
---
## Part 1: Create `Apps/snacks/`
### Steps
1. **Create `Apps/snacks/` directory** from `_template`
2. **`docker-compose.yaml`** — adapt upstream `deploy-compose.yml`:
| Field | Upstream value | ZimaOS target |
|---|---|---|
| `name` | n/a | `snacks` |
| `image` | `derekshreds/snacks-docker:latest` | **Pinned version** — fetch latest release tag from GitHub, verify manifest exists |
| `network_mode` | `host` | Keep `host` — required for cluster UDP broadcast discovery |
| `privileged` | `true` | Keep `true` — required for `/dev/dri` access on QNAP/ZimaOS |
| `devices` | `/dev/dri:/dev/dri` | Keep — VAAPI/QSV hardware acceleration |
| `ports` | none (host mode) | Add `6767:6767` for web UI |
| `volumes` | QNAP-specific paths | Parameterize as `/DATA/AppData/$AppID/...` |
| `environment` | QNAP-specific ffmpeg path | Use default ffmpeg path; make jellyfin-ffmpeg path configurable |
Security baseline (MUST):
- `security_opt: ["no-new-privileges:true"]`
- `cap_drop: ["ALL"]`
- `deploy.resources.reservations` set to appropriate value
High-risk settings that MUST be documented in README:
- `network_mode: host` — required for cluster UDP broadcast
- `privileged: true` — required for `/dev/dri` access
- Device mount `/dev/dri` — GPU acceleration
3. **`README.md`** — document:
- Purpose: automated video library encoder with hardware acceleration
- Port: 6767 (web UI)
- Volumes: media library, logs, config
- High-risk settings with justification, alternatives evaluated, and risks
- Hardware acceleration options (VAAPI, QSV, NVENC)
- Cluster mode (UDP broadcast requirement)
- Health check endpoint
4. **Image pinning**: Before merge, verify the image tag exists in Docker Hub registry (manifest check)
5. **Run validation**: `./scripts/validate-appstore.sh`
6. **Optional**: `HOW_TO_VERIFY.md` with integration test cases
### Risk Assessment
- **High risk** due to `network_mode: host`, `privileged: true`, and device mounts
- Must document all three in README per AGENTS.md §3
- Image must be pinned — no `:latest`
### Branch name
`snacks/initial/add-video-encoder`
---
## Part 2: Update `apps.md`
### Changes
Replace current content with a table format that includes:
- Done/pending checkbox
- App name
- Source URL
- Brief description
- Agent instructions column (how to pick up this item)
The file should serve as an agent-facing backlog — clear enough that an agent can read it, understand what is needed, and execute without additional prompting.
### Suggested format
```markdown
## Backlog
| # | Done | Name | Source | What | Agent instructions |
|---|---|---|---|---|---|
| 1 | [ ] | Snacks | https://github.com/derekshreds/snacks | Automated video library encoder | Pick up, follow AGENTS.md §9 workflow, branch `snacks/initial/add-video-encoder` |
## Adding a new app
1. Copy `Apps/_template/``Apps/<app-id>/`
2. Set `name` in compose (lowercase + hyphen only)
3. Pin image to explicit version/tag (no `:latest`)
4. Add `x-casaos` metadata
5. Write `README.md` with purpose, ports, volumes, and risk justifications
6. Validate: `./scripts/validate-appstore.sh`
7. Run final validation before release: `./scripts/validate-appstore.sh --enforce-risk-docs`
```
---
## Verification
- `docker-compose -f Apps/snacks/docker-compose.yaml config` passes (no syntax errors)
- No `:latest` references
- `x-casaos` metadata complete
- README documents all high-risk settings with justification
- `./scripts/validate-appstore.sh` reports `Validation OK`
+192
View File
@@ -0,0 +1,192 @@
# Plan: Local LLM Zima App (Intel NUC8)
## Context
- **Hardware**: Intel NUC8 i7, 16GB RAM, 500GB SSD
- **Goal**: Zima app for local LLM inference with web UI
- **Constraints**: Intel Iris GPU cannot be used for LLM offload; CPU-only inference
- **Decisions**:
- Include OpenWebUI (two-container solution)
- 8G memory reservation (allows 7B Q4 models)
- App name: `llama-server`
---
## Technology Decision
### vLLM — **REJECTED**
- Requires NVIDIA CUDA GPU
- Cannot run on Intel NUC
### llama.cpp (llama-server) — **SELECTED**
- CPU-only, AVX2/AVX512 optimized
- Built-in REST API server
- Minimal footprint, fast for quantized models
- Best fit for NUC8 constraints
### LocalAI — **BACKUP OPTION**
- More features (TTS, image gen, multi-model)
- Can backend to llama.cpp
- Heavier; only choose if extra features needed
### OpenWebUI — **RECOMMENDED COMPANION**
- Modern chat UI for LLM
- Docker-based, easy to deploy alongside
- Can be separate Zima app or documented companion
---
## Architecture: Two Zima Apps
```
┌─────────────────────────┐ ┌─────────────────────────┐
│ llama-server │ │ open-webui │
│ - REST API :8080 │────▶│ - Chat UI :3000 │
│ - Serves model │ │ - Connects to LLM API │
└─────────────────────────┘ └─────────────────────────┘
```
Both are separate Zima apps, deployed independently. OpenWebUI references `http://llama-server:8080` via Docker internal networking.
### App 1: `llama-server`
- Container: `ghcr.io/ggerganov/llama.cpp:server`
- Port: 8080
- Memory: 8G reservation
### App 2: `open-webui`
- Container: `ghcr.io/open-webui/open-webui:main`
- Port: 3000
- Memory: 2G reservation
- Environment: `OLLAMA_BASE_URL=http://llama-server:8080`
---
## App: `llama-server`
### Container: `ghcr.io/ggerganov/llama.cpp:server`
**Environment Variables**:
| Variable | Default | Description |
|----------|---------|-------------|
| `MODEL` | (required) | Model filename in `/models` |
| `CTX_SIZE` | 2048 | Context window size |
| `N_THREADS` | auto | CPU threads (auto = all) |
| `HOST` | 0.0.0.0 | Listen address |
| `PORT` | 8080 | API port |
| `MAX_TOKENS` | 512 | Max tokens to generate |
**Volumes**:
| Container | Description |
|-----------|-------------|
| `/models` | Model files (GGUF format) |
| `/DATA/AppData/$AppID/logs` | Server logs |
**Ports**:
| Container | Protocol | Description |
|-----------|----------|-------------|
| 8080 | TCP | llama.cpp REST API |
**Resources**:
- Memory reservation: **8G** (allows 7B Q4 models)
**Security**:
- `security_opt: no-new-privileges:true`
- `cap_drop: ALL`
- No privileged needed (CPU-only)
### Model Download (Documented in README)
Users download models manually:
```bash
# Example: Download Llama 3.2 3B Q4_K_M
curl -L -o /DATA/AppData/llama-server/models/llama-3.2-3b-q4_k_m.gguf \
"https://huggingface.co/QuantFactory/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct.Q4_K_M.gguf"
```
**Recommended Models for 16GB RAM**:
| Model | Size | Quant | RAM Needed | Speed (est) |
|-------|------|-------|------------|-------------|
| Llama 3.2 3B | 1.8GB | Q4_K_M | ~4GB | ~15-20 tok/s |
| Phi-3.5 Mini 3B | 1.8GB | Q4_K_M | ~4GB | ~15-20 tok/s |
| Mistral 7B | 4.1GB | Q4_K_M | ~6-7GB | ~8-12 tok/s |
| Qwen 2.5 7B | 4.4GB | Q4_K_M | ~6-7GB | ~8-12 tok/s |
---
## App: `open-webui`
### Container: `ghcr.io/open-webui/open-webui:main`
**Environment Variables**:
| Variable | Default | Description |
|----------|---------|-------------|
| `OLLAMA_BASE_URL` | http://llama-server:8080 | LLM API endpoint |
| `WEBUI_PORT` | 3000 | Web UI port |
**Ports**:
| Container | Protocol | Description |
|-----------|----------|-------------|
| 3000 | TCP | OpenWebUI |
**Resources**:
- Memory reservation: **2G**
**Notes**:
- Connects to `http://llama-server:8080` via Docker internal networking
- Requires `llama-server` app to be running first
---
## File Structure
```
Apps/llama-server/
├── docker-compose.yaml
├── README.md
└── HOW_TO_VERIFY.md (optional)
Apps/open-webui/
├── docker-compose.yaml
├── README.md
└── HOW_TO_VERIFY.md (optional)
```
---
## Implementation Steps
### llama-server
1. Create `Apps/llama-server/` directory
2. Write `docker-compose.yaml` with:
- Image: `ghcr.io/ggerganov/llama.cpp:server`
- 8G memory reservation
- Port 8080
- Model volume at `/models`
- Env vars: MODEL, CTX_SIZE, N_THREADS, HOST, PORT
3. Write `README.md` with:
- Model download instructions
- First-run setup
- API testing examples
- Performance tips for NUC8
4. Validate with `./scripts/validate-appstore.sh`
### open-webui
1. Create `Apps/open-webui/` directory
2. Write `docker-compose.yaml` with:
- Image: `ghcr.io/open-webui/open-webui:main`
- 2G memory reservation
- Port 3000
- Environment: `OLLAMA_BASE_URL=http://llama-server:8080`
3. Write `README.md` with:
- Prerequisites (llama-server must be running first)
- How to access
- Troubleshooting connection issues
4. Validate with `./scripts/validate-appstore.sh`
---
## Risk Assessment
| Risk | Level | Mitigation |
|------|-------|------------|
| NUC8 RAM insufficient for 7B with other apps | Medium | 8G reservation; close other apps for 7B |
| Model download issues | Low | Provide direct HF links in README |
| OpenWebUI API compatibility | Low | llama.cpp v1 API is OpenAI-compatible |
| Intel AVX2 performance | Low | llama.cpp auto-detects and uses AVX2 |
+93
View File
@@ -0,0 +1,93 @@
# Plan: Update AGENTS.md with Commit/Test/Build/Push Workflow
## Context
The repo's AGENTS.md (section 4 "Arbetsflöde för ändringar") currently only mentions `./scripts/validate-appstore.sh` as a loose recommendation. It lacks detailed guidance on the full development lifecycle: committing, testing images, building the appstore zip, and pushing.
## Goal
Add a new section to AGENTS.md (or expand section 4) covering the full workflow:
### A. Branch & Commit Workflow
- Branch naming per existing section 8
- Single-focus commits (one logical change per commit)
- Commit message format: short summary + bullet points for details
- What files can be committed (Apps/ scope rules from section 9)
### B. Image Verification (Pre-commit)
- Before committing compose changes, verify images are online:
```bash
docker manifest inspect <image:tag@sha256:...>
```
- Alternative: use build-appstore-zip.sh which does this automatically
### C. Local Validation
- Always run before push/PR:
```bash
./scripts/validate-appstore.sh
```
### D. Building the Appstore Package
- Script: `./scripts/build-appstore-zip.sh`
- Outputs to `dist/phirna-appstore.zip`
- Auto-generates SHA256 checksum
- Verifies all images online before building
- Commits `dist/` separately from app changes
### E. Push & Release
- Push order: app commits → build zip → commit zip → push
- PR description must include (per existing section 6):
- Affected app IDs
- Security risk level
- High-risk settings changes
## Proposed New Section (12) in AGENTS.md
```
## 12) Release- och publiceringsarbetsflöde
### Steg 1: Branch
Skapa branch enligt format i sektion 8:
<appnamn>/<initial|bugfix|update>/<detalj>
### Steg 2: Verifiera images (innan commit)
Kontrollera att alla Docker-images är tillgängliga online:
docker manifest inspect <image:tag@sha256:...>
### Steg 3: Validera lokalt
Kör validering innan commit:
./scripts/validate-appstore.sh
### Steg 4: Committa ändringar
- Små, reviewbara commits
- Separera appfiler från dist/-filer
- Commit-meddelande: rubrik + bulletpunkter
### Steg 5: Bygg appstore-zip
./scripts/build-appstore-zip.sh
- Skapar dist/phirna-appstore.zip
- Verifierar alla images
- Genererar SHA256
### Steg 6: Committa dist/
Separer commit för dist/ från appfiler:
git add dist/ && git commit -m "Build appstore zip"
### Steg 7: Push och PR
git push -u origin <branch>
Skapa PR med:
- Vilka app-id som påverkas
- Säkerhetsrisk (låg/medel/hög)
- Högrisk-inställningar vid introduktion
```
## Implementation
1. Read current AGENTS.md
2. Insert new section 12 after existing section 11
3. Renumber subsequent sections (12 → 13, etc.)
## Questions for User
- Should this be a new numbered section or expand existing section 4?
- Is `dist/phirna-appstore.zip` the correct output name for all repos, or should this be configurable?
+170
View File
@@ -0,0 +1,170 @@
# Plan: Gitea Bot User Setup for Tea CLI
## Context
Enable the agent (Kilo) to interact with Gitea (git.phirna.uk) via the `tea` CLI for:
- Creating branches
- Committing and pushing changes
- Creating pull requests
- Managing issues and labels
## Step 1: Username Suggestion
**Suggested username: `kilo-bot`**
While not directly Norse mythology, "Kilo" evokes the Norse root meaning "coal" or "torch". Alternatives if you prefer pure mythology:
| Username | Origin |
|----------|--------|
| `kilo-bot` | Kilo = "torch of life" from Old Norse "Kjöl" |
| `mimir-bot` | Mimir - Norse god of wisdom, keeper of knowledge |
| `hnir-bio` | Hnir - "breath" in Old Norse |
| `sowilo-bot` | Sowilo - the S rune, meaning "sun" |
**Recommendation**: `kilo-bot` — maintains brand consistency with the agent name "Kilo".
## Step 2: Required Permissions
Based on Gitea granular scopes, the bot needs:
| Scope | Reason |
|-------|--------|
| `write:repository` | Create branches, push commits, create PRs |
| `read:repository` | Read branches, commits, repos |
| `read:user` | Identify authenticated user |
| `write:issue` | Create/update issues if needed |
| `read:org` | Read org membership if needed |
**Alternative**: Use `write:repository, read:user` for minimal permissions covering all git operations.
**NOT needed**: `admin` (would allow deleting repos, managing orgs, etc.)
## Step 3: Create the Bot User
Requires admin access on git.phirna.uk. Steps:
1. Go to `https://git.phirna.uk/admin/users/new` (or use `tea admin`)
2. Create user `kilo-bot` with email (e.g., `kilo-bot@phirna.uk`)
3. Set a strong random password (store in password manager)
4. Optionally add to relevant organization(s)
## Step 4: Generate Access Token
1. Login as `kilo-bot`
2. Go to Settings → Applications → "Manage Access Tokens"
3. Create token with name `kilo-cli` and scopes:
- `repository:write`
- `user:read`
4. Copy the generated token securely
## Step 5: Configure Tea
```bash
tea logins add --name kilo-bot --url https://git.phirna.uk --token <generated-token>
```
Or set environment variable `GITEA_TOKEN` or configure in `~/.config/tea/config.yml`.
## Step 6: Update AGENTS.md
Add new section or subsection covering:
- Bot user credentials (token) storage approach
- Expected token scopes
- tea command patterns for common operations
- Security considerations (bot has limited scope)
## Step 7: Create Skill (optional but recommended)
Create `.kilo/.skills/gitea-agent.md`:
- Standardized tea commands for branch creation
- Commit/push workflow via tea
- PR creation commands
- Issue management shortcuts
## Security Considerations
- Bot should **NOT** be admin
- Token should be scoped to `write:repository` + `read:user` only
- Token stored in environment or secured config, NOT in repo
- Consider setting bot's `MaxTokenPermissions` at org level if supported
## Verification Steps
After setup, test:
```bash
# Verify identity
tea whoami --login kilo-bot
# List repos (should see assigned repos)
tea repos list --login kilo-bot
# Create a test branch
git checkout -b test/tea-cli-test
git push -u origin test/tea-cli-test --force
```
## Confirmed Decisions
| Decision | Choice |
|----------|--------|
| Username | `mimir` (already created on Gitea) |
| Token storage | Tea login system (`tea logins add`) |
| Access scope | All repos on git.phirna.uk |
## Implementation Steps
### Step 1: Generate Access Token for mimir
1. Login to `https://git.phirna.uk` as `mimir`
2. Go to Settings → Applications → "Manage Access Tokens"
3. Create token with name `kilo-cli` and scopes:
- `repository:write`
- `user:read`
4. Copy the generated token securely
### Step 2: Add Tea Login
```bash
tea logins add --name mimir --url https://git.phirna.uk --token <generated-token>
```
Verify with:
```bash
tea whoami --login mimir
```
### Step 3: Update AGENTS.md
Add new section under "Repo-invarianter" or create new section:
```
## X) Gitea Bot (mimir)
För att kunna skapa branches, commits och PRs via tea:
- Användare: `mimir` på git.phirna.uk
- Token lagras i tea's login system (`tea logins add`)
- Scopes: `repository:write`, `user:read`
- Säkerhet: mimir är inte admin, token har begränsade scopes
Exempel-kommandon:
tea pr create --login mimir --owner <owner> --repo <repo>
tea pulls list --login mimir --owner <owner> --repo <repo>
```
### Step 4: Create Skill (optional)
Create `.kilo/.skills/gitea-agent.md` with tea command patterns.
### Step 5: Verify Access
```bash
# Should show mimir's identity
tea whoami --login mimir
# Should list repos mimir can access
tea repos list --login mimir
```
+99 -1
View File
@@ -16,7 +16,9 @@ Fokus: korrekthet, låg risk och underhållbarhet.
- `docker-compose.yaml` - `docker-compose.yaml`
- `README.md` (syfte, portar, volymer, privilegier, risker) - `README.md` (syfte, portar, volymer, privilegier, risker)
- Compose ska ha giltig top-level `name` (gemener + bindestreck). - Compose ska ha giltig top-level `name` (gemener + bindestreck).
- Om `container_name` används måste den vara max 32 tecken och bara innehålla `0-9`, `a-z`, `A-Z`, `_`, `-`.
- Endast `.yaml` används i repot (aldrig `.yml`). - Endast `.yaml` används i repot (aldrig `.yml`).
- Swap-filer (t.ex. `*.swp`, `*.swo`) ska alltid ignoreras och får aldrig committas.
## 3) Säkerhetsbaseline (Compose) ## 3) Säkerhetsbaseline (Compose)
@@ -24,6 +26,9 @@ MUST:
- Pinna images till explicit version eller digest. - Pinna images till explicit version eller digest.
- Inte använda `:latest`. - Inte använda `:latest`.
- Hålla volymer snäva (`/DATA/AppData/$AppID/...`). - Hålla volymer snäva (`/DATA/AppData/$AppID/...`).
- Vid förslag/byte av `image:` måste imagen verifieras online (manifest/tag finns i registry) innan merge.
- Använd App/_template som grund om inget annat anges.
- deply.resources.reservations ska sättas till något passande. Detta kommer även parsas av ZimaOS webui
SHOULD: SHOULD:
- `security_opt: ["no-new-privileges:true"]` - `security_opt: ["no-new-privileges:true"]`
@@ -97,6 +102,11 @@ Regler för `<detalj>`:
- När en ny app skapas ska en ny mapp alltid skapas under `Apps/<app-id>/`. - När en ny app skapas ska en ny mapp alltid skapas under `Apps/<app-id>/`.
- Flera agenter kan arbeta samtidigt. Fråga alltid innan du skapar ny branch eller byter branch, för att undvika krockar. - Flera agenter kan arbeta samtidigt. Fråga alltid innan du skapar ny branch eller byter branch, för att undvika krockar.
- `git add` och `git commit` får endast omfatta filer i den egna appens undermapp under `Apps/<app-id>/`. - `git add` och `git commit` får endast omfatta filer i den egna appens undermapp under `Apps/<app-id>/`.
- För varje fil som en agent tänker ändra ska agenten först skapa en tom låsfil: `<filnamn>.agent_wip`.
- Om `<filnamn>.agent_wip` redan finns ska agenten vänta 60 sekunder och kontrollera igen.
- Upprepa väntan i upp till 10 minuter (10 försök). Om låsfilen fortfarande finns efter 10 minuter: avbryt och be användaren om beslut.
- När agenten är klar med filen ska motsvarande `<filnamn>.agent_wip` tas bort direkt.
- Låsfilen ska tas bort även vid fel/avbrott så långt det går, för att undvika falska lås.
## 10) Verifieringsdokument när app är redo ## 10) Verifieringsdokument när app är redo
@@ -112,9 +122,97 @@ Filen ska vara praktiskt körbar och innehålla:
- exakta kommandon för att verifiera DNS, nät och TLS, - exakta kommandon för att verifiera DNS, nät och TLS,
- en sektion \"data att samla\" för snabb Codex-felsökning. - en sektion \"data att samla\" för snabb Codex-felsökning.
Sektionen \"data att samla\" ska minst täcka: Sektionen "data att samla" ska minst täcka:
- versions-/buildinfo (appversion, branch/commit eller zip + checksum), - versions-/buildinfo (appversion, branch/commit eller zip + checksum),
- relevant konfiguration (med maskade secrets), - relevant konfiguration (med maskade secrets),
- loggar från berörda containers, - loggar från berörda containers,
- konkreta felobservationer (hostname, tidpunkt, förväntat vs faktiskt beteende). - konkreta felobservationer (hostname, tidpunkt, förväntat vs faktiskt beteende).
<<<<<<< HEAD
## 11) Release- och publiceringsarbetsflöde
### Steg 1: Branch
Skapa branch enligt format i sektion 8:
`<appnamn>/<initial|bugfix|update>/<detalj>`
### Steg 2: Verifiera images (innan commit)
Kontrollera att alla Docker-images är tillgängliga online. Scriptet `build-appstore-zip.sh` verifierar automatiskt -- kör det för att kontrollera, eller använd:
```bash
docker manifest inspect <image:tag@sha256:...>
```
### Steg 3: Validera lokalt
Kör validering innan commit:
```bash
./scripts/validate-appstore.sh
```
### Steg 4: Committa ändringar
- Små, reviewbara commits.
- Separera appfiler från `dist/`-filer.
- Commit-meddelande: rubrik + bulletpunkter.
### Steg 5: Bygg appstore-zip
```bash
./scripts/build-appstore-zip.sh
```
- Skapar `dist/phirna-appstore.zip`.
- Verifierar alla images online automatiskt.
- Genererar SHA256 checksum.
- Med `CI=true` eller `--strict-images` misslyckas bygget om en image saknas.
### Steg 6: Committa dist/
Separer commit för `dist/` från appfiler:
```bash
git add dist/ && git commit -m "Build appstore zip"
```
### Steg 7: Push och PR
```bash
git push -u origin <branch>
```
PR ska inkludera:
- Vilka app-id som påverkas.
- Säkerhetsrisk (låg/medel/hög).
- Högrisk-inställningar vid introduktion eller förändring.
## 11) Gitea Bot (mimir)
För att kunna skapa branches, commits och PRs via tea-CLI:
- **Användare**: `mimir` på git.phirna.uk
- **Token**: Lagras i tea's login-system via `tea logins add`
- **Scopes**: `repository:write`, `user:read`
- **Säkerhet**: mimir är inte admin, token har begränsade scopes
### Vanliga kommandon
```bash
# Sätt aktiv login
export GITEA_LOGIN=mimir
# Lista repos
tea repos list --login mimir
# Skapa branch och push
git checkout -b <branch-name>
git push -u origin <branch-name>
# Skapa PR
tea pulls create --login mimir --owner <owner> --repo <repo> --head <branch> --base <target>
# Lista öppna PRs
tea pulls list --login mimir --owner <owner> --repo <repo>
# Hantera issues
tea issues list --login mimir --owner <owner> --repo <repo>
tea issues create --login mimir --owner <owner> --repo <repo> --title "Titel" --body "Body"
```
+16 -11
View File
@@ -2,19 +2,24 @@ name: sample-app
services: services:
app: app:
image: ghcr.io/example/sample-app:1.0.0 # Setting this will allow the Zima webui to display the field "REQUIRE MEMORY"
deploy:
resources:
reservations:
memory: 64M
image: joafri/sample-app-app:main
container_name: sample-app container_name: sample-app
restart: unless-stopped restart: unless-stopped
environment: environment:
TZ: ${TZ} TZ: Europe/Stockholm
PUID: ${PUID} PUID: "1000"
PGID: ${PGID} PGID: "1000"
WEBUI_PORT: ${WEBUI_PORT:-8080} WEBUI_PORT: "8080"
ports: ports:
- target: 8080 - target: 8080
published: ${WEBUI_PORT:-8080} published: "8080"
protocol: tcp protocol: tcp
volumes: volumes:
@@ -53,10 +58,10 @@ x-casaos:
- amd64 - amd64
- arm64 - arm64
main: app main: app
category: Utilities category: phirna
author: Zima Apps Team author: Joachim Friberg
developer: example developer: Joachim Friberg
icon: https://example.invalid/icon.png icon: https://cdn.simpleicons.org/docker
tagline: tagline:
en_US: Replace with a short one-line value proposition en_US: Replace with a short one-line value proposition
description: description:
@@ -64,4 +69,4 @@ x-casaos:
title: title:
en_US: Sample App en_US: Sample App
index: / index: /
port_map: ${WEBUI_PORT:-8080} port_map: "8080"
+130
View File
@@ -0,0 +1,130 @@
# ARK: Survival Ascended Server
ARK: Survival Ascended dedicated server container for ZimaOS.
## Overview
This app provides a dedicated server for ARK: Survival Ascended, allowing you to host your own game server with automatic updates and remote administration capabilities.
## Features
- **Automatic Updates**: Server files are automatically updated on startup
- **Cluster Support**: Configure multiple servers to work together
- **Remote Administration**: RCON access for server management
- **Customizable**: Adjust game parameters, player limits, and maps
- **Secure by Default**: Runs with minimal privileges and capabilities
## Configuration
### Ports
- **7777/udp**: Game port for player connections
- **27020/tcp**: RCON port for remote server administration
### Volumes
All volumes are mounted under `/DATA/AppData/ark-suac/`:
- `steam`: Steam runtime files
- `steamcmd`: SteamCMD installation
- `server-files`: ARK server files and save data
- `cluster-shared`: Shared data for cluster setups
### Environment Variables
- `TZ`: Timezone (default: Europe/Stockholm)
- `ASA_START_PARAMS`: Server start parameters and map configuration
- `ENABLE_DEBUG`: Set to "1" to enable debug mode (prevents server startup)
## Security
### Baseline Security
- Runs as non-root user `gameserver`
- All capabilities dropped (`cap_drop: ALL`)
- `no-new-privileges` security option enabled
- No privileged mode required
- No Docker socket access
### Permission Bootstrap
The compose includes a one-shot helper service `set-permissions` that runs as `root`
before the game server starts. It only performs `chown` on app-specific bind mounts so
the main `ark-asa-server` service can run as non-root `gameserver`.
### Data Persistence
All game data, including server files, save games, and configuration, is persisted in the bound volumes. The container will automatically download and update server files on startup.
## Usage
### First Start
1. Install the app through ZimaOS App Store
2. Configure desired map and settings via `ASA_START_PARAMS`
3. Start the container - first startup will download ~11GB of server files
4. Connect to your server via the ARK game client
### Map Configuration
Change the map by modifying `ASA_START_PARAMS`. Example maps:
- `TheIsland_WP` (default)
- `ScorchedEarth_WP`
- `Ragnarok_WP`
- `Valguero_WP`
### Cluster Setup
For multi-server clusters:
1. Use the same `clusterid` parameter for all servers
2. Share the `cluster-shared` volume between containers
3. Configure different ports for each server instance
### Remote Administration
Connect to RCON on port 27020 using tools like:
- [ARK Server API](https://ark-server-api.com/)
- [Battlemetrics RCON](https://www.battlemetrics.com/rcon)
- Command line: `nc hostname 27020`
## Performance Requirements
- **RAM**: ~13GB per server instance
- **Disk**: ~11GB for server files + space for save games
- **CPU**: Modern multi-core processor recommended
## Updates
Server files are automatically updated on container startup. To update the container image itself, pull the new version through ZimaOS.
## Troubleshooting
### Server Not Visible
- Verify port 7777/udp is properly forwarded
- Check firewall settings
- Ensure `ASA_START_PARAMS` includes `?listen` parameter
### Connection Timeouts
- Verify your network configuration
- Check that UDP traffic is not being blocked
- Ensure the server has completed initialization (can take several minutes)
### Debug Mode
Set `ENABLE_DEBUG=1` to prevent server startup and investigate container issues.
## References
- [Upstream Repository](https://github.com/mschnitzer/asa-linux-server)
- [ARK Official Website](https://survivetheark.com/)
- [SteamCMD Documentation](https://developer.valvesoftware.com/wiki/SteamCMD)
## Support
For issues specific to this ZimaOS integration, please contact the Zima Apps Team.
For game server issues, refer to the upstream repository or ARK official support.
@@ -0,0 +1 @@
root/usr/share/proton/GE-Proton8-21.tar.gz filter=lfs diff=lfs merge=lfs -text
@@ -0,0 +1,20 @@
.PHONY: prepare build build-development build-beta load
GLOBAL_BUILD_DIR = /tmp/.kiwi-build-results
TARGET_DIR = $(GLOBAL_BUILD_DIR)/ark-survival-ascended-linux-container-image
prepare:
- sudo rm -rf $(TARGET_DIR)
- mkdir -p $(GLOBAL_BUILD_DIR)
build: prepare
- sudo kiwi-ng --profile stable --color-output --debug system build --target-dir $(TARGET_DIR) --description .
build-development: prepare
- sudo kiwi-ng --profile development --color-output --debug system build --target-dir $(TARGET_DIR) --description .
build-beta: prepare
- sudo kiwi-ng --profile beta --color-output --debug system build --target-dir $(TARGET_DIR) --description .
load:
- sudo docker load -i $(TARGET_DIR)/*.xz
@@ -0,0 +1,592 @@
# ARK: Survival Ascended - Dedicated Linux Server - Docker Image
This repository provides a step by step guide for Linux administrators to host ARK: Survival Ascended servers on Linux using a docker image.
## Table of Contents
* [Hardware Requirements](#hardware-requirements)
* [Installation](#installation)
* [Install Docker & Docker Compose](#1-install-docker--docker-compose)
* [Start docker daemon](#2-start-docker-daemon)
* [Create the Docker Compose config](#3-create-the-docker-compose-config)
* [First server start](#4-first-server-start)
* [Server configuration](#5-server-configuration)
* [Changing the start parameters AND the player limit](#6-changing-the-start-parameters-and-the-player-limit)
* [Port forwarding?](#port-forwarding)
* [Changing the game port and RCON port](#changing-the-game-port-and-rcon-port)
* [Start/Restart/Stop](#startrestartstop)
* [Server Administration](#server-administration)
* [Debug Mode](#debug-mode)
* [Applying server updates](#applying-server-updates)
* [Daily restarts](#daily-restarts)
* [Executing RCON commands](#executing-rcon-commands)
* [Setting up a second server / cluster](#setting-up-a-second-server--cluster)
* [Adding Mods](#adding-mods)
* [Adding Mod Maps](#adding-mod-maps)
* [Adding Plugins](#adding-plugins)
* [Map Names](#map-names)
* [Updating the Container Image](#updating-the-container-image)
* [Common Issues](#common-issues)
* [Server is not visible in server browser](#server-is-not-visible-in-server-browser)
* [Addressing "Connection Timeout" issues](#addressing-connection-timeout-issues)
* [Your server has multiple IPv4 addresses](#your-server-has-multiple-ipv4-addresses)
* [Debugging with curl](#debugging-with-curl)
* [How to customize your routing?](#how-to-customize-your-routing)
* [Making your iptable rules persistent](#making-your-iptable-rules-persistent)
* [Found an Issue or Bug?](#found-an-issue-or-bug)
* [Credits](#credits)
## Hardware Requirements
The hardware requirements might change over time, but as of today you can expect:
* ~13 GB RAM usage per server instance
* ~11 GB disk space (the server files alone, without any savegames)
I cannot tell you what CPU to use, as I didn't do any testing on this, but this is the hardware I'm running one ASA server on:
* Intel Xeon E3-1275v5
* 2x SSD M.2 NVMe 512 GB
* 4x RAM 16384 MB DDR4 ECC
The server runs next to other services and it runs pretty well.
## Installation
Required Linux experience: **Beginner**
In theory, you can use these steps on any Linux system where Docker is installed. It has been tested with:
* openSUSE Leap 15.6 (>= 16.0 won't work due to missing 32bit support)
* Debian 12 (bookworm)
* **NOT WORKING:** Ubuntu 22.04.x LTS (Jammy Jellyfish) [As of March 28th 2025, a recent distro update causes the container to have a constant high CPU usage, well beyond 400% and the server won't launch. Use Ubuntu 24.04.x if you can]
* Ubuntu 24.04.1 (Noble Numbat)
You need to be root user (`su root`) to perform these steps, but don't worry, the ASA server itself will run rootless.
### 1. Install Docker & Docker Compose
#### openSUSE Leap 15.6:
```
zypper in -y docker docker-compose
```
#### Debian 12
It is recommended to install the docker engine from Docker's official repository. Follow the instructions in [this guide](https://docs.docker.com/engine/install/debian/#install-using-the-repository)
and refer to the "Install using the apt repository" section.
#### Ubuntu (24.04.x):
The docker engine is not part of the official Ubuntu 24.x repositories, thus you need to install it from the Docker's repository instead. Please refer to
[this guide](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository) and follow the steps outlined in the "Install using the apt repository" section.
### 2. Start docker daemon
```
systemctl start docker
systemctl enable docker
```
### 3. Create the Docker Compose config
Create a directory called `asa-server` wherever you like and download [my docker-compose.yml](https://github.com/mschnitzer/ark-survival-ascended-linux-container-image/blob/main/docker-compose.yml) example.
```
mkdir asa-server
cd asa-server
wget https://raw.githubusercontent.com/mschnitzer/ark-survival-ascended-linux-container-image/main/docker-compose.yml
```
### 4. First server start
Now start the server for the first time. It will install Steam, Proton, and downloads the ARK: Survival Ascended server files.
Go to the directory of your `docker-compose.yml` file and execute the following command:
```
docker compose up -d
```
It will download my docker image and then spins up a container called `asa-server-1` (defined in `docker-compose.yml`). You can follow the installation and the start of your server by running:
```
docker logs -f asa-server-1
```
(Note: You can safely run `CTRL + C` to exit the log window again without causing the server to stop)
Once the log shows the following line:
```
Starting the ARK: Survival Ascended dedicated server...
```
... the server should be reachable and discoverable through the server browser in ~2-5 minutes.
The server name is randomly generated upon the first start. Please execute the following command to see under which name the server is discoverable in the server browser:
```
docker exec asa-server-1 cat server-files/ShooterGame/Saved/Config/WindowsServer/GameUserSettings.ini | grep SessionName
```
If the command fails in execution and reports an `No such file or directory` error, just wait some more minutes and it should eventually work. Once the command executed successfully, it should output something like this:
```
SessionName=ARK #334850
```
Now try to find the server by its name. Just search in the "Unofficial" section in ASA for the number of the server. In my case it is `334850`. If you are not able to connect to it right away, wait up to 5 more minutes and
try it again. If it's still not possible, [open an issue on GitHub](https://github.com/mschnitzer/ark-survival-ascended-linux-container-image/issues/new) to get help.
Once confirmed that you are able to connect, stop the server again:
```
docker stop asa-server-1
```
### 5. Server configuration
The `docker-compose.yml` config defines three docker volumes, which serve as a storage for your server files, Steam, and Proton. They are directly mounted to the docker container and can be edited outside of the container. The
location of these volumes is `/var/lib/docker/volumes`. If you followed the steps 1:1, then you should find the following directories at that location:
```
asa-server_cluster-shared/
asa-server_server-files-1/
asa-server_steam-1/
asa-server_steamcmd-1/
```
The prefix `asa-server` is defined by the directory name of your `docker-compose.yml` file.
You can ignore `asa-server_steam-1` and `asa-server_steamcmd-1`, these volumes are being used by the container to avoid setting up `Steam` and `steamcmd` on every launch again. Server files including config files are stored at `asa-server_server-files-1`. `asa-server_cluster-shared` provides support for server clusters, so that survivors can travel between your servers with their characters and dinos.
The `GameUserSettings.ini` and `Game.ini` file can be found at `/var/lib/docker/volumes/asa-server_server-files-1/_data/ShooterGame/Saved/Config/WindowsServer`. The `Game.ini` file is not there by default, so you might want to create it yourself.
You don't need to worry about file permissions. The `docker-compose.yml` is running a container before starting the ASA server and adjusts the file permissions to `25000:25000`, which is the user id and group id the server starts with. These ids are not bound to any user on your system and that's fine and not an issue.
### 6. Changing the start parameters AND the player limit
Start parameters are defined in the `docker-compose.yml`:
```yml
...
environment:
- ASA_START_PARAMS=TheIsland_WP?listen?Port=7777?RCONPort=27020?RCONEnabled=True -WinLiveMaxPlayers=50
...
```
Please note:
* The value before `?listen` is the name of the map the server launches with. ([See all official map names](#map-names))
* Please do not remove `?listen` from the parameters, otherwise the server is not binding ports
* `?Port=` is the server port players connect to
* `?RCONPort=` is the port of the RCON server that allows remote administration of the server
* The player limit is set by `-WinLiveMaxPlayers`. Please note that for ASA servers, editing the player limit via `GameUserSettings.ini` is not working.
## Port forwarding?
There should not be the need to forward any ports if your server runs in a public cloud. This is because docker configures `iptables` by itself. In a home setup, where a router is in between, it is very likely that you need to forward ports.
In any case, you ONLY need to forward the following ports:
```
7777 (UDP only - This is the game port to allow players to connect to the server)
27020 (TCP only - This is the port to connect through RCON and is therefore optional to forward)
```
As of today, ASA does no longer offer a way to query the server, so there's no query port and you won't be able to find your server through the Steam server browser, only via the ingame browser.
## Changing the game port and RCON port
You already learned that ports are defined by `ASA_START_PARAMS` in the `docker-compose.yml` file. This just tells the ASA server what ports to bind.
As a first step for port changes adjust the start parameters accordingly.
E. g. if you want to change the game port from `7777` to `7755` your new start parameters would look like this:
```yml
...
environment:
- ASA_START_PARAMS=TheIsland_WP?listen?Port=7755?RCONPort=27020?RCONEnabled=True -WinLiveMaxPlayers=50 -clusterid=default -ClusterDirOverride="/home/gameserver/cluster-shared"
- ENABLE_DEBUG=0
...
```
But this alone is not enough and you need to apply the following changes as well.
Open the `docker-compose.yml` file again and edit the lines where the container ports are defined:
```yml
...
ports:
# Game port for player connections through the server browser
- 0.0.0.0:7777:7777/udp
# RCON port for remote server administration
- 0.0.0.0:27020:27020/tcp
...
```
Adjust the port to your liking, but make sure that you change both numbers (the one before and after the `:`). Assuming the above game port changes to `7755` this would be the result:
```yml
...
ports:
# Game port for player connections through the server browser
- 0.0.0.0:7755:7755/udp
# RCON port for remote server administration
- 0.0.0.0:27020:27020/tcp
...
```
Now that your port changes are set, you have to recreate your container. Therefore you need to use `docker compose up -d` in order to apply your port changes.
## Start/Restart/Stop
To perform any of the actions, execute the following commands (you need to be in the directory of the `docker-compose.yml` file):
```
docker compose start asa-server-1
docker compose restart asa-server-1
docker compose stop asa-server-1
```
You can also use the native docker commands, where you do not need to be in the directory of the `docker-compose.yml` file. However using this method would not check for changes in your `docker-compose.yml` file.
So in case you edited the `docker-compose.yml` file (e.g. because you adjusted the start parameters), you need to use `docker compose` commands instead.
```
docker start/restart/stop asa-server-1
```
## Server Administration
### Debug Mode
Sometimes you want to test something inside the container without starting the ASA server. The debug mode can be enabled by changing `- ENABLE_DEBUG=0` to `1` in the `docker-compose.yml` file.
Once done, the result will look like this:
```yml
...
version: "3.3"
services:
asa-server-1:
container_name: asa-server-1
hostname: asa-server-1
entrypoint: "/usr/bin/start_server"
user: gameserver
image: "mschnitzer/asa-linux-server:latest"
environment:
- ASA_START_PARAMS=TheIsland_WP?listen?Port=7777?RCONPort=27020?RCONEnabled=True -WinLiveMaxPlayers=50
- ENABLE_DEBUG=1
...
```
Now run `docker compose up -d` and the container will just start without launching the server or validating server files.
Check if the container launched in debug mode by running `docker logs -f asa-server-1` and check whether it's saying "Entering debug mode...". If that's the case, you are good.
You can enter the shell of your server by running
```
docker exec -ti asa-server-1 bash
```
If you need root access run
```
docker exec -ti -u root asa-server-1 bash
```
### Applying server updates
Updates will be automatically downloaded or applied once you restart the container with ...
```
docker restart asa-server-1
```
It is totally possible that after a restart and applying all updates, the client is still one or more versions ahead. This is because Wildcard does sometimes run client-only updates, since not all
updates are affecting the server software. This is not a problem at all. As long as you can connect to your server, everything is fine. The server software checks for incompatible client
versions anyway.
In general you can check when the latest server update was published by Wildcard, by checking [this link](https://steamdb.info/app/2430930/depots/). The section mentioning the last update of the `public` branch
tells you when the last update was rolled out for the server software.
If you have any doubts on this, open a GitHub issue.
### Daily restarts
As `root` user of your server (or any other user that is member of the `docker` group) open your crontab configuration:
```
crontab -e
```
Add the following lines to it:
```
30 3 * * * docker exec asa-server-1 asa-ctrl rcon --exec 'serverchat Server restart in 30 minutes'
50 3 * * * docker exec asa-server-1 asa-ctrl rcon --exec 'serverchat Server restart in 10 minutes'
57 3 * * * docker exec asa-server-1 asa-ctrl rcon --exec 'serverchat Server restart in 3 minutes'
58 3 * * * docker exec asa-server-1 asa-ctrl rcon --exec 'saveworld'
0 4 * * * docker restart asa-server-1
```
Explanation:
* Line 1: Every day at 03:30am of your server's timezone, a message will be sent to all players announcing a restart in 30 minutes.
* Line 2: Every day at 03:50am of your server's timezone, a message will be sent to all players announcing a restart in 10 minutes.
* Line 3: Every day at 03:57am of your server's timezone, a message will be sent to all players announcing a restart in 3 minutes.
* Line 4: Every day at 03:58am of your server's timezone, the server saves the world before the restart happens.
* Line 5: Every day at 04:00am of your server's timezone, the ASA server gets restarted and installs pending updates from Steam.
Read more about the crontab syntax [here](https://www.adminschoice.com/crontab-quick-reference).
**NOTE:** The first 4 lines execute RCON commands, which requires you to have a working RCON setup. Please follow the instructions in section "[Executing RCON commands](#executing-rcon-commands)" to
ensure you can execute RCON commands.
### Executing RCON commands
You can run RCON commands by accessing the `rcon` subcommand of the `asa-ctrl` tool which is shipped with the container image. There's no need to enter your server password, IP, or RCON port manually. As long as
you have set your RCON password and port, either as a start parameter or in the `GameUserSettings.ini` file of your server, `asa-ctrl` is able to figure those details out by itself.
The following variables need to be present in `GameUserSettings.ini` under the `[ServerSettings]` section:
```
RCONEnabled=True
ServerAdminPassword=mysecretpass
RCONPort=27020
```
**NOTE:** There can be issues setting `ServerAdminPassword` as command line option. I'd suggest to set it in the `GameUserSettings.ini` file only.
Example:
```
docker exec -t asa-server-1 asa-ctrl rcon --exec 'saveworld'
```
**NOTE:** As opposed to ingame cheat commands, you must not put `admincheat` or `cheat` in front of the command.
## Setting up a second server / cluster
Setting up a second server is quite easy and you can easily add more if you want (given that your hardware is capable of running multiple instances). There's already a definition for a second server in the `docker-compose.yml` file,
but the definition is commented out by a leading `#`. If you remove these `#`, and run `docker compose up -d` again, then the second server should start and it will listen on the game port `7778` and the query port `27021`. Please note that
the server files, as well as Steam, and steamcmd will be downloaded again and the first start can take a while.
You can edit the start parameters in the same way like for the first server and the files of the second server are located at the same location, except that the second server has its suffix changed from `-1` to `-2`. The directories will therefore,
named like this:
```
asa-server_server-files-2/
asa-server_steam-2/
asa-server_steamcmd-2/
```
That's it! Your second server is now running in a cluster setup. This means that travelling between your servers is possible through Obelisks. If you do not want players to travel between your servers, you need to remove the `-clusterid` option
from the start parameters. It's advised to change the `-clusterid` parameter for all of your servers to a random string and keep it secret (e.g. `-clusterid=aSM42F6PLaPk` as opposed to `-clusterid=default`). The reason for that is that you will
end up seeing also other servers from the community that use `default` as their `clusterid`. If you only want players to travel between your own servers, then the `clusterid` must be different.
If you want to spin up more servers, you need to add more entries to the `docker-compose.yml` file. The following sections need to be edited: `services` and `volumes`. Make sure that you adjust all suffixes and replace them with a new one
(e.g. `-3` now) for the newly added entries.
## Adding Mods
Mods can be added by adjusting the `docker-compose.yml` file and adding a `-mods` option to the start parameters.
e.g.
```
[...]
- ASA_START_PARAMS=TheIsland_WP?listen?Port=7777?RCONPort=27020?RCONEnabled=True -WinLiveMaxPlayers=50 -mods=12345,67891
[...]
```
Once done, restart the server using `docker compose up -d`. It might take longer until the server comes up, because the server has to download the mods first.
Mod IDs are usually somewhere listed on the mod page of a mod on curseforge.com.
### Adding Mod Maps
Search for a map on curseforge.com and find out what mod id the map has and what the map name is. For the map [Svartalfheim](https://www.curseforge.com/ark-survival-ascended/mods/svartalfheim) the map name
is `Svartalfheim_WP` and the mod id is `893657`.
Once you found out the information you need, you need to adjust your start parameters in the `docker-compose.yml` file and add the map name, as well as the `-mods` option.
e.g.
```
[...]
- ASA_START_PARAMS=Svartalfheim_WP?listen?Port=7777?RCONPort=27020?RCONEnabled=True -WinLiveMaxPlayers=50 -mods=893657
[...]
```
Restart your server using `docker compose up -d`. It may take a while, as the server has to download the map, so be patient.
## Adding Plugins
Plugin support was introduced by version 1.4.0 of this container image. So make sure that you updated to the latest version of the container image or to version 1.4.0 as described [here](#updating-the-container-image).
There's a project ([see here](https://gameservershub.com/forums/resources/ark-survival-ascended-serverapi-crossplay-supported.683/)) that allows you to load plugins on your server (e.g. permission handling). To install the plugin loader, please visit [gameservershub.com](https://gameservershub.com/forums/resources/ark-survival-ascended-serverapi-crossplay-supported.683/) and refer to the "ServerAPI Installation Steps" section and download the zip archive. A `gameservershub.com` account is required in order to download the plugin loader.
When the download of the zip archive is completed, follow these steps to install the plugin loader:
1. Make sure that you launched the ASA server at least once without the plugin loader.
2. Stop the ASA server container by running `docker stop asa-server-1`
3. Enter the server files binary directory as `root` user: `cd /var/lib/docker/volumes/asa-server_server-files-1/_data/ShooterGame/Binaries/Win64`
4. Place the downloaded zip archive in that directory (the name of the archive must start with `AsaApi_`). Do not unzip the content.
5. Restart your server using `docker compose up -d`
The installation happens automatically by the container start script. You can follow the installation process by running `docker logs -f asa-server-1`. Once the log says "Detected ASA Server API loader. Launching server through AsaApiLoader.exe",
the installation is complete. In the following log lines your should see the start process of the plugin loader.
How to install plugins is described on gameservershub.com, from which you obtained the plugin loader. Please refer to their guide instead.
## Map Names
This is a list of all official map names with their map id. The map id is used as start parameter in the `docker-compose.yml` file. ([click](#6-changing-the-start-parameters-and-the-player-limit))
| Map Name | Map ID (for the start parameter) |
| ------------- | ------------- |
| The Island | TheIsland_WP |
| Scorched Earth | ScorchedEarth_WP |
| The Center | TheCenter_WP |
| Aberration | Aberration_WP |
| Extinction | Extinction_WP |
| Ragnarok | Ragnarok_WP |
| Astraeos | Astraeos_WP |
| Valguero | Valguero_WP |
| Lost Colony | LostColony_WP |
**NOTE:** Mod Maps have their own id! ([click](#adding-mod-maps))
## Updating the Container Image
The container image will be updated from time to time. In general, we try to not break previous installations by an update, but to add certain features, it might be necessary to introduce backward incompatibilities.
The default `docker-compose.yml` file suggests to use the `latest` branch of the container image. If you want to stay on one specific version, you can force the container image to launch with that said version, by
changing `image: "mschnitzer/asa-linux-server:latest"` in your `docker-compose.yml` file (as outlined below) to whatever version suits you. A list of all versions can be
found [here](https://hub.docker.com/r/mschnitzer/asa-linux-server/tags).
For example:
If you want to stay on version `1.4.0` for your ASA server, you must change `image: "mschnitzer/asa-linux-server:latest"` to `image: "mschnitzer/asa-linux-server:1.4.0"`.
Even if you stay on branch `latest`, your container image won't be updated automatically if we roll out an update. You explicitly need to run `docker pull mschnitzer/asa-linux-server:latest` to obtain the newest version.
We strongly suggest to read through the [releases page](https://github.com/mschnitzer/ark-survival-ascended-linux-container-image/releases) of this repository to see what has changed between versions. If there's
a backward incompatibility being introduced, it will be mentioned there with an explanation what to change.
## Common Issues
### Server is not visible in server browser
If you cannot discover your server in the server browser, it's most likely due to at least one of the following reasons:
* Your server is still booting up, give it ~5 minutes
* You are not looking at the "Unofficial" server browser list
* Your filter settings in the server browser exclude your server
* You forgot clicking the "Show player server settings". ([view screenshot](https://raw.githubusercontent.com/mschnitzer/ark-survival-ascended-linux-container-image/main/assets/show-player-servers.jpg)) By default, only Nitrado servers are shown to players when searching for unofficial servers, unfortunately.
## Addressing "Connection Timeout" issues
First of all, try to connect through the ingame console to your server. In many cases this works, but only connecting through the server browser causes an issue. Try to run the command `open $IP:$PORT` and test whether you
can connect to it.
If that is NOT working and you are having a home setup and not a VPS cloud setup, make sure your ports are REALLY open. This needs to be configured on your router. The ports that need to be opened are listed above in this README.
Please refer to the documentation of your router how to configure port forwarding properly.
If you can connect to your server through the console command, but not via the sever browser, it is very likely that you are running into one of these issues:
### Your server has multiple IPv4 addresses
If your server has multiple IPv4 addresses and you bound your ASA server to one of the secondary ones, by default, docker routes your traffic always through your primary network interface, which would cause the server browser to list your
server under the wrong IP address.
For example:
Your primary IP is: `255.255.300.300`
Your secondary one is: `255.255.400.400`
You adjusted the `docker-compose.yml` file in a way where it binds the ports on interface `255.255.400.400`. However, if your ASA server communicates with the internet and announces itself to the ASA server list, the ASA master server that manages the
server browser entries, would see the requests coming from `255.255.300.300` as this is your primary network interface.
This issue can be solved by forcing the traffic to be routed manually through your secondary network interface.
But before we start fixing it, you should make sure that this is really the issue.
#### Debugging with curl
1. Log in to the container `docker exec -ti -u root asa-server-1 bash`
2. Run `zypper --no-gpg-checks ref`
3. Install curl `zypper in -y curl`
4. Run `curl icanhazip.com` (`icanhazip.com` is a service that tells you from what ip address it received traffic from)
If the service responds with an IP that you have not assigned to the ASA server in the `docker-compose.yml` file, then it's very likely that this is the reason why you are getting a "Connection Timeout" error.
Please continue following the instructions below.
#### How to customize your routing?
You need to adjust the `docker-compose.yml` file and add `com.docker.network.bridge.enable_ip_masquerade: 'false'` to the `networks` section, so that it looks like this:
```yml
networks:
asa-network:
attachable: true
driver: bridge
driver_opts:
com.docker.network.bridge.name: 'asanet'
com.docker.network.bridge.enable_ip_masquerade: 'false'
```
Now stop the ASA server if it's running:
```
docker stop asa-server-1
```
Delete the docker network interface and the container, so that they can be recreated:
```
docker rm asa-server-1
docker network rm asa-server_asa-network
```
Now run `docker compose up -d` from within the directory where your `docker-compose.yml` is located at.
Once done and the container is up again, inspect the network to find its subnet:
```
docker network inspect asa-server_asa-network | grep Subnet
```
Now customize the routing of the container through `iptables`, where `$SUBNET` needs to be replaced with the subnet from the previous command (including the `/24` or `/16` - whatever it is in your case):
```
iptables -t nat -A POSTROUTING -s $SUBNET ! -o asanet -j SNAT --to-source $YOUR_SECONDARY_IP_USED_BY_ASA
```
Once done, connect to your container and test that the remote IP is the right one, by following the steps with `curl` again.
Now try to connect to your server through the server browser. If that is not solving your problem or if the IP is still the wrong one, open a GitHub issue. If it solves your problem, continue with the
next section to make the `iptables` adjustments persistent after reboot.
#### Making your iptable rules persistent
Changes to the `iptables` will get reverted after reboot. You can make them persistent by saving the current state:
```
iptables-save > /root/iptables
```
Now run `crontab -e` and add the following entry:
```
@reboot /bin/bash -c 'sleep 15 ; /usr/sbin/iptables-restore < /root/iptables'
```
Save the cronjob and test it by rebooting your system. You can test whether it has worked by following the `curl` steps from above again.
## Found an Issue or Bug?
Create a ticket on GitHub, I will do my best to fix it. Feel free to open a pull request as well.
## Credits
* Glorius Eggroll - For his version of Proton to run the ARK Windows binaries on Linux ([click](https://github.com/GloriousEggroll/proton-ge-custom))
* cdp1337 - For his Linux guide of installing Proton and running ARK on Linux ([click](https://github.com/cdp1337/ARKSurvivalAscended-Linux))
* tesfabpel - For his Valve RCON implementation in Ruby ([click](https://github.com/tesfabpel/srcon-rb))
@@ -0,0 +1,91 @@
<?xml version="1.0" encoding="utf-8"?>
<image schemaversion="7.4" name="asa-linux-server">
<description type="system">
<author>Manuel Schnitzer</author>
<contact>github@mschnitzer.de</contact>
<specification>ARK: Survival Ascended - Dedicated Server (Linux)</specification>
</description>
<profiles>
<profile name="stable" description="building docker tag: latest"/>
<profile name="beta" description="building docker tag: beta"/>
<profile name="development" description="building docker tag: development"/>
</profiles>
<preferences>
<version>1.5.1</version>
<packagemanager>zypper</packagemanager>
<rpm-excludedocs>true</rpm-excludedocs>
<rpm-check-signatures>false</rpm-check-signatures>
<locale>en_US</locale>
<keytable>us</keytable>
</preferences>
<preferences profiles="stable">
<type image="docker">
<containerconfig tag="latest" workingdir="/home/gameserver" name="mschnitzer/asa-linux-server"/>
</type>
</preferences>
<preferences profiles="beta">
<type image="docker">
<containerconfig tag="beta" workingdir="/home/gameserver" name="mschnitzer/asa-linux-server"/>
</type>
</preferences>
<preferences profiles="development">
<type image="docker">
<containerconfig tag="development" workingdir="/home/gameserver" name="mschnitzer/asa-linux-server"/>
</type>
</preferences>
<users>
<user name="gameserver" password="" home="/home/gameserver" groups="gameserver" id="25000"/>
</users>
<repository type="rpm-md" alias="repo-oss" imageinclude="true">
<source path="http://cdn.opensuse.org/distribution/leap/16.0/repo/oss/x86_64"/>
</repository>
<packages type="image">
<package name="patterns-base-base"/>
<package name="glibc-locale-base"/>
<package name="timezone"/>
<!-- application dependencies: ASA -->
<package name="libgcc_s1-32bit"/>
<package name="python313"/>
<package name="wget"/>
<package name="tar"/>
<package name="libfreetype6"/>
<!-- application dependencies: ASA server api plugin loader -->
<package name="unzip"/>
<!-- application dependencies: asa-ctrl -->
<package name="ruby3.4"/>
<package name="ruby3.4-devel"/>
<package name="make"/>
<package name="gcc-c++"/>
</packages>
<packages type="image" profiles="development">
<package name="vim"/>
<package name="vim-data"/>
<package name="vim-data-common"/>
</packages>
<packages type="bootstrap">
<package name="udev"/>
<package name="filesystem"/>
<package name="glibc-locale"/>
<package name="cracklib-dict-small"/>
<package name="ca-certificates"/>
<package name="ca-certificates-mozilla"/>
</packages>
<packages type="delete">
<package name="dbus-1"/>
<package name="kbd"/>
<package name="kmod"/>
<package name="ncurses-utils"/>
<package name="pinentry"/>
<package name="udev"/>
<package name="glibc-locale"/>
<package name="systemd"/>
</packages>
<packages type="delete" profiles="stable,beta">
<package name="ruby3.4-devel"/>
<package name="make"/>
<package name="gcc-c++"/>
<package name="gcc15"/>
<package name="gcc15-c++"/>
<package name="libstdc++6-devel-gcc15"/>
</packages>
</image>
Binary file not shown.

After

Width:  |  Height:  |  Size: 227 KiB

@@ -0,0 +1,29 @@
#!/bin/bash
rm -f /etc/localtime
zypper --gpg-auto-import-keys ref
groupmod -g 25000 gameserver
usermod -u 25000 gameserver
chmod 0755 /usr/bin/start_server
chmod 0755 /usr/bin/cli-asa-mods
# install ruby gems
cd /usr/share/asa-ctrl
bundle.ruby3.4
if [ "$kiwi_profiles" = "development" ]; then
# will be mounted to ease development
rm -r /usr/share/asa-ctrl
gem.ruby3.4 install byebug
else
chmod 0755 /usr/share/asa-ctrl/main.rb
fi
ln -s /usr/share/asa-ctrl/main.rb /usr/bin/asa-ctrl
# This fixes a warning with Proton. The warning confused some people, but it haven't had any effect on the ASA server.
echo "d5b7b5ed-1674-497d-ad98-7437a6543312" > /etc/machine-id
chmod 644 /etc/machine-id
exit 0
@@ -0,0 +1,89 @@
###
### THIS IS THE DEVELOPMENT COMPOSE FILE!
### You are most likely looking for the 'docker-compose.yml' file instead!
###
version: "3.3"
services:
asa-server-1:
container_name: asa-server-1
hostname: asa-server-1
entrypoint: "/usr/bin/start_server"
user: gameserver
image: "mschnitzer/asa-linux-server:development"
tty: true
environment:
- ASA_START_PARAMS=TheIsland_WP?listen?Port=7777?RCONPort=27020?RCONEnabled=True -WinLiveMaxPlayers=50 -clusterid=default -ClusterDirOverride="/home/gameserver/cluster-shared"
- ENABLE_DEBUG=0
ports:
# Game port for player connections through the server browser
- 0.0.0.0:7777:7777/udp
# RCON port for remote server administration
- 0.0.0.0:27020:27020/tcp
depends_on:
- set-permissions-1
volumes:
- steam-1:/home/gameserver/Steam:rw
- steamcmd-1:/home/gameserver/steamcmd:rw
- server-files-1:/home/gameserver/server-files:rw
- cluster-shared:/home/gameserver/cluster-shared:rw
- /etc/localtime:/etc/localtime:ro
- ./root/usr/share/asa-ctrl:/usr/share/asa-ctrl:rw
networks:
asa-network:
set-permissions-1:
entrypoint: "/bin/bash -c 'chown -R 25000:25000 /steam ; chown -R 25000:25000 /steamcmd ; chown -R 25000:25000 /server-files ; chown -R 25000:25000 /cluster-shared'"
user: root
image: "opensuse/leap"
volumes:
- steam-1:/steam:rw
- steamcmd-1:/steamcmd:rw
- server-files-1:/server-files:rw
- cluster-shared:/cluster-shared:rw
# asa-server-2:
# container_name: asa-server-2
# hostname: asa-server-2
# entrypoint: "/usr/bin/start_server"
# user: gameserver
# image: "mschnitzer/asa-linux-server:development"
# tty: true
# environment:
# - ASA_START_PARAMS=ScorchedEarth_WP?listen?Port=7778?RCONPort=27021?RCONEnabled=True -WinLiveMaxPlayers=50 -clusterid=default -ClusterDirOverride="/home/gameserver/cluster-shared"
# ports:
# # Game port for player connections through the server browser
# - 0.0.0.0:7778:7778/udp
# # RCON port for remote server administration
# - 0.0.0.0:27021:27021/tcp
# depends_on:
# - set-permissions-2
# volumes:
# - steam-2:/home/gameserver/Steam:rw
# - steamcmd-2:/home/gameserver/steamcmd:rw
# - server-files-2:/home/gameserver/server-files:rw
# - cluster-shared:/home/gameserver/cluster-shared:rw
# - /etc/localtime:/etc/localtime:ro
# - ./root/usr/share/asa-ctrl:/usr/share/asa-ctrl:rw
# networks:
# asa-network:
# set-permissions-2:
# entrypoint: "/bin/bash -c 'chown -R 25000:25000 /steam ; chown -R 25000:25000 /steamcmd ; chown -R 25000:25000 /server-files ; chown -R 25000:25000 /cluster-shared'"
# user: root
# image: "opensuse/leap"
# volumes:
# - steam-2:/steam:rw
# - steamcmd-2:/steamcmd:rw
# - server-files-2:/server-files:rw
# - cluster-shared:/cluster-shared:rw
volumes:
cluster-shared:
steam-1:
steamcmd-1:
server-files-1:
# steam-2:
# steamcmd-2:
# server-files-2:
networks:
asa-network:
attachable: true
driver: bridge
driver_opts:
com.docker.network.bridge.name: 'asanet'
@@ -0,0 +1,83 @@
version: "3.3"
services:
asa-server-1:
container_name: asa-server-1
hostname: asa-server-1
entrypoint: "/usr/bin/start_server"
user: gameserver
image: "mschnitzer/asa-linux-server:latest"
tty: true
environment:
- ASA_START_PARAMS=TheIsland_WP?listen?Port=7777?RCONPort=27020?RCONEnabled=True -WinLiveMaxPlayers=50 -clusterid=default -ClusterDirOverride="/home/gameserver/cluster-shared"
- ENABLE_DEBUG=0
ports:
# Game port for player connections through the server browser
- 0.0.0.0:7777:7777/udp
# RCON port for remote server administration
- 0.0.0.0:27020:27020/tcp
depends_on:
- set-permissions-1
volumes:
- steam-1:/home/gameserver/Steam:rw
- steamcmd-1:/home/gameserver/steamcmd:rw
- server-files-1:/home/gameserver/server-files:rw
- cluster-shared:/home/gameserver/cluster-shared:rw
- /etc/localtime:/etc/localtime:ro
networks:
asa-network:
set-permissions-1:
entrypoint: "/bin/bash -c 'chown -R 25000:25000 /steam ; chown -R 25000:25000 /steamcmd ; chown -R 25000:25000 /server-files ; chown -R 25000:25000 /cluster-shared'"
user: root
image: "opensuse/leap"
volumes:
- steam-1:/steam:rw
- steamcmd-1:/steamcmd:rw
- server-files-1:/server-files:rw
- cluster-shared:/cluster-shared:rw
# asa-server-2:
# container_name: asa-server-2
# hostname: asa-server-2
# entrypoint: "/usr/bin/start_server"
# user: gameserver
# image: "mschnitzer/asa-linux-server:latest"
# tty: true
# environment:
# - ASA_START_PARAMS=ScorchedEarth_WP?listen?Port=7778?RCONPort=27021?RCONEnabled=True -WinLiveMaxPlayers=50 -clusterid=default -ClusterDirOverride="/home/gameserver/cluster-shared"
# ports:
# # Game port for player connections through the server browser
# - 0.0.0.0:7778:7778/udp
# # RCON port for remote server administration
# - 0.0.0.0:27021:27021/tcp
# depends_on:
# - set-permissions-2
# volumes:
# - steam-2:/home/gameserver/Steam:rw
# - steamcmd-2:/home/gameserver/steamcmd:rw
# - server-files-2:/home/gameserver/server-files:rw
# - cluster-shared:/home/gameserver/cluster-shared:rw
# - /etc/localtime:/etc/localtime:ro
# networks:
# asa-network:
# set-permissions-2:
# entrypoint: "/bin/bash -c 'chown -R 25000:25000 /steam ; chown -R 25000:25000 /steamcmd ; chown -R 25000:25000 /server-files ; chown -R 25000:25000 /cluster-shared'"
# user: root
# image: "opensuse/leap"
# volumes:
# - steam-2:/steam:rw
# - steamcmd-2:/steamcmd:rw
# - server-files-2:/server-files:rw
# - cluster-shared:/cluster-shared:rw
volumes:
cluster-shared:
steam-1:
steamcmd-1:
server-files-1:
# steam-2:
# steamcmd-2:
# server-files-2:
networks:
asa-network:
attachable: true
driver: bridge
driver_opts:
com.docker.network.bridge.name: 'asanet'
@@ -0,0 +1,34 @@
#!/usr/bin/ruby.ruby3.4
require 'json'
db_path = '/home/gameserver/server-files/mods.json'
unless File.exist?(db_path)
print ""
exit! 0
end
begin
mods = JSON.parse(File.read(db_path))
args = "-mods="
counter = 0
mods.each do |mod|
if mod['enabled']
args += ',' if counter > 0
args += mod['mod_id'].to_s
counter += 1
end
end
if counter > 0
print args
end
rescue JSON::ParserError
File.write('/tmp/mod-read-error', 'mods.json is corrupted')
print ""
rescue => err
File.write('/tmp/mod-read-error', err.to_s)
print ""
end
@@ -0,0 +1,91 @@
#!/bin/bash
if [ "$ENABLE_DEBUG" = "1" ]; then
echo "Entering debug mode..."
sleep 999999999999
exit 0
fi
# download steamcmd if necessary
if [ ! -d "/home/gameserver/steamcmd/linux32" ]; then
cd /home/gameserver/steamcmd
wget https://steamcdn-a.akamaihd.net/client/installer/steamcmd_linux.tar.gz
tar xfvz steamcmd_linux.tar.gz
fi
# download/update server files
cd /home/gameserver/steamcmd
./steamcmd.sh +force_install_dir /home/gameserver/server-files +login anonymous +app_update 2430930 validate +quit
PROTON_VERSION="10-17"
PROTON_DIR_NAME="GE-Proton$PROTON_VERSION"
PROTON_ARCHIVE_NAME="$PROTON_DIR_NAME.tar.gz"
STEAM_COMPAT_DATA=/home/gameserver/server-files/steamapps/compatdata
STEAM_COMPAT_DIR=/home/gameserver/Steam/compatibilitytools.d
ASA_COMPAT_DATA=$STEAM_COMPAT_DATA/2430930
ASA_BINARY_DIR="/home/gameserver/server-files/ShooterGame/Binaries/Win64"
START_PARAMS_FILE="/home/gameserver/server-files/start-parameters"
MODS="$(/usr/bin/cli-asa-mods)"
ASA_START_PARAMS="$ASA_START_PARAMS $MODS"
ASA_BINARY_NAME="ArkAscendedServer.exe"
ASA_PLUGIN_BINARY_NAME="AsaApiLoader.exe"
ASA_PLUGIN_LOADER_ARCHIVE_NAME=$(basename $ASA_BINARY_DIR/AsaApi_*.zip)
ASA_PLUGIN_LOADER_ARCHIVE_PATH="$ASA_BINARY_DIR/$ASA_PLUGIN_LOADER_ARCHIVE_NAME"
ASA_PLUGIN_BINARY_PATH="$ASA_BINARY_DIR/$ASA_PLUGIN_BINARY_NAME"
LAUNCH_BINARY_NAME="$ASA_BINARY_NAME"
# install proton if necessary
if [ ! -d "$STEAM_COMPAT_DIR/$PROTON_DIR_NAME" ]; then
mkdir -p $STEAM_COMPAT_DIR
echo "Downloading Proton version $PROTON_VERSION... This might take a while"
wget -P /tmp https://github.com/GloriousEggroll/proton-ge-custom/releases/download/GE-Proton$PROTON_VERSION/GE-Proton$PROTON_VERSION.tar.gz
EXIT_CODE=$?
if [ $EXIT_CODE -ne 0 ]; then
echo "Error: Error while downloading Proton ($EXIT_CODE)"
exit 200
fi
echo "Download finished, comparing checksums..."
sha512sum -c /usr/share/proton/GE-Proton$PROTON_VERSION.sha512sum
if [ $? -ne 0 ]; then
echo "Error: Proton checksum mismatch!"
exit 201
fi
tar -xf /tmp/$PROTON_ARCHIVE_NAME -C $STEAM_COMPAT_DIR
rm /tmp/$PROTON_ARCHIVE_NAME
fi
# install proton compat game data
if [ ! -d "$ASA_COMPAT_DATA" ]; then
mkdir -p $STEAM_COMPAT_DATA
cp -r $STEAM_COMPAT_DIR/$PROTON_DIR_NAME/files/share/default_pfx $ASA_COMPAT_DATA
fi
echo "Starting the ARK: Survival Ascended dedicated server..."
echo "Start parameters: $ASA_START_PARAMS"
export XDG_RUNTIME_DIR=/run/user/$(id -u)
export STEAM_COMPAT_CLIENT_INSTALL_PATH=/home/gameserver/Steam
export STEAM_COMPAT_DATA_PATH=$ASA_COMPAT_DATA
cd "$ASA_BINARY_DIR"
# unzip the asa plugin api archive if it exists. delete it afterwards
if [ -f "$ASA_PLUGIN_LOADER_ARCHIVE_PATH" ]; then
unzip -o $ASA_PLUGIN_LOADER_ARCHIVE_NAME
rm $ASA_PLUGIN_LOADER_ARCHIVE_NAME
fi
if [ -f "$ASA_PLUGIN_BINARY_PATH" ]; then
echo "Detected ASA Server API loader. Launching server through $ASA_PLUGIN_BINARY_NAME"
LAUNCH_BINARY_NAME="$ASA_PLUGIN_BINARY_NAME"
fi
# Remove steamclient64.dll to prevent server from crashing.
# File is not needed and was probably accidentally committed to Steam.
# See: https://github.com/mschnitzer/ark-survival-ascended-linux-container-image/issues/123
rm -f /home/gameserver/server-files/ShooterGame/Binaries/Win64/steamclient64.dll
$STEAM_COMPAT_DIR/$PROTON_DIR_NAME/proton run $LAUNCH_BINARY_NAME $ASA_START_PARAMS
@@ -0,0 +1,4 @@
source 'https://rubygems.org'
gem 'slop', '= 4.10.1'
gem 'iniparse', '= 1.5.0'
@@ -0,0 +1,15 @@
GEM
remote: https://rubygems.org/
specs:
iniparse (1.5.0)
slop (4.10.1)
PLATFORMS
x86_64-linux-gnu
DEPENDENCIES
iniparse (= 1.5.0)
slop (= 4.10.1)
BUNDLED WITH
2.5.0.dev
@@ -0,0 +1,15 @@
module AsaCtrl
module Cli
class CliInterface
def initialize(opts)
@opts = opts
print_help! if opts[:help]
end
def print_help!
raise "Help not implemented!"
end
end
end
end
@@ -0,0 +1,34 @@
module AsaCtrl
module Cli
class ModsInterface < CliInterface
def initialize(opts)
super(opts)
execute!
end
def execute!
if @opts[:enable]
enable_mod!
end
exit! AsaCtrl::ExitCodes::OK
end
def enable_mod!
mod_id = @opts[:enable]
AsaCtrl::Mods::Database.get_instance.enable_mod!(mod_id)
puts "Enabled mod id '#{mod_id}' successfully. The server will download the mod upon startup."
rescue AsaCtrl::Errors::ModAlreadyEnabledError
AsaCtrl::Cli.exit_with_error!("This mod is already enabled! Use 'asa-ctrl mods --list' to see what mods are currently enabled.",
AsaCtrl::ExitCodes::MOD_ALREADY_ENABLED)
end
def print_help!
puts "Usage: asa-ctrl mods [--install] (--dry-run)"
exit! AsaCtrl::ExitCodes::OK
end
end
end
end
@@ -0,0 +1,42 @@
module AsaCtrl
module Cli
class RconInterface < CliInterface
def initialize(opts)
super(opts)
execute!
end
def execute!
if @opts[:exec]
run_command!
end
exit! AsaCtrl::ExitCodes::OK
end
def run_command!
rcon_command = @opts[:exec]
response = AsaCtrl::Rcon.exec_command!('127.0.0.1', AsaCtrl::Rcon.identify_port, rcon_command, AsaCtrl::Rcon.identify_password)
if response[:id] == AsaCtrl::Rcon::PacketTypes::RESPONSE_VALUE
puts response[:body]
else
AsaCtrl::Cli.exit_with_error!("Rcon command execution failed: #{response}",
AsaCtrl::ExitCodes::RCON_COMMAND_EXECUTION_FAILED)
end
rescue AsaCtrl::Errors::RconPasswordNotFoundError
AsaCtrl::Cli.exit_with_error!("Could not read RCON password. Make sure it is properly configured, either as start parameter ?ServerAdminPassword=mypass or " \
"in GameUserSettings.ini in the [ServerSettings] section as ServerAdminPassword=mypass", AsaCtrl::ExitCodes::RCON_PASSWORD_NOT_FOUND)
rescue AsaCtrl::Errors::RconAuthenticationError
AsaCtrl::Cli.exit_with_error!("Could not execute this RCON command. Authentication failed (wrong server password).",
AsaCtrl::ExitCodes::RCON_PASSWORD_WRONG)
end
def print_help!
puts "Usage: asa-ctrl rcon [--exec]"
exit! AsaCtrl::ExitCodes::OK
end
end
end
end
@@ -0,0 +1,23 @@
module AsaCtrl
module Cli
HELP_ARGUMENT = '--help'
HELP_DESCRIPTION = 'Prints a help message'
def self.passed_command(args)
if ARGV.size == 0
[]
else
[ARGV[0]]
end
end
def self.print_usage
puts "Usage: asa-ctrl [rcon] (--help)"
end
def self.exit_with_error!(message, code)
$stderr.puts "Error: #{message}"
exit! code
end
end
end
@@ -0,0 +1,5 @@
module AsaCtrl
module Errors
class BaseError < StandardError; end
end
end
@@ -0,0 +1,5 @@
require_relative './base_error.rb'
require_relative './mod_already_enabled_error.rb'
require_relative './rcon_authentication_error.rb'
require_relative './rcon_password_not_found_error.rb'
require_relative './rcon_port_not_found_error.rb'
@@ -0,0 +1,5 @@
module AsaCtrl
module Errors
class ModAlreadyEnabledError < BaseError; end
end
end
@@ -0,0 +1,5 @@
module AsaCtrl
module Errors
class RconAuthenticationError < BaseError; end
end
end
@@ -0,0 +1,5 @@
module AsaCtrl
module Errors
class RconPasswordNotFoundError < BaseError; end
end
end
@@ -0,0 +1,5 @@
module AsaCtrl
module Errors
class RconPortNotFoundError < BaseError; end
end
end
@@ -0,0 +1,10 @@
module AsaCtrl
module ExitCodes
OK = 0
CORRUPTED_MODS_DATABASE = 1
MOD_ALREADY_ENABLED = 2
RCON_PASSWORD_NOT_FOUND = 3
RCON_PASSWORD_WRONG = 4
RCON_COMMAND_EXECUTION_FAILED = 5
end
end
@@ -0,0 +1,2 @@
require_relative './start_params_helper.rb'
require_relative './ini_config_helper.rb'
@@ -0,0 +1,17 @@
module AsaCtrl
module IniConfigHelper
def self.game_user_settings_ini
self.parse('/home/gameserver/server-files/ShooterGame/Saved/Config/WindowsServer/GameUserSettings.ini')
end
def self.game_ini
self.parse('/home/gameserver/server-files/ShooterGame/Saved/Config/WindowsServer/Game.ini')
end
def self.parse(path)
return unless File.exist?(path)
IniParse.parse(File.read(path))
end
end
end
@@ -0,0 +1,22 @@
module AsaCtrl
module StartParamsHelper
def self.get_value(start_params, key)
return unless start_params
value = ''
offset = start_params.index("#{key}=")
return unless offset
offset += "#{key}=".length
start_params[offset..-1].each_char do |char|
break if char == ' ' || char == '?'
value += char
end
value
end
end
end
@@ -0,0 +1,36 @@
#!/usr/bin/ruby.ruby3.4
require 'json'
require 'slop'
require 'iniparse'
require 'socket'
if ENV['DEV'] == '1'
require 'byebug'
end
require_relative './exit_codes.rb'
require_relative './errors/errors.rb'
require_relative './helpers/helpers.rb'
require_relative './mods/database.rb'
require_relative './rcon/rcon.rb'
require_relative './cli/utils.rb'
require_relative './cli/interfaces/cli_interface.rb'
require_relative './cli/interfaces/mods_interface.rb'
require_relative './cli/interfaces/rcon_interface.rb'
main_args = Slop.parse(AsaCtrl::Cli.passed_command(ARGV)) do |args|
args.on 'rcon', 'Interface for RCON command execution' do
opts = Slop.parse(ARGV[1..-1]) do |opt|
opt.string '--exec', 'An RCON command to execute'
opt.bool AsaCtrl::Cli::HELP_ARGUMENT, AsaCtrl::Cli::HELP_DESCRIPTION
end
AsaCtrl::Cli::RconInterface.new(opts)
end
args.on AsaCtrl::Cli::HELP_ARGUMENT, AsaCtrl::Cli::HELP_DESCRIPTION do
# handled once slop exits
end
end
AsaCtrl::Cli.print_usage
@@ -0,0 +1,66 @@
module AsaCtrl
module Mods
MOD_DATABASE_PATH = '/home/gameserver/server-files/mods.json'
class Database
@@singleton_reference = nil
def initialize(database_path)
@database_path = database_path
ensure_database_presence!
load_database
end
def self.get_instance
return @@singleton_reference if @@singleton_reference
@@singleton_reference = Database.new(MOD_DATABASE_PATH)
end
def enable_mod!(mod_id)
@database.each do |record|
if record['mod_id'].to_i == mod_id.to_i
raise AsaCtrl::Errors::ModAlreadyEnabledError if record['enabled']
record['enabled'] = true
write_database!
return
end
end
add_new_record!(mod_id, 'unknown', true, false)
end
def add_new_record!(mod_id, name, enabled, scanned)
@database << {
mod_id: mod_id.to_i,
name: name,
enabled: enabled,
scanned: scanned
}
write_database!
end
def write_database!
File.write(@database_path, JSON.pretty_generate(@database))
end
def ensure_database_presence!
return if File.exist?(@database_path)
@database = []
write_database!
end
def load_database
@database = JSON.parse(File.read(@database_path))
rescue JSON::ParserError
# we do not want to delete the file for the user, as they might want to save its content first
AsaCtrl::Cli.exit_with_error!("mods.json file is corrupted and cannot be parsed, please delete this file " \
"manually. It can be found in the server files root directory.", AsaCtrl::ExitCodes::CORRUPTED_MODS_DATABASE)
end
end
end
end
@@ -0,0 +1,69 @@
module AsaCtrl
module Rcon
module PacketTypes
RESPONSE_VALUE = 0
EXEC_COMMAND = 2
AUTH_RESPONSE = 2
AUTH = 3
end
Packet = Struct.new(:size, :id, :type, :body)
def self.exec_command!(server_ip, rcon_port, rcon_command, password)
socket = TCPSocket.new(server_ip, rcon_port)
raise AsaCtrl::Errors::RconAuthenticationError unless self.authenticate!(socket, password)
self.send_packet!(socket, rcon_command, PacketTypes::EXEC_COMMAND)
end
def self.authenticate!(socket, password)
response = self.send_packet!(socket, password, PacketTypes::AUTH)
response[:id] != -1
end
def self.send_packet!(socket, data, packet_id)
packet = Packet.new(10+data.bytesize, 0, packet_id, data)
self.send_to(packet, socket)
self.recv_from(socket)
end
def self.send_to(packet, socket)
szb = [packet[:size]].pack 'l<'
idb = [packet[:id]].pack 'l<'
type_b = [packet[:type]].pack 'l<'
body_b = [packet[:body]].pack 'Z*'
data = szb + idb + type_b + body_b + "\0"
socket.sendmsg(data)
end
def self.recv_from(socket)
msg_ary = socket.recvmsg
msg = msg_ary[0]
ary = msg.unpack('l<l<l<Z*')
Packet.new(ary[0], ary[1], ary[2], ary[3])
end
def self.identify_password
password = AsaCtrl::StartParamsHelper.get_value(ENV['ASA_START_PARAMS'], 'ServerAdminPassword')
return password if password
config = AsaCtrl::IniConfigHelper.game_user_settings_ini
return config['ServerSettings']['ServerAdminPassword'] if config['ServerSettings'] && config['ServerSettings']['ServerAdminPassword']
raise AsaCtrl::Errors::RconPasswordNotFoundError
end
def self.identify_port
port = AsaCtrl::StartParamsHelper.get_value(ENV['ASA_START_PARAMS'], 'RCONPort')
return port.to_i if port
config = AsaCtrl::IniConfigHelper.game_user_settings_ini
return config['ServerSettings']['RCONPort'].to_i if config['ServerSettings'] && config['ServerSettings']['RCONPort']
raise AsaCtrl::Errors::RconPortNotFoundError
end
end
end
@@ -0,0 +1 @@
d21bd48479ab35213a7bf5b7eb87d0f156da16891cdc6bed3665e268811304c20cc9834ff901b9d4e31abc6d10bf6b8066276d32f7c487607270a3a9f975ae2c /tmp/GE-Proton10-17.tar.gz
+129
View File
@@ -0,0 +1,129 @@
name: ark-suac
services:
ark-asa-server:
image: mschnitzer/asa-linux-server:1.4.0
container_name: ark-asa-server
hostname: ark-asa-server
entrypoint: /usr/bin/start_server
user: gameserver
tty: true
restart: unless-stopped
environment:
TZ: Europe/Stockholm
ASA_START_PARAMS: 'TheIsland_WP?listen?Port=7777?RCONPort=27020?RCONEnabled=True -WinLiveMaxPlayers=50 -clusterid=default -ClusterDirOverride="/home/gameserver/cluster-shared"'
ENABLE_DEBUG: "0"
ports:
- target: 7777
published: "7777"
protocol: udp
- target: 27020
published: "27020"
protocol: tcp
depends_on:
- set-permissions
volumes:
- type: bind
source: /DATA/AppData/$AppID/steam
target: /home/gameserver/Steam
- type: bind
source: /DATA/AppData/$AppID/steamcmd
target: /home/gameserver/steamcmd
- type: bind
source: /DATA/AppData/$AppID/server-files
target: /home/gameserver/server-files
- type: bind
source: /DATA/AppData/$AppID/cluster-shared
target: /home/gameserver/cluster-shared
- type: bind
source: /etc/localtime
target: /etc/localtime
read_only: true
# Secure-by-default baseline. Relax only if appen kräver det.
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
x-casaos:
envs:
- container: TZ
description:
en_us: Timezone, for example Europe/Stockholm
- container: ASA_START_PARAMS
description:
en_us: ARK start params, including map and ports
- container: ENABLE_DEBUG
description:
en_us: Set 1 to enter debug mode without starting the server
ports:
- container: "7777"
description:
en_us: Game port for player connections (UDP)
- container: "27020"
description:
en_us: RCON admin port (TCP)
volumes:
- container: /home/gameserver/Steam
description:
en_us: Steam runtime cache
- container: /home/gameserver/steamcmd
description:
en_us: SteamCMD cache
- container: /home/gameserver/server-files
description:
en_us: ARK server files and saved data
- container: /home/gameserver/cluster-shared
description:
en_us: Shared cluster transfer data
set-permissions:
image: opensuse/leap:15.6
container_name: ark-asa-set-permissions
user: root
restart: "no"
entrypoint:
- /bin/bash
- -c
- chown -R 25000:25000 /steam /steamcmd /server-files /cluster-shared
volumes:
- type: bind
source: /DATA/AppData/$AppID/steam
target: /steam
- type: bind
source: /DATA/AppData/$AppID/steamcmd
target: /steamcmd
- type: bind
source: /DATA/AppData/$AppID/server-files
target: /server-files
- type: bind
source: /DATA/AppData/$AppID/cluster-shared
target: /cluster-shared
x-casaos:
architectures:
- amd64
- arm64
main: ark-asa-server
category: phirna
author: Joachim Friberg
developer: Joachim Friberg
icon: https://cdn.jsdelivr.net/gh/selfhst/icons/png/ark-survival-ascended.png
tagline:
en_us: ARK Survival Ascended dedicated server for ZimaOS
description:
en_us: |
Host your own ARK: Survival Ascended dedicated server.
Minimum requirements:
- RAM: 16 GB (24 GB recommended)
- Disk: 20 GB+ for server files and updates
title:
en_us: ARK Survival Ascended Server
index: /
port_map: "7777"
+18 -25
View File
@@ -27,32 +27,22 @@ Konfigurera lokal DNS (AdGuard Home):
- `A`/rewrite för `*.home.example.com` -> ZimaOS LAN-IP - `A`/rewrite för `*.home.example.com` -> ZimaOS LAN-IP
- `A`/rewrite för `home.example.com` -> ZimaOS LAN-IP - `A`/rewrite för `home.example.com` -> ZimaOS LAN-IP
## 3) Positivt test: endpoint skapas korrekt ## 3) Positivt test: status-UI + endpoint
Exempel med Frigate (web UI på 5000):
Sätt env-vars på Frigate-containern:
```text
LABEL_CADDY_ENABLE=true
LABEL_CADDY_TARGET_PORT=5000
LABEL_CADDY_HOST=frigate
LABEL_CADDY_SCHEME=http
LABEL_CADDY_PATH=/
```
Verifiera:
1. Vänta 15-30 sekunder (default polling). 1. Vänta 15-30 sekunder (default polling).
2. Öppna `https://frigate.home.example.com`. 2. Öppna `http://<zima-lan-ip>:31820/`.
3. Kontrollera att sidan svarar med giltigt certifikat. 3. Verifiera i UI:
4. Kontrollera att media-portar (8554/8555) inte blivit egna hostnamn/endpoints. - `Caddy Apply = OK`
- `Cloudflare Reachability = OK`
- `Cloudflare Token = OK`
- minst en route visas under `Generated Hosts`.
4. Öppna en exponerad host, t.ex. `https://frigate.home.example.com`.
Förväntat resultat: Förväntat resultat:
- Endpoint finns för web UI. - UI laddar och visar aktuell status.
- Endpoint finns för web UI-hosten.
- Certifikat utfärdas via Let's Encrypt DNS-01. - Certifikat utfärdas via Let's Encrypt DNS-01.
- Endast explicit målport routas.
## 4) Negativt test: fail-closed ## 4) Negativt test: fail-closed
@@ -60,24 +50,26 @@ Scenario A, ingen opt-in:
1. Ta bort `LABEL_CADDY_ENABLE=true` från målcontainern. 1. Ta bort `LABEL_CADDY_ENABLE=true` från målcontainern.
2. Vänta en polling-cykel. 2. Vänta en polling-cykel.
3. Kontrollera att endpointen försvinner/slutar routas. 3. Verifiera i status-UI att routen försvinner.
Scenario B, saknad Cloudflare-token: Scenario B, saknad/ogiltig Cloudflare-token:
1. Sätt `REQUIRE_CLOUDFLARE=true`. 1. Sätt `REQUIRE_CLOUDFLARE=true`.
2. Ta bort eller ogiltiggör `CLOUDFLARE_API_TOKEN`. 2. Ta bort eller ogiltiggör `CLOUDFLARE_API_TOKEN`.
3. Verifiera att ingen ny extern route publiceras. 3. Verifiera i status-UI att Cloudflare Token inte är OK.
4. Verifiera att ingen ny extern route publiceras.
Förväntat resultat: Förväntat resultat:
- Appen exponerar inte tjänster oavsiktligt. - Appen exponerar inte tjänster oavsiktligt.
- Loggar visar tydligt fel kring token/Cloudflare. - Loggar visar tydligt token/Cloudflare-relaterat fel.
## 5) Kommandon för snabb verifiering ## 5) Kommandon för snabb verifiering
Byt `<domain>` och `<host>` enligt din miljö. Byt `<domain>`, `<host>` och `<zima-lan-ip>` enligt din miljö.
```bash ```bash
curl -sS http://<zima-lan-ip>:31820/status.json | jq .
nslookup frigate.home.example.com nslookup frigate.home.example.com
curl -vk https://frigate.home.example.com/ curl -vk https://frigate.home.example.com/
openssl s_client -connect frigate.home.example.com:443 -servername frigate.home.example.com </dev/null openssl s_client -connect frigate.home.example.com:443 -servername frigate.home.example.com </dev/null
@@ -106,6 +98,7 @@ Samla detta först. Det kortar felsökningstiden avsevärt.
- Målcontainerns publicerade portar. - Målcontainerns publicerade portar.
3. Nät/TLS-bevis: 3. Nät/TLS-bevis:
- `curl -sS http://<zima-lan-ip>:31820/status.json | jq .`.
- `nslookup`-svar från en LAN-klient. - `nslookup`-svar från en LAN-klient.
- `curl -vk` output mot endpoint. - `curl -vk` output mot endpoint.
- `openssl s_client` sammanfattning av cert/issuer. - `openssl s_client` sammanfattning av cert/issuer.
+27 -2
View File
@@ -8,6 +8,25 @@ Arkitektur:
- Endast explicit opt-in exponeras (`LABEL_CADDY_ENABLE=true`). - Endast explicit opt-in exponeras (`LABEL_CADDY_ENABLE=true`).
- Caddy config laddas dynamiskt via Caddy Admin API (`POST /load`). - Caddy config laddas dynamiskt via Caddy Admin API (`POST /load`).
- TLS sker med Let's Encrypt DNS-01 via Cloudflare. - TLS sker med Let's Encrypt DNS-01 via Cloudflare.
- Ett enkelt status-UI finns på port `31820` (LAN/private ranges only).
## Status-UI (v1)
Status-UI visar:
- senaste config apply-status,
- Cloudflare reachability + token-validitet,
- genererade hosts/routes,
- om Let's Encrypt-cert verkar finnas för varje host.
URL (default):
- `http://<zima-lan-ip>:31820/`
Obs:
- "Hosts" i UI betyder **genererade Caddy-routes**, inte att appen skapar A/CNAME-records i Cloudflare-zonen.
- Certstatus bygger på Caddys lokala certificate storage och är en praktisk v1-signal.
## Säkerhetsmodell ## Säkerhetsmodell
@@ -15,6 +34,7 @@ Arkitektur:
- Ingen auto-exponering av alla portar. - Ingen auto-exponering av alla portar.
- Endast HTTP(S)-upstreams stöds i v1. - Endast HTTP(S)-upstreams stöds i v1.
- Docker socket exponeras inte direkt till agenten, endast via `socket-proxy` med begränsade endpoint-flaggor. - Docker socket exponeras inte direkt till agenten, endast via `socket-proxy` med begränsade endpoint-flaggor.
- Status-UI är LAN-begränsat (`remote_ip private_ranges`).
## Miljövariabler (appnivå) ## Miljövariabler (appnivå)
@@ -24,6 +44,9 @@ Arkitektur:
- `REQUIRE_CLOUDFLARE` (default `true`) - `REQUIRE_CLOUDFLARE` (default `true`)
- `ALLOW_INTERNAL_TLS_FALLBACK` (default `false`) - `ALLOW_INTERNAL_TLS_FALLBACK` (default `false`)
- `POLL_SECONDS` (default `15`) - `POLL_SECONDS` (default `15`)
- `STATUS_BIND` (default `0.0.0.0:8089`, intern agent endpoint)
- `STATUS_UI_PORT` (default `31820`)
- `CF_VERIFY_URL` (default Cloudflare token verify endpoint)
Cloudflare-token bör vara scoped till minsta möjliga rättigheter: Cloudflare-token bör vara scoped till minsta möjliga rättigheter:
@@ -97,10 +120,12 @@ Kör lokala integrationstester för discovery-agenten:
./Apps/caddy-autogen/tests/run_integration_tests.sh ./Apps/caddy-autogen/tests/run_integration_tests.sh
``` ```
Testerna mockar Docker API-svar och Caddy `/load`-anrop och verifierar: Testerna mockar Docker API-svar och Cloudflare verify och verifierar:
- opt-in-regler (endast markerade containers exponeras), - opt-in-regler (endast markerade containers exponeras),
- säker portselektionslogik (inga media/UDP-portar routas av misstag), - säker portselektionslogik (inga media/UDP-portar routas av misstag),
- fail-closed beteende när Cloudflare-token saknas. - fail-closed beteende när Cloudflare-token saknas,
- status-UI-block i genererad Caddy-konfiguration,
- cert-matchning mot lokal Caddy certificate storage.
För verifiering i riktig ZimaOS-miljö, se `HOW_TO_VERIFY.md` i samma mapp. För verifiering i riktig ZimaOS-miljö, se `HOW_TO_VERIFY.md` i samma mapp.
+271 -7
View File
@@ -4,10 +4,12 @@ import json
import os import os
import re import re
import sys import sys
import threading
import time import time
import urllib.error import urllib.error
import urllib.parse import urllib.parse
import urllib.request import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
TRUE_VALUES = {"1", "true", "yes", "on"} TRUE_VALUES = {"1", "true", "yes", "on"}
@@ -116,7 +118,15 @@ def _build_fqdn(host_hint: str, base_domain: str) -> str:
return fqdn return fqdn
def _collect_routes(docker_api_url: str, env_prefix: str, denylist: set, base_domain: str, default_scheme: str, default_path: str, default_health_uri: str): def _collect_routes(
docker_api_url: str,
env_prefix: str,
denylist: set,
base_domain: str,
default_scheme: str,
default_path: str,
default_health_uri: str,
):
routes = [] routes = []
containers = _get_json(f"{docker_api_url}/containers/json?all=0") containers = _get_json(f"{docker_api_url}/containers/json?all=0")
for c in containers: for c in containers:
@@ -175,7 +185,32 @@ def _collect_routes(docker_api_url: str, env_prefix: str, denylist: set, base_do
return routes return routes
def _generate_caddyfile(routes, token: str, require_cloudflare: bool, allow_internal_tls_fallback: bool, wildcard_domain: str, cert_email: str): def _append_status_site(out: list[str], status_ui_port: int, status_upstream: str) -> None:
out.append(f":{status_ui_port} {{")
out.append(" @allowed remote_ip private_ranges")
out.append(" handle @allowed {")
out.append(" @status_json path /status.json")
out.append(" handle @status_json {")
out.append(f" reverse_proxy {status_upstream}")
out.append(" }")
out.append(" root * /srv/status")
out.append(" file_server")
out.append(" }")
out.append(" respond \"forbidden\" 403")
out.append("}")
out.append("")
def _generate_caddyfile(
routes,
token: str,
require_cloudflare: bool,
allow_internal_tls_fallback: bool,
wildcard_domain: str,
cert_email: str,
status_ui_port: int,
status_upstream: str,
):
if require_cloudflare and not token: if require_cloudflare and not token:
raise RuntimeError("CLOUDFLARE_API_TOKEN is required in fail-closed mode") raise RuntimeError("CLOUDFLARE_API_TOKEN is required in fail-closed mode")
@@ -189,6 +224,8 @@ def _generate_caddyfile(routes, token: str, require_cloudflare: bool, allow_inte
out.append("}") out.append("}")
out.append("") out.append("")
_append_status_site(out, status_ui_port=status_ui_port, status_upstream=status_upstream)
if wildcard_domain and token: if wildcard_domain and token:
out.append(f"{wildcard_domain}, *.{wildcard_domain} {{") out.append(f"{wildcard_domain}, *.{wildcard_domain} {{")
out.append(" tls {") out.append(" tls {")
@@ -237,6 +274,147 @@ def _generate_caddyfile(routes, token: str, require_cloudflare: bool, allow_inte
return "\n".join(out) return "\n".join(out)
def _parse_bind_addr(value: str, default_host: str = "0.0.0.0", default_port: int = 8089) -> tuple[str, int]:
raw = str(value or "").strip()
if not raw:
return default_host, default_port
if ":" not in raw:
raise ValueError(f"invalid bind address '{raw}', expected host:port")
host, port_raw = raw.rsplit(":", 1)
if not host:
host = default_host
try:
port = int(port_raw)
except ValueError as exc:
raise ValueError(f"invalid port in bind address '{raw}'") from exc
if port < 1 or port > 65535:
raise ValueError(f"port out of range in bind address '{raw}'")
return host, port
def _verify_cloudflare_token(verify_url: str, token: str) -> dict:
now = int(time.time())
if not token:
return {
"reachable": False,
"token_valid": False,
"last_check_ts": now,
"error": "CLOUDFLARE_API_TOKEN is missing",
}
req = urllib.request.Request(
verify_url,
headers={
"Authorization": f"Bearer {token}",
"Accept": "application/json",
},
)
try:
with urllib.request.urlopen(req, timeout=10) as resp:
payload = json.loads(resp.read().decode("utf-8"))
return {
"reachable": True,
"token_valid": bool(payload.get("success", False)),
"last_check_ts": now,
"error": "",
}
except urllib.error.HTTPError as exc:
return {
"reachable": True,
"token_valid": False,
"last_check_ts": now,
"error": f"HTTP {exc.code}",
}
except urllib.error.URLError as exc:
return {
"reachable": False,
"token_valid": False,
"last_check_ts": now,
"error": f"connection failure: {exc}",
}
except Exception as exc:
return {
"reachable": False,
"token_valid": False,
"last_check_ts": now,
"error": str(exc),
}
def _collect_letsencrypt_hosts(caddy_data_dir: str) -> set[str]:
results: set[str] = set()
cert_root = os.path.join(caddy_data_dir, "caddy", "certificates")
if not os.path.isdir(cert_root):
return results
for root, _dirs, files in os.walk(cert_root):
if "letsencrypt" not in root.lower():
continue
for filename in files:
if not filename.endswith(".crt"):
continue
host = filename[:-4].lower()
if host.startswith("_."):
host = "*." + host[2:]
if host and host != "*" and (host.startswith("*.") or HOST_RE.match(host)):
results.add(host)
return results
def _has_matching_le_cert(route_fqdn: str, cert_hosts: set[str]) -> bool:
if route_fqdn in cert_hosts:
return True
for cert_host in cert_hosts:
if not cert_host.startswith("*."):
continue
suffix = cert_host[2:]
if route_fqdn == suffix or route_fqdn.endswith("." + suffix):
return True
return False
def _build_status_payload(state: dict) -> bytes:
body = json.dumps(state, separators=(",", ":"), sort_keys=True).encode("utf-8")
return body
def _start_status_server(bind_addr: str, snapshot: dict, lock: threading.Lock):
host, port = _parse_bind_addr(bind_addr)
class _Handler(BaseHTTPRequestHandler):
def do_GET(self):
if self.path not in {"/status.json", "/healthz"}:
self.send_response(404)
self.end_headers()
return
if self.path == "/healthz":
self.send_response(200)
self.send_header("Content-Type", "text/plain; charset=utf-8")
self.end_headers()
self.wfile.write(b"ok")
return
with lock:
payload = _build_status_payload(snapshot)
self.send_response(200)
self.send_header("Content-Type", "application/json; charset=utf-8")
self.send_header("Cache-Control", "no-store")
self.send_header("Content-Length", str(len(payload)))
self.end_headers()
self.wfile.write(payload)
def log_message(self, fmt, *args):
return
server = ThreadingHTTPServer((host, port), _Handler)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
_log(f"INFO: status endpoint listening on {host}:{port}")
return server, thread
def main(): def main():
defaults = _read_simple_yaml(os.getenv("CONFIG_FILE", "/app/config/defaults.yaml")) defaults = _read_simple_yaml(os.getenv("CONFIG_FILE", "/app/config/defaults.yaml"))
@@ -249,6 +427,25 @@ def main():
default_scheme = str(_cfg("DEFAULT_SCHEME", defaults, "default_scheme", "http")).strip().lower() default_scheme = str(_cfg("DEFAULT_SCHEME", defaults, "default_scheme", "http")).strip().lower()
default_path = str(_cfg("DEFAULT_PATH", defaults, "default_path", "/")).strip() or "/" default_path = str(_cfg("DEFAULT_PATH", defaults, "default_path", "/")).strip() or "/"
default_health_uri = str(_cfg("DEFAULT_HEALTH_URI", defaults, "default_health_uri", "")).strip() default_health_uri = str(_cfg("DEFAULT_HEALTH_URI", defaults, "default_health_uri", "")).strip()
status_bind = str(_cfg("STATUS_BIND", defaults, "status_bind", "0.0.0.0:8089")).strip()
status_ui_port_raw = _cfg("STATUS_UI_PORT", defaults, "status_ui_port", 31820)
status_upstream = str(_cfg("STATUS_UPSTREAM", defaults, "status_upstream", "discovery-agent:8089")).strip()
cf_verify_url = str(
_cfg(
"CF_VERIFY_URL",
defaults,
"cf_verify_url",
"https://api.cloudflare.com/client/v4/user/tokens/verify",
)
).strip()
caddy_data_dir = str(_cfg("CADDY_DATA_DIR", defaults, "caddy_data_dir", "/caddy-data")).strip()
try:
status_ui_port = int(status_ui_port_raw)
if status_ui_port < 1 or status_ui_port > 65535:
raise ValueError
except (TypeError, ValueError):
status_ui_port = 31820
poll_seconds_raw = _cfg("POLL_SECONDS", defaults, "poll_seconds", 15) poll_seconds_raw = _cfg("POLL_SECONDS", defaults, "poll_seconds", 15)
try: try:
@@ -268,12 +465,37 @@ def main():
token = os.getenv("CLOUDFLARE_API_TOKEN", "").strip() token = os.getenv("CLOUDFLARE_API_TOKEN", "").strip()
last_digest = "" last_digest = ""
snapshot_lock = threading.Lock()
snapshot = {
"app": {
"name": "caddy-autogen",
"status_ui_port": status_ui_port,
"status_upstream": status_upstream,
"require_cloudflare": require_cloudflare,
"allow_internal_tls_fallback": allow_internal_tls_fallback,
},
"last_tick_ts": 0,
"last_apply_ok": False,
"last_apply_http_status": 0,
"last_error": "not started",
"routes": [],
"cloudflare": {
"reachable": False,
"token_valid": False,
"last_check_ts": 0,
"error": "not checked",
},
"certs": [],
}
_start_status_server(status_bind, snapshot, snapshot_lock)
_log( _log(
"INFO: starting caddy-autogen discovery-agent " "INFO: starting caddy-autogen discovery-agent "
f"(docker_api_url={docker_api_url}, caddy_load_url={caddy_load_url}, poll_seconds={poll_seconds})" f"(docker_api_url={docker_api_url}, caddy_load_url={caddy_load_url}, poll_seconds={poll_seconds})"
) )
while True: while True:
tick_ts = int(time.time())
try: try:
routes = _collect_routes( routes = _collect_routes(
docker_api_url=docker_api_url, docker_api_url=docker_api_url,
@@ -284,6 +506,16 @@ def main():
default_path=default_path, default_path=default_path,
default_health_uri=default_health_uri, default_health_uri=default_health_uri,
) )
cloudflare_status = _verify_cloudflare_token(cf_verify_url, token)
cert_hosts = _collect_letsencrypt_hosts(caddy_data_dir)
cert_rows = [
{
"fqdn": route["fqdn"],
"letsencrypt_present": _has_matching_le_cert(route["fqdn"], cert_hosts),
}
for route in routes
]
caddyfile = _generate_caddyfile( caddyfile = _generate_caddyfile(
routes=routes, routes=routes,
token=token, token=token,
@@ -291,21 +523,53 @@ def main():
allow_internal_tls_fallback=allow_internal_tls_fallback, allow_internal_tls_fallback=allow_internal_tls_fallback,
wildcard_domain=wildcard_domain, wildcard_domain=wildcard_domain,
cert_email=cert_email, cert_email=cert_email,
status_ui_port=status_ui_port,
status_upstream=status_upstream,
) )
digest = hashlib.sha256(caddyfile.encode("utf-8")).hexdigest() digest = hashlib.sha256(caddyfile.encode("utf-8")).hexdigest()
apply_ok = True
apply_status = 0
last_error = ""
if digest != last_digest: if digest != last_digest:
status = _post_caddyfile(caddy_load_url, caddyfile) apply_status = _post_caddyfile(caddy_load_url, caddyfile)
_log(f"INFO: applied config (routes={len(routes)}, status={status})") _log(f"INFO: applied config (routes={len(routes)}, status={apply_status})")
last_digest = digest last_digest = digest
else: else:
_log(f"INFO: no config changes (routes={len(routes)})") _log(f"INFO: no config changes (routes={len(routes)})")
with snapshot_lock:
snapshot["last_tick_ts"] = tick_ts
snapshot["last_apply_ok"] = apply_ok
snapshot["last_apply_http_status"] = apply_status
snapshot["last_error"] = last_error
snapshot["routes"] = routes
snapshot["cloudflare"] = cloudflare_status
snapshot["certs"] = cert_rows
except urllib.error.HTTPError as e: except urllib.error.HTTPError as e:
_log(f"ERROR: http failure {e.code} {e.reason}") err = f"http failure {e.code} {e.reason}"
_log(f"ERROR: {err}")
with snapshot_lock:
snapshot["last_tick_ts"] = tick_ts
snapshot["last_apply_ok"] = False
snapshot["last_apply_http_status"] = 0
snapshot["last_error"] = err
except urllib.error.URLError as e: except urllib.error.URLError as e:
_log(f"ERROR: connection failure: {e}") err = f"connection failure: {e}"
_log(f"ERROR: {err}")
with snapshot_lock:
snapshot["last_tick_ts"] = tick_ts
snapshot["last_apply_ok"] = False
snapshot["last_apply_http_status"] = 0
snapshot["last_error"] = err
except Exception as e: except Exception as e:
_log(f"ERROR: {e}") err = str(e)
_log(f"ERROR: {err}")
with snapshot_lock:
snapshot["last_tick_ts"] = tick_ts
snapshot["last_apply_ok"] = False
snapshot["last_apply_http_status"] = 0
snapshot["last_error"] = err
time.sleep(poll_seconds) time.sleep(poll_seconds)
+13
View File
@@ -2,6 +2,19 @@
admin {$CADDY_ADMIN} admin {$CADDY_ADMIN}
} }
:31820 {
@allowed remote_ip private_ranges
handle @allowed {
@status_json path /status.json
handle @status_json {
reverse_proxy discovery-agent:8089
}
root * /srv/status
file_server
}
respond "forbidden" 403
}
:80 { :80 {
respond "caddy-autogen initializing" 200 respond "caddy-autogen initializing" 200
} }
+1
View File
@@ -5,3 +5,4 @@ RUN xcaddy build \
FROM caddy:2.10.2-alpine FROM caddy:2.10.2-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy COPY --from=builder /usr/bin/caddy /usr/bin/caddy
COPY Caddyfile /etc/caddy/Caddyfile COPY Caddyfile /etc/caddy/Caddyfile
COPY status/ /srv/status/
+186
View File
@@ -0,0 +1,186 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Caddy AutoGen Status</title>
<style>
:root {
--bg: #0f172a;
--panel: #111827;
--muted: #9ca3af;
--ok: #22c55e;
--warn: #f59e0b;
--bad: #ef4444;
--text: #e5e7eb;
}
body {
margin: 0;
font-family: "Iosevka", "Menlo", "Consolas", monospace;
background: radial-gradient(circle at top, #1e293b, #0b1020 60%);
color: var(--text);
}
.wrap {
max-width: 980px;
margin: 0 auto;
padding: 24px;
}
h1 { margin: 0 0 8px; }
.subtitle { color: var(--muted); margin-bottom: 16px; }
.grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(240px, 1fr));
gap: 12px;
margin-bottom: 12px;
}
.card {
background: color-mix(in oklab, var(--panel), black 8%);
border: 1px solid #1f2937;
border-radius: 10px;
padding: 12px;
}
.label { color: var(--muted); font-size: 12px; text-transform: uppercase; letter-spacing: 0.07em; }
.value { font-size: 20px; margin-top: 6px; }
.ok { color: var(--ok); }
.warn { color: var(--warn); }
.bad { color: var(--bad); }
table {
width: 100%;
border-collapse: collapse;
}
th, td {
text-align: left;
border-bottom: 1px solid #1f2937;
padding: 8px 6px;
font-size: 14px;
}
.raw {
white-space: pre-wrap;
color: var(--muted);
font-size: 12px;
}
.topline {
display: flex;
justify-content: space-between;
align-items: baseline;
gap: 8px;
}
button {
background: #1d4ed8;
color: white;
border: 0;
border-radius: 8px;
padding: 8px 12px;
cursor: pointer;
}
button:hover { background: #2563eb; }
</style>
</head>
<body>
<div class="wrap">
<div class="topline">
<h1>Caddy AutoGen Status</h1>
<button id="refresh">Refresh</button>
</div>
<div class="subtitle" id="updated">Loading...</div>
<div class="grid">
<div class="card">
<div class="label">Caddy Apply</div>
<div class="value" id="apply-status">-</div>
</div>
<div class="card">
<div class="label">Cloudflare Reachability</div>
<div class="value" id="cf-reachable">-</div>
</div>
<div class="card">
<div class="label">Cloudflare Token</div>
<div class="value" id="cf-token">-</div>
</div>
<div class="card">
<div class="label">Routes</div>
<div class="value" id="route-count">-</div>
</div>
</div>
<div class="card" style="margin-bottom: 12px;">
<div class="label">Generated Hosts</div>
<table>
<thead>
<tr>
<th>Host</th>
<th>Upstream</th>
<th>Scheme</th>
<th>Path</th>
<th>LE Cert</th>
</tr>
</thead>
<tbody id="routes-body"></tbody>
</table>
</div>
<div class="card">
<div class="label">Last Error</div>
<div id="last-error" class="raw">-</div>
</div>
</div>
<script>
function stateText(ok) {
return ok ? ["ok", "OK"] : ["bad", "FAIL"];
}
function setEl(id, text, cls) {
const el = document.getElementById(id);
el.textContent = text;
el.classList.remove("ok", "warn", "bad");
if (cls) el.classList.add(cls);
}
function fmtTs(ts) {
if (!ts) return "-";
return new Date(ts * 1000).toLocaleString();
}
async function loadStatus() {
const res = await fetch('/status.json', { cache: 'no-store' });
if (!res.ok) throw new Error('status fetch failed: ' + res.status);
const data = await res.json();
const applyPair = stateText(Boolean(data.last_apply_ok));
setEl('apply-status', applyPair[1] + (data.last_apply_http_status ? ` (${data.last_apply_http_status})` : ''), applyPair[0]);
const reachablePair = stateText(Boolean(data.cloudflare && data.cloudflare.reachable));
setEl('cf-reachable', reachablePair[1], reachablePair[0]);
const tokenPair = stateText(Boolean(data.cloudflare && data.cloudflare.token_valid));
setEl('cf-token', tokenPair[1], tokenPair[0]);
setEl('route-count', String((data.routes || []).length));
setEl('updated', `Last tick: ${fmtTs(data.last_tick_ts)} | Cloudflare check: ${fmtTs(data.cloudflare && data.cloudflare.last_check_ts)}`);
const certByHost = new Map((data.certs || []).map((c) => [c.fqdn, c.letsencrypt_present]));
const rows = (data.routes || []).map((r) => {
const cert = certByHost.get(r.fqdn) ? 'yes' : 'no';
return `<tr><td>${r.fqdn}</td><td>${r.upstream}</td><td>${r.scheme}</td><td>${r.path}</td><td>${cert}</td></tr>`;
});
document.getElementById('routes-body').innerHTML = rows.join('') || '<tr><td colspan="5">No routes</td></tr>';
const err = data.last_error || data.cloudflare?.error || '';
document.getElementById('last-error').textContent = err || '(none)';
}
async function refresh() {
try {
await loadStatus();
} catch (err) {
setEl('updated', 'Error: ' + err.message, 'bad');
}
}
document.getElementById('refresh').addEventListener('click', refresh);
refresh();
setInterval(refresh, 15000);
</script>
</body>
</html>
+5
View File
@@ -10,3 +10,8 @@ poll_seconds: 15
require_cloudflare: true require_cloudflare: true
allow_internal_tls_fallback: false allow_internal_tls_fallback: false
container_name_denylist: "caddy-autogen,caddy-autogen-discovery,caddy-autogen-socket-proxy" container_name_denylist: "caddy-autogen,caddy-autogen-discovery,caddy-autogen-socket-proxy"
status_bind: "0.0.0.0:8089"
status_ui_port: 31820
status_upstream: "discovery-agent:8089"
cf_verify_url: "https://api.cloudflare.com/client/v4/user/tokens/verify"
caddy_data_dir: "/caddy-data"
+45 -28
View File
@@ -2,21 +2,25 @@ name: caddy-autogen
services: services:
caddy: caddy:
image: joafri/caddy-autogen-caddy:main
build: build:
context: ./caddy context: ./caddy
dockerfile: Dockerfile dockerfile: Dockerfile
container_name: caddy-autogen container_name: caddy-autogen
restart: unless-stopped restart: unless-stopped
environment: environment:
TZ: ${TZ} TZ: Europe/Stockholm
CADDY_ADMIN: ${CADDY_ADMIN:-0.0.0.0:2019} CADDY_ADMIN: 0.0.0.0:2019
CLOUDFLARE_API_TOKEN: ${CLOUDFLARE_API_TOKEN} CLOUDFLARE_API_TOKEN: CHANGE_ME
ports: ports:
- target: 80 - target: 80
published: ${HTTP_PORT:-80} published: 81
protocol: tcp protocol: tcp
- target: 443 - target: 443
published: ${HTTPS_PORT:-443} published: 4431
protocol: tcp
- target: 31820
published: 31820
protocol: tcp protocol: tcp
volumes: volumes:
- type: bind - type: bind
@@ -44,6 +48,9 @@ services:
- container: "443" - container: "443"
description: description:
en_us: HTTPS ingress en_us: HTTPS ingress
- container: "31820"
description:
en_us: Local/LAN status UI
volumes: volumes:
- container: /data - container: /data
description: description:
@@ -53,11 +60,11 @@ services:
en_us: Caddy configuration state en_us: Caddy configuration state
socket-proxy: socket-proxy:
image: lscr.io/linuxserver/socket-proxy:version-24.02.26 image: lscr.io/linuxserver/socket-proxy:version-3.0.9-r0
container_name: caddy-autogen-socket-proxy container_name: caddy-autogen-socket-proxy
restart: unless-stopped restart: unless-stopped
environment: environment:
TZ: ${TZ} TZ: Europe/Stockholm
CONTAINERS: 1 CONTAINERS: 1
EVENTS: 1 EVENTS: 1
INFO: 1 INFO: 1
@@ -79,6 +86,7 @@ services:
- ALL - ALL
discovery-agent: discovery-agent:
image: joafri/caddy-autogen-discovery-agent:main
build: build:
context: ./agent context: ./agent
dockerfile: Dockerfile dockerfile: Dockerfile
@@ -88,27 +96,36 @@ services:
- caddy - caddy
- socket-proxy - socket-proxy
environment: environment:
TZ: ${TZ} TZ: Europe/Stockholm
DOCKER_API_URL: ${DOCKER_API_URL:-http://socket-proxy:2375} DOCKER_API_URL: http://socket-proxy:2375
CADDY_LOAD_URL: ${CADDY_LOAD_URL:-http://caddy:2019/load} CADDY_LOAD_URL: http://caddy:2019/load
BASE_DOMAIN: ${BASE_DOMAIN} BASE_DOMAIN: example.com
WILDCARD_DOMAIN: ${WILDCARD_DOMAIN:-} WILDCARD_DOMAIN: ""
CLOUDFLARE_API_TOKEN: ${CLOUDFLARE_API_TOKEN} CLOUDFLARE_API_TOKEN: CLOUDFLARE_API_TOKEN
CERT_EMAIL: ${CERT_EMAIL:-} CERT_EMAIL: ""
REQUIRE_CLOUDFLARE: ${REQUIRE_CLOUDFLARE:-true} REQUIRE_CLOUDFLARE: false
ALLOW_INTERNAL_TLS_FALLBACK: ${ALLOW_INTERNAL_TLS_FALLBACK:-false} ALLOW_INTERNAL_TLS_FALLBACK: false
ENV_PREFIX: ${ENV_PREFIX:-LABEL_CADDY_} ENV_PREFIX: LABEL_CADDY_
POLL_SECONDS: ${POLL_SECONDS:-15} POLL_SECONDS: 15
CONTAINER_NAME_DENYLIST: ${CONTAINER_NAME_DENYLIST:-caddy-autogen,caddy-autogen-discovery,caddy-autogen-socket-proxy} CONTAINER_NAME_DENYLIST: caddy-autogen,caddy-autogen-discovery,caddy-autogen-socket-proxy
DEFAULT_SCHEME: ${DEFAULT_SCHEME:-http} DEFAULT_SCHEME: http
DEFAULT_PATH: ${DEFAULT_PATH:-/} DEFAULT_PATH: /
DEFAULT_HEALTH_URI: ${DEFAULT_HEALTH_URI:-} DEFAULT_HEALTH_URI:
CONFIG_FILE: ${CONFIG_FILE:-/app/config/defaults.yaml} STATUS_BIND: 0.0.0.0:8089
STATUS_UI_PORT: 31820
STATUS_UPSTREAM: discovery-agent:8089
CF_VERIFY_URL: https://api.cloudflare.com/client/v4/user/tokens/verify
CADDY_DATA_DIR: /caddy-data
CONFIG_FILE: /app/config/defaults.yaml
volumes: volumes:
- type: bind - type: bind
source: /DATA/AppData/$AppID/config source: /DATA/AppData/$AppID/config
target: /app/config target: /app/config
read_only: true read_only: true
- type: bind
source: /DATA/AppData/$AppID/caddy/data
target: /caddy-data
read_only: true
read_only: true read_only: true
tmpfs: tmpfs:
- /tmp - /tmp
@@ -144,10 +161,10 @@ x-casaos:
- arm64 - arm64
- arm - arm
main: caddy main: caddy
category: Network category: phirna
author: Zima Apps Team author: Joachim Friberg
developer: Zima Apps Team developer: Joachim Friberg
icon: https://caddyserver.com/resources/images/caddy-circle-lock.svg icon: https://cdn.simpleicons.org/caddy
tagline: tagline:
en_us: Auto-generate Caddy endpoints from running containers en_us: Auto-generate Caddy endpoints from running containers
description: description:
@@ -158,5 +175,5 @@ x-casaos:
title: title:
en_us: Caddy AutoGen en_us: Caddy AutoGen
index: / index: /
port_map: ${HTTPS_PORT:-443} port_map: "443"
scheme: https scheme: https
@@ -1,5 +1,7 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
import importlib.util import importlib.util
import json
import tempfile
from pathlib import Path from pathlib import Path
@@ -93,8 +95,13 @@ def test_optin_route_selection(module):
allow_internal_tls_fallback=False, allow_internal_tls_fallback=False,
wildcard_domain="home.example.test", wildcard_domain="home.example.test",
cert_email="", cert_email="",
status_ui_port=31820,
status_upstream="discovery-agent:8089",
) )
assert_true(":31820" in caddyfile, "expected status UI server block")
assert_true("remote_ip private_ranges" in caddyfile, "expected LAN-only restriction")
assert_true("reverse_proxy discovery-agent:8089" in caddyfile, "expected status upstream")
assert_true("frigate.home.example.test" in caddyfile, "expected frigate host in caddyfile") assert_true("frigate.home.example.test" in caddyfile, "expected frigate host in caddyfile")
assert_true("reverse_proxy http://host.docker.internal:5000" in caddyfile, "expected web port route") assert_true("reverse_proxy http://host.docker.internal:5000" in caddyfile, "expected web port route")
assert_true("8554" not in caddyfile and "8555" not in caddyfile, "media ports must not be routed") assert_true("8554" not in caddyfile and "8555" not in caddyfile, "media ports must not be routed")
@@ -122,6 +129,8 @@ def test_fail_closed_and_internal_fallback(module):
allow_internal_tls_fallback=False, allow_internal_tls_fallback=False,
wildcard_domain="", wildcard_domain="",
cert_email="", cert_email="",
status_ui_port=31820,
status_upstream="discovery-agent:8089",
) )
except RuntimeError as exc: except RuntimeError as exc:
failed = True failed = True
@@ -135,15 +144,60 @@ def test_fail_closed_and_internal_fallback(module):
allow_internal_tls_fallback=True, allow_internal_tls_fallback=True,
wildcard_domain="", wildcard_domain="",
cert_email="", cert_email="",
status_ui_port=31820,
status_upstream="discovery-agent:8089",
) )
assert_true("local_certs" in fallback_caddyfile, "expected local_certs in fallback mode") assert_true("local_certs" in fallback_caddyfile, "expected local_certs in fallback mode")
assert_true("tls internal" in fallback_caddyfile, "expected internal tls in fallback mode") assert_true("tls internal" in fallback_caddyfile, "expected internal tls in fallback mode")
def test_cloudflare_verify_and_cert_discovery(module):
class FakeResponse:
def __init__(self, payload):
self._payload = payload
def read(self):
return json.dumps(self._payload).encode("utf-8")
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
return False
def fake_urlopen(req, timeout=0):
_ = req
_ = timeout
return FakeResponse({"success": True})
original_urlopen = module.urllib.request.urlopen
module.urllib.request.urlopen = fake_urlopen
try:
status = module._verify_cloudflare_token("https://api.cloudflare.com/client/v4/user/tokens/verify", "token")
finally:
module.urllib.request.urlopen = original_urlopen
assert_true(status["reachable"] is True, "cloudflare should be reachable in mocked success")
assert_true(status["token_valid"] is True, "token should be valid in mocked success")
with tempfile.TemporaryDirectory() as td:
cert_dir = Path(td) / "caddy" / "certificates" / "acme-v02.api.letsencrypt.org-directory" / "example.com"
cert_dir.mkdir(parents=True, exist_ok=True)
(cert_dir / "demo.home.example.test.crt").write_text("fake", encoding="utf-8")
(cert_dir / "_.home.example.test.crt").write_text("fake", encoding="utf-8")
hosts = module._collect_letsencrypt_hosts(td)
assert_true("demo.home.example.test" in hosts, "expected concrete cert host")
assert_true("*.home.example.test" in hosts, "expected wildcard cert host conversion")
assert_true(module._has_matching_le_cert("api.home.example.test", hosts), "wildcard should match")
assert_true(module._has_matching_le_cert("demo.home.example.test", hosts), "exact cert should match")
def main(): def main():
module = load_agent_module() module = load_agent_module()
test_optin_route_selection(module) test_optin_route_selection(module)
test_fail_closed_and_internal_fallback(module) test_fail_closed_and_internal_fallback(module)
test_cloudflare_verify_and_cert_discovery(module)
print("Integration tests passed") print("Integration tests passed")
+45 -5
View File
@@ -67,7 +67,27 @@ Förväntat resultat:
- posten med `10.0.4.2` har `used=true`. - posten med `10.0.4.2` har `used=true`.
- `containers` innehåller `ip-verify-nginx`. - `containers` innehåller `ip-verify-nginx`.
### Test C: Disable/Delete efter frigöring ### Test C: DNS create när posten är enabled + used
Förutsätter DNS config i appen, exempel för AdGuard:
- `DNS_PROVIDER=adguard`
- `DNS_BASE_DOMAIN=home.arpa`
- `ADGUARD_URL=http://<adguard-ip>:3000`
- `ADGUARD_USERNAME=<user>`
- `ADGUARD_PASSWORD=<password>`
Verifiera att record skapats:
```bash
dig +short lan-test.home.arpa @<adguard-ip>
```
Förväntat resultat:
- returnerar `10.0.4.2`.
### Test D: Disable/Delete efter frigöring
Stoppa testcontainer: Stoppa testcontainer:
@@ -103,7 +123,7 @@ Förväntat resultat:
## 3) Negativt / fail-closed testfall ## 3) Negativt / fail-closed testfall
### Test D: Blockera disable när IP används ### Test E: Blockera disable när IP används
1. Skapa + enable som i Test A. 1. Skapa + enable som i Test A.
2. Starta container som i Test B. 2. Starta container som i Test B.
@@ -120,19 +140,39 @@ Förväntat resultat:
- HTTP `409`. - HTTP `409`.
- feltext som anger att posten används av container. - feltext som anger att posten används av container.
### Test F: Fail-closed vid DNS-fel
1. Se till att en post är `enabled` och `used` (Test A+B).
2. Sabotera DNS-auth tillfälligt, exempel:
- ändra `ADGUARD_PASSWORD` till fel värde och starta om appen.
3. Försök disable/delete på posten.
```bash
curl -sS -o /tmp/dns-fail.out -w '%{http_code}\n' \
-X POST "http://127.0.0.1:31810/api/entries/${ENTRY_ID}/disable"
cat /tmp/dns-fail.out
```
Förväntat resultat:
- HTTP `409` eller `503`.
- feltext som indikerar DNS-synkfel.
- posten ska inte lämna systemet i delvis uppdaterat läge.
## 4) DNS / nät / TLS verifiering ## 4) DNS / nät / TLS verifiering
### DNS (om hostname används i LAN) ### DNS (om hostname används i LAN)
```bash ```bash
DNS_SERVER="<dns-server-ip>" DNS_SERVER="<dns-server-ip>"
HOSTNAME_TO_TEST="<hostname-i-lan>" HOSTNAME_TO_TEST="lan-test.home.arpa"
dig +short "${HOSTNAME_TO_TEST}" @"${DNS_SERVER}" dig +short "${HOSTNAME_TO_TEST}" @"${DNS_SERVER}"
``` ```
Förväntat resultat: Förväntat resultat:
- returnerar avsedd LAN-IP. - returnerar avsedd LAN-IP när posten är `enabled && used`.
- ingen träff när posten inte längre är `used` eller är `disabled`.
### Nätverk (lyssning och routning) ### Nätverk (lyssning och routning)
@@ -183,7 +223,7 @@ docker inspect docker-ip-addr-manager | jq '.[0].Config.Env'
```bash ```bash
docker logs --tail 200 docker-ip-addr-manager docker logs --tail 200 docker-ip-addr-manager
docker logs --tail 200 docker-ip-addr-manager-socket-proxy docker logs --tail 200 docker-ip-addr-manager-proxy
``` ```
- Konkreta felobservationer: - Konkreta felobservationer:
+37 -4
View File
@@ -13,12 +13,16 @@ Exempel: istället för att köra `ip addr add 10.0.4.2/16 dev eth0` via SSH, ka
- Sorterbar tabell: namn, IP-adress, used/unused, containernamn, device, enable/disable. - Sorterbar tabell: namn, IP-adress, used/unused, containernamn, device, enable/disable.
- Used/unused-kontroll via Docker API (`NetworkSettings.Ports`) med exakt `HostIp`-match. - Used/unused-kontroll via Docker API (`NetworkSettings.Ports`) med exakt `HostIp`-match.
- Include stopped containers i used-kontroll. - Include stopped containers i used-kontroll.
- DNS-livscykel (opt-in): skapar A-record när `enabled=true` och `used=true`, tar bort record när villkoret inte längre gäller.
- DNS-namn byggs från `name` + `DNS_BASE_DOMAIN` => `<name>.<base-domain>` (DNS-säkrad label).
- Fail-closed: - Fail-closed:
- disable blockeras om IP används av minst en container, - disable blockeras om IP används av minst en container,
- delete blockeras om posten är enabled eller used, - delete blockeras om posten är enabled eller used,
- disable/delete blockeras om Docker-usage inte kan verifieras. - disable/delete blockeras om Docker-usage inte kan verifieras,
- state-ändringar blockeras om nödvändig DNS-synk misslyckas.
- Startup reconcile: enabled-poster återappliceras vid appstart. - Startup reconcile: enabled-poster återappliceras vid appstart.
- Manuell refresh-knapp (ingen websocket i v1). - DNS reconcile körs i bakgrunden med poll-interval.
- Manuell refresh-knapp för UI-status (ingen websocket i v1).
## Portar ## Portar
@@ -77,6 +81,33 @@ Viktiga environment-variabler:
- alternativt `http://127.0.0.1:2375` för socket-proxy. - alternativt `http://127.0.0.1:2375` för socket-proxy.
- `DOCKER_TIMEOUT_SECONDS` (default `3`) - `DOCKER_TIMEOUT_SECONDS` (default `3`)
- `STATE_FILE` (default `/data/entries.json`) - `STATE_FILE` (default `/data/entries.json`)
- `DNS_PROVIDER` (`none`, `adguard`, `rfc2136`; default `none`)
- `DNS_BASE_DOMAIN` (exempel: `home.arpa`)
- `DNS_TTL_SECONDS` (default `120`)
- `DNS_SYNC_INTERVAL_SECONDS` (default `15`)
AdGuard (`DNS_PROVIDER=adguard`):
- `ADGUARD_URL` (exempel: `http://127.0.0.1:3000`)
- `ADGUARD_USERNAME`
- `ADGUARD_PASSWORD`
- `ADGUARD_API_TOKEN` (framtida alternativ, inte aktiv auth-väg i v1)
RFC2136 (`DNS_PROVIDER=rfc2136`):
- `RFC2136_SERVER`
- `RFC2136_ZONE`
- `RFC2136_PORT` (default `53`)
- `RFC2136_TSIG_KEY_NAME` (valfri om osignerade updates tillåts)
- `RFC2136_TSIG_SECRET` (base64, valfri utan TSIG)
- `RFC2136_TSIG_ALGORITHM` (default `hmac-sha256`)
## DNS-beteende
- Villkor för record: endast när posten är `enabled` och `used`.
- När posten inte längre är `used` tas DNS-record bort i bakgrundsreconcile.
- Vid enable/disable/delete görs direkt DNS-synk och operationen failar vid synkfel (fail-closed).
- Om Docker usage-kontroll är okänd i bakgrundsloop görs inga DNS-mutationer i den cykeln.
## Integrationstester ## Integrationstester
@@ -91,7 +122,9 @@ Testerna mockar Docker API och `ip`-kommandoflöde och verifierar:
- exakt `HostIp`-matchning, - exakt `HostIp`-matchning,
- fail-closed disable/delete, - fail-closed disable/delete,
- blockering vid enabled/used, - blockering vid enabled/used,
- startup reconcile av enabled-poster. - startup reconcile av enabled-poster,
- DNS create/delete på `enabled && used`,
- fail-closed rollback vid DNS-synkfel.
## Auth-notis ## Auth-notis
@@ -102,4 +135,4 @@ Auth/autorisering ska implementeras i en senare version och är en uttalad roadm
## Roadmap (ej v1) ## Roadmap (ej v1)
- WebSocket-baserad live-uppdatering av used-status. - WebSocket-baserad live-uppdatering av used-status.
- DNS-integration (Cloudflare/lokal DNS) kopplat till IP-poster och hostnamn. - Alternativ auth för AdGuard via API-token.
@@ -10,6 +10,20 @@ class Settings:
docker_api_url: str docker_api_url: str
docker_timeout_seconds: float docker_timeout_seconds: float
app_port: int app_port: int
dns_provider: str
dns_base_domain: str
dns_ttl_seconds: int
dns_sync_interval_seconds: float
adguard_url: str
adguard_username: str
adguard_password: str
adguard_api_token: str
rfc2136_server: str
rfc2136_zone: str
rfc2136_port: int
rfc2136_tsig_key_name: str
rfc2136_tsig_secret: str
rfc2136_tsig_algorithm: str
def get_settings() -> Settings: def get_settings() -> Settings:
@@ -18,4 +32,18 @@ def get_settings() -> Settings:
docker_api_url=os.getenv("DOCKER_API_URL", "unix:///var/run/docker.sock"), docker_api_url=os.getenv("DOCKER_API_URL", "unix:///var/run/docker.sock"),
docker_timeout_seconds=float(os.getenv("DOCKER_TIMEOUT_SECONDS", "3")), docker_timeout_seconds=float(os.getenv("DOCKER_TIMEOUT_SECONDS", "3")),
app_port=int(os.getenv("APP_PORT", "31810")), app_port=int(os.getenv("APP_PORT", "31810")),
dns_provider=os.getenv("DNS_PROVIDER", "none"),
dns_base_domain=os.getenv("DNS_BASE_DOMAIN", ""),
dns_ttl_seconds=int(os.getenv("DNS_TTL_SECONDS", "120")),
dns_sync_interval_seconds=float(os.getenv("DNS_SYNC_INTERVAL_SECONDS", "15")),
adguard_url=os.getenv("ADGUARD_URL", ""),
adguard_username=os.getenv("ADGUARD_USERNAME", ""),
adguard_password=os.getenv("ADGUARD_PASSWORD", ""),
adguard_api_token=os.getenv("ADGUARD_API_TOKEN", ""),
rfc2136_server=os.getenv("RFC2136_SERVER", ""),
rfc2136_zone=os.getenv("RFC2136_ZONE", ""),
rfc2136_port=int(os.getenv("RFC2136_PORT", "53")),
rfc2136_tsig_key_name=os.getenv("RFC2136_TSIG_KEY_NAME", ""),
rfc2136_tsig_secret=os.getenv("RFC2136_TSIG_SECRET", ""),
rfc2136_tsig_algorithm=os.getenv("RFC2136_TSIG_ALGORITHM", "hmac-sha256"),
) )
@@ -0,0 +1,309 @@
from __future__ import annotations
from dataclasses import dataclass
import base64
import http.client
import json
from typing import Protocol
from urllib.parse import urlparse
class DnsSyncError(RuntimeError):
pass
class DnsProvider(Protocol):
def upsert_a_record(self, fqdn: str, ip: str, ttl: int) -> None:
raise NotImplementedError
def delete_a_record(self, fqdn: str) -> None:
raise NotImplementedError
def to_fqdn(entry_name: str, base_domain: str) -> str:
label = _sanitize_label(entry_name)
domain = base_domain.strip().lower().strip(".")
if not domain:
raise DnsSyncError("DNS_BASE_DOMAIN is required when DNS is enabled")
return f"{label}.{domain}"
def _sanitize_label(value: str) -> str:
source = value.strip().lower()
if not source:
raise DnsSyncError("Entry name is required to create DNS record")
cleaned: list[str] = []
prev_dash = False
for ch in source:
if "a" <= ch <= "z" or "0" <= ch <= "9":
cleaned.append(ch)
prev_dash = False
continue
if ch in {" ", "_", "-"} and not prev_dash:
cleaned.append("-")
prev_dash = True
label = "".join(cleaned).strip("-")
if not label:
raise DnsSyncError(f"Entry name cannot produce DNS-safe label: {value!r}")
if len(label) > 63:
raise DnsSyncError("DNS label derived from entry name is too long (max 63)")
return label
@dataclass(frozen=True)
class AdguardConfig:
url: str
username: str
password: str
timeout_seconds: float
class AdguardDnsProvider:
def __init__(self, config: AdguardConfig):
parsed = urlparse(config.url)
if parsed.scheme not in {"http", "https"}:
raise ValueError("ADGUARD_URL must use http or https")
if not parsed.netloc:
raise ValueError("ADGUARD_URL must include host")
self._https = parsed.scheme == "https"
self._host = parsed.hostname or "localhost"
self._port = parsed.port
self._base_path = parsed.path.rstrip("/")
self._username = config.username
self._password = config.password
self._timeout = config.timeout_seconds
self._session_cookie: str | None = None
def upsert_a_record(self, fqdn: str, ip: str, ttl: int) -> None:
del ttl # AdGuard rewrite records do not expose TTL controls.
rewrites = self._list_rewrites()
for item in rewrites:
if item.get("domain") == fqdn and item.get("answer") == ip:
return
if item.get("domain") == fqdn and item.get("answer") != ip:
self._request("POST", "/control/rewrite/delete", {"domain": fqdn, "answer": item.get("answer", "")})
self._request("POST", "/control/rewrite/add", {"domain": fqdn, "answer": ip})
def delete_a_record(self, fqdn: str) -> None:
rewrites = self._list_rewrites()
for item in rewrites:
if item.get("domain") != fqdn:
continue
self._request("POST", "/control/rewrite/delete", {"domain": fqdn, "answer": item.get("answer", "")})
def _list_rewrites(self) -> list[dict]:
payload = self._request("GET", "/control/rewrite/list", None)
if not isinstance(payload, list):
raise DnsSyncError("AdGuard returned unexpected rewrite list format")
output: list[dict] = []
for item in payload:
if isinstance(item, dict):
output.append(item)
return output
def _request(self, method: str, path: str, payload: dict | None) -> object:
if self._session_cookie is None:
self._login()
return self._request_with_session(method, path, payload, retry_on_auth=True)
def _login(self) -> None:
body = {"name": self._username, "password": self._password}
payload, headers = self._raw_request("POST", "/control/login", body, include_auth=False)
if headers is None:
raise DnsSyncError("AdGuard login failed: missing response headers")
cookie = headers.get("set-cookie", "")
session = ""
for piece in cookie.split(";"):
piece = piece.strip()
if piece.startswith("agh_session="):
session = piece
break
if not session:
raise DnsSyncError("AdGuard login failed: no agh_session cookie")
self._session_cookie = session
del payload
def _request_with_session(self, method: str, path: str, payload: dict | None, retry_on_auth: bool) -> object:
body, _ = self._raw_request(method, path, payload, include_auth=True)
if isinstance(body, dict) and body.get("message") == "unauthorized":
if retry_on_auth:
self._session_cookie = None
self._login()
return self._request_with_session(method, path, payload, retry_on_auth=False)
raise DnsSyncError("AdGuard request unauthorized")
return body
def _raw_request(
self, method: str, path: str, payload: dict | None, include_auth: bool
) -> tuple[object, dict[str, str] | None]:
conn: http.client.HTTPConnection | http.client.HTTPSConnection
if self._https:
conn = http.client.HTTPSConnection(self._host, self._port, timeout=self._timeout)
else:
conn = http.client.HTTPConnection(self._host, self._port, timeout=self._timeout)
request_path = f"{self._base_path}{path}"
raw = ""
headers = {"Content-Type": "application/json"}
if include_auth and self._session_cookie:
headers["Cookie"] = self._session_cookie
if payload is not None:
raw = json.dumps(payload)
try:
conn.request(method, request_path, body=raw, headers=headers)
response = conn.getresponse()
body_text = response.read().decode("utf-8", errors="replace")
response_headers = {k.lower(): v for k, v in response.getheaders()}
except OSError as exc:
raise DnsSyncError(f"AdGuard request failed for {path}: {exc}") from exc
finally:
conn.close()
if response.status < 200 or response.status >= 300:
raise DnsSyncError(
f"AdGuard request failed for {path}: HTTP {response.status} {response.reason}; body={body_text[:400]}"
)
if not body_text.strip():
return {}, response_headers
try:
return json.loads(body_text), response_headers
except json.JSONDecodeError:
return body_text, response_headers
@dataclass(frozen=True)
class Rfc2136Config:
server: str
zone: str
port: int
timeout_seconds: float
tsig_key_name: str
tsig_secret: str
tsig_algorithm: str
class Rfc2136DnsProvider:
def __init__(self, config: Rfc2136Config):
if not config.server.strip():
raise ValueError("RFC2136_SERVER is required")
if not config.zone.strip():
raise ValueError("RFC2136_ZONE is required")
self._server = config.server.strip()
self._zone = config.zone.strip().rstrip(".")
self._port = config.port
self._timeout = config.timeout_seconds
self._key_name = config.tsig_key_name.strip()
self._secret = config.tsig_secret.strip()
self._algorithm = config.tsig_algorithm.strip() or "hmac-sha256"
def upsert_a_record(self, fqdn: str, ip: str, ttl: int) -> None:
rcode, tsigkeyring, update, query = self._dns_modules()
zone_text = self._zone_with_dot()
keyring = self._keyring_or_none(tsigkeyring)
target = self._absolute_name(fqdn)
try:
req = update.Update(zone_text, keyring=keyring, keyname=self._key_name or None, keyalgorithm=self._algorithm)
req.delete(target, "A")
req.add(target, int(ttl), "A", ip)
response = query.tcp(req, self._server, port=self._port, timeout=self._timeout)
except Exception as exc: # noqa: BLE001
raise DnsSyncError(f"RFC2136 upsert failed for {fqdn} -> {ip}: {exc}") from exc
if response.rcode() != rcode.NOERROR:
text = rcode.to_text(response.rcode())
raise DnsSyncError(f"RFC2136 upsert failed for {fqdn}: {text}")
def delete_a_record(self, fqdn: str) -> None:
rcode, tsigkeyring, update, query = self._dns_modules()
zone_text = self._zone_with_dot()
keyring = self._keyring_or_none(tsigkeyring)
target = self._absolute_name(fqdn)
try:
req = update.Update(zone_text, keyring=keyring, keyname=self._key_name or None, keyalgorithm=self._algorithm)
req.delete(target, "A")
response = query.tcp(req, self._server, port=self._port, timeout=self._timeout)
except Exception as exc: # noqa: BLE001
raise DnsSyncError(f"RFC2136 delete failed for {fqdn}: {exc}") from exc
if response.rcode() != rcode.NOERROR:
text = rcode.to_text(response.rcode())
raise DnsSyncError(f"RFC2136 delete failed for {fqdn}: {text}")
def _dns_modules(self):
try:
import dns.query as query
import dns.rcode as rcode
import dns.tsigkeyring as tsigkeyring
import dns.update as update
except ImportError as exc:
raise DnsSyncError("dnspython is required for RFC2136 mode") from exc
return rcode, tsigkeyring, update, query
def _keyring_or_none(self, tsigkeyring):
if not self._key_name and not self._secret:
return None
if not self._key_name or not self._secret:
raise DnsSyncError("RFC2136 TSIG requires both key name and secret")
key_name = self._key_name if self._key_name.endswith(".") else f"{self._key_name}."
try:
base64.b64decode(self._secret, validate=True)
except Exception as exc: # noqa: BLE001
raise DnsSyncError("RFC2136_TSIG_SECRET must be valid base64") from exc
if self._algorithm not in {"hmac-sha256", "hmac-sha512", "hmac-sha1", "hmac-md5.sig-alg.reg.int"}:
raise DnsSyncError(f"Unsupported TSIG algorithm: {self._algorithm}")
return tsigkeyring.from_text({key_name: self._secret})
def _zone_with_dot(self) -> str:
return self._zone if self._zone.endswith(".") else f"{self._zone}."
def _absolute_name(self, fqdn: str) -> str:
return fqdn if fqdn.endswith(".") else f"{fqdn}."
def build_dns_provider(
provider_name: str,
*,
adguard_url: str,
adguard_username: str,
adguard_password: str,
rfc2136_server: str,
rfc2136_zone: str,
rfc2136_port: int,
rfc2136_tsig_key_name: str,
rfc2136_tsig_secret: str,
rfc2136_tsig_algorithm: str,
timeout_seconds: float,
) -> DnsProvider | None:
mode = provider_name.strip().lower()
if not mode or mode == "none":
return None
if mode == "adguard":
if not adguard_url.strip():
raise DnsSyncError("ADGUARD_URL is required for DNS_PROVIDER=adguard")
if not adguard_username.strip() or not adguard_password.strip():
raise DnsSyncError("ADGUARD_USERNAME and ADGUARD_PASSWORD are required for DNS_PROVIDER=adguard")
return AdguardDnsProvider(
AdguardConfig(
url=adguard_url,
username=adguard_username,
password=adguard_password,
timeout_seconds=timeout_seconds,
)
)
if mode == "rfc2136":
return Rfc2136DnsProvider(
Rfc2136Config(
server=rfc2136_server,
zone=rfc2136_zone,
port=rfc2136_port,
timeout_seconds=timeout_seconds,
tsig_key_name=rfc2136_tsig_key_name,
tsig_secret=rfc2136_tsig_secret,
tsig_algorithm=rfc2136_tsig_algorithm,
)
)
raise DnsSyncError(f"Unsupported DNS_PROVIDER: {provider_name}")
@@ -1,6 +1,7 @@
from __future__ import annotations from __future__ import annotations
from pathlib import Path from pathlib import Path
import threading
from fastapi import FastAPI, HTTPException from fastapi import FastAPI, HTTPException
from fastapi.responses import FileResponse, JSONResponse from fastapi.responses import FileResponse, JSONResponse
@@ -8,6 +9,7 @@ from fastapi.staticfiles import StaticFiles
from app.config import get_settings from app.config import get_settings
from app.docker_api import DockerApiClient, DockerApiError, DockerUsageResolver from app.docker_api import DockerApiClient, DockerApiError, DockerUsageResolver
from app.dns_sync import DnsSyncError, build_dns_provider
from app.ip_commands import CommandError, IpAddressManager from app.ip_commands import CommandError, IpAddressManager
from app.service import ( from app.service import (
ConflictError, ConflictError,
@@ -25,11 +27,34 @@ def build_service() -> EntryService:
docker_client = DockerApiClient(settings.docker_api_url, timeout_seconds=settings.docker_timeout_seconds) docker_client = DockerApiClient(settings.docker_api_url, timeout_seconds=settings.docker_timeout_seconds)
usage_resolver = DockerUsageResolver(docker_client) usage_resolver = DockerUsageResolver(docker_client)
ip_manager = IpAddressManager() ip_manager = IpAddressManager()
return EntryService(storage=storage, usage_resolver=usage_resolver, ip_manager=ip_manager) dns_provider = build_dns_provider(
settings.dns_provider,
adguard_url=settings.adguard_url,
adguard_username=settings.adguard_username,
adguard_password=settings.adguard_password,
rfc2136_server=settings.rfc2136_server,
rfc2136_zone=settings.rfc2136_zone,
rfc2136_port=settings.rfc2136_port,
rfc2136_tsig_key_name=settings.rfc2136_tsig_key_name,
rfc2136_tsig_secret=settings.rfc2136_tsig_secret,
rfc2136_tsig_algorithm=settings.rfc2136_tsig_algorithm,
timeout_seconds=settings.docker_timeout_seconds,
)
return EntryService(
storage=storage,
usage_resolver=usage_resolver,
ip_manager=ip_manager,
dns_provider=dns_provider,
dns_base_domain=settings.dns_base_domain,
dns_ttl_seconds=settings.dns_ttl_seconds,
)
service = build_service() service = build_service()
app = FastAPI(title="Docker IP Addr Manager", version="0.1.0") app = FastAPI(title="Docker IP Addr Manager", version="0.1.0")
settings = get_settings()
stop_event = threading.Event()
background_thread: threading.Thread | None = None
static_dir = Path(__file__).parent / "static" static_dir = Path(__file__).parent / "static"
app.mount("/static", StaticFiles(directory=static_dir), name="static") app.mount("/static", StaticFiles(directory=static_dir), name="static")
@@ -41,6 +66,39 @@ def startup_reconcile() -> None:
if errors: if errors:
for error in errors: for error in errors:
print(f"[startup-reconcile] {error}") print(f"[startup-reconcile] {error}")
dns_errors = service.reconcile_dns_records()
if dns_errors:
for error in dns_errors:
print(f"[dns-reconcile-startup] {error}")
_start_dns_background_loop()
@app.on_event("shutdown")
def shutdown_reconcile() -> None:
stop_event.set()
if background_thread and background_thread.is_alive():
background_thread.join(timeout=2.0)
def _dns_background_worker(interval_seconds: float) -> None:
while not stop_event.wait(interval_seconds):
errors = service.reconcile_dns_records()
for error in errors:
print(f"[dns-reconcile] {error}")
def _start_dns_background_loop() -> None:
global background_thread
if settings.dns_provider.strip().lower() in {"", "none"}:
return
if background_thread and background_thread.is_alive():
return
background_thread = threading.Thread(
target=_dns_background_worker,
args=(max(settings.dns_sync_interval_seconds, 1.0),),
daemon=True,
)
background_thread.start()
@app.get("/") @app.get("/")
@@ -139,3 +197,8 @@ def delete_entry(entry_id: str) -> dict:
@app.exception_handler(DockerApiError) @app.exception_handler(DockerApiError)
async def docker_error_handler(_, exc: DockerApiError): async def docker_error_handler(_, exc: DockerApiError):
return JSONResponse(status_code=503, content={"detail": str(exc)}) return JSONResponse(status_code=503, content={"detail": str(exc)})
@app.exception_handler(DnsSyncError)
async def dns_error_handler(_, exc: DnsSyncError):
return JSONResponse(status_code=503, content={"detail": str(exc)})
@@ -46,6 +46,8 @@ class EntryView:
used: bool used: bool
containers: list[str] containers: list[str]
usage_known: bool usage_known: bool
dns_desired: bool = False
dns_last_error: str | None = None
def to_dict(self) -> dict: def to_dict(self) -> dict:
return { return {
@@ -59,4 +61,6 @@ class EntryView:
"used": self.used, "used": self.used,
"containers": self.containers, "containers": self.containers,
"usage_known": self.usage_known, "usage_known": self.usage_known,
"dns_desired": self.dns_desired,
"dns_last_error": self.dns_last_error,
} }
@@ -7,6 +7,7 @@ from typing import Callable
from uuid import uuid4 from uuid import uuid4
from app.docker_api import DockerApiError, DockerUsageResolver from app.docker_api import DockerApiError, DockerUsageResolver
from app.dns_sync import DnsProvider, DnsSyncError, to_fqdn
from app.interfaces import list_host_interfaces from app.interfaces import list_host_interfaces
from app.ip_commands import CommandError, IpAddressManager from app.ip_commands import CommandError, IpAddressManager
from app.models import EntryView, IpEntry from app.models import EntryView, IpEntry
@@ -50,12 +51,19 @@ class EntryService:
usage_resolver: DockerUsageResolver, usage_resolver: DockerUsageResolver,
ip_manager: IpAddressManager, ip_manager: IpAddressManager,
interface_provider: Callable[[], list[str]] = list_host_interfaces, interface_provider: Callable[[], list[str]] = list_host_interfaces,
dns_provider: DnsProvider | None = None,
dns_base_domain: str = "",
dns_ttl_seconds: int = 120,
): ):
self._storage = storage self._storage = storage
self._usage_resolver = usage_resolver self._usage_resolver = usage_resolver
self._ip_manager = ip_manager self._ip_manager = ip_manager
self._interface_provider = interface_provider self._interface_provider = interface_provider
self._dns_provider = dns_provider
self._dns_base_domain = dns_base_domain
self._dns_ttl_seconds = dns_ttl_seconds
self._lock = threading.Lock() self._lock = threading.Lock()
self._dns_errors_by_id: dict[str, str] = {}
def list_interfaces(self) -> list[str]: def list_interfaces(self) -> list[str]:
interfaces = self._interface_provider() interfaces = self._interface_provider()
@@ -89,6 +97,8 @@ class EntryService:
used=used, used=used,
containers=containers, containers=containers,
usage_known=usage_known, usage_known=usage_known,
dns_desired=bool(self._dns_provider) and usage_known and used and entry.enabled,
dns_last_error=self._dns_errors_by_id.get(entry.id),
) )
) )
@@ -100,6 +110,7 @@ class EntryService:
entries = self._storage.list_entries() entries = self._storage.list_entries()
self._assert_device_exists(parsed["device"]) self._assert_device_exists(parsed["device"])
self._assert_unique_binding(entries, ip=parsed["ip"], cidr=parsed["cidr"], device=parsed["device"]) self._assert_unique_binding(entries, ip=parsed["ip"], cidr=parsed["cidr"], device=parsed["device"])
self._assert_unique_name(entries, name=parsed["name"])
created = IpEntry( created = IpEntry(
id=uuid4().hex, id=uuid4().hex,
@@ -129,6 +140,7 @@ class EntryService:
device=parsed["device"], device=parsed["device"],
ignore_entry_id=entry_id, ignore_entry_id=entry_id,
) )
self._assert_unique_name(entries, name=parsed["name"], ignore_entry_id=entry_id)
updated = IpEntry( updated = IpEntry(
id=current.id, id=current.id,
name=parsed["name"], name=parsed["name"],
@@ -145,6 +157,7 @@ class EntryService:
with self._lock: with self._lock:
entries = self._storage.list_entries() entries = self._storage.list_entries()
index, entry = _find_entry(entries, entry_id) index, entry = _find_entry(entries, entry_id)
previous_enabled = entry.enabled
if enabled: if enabled:
self._ip_manager.ensure_present(entry.ip, entry.cidr, entry.device) self._ip_manager.ensure_present(entry.ip, entry.cidr, entry.device)
@@ -154,6 +167,12 @@ class EntryService:
self._ip_manager.ensure_absent(entry.ip, entry.cidr, entry.device) self._ip_manager.ensure_absent(entry.ip, entry.cidr, entry.device)
entry.enabled = False entry.enabled = False
try:
self._sync_dns_for_entry_locked(entry, strict=True)
except Exception: # noqa: BLE001
self._rollback_enable_change(entry, previous_enabled)
raise
entries[index] = entry entries[index] = entry
self._storage.save_entries(entries) self._storage.save_entries(entries)
return entry return entry
@@ -166,8 +185,10 @@ class EntryService:
raise ConflictError("Disable entry before deleting") raise ConflictError("Disable entry before deleting")
self._assert_not_used(entry) self._assert_not_used(entry)
self._delete_dns_for_entry_locked(entry, strict=True)
remaining = [candidate for candidate in entries if candidate.id != entry_id] remaining = [candidate for candidate in entries if candidate.id != entry_id]
self._storage.save_entries(remaining) self._storage.save_entries(remaining)
self._dns_errors_by_id.pop(entry.id, None)
def reconcile_enabled_entries(self) -> list[str]: def reconcile_enabled_entries(self) -> list[str]:
errors: list[str] = [] errors: list[str] = []
@@ -186,6 +207,84 @@ class EntryService:
self._storage.save_entries(entries) self._storage.save_entries(entries)
return errors return errors
def reconcile_dns_records(self) -> list[str]:
if not self._dns_provider:
return []
errors: list[str] = []
with self._lock:
entries = self._storage.list_entries()
usage_map, usage_known, usage_error = self._resolve_usage(entries)
if not usage_known:
msg = f"Docker usage check failed for DNS reconcile: {usage_error or 'unknown error'}"
for entry in entries:
self._dns_errors_by_id[entry.id] = msg
return [msg]
for entry in entries:
used = bool(usage_map.get(entry.ip, set()))
desired = entry.enabled and used
try:
self._apply_dns_state_locked(entry, desired)
self._dns_errors_by_id.pop(entry.id, None)
except (DnsSyncError, DependencyError, ConflictError) as exc:
self._dns_errors_by_id[entry.id] = str(exc)
errors.append(f"{entry.name}: {exc}")
return errors
def _rollback_enable_change(self, entry: IpEntry, previous_enabled: bool) -> None:
try:
if previous_enabled:
self._ip_manager.ensure_present(entry.ip, entry.cidr, entry.device)
entry.enabled = True
else:
self._ip_manager.ensure_absent(entry.ip, entry.cidr, entry.device)
entry.enabled = False
except CommandError:
pass
def _sync_dns_for_entry_locked(self, entry: IpEntry, strict: bool) -> None:
if not self._dns_provider:
return
usage_map, usage_known, usage_error = self._resolve_usage([entry])
if not usage_known:
msg = f"Docker usage check failed: {usage_error or 'unknown error'}"
self._dns_errors_by_id[entry.id] = msg
if strict:
raise DependencyError(msg)
return
desired = entry.enabled and bool(usage_map.get(entry.ip, set()))
self._apply_dns_state_locked(entry, desired)
self._dns_errors_by_id.pop(entry.id, None)
def _delete_dns_for_entry_locked(self, entry: IpEntry, strict: bool) -> None:
if not self._dns_provider:
return
try:
fqdn = to_fqdn(entry.name, self._dns_base_domain)
self._dns_provider.delete_a_record(fqdn)
self._dns_errors_by_id.pop(entry.id, None)
except DnsSyncError as exc:
self._dns_errors_by_id[entry.id] = str(exc)
if strict:
raise ConflictError(f"DNS delete failed for {entry.name}: {exc}") from exc
def _apply_dns_state_locked(self, entry: IpEntry, desired: bool) -> None:
if not self._dns_provider:
return
try:
fqdn = to_fqdn(entry.name, self._dns_base_domain)
if desired:
self._dns_provider.upsert_a_record(fqdn, entry.ip, self._dns_ttl_seconds)
else:
self._dns_provider.delete_a_record(fqdn)
except DnsSyncError as exc:
raise ConflictError(f"DNS sync failed for {entry.name}: {exc}") from exc
def _assert_not_used(self, entry: IpEntry) -> None: def _assert_not_used(self, entry: IpEntry) -> None:
try: try:
usage = self._usage_resolver.resolve_ip_usage({entry.ip}) usage = self._usage_resolver.resolve_ip_usage({entry.ip})
@@ -223,6 +322,14 @@ class EntryService:
if entry.ip == ip and entry.cidr == cidr and entry.device == device: if entry.ip == ip and entry.cidr == cidr and entry.device == device:
raise ConflictError("Entry with same ip/cidr/device already exists") raise ConflictError("Entry with same ip/cidr/device already exists")
def _assert_unique_name(self, entries: list[IpEntry], name: str, ignore_entry_id: str | None = None) -> None:
target = name.strip().lower()
for entry in entries:
if ignore_entry_id and entry.id == ignore_entry_id:
continue
if entry.name.strip().lower() == target:
raise ConflictError("Entry name must be unique")
def _assert_device_exists(self, device: str) -> None: def _assert_device_exists(self, device: str) -> None:
interfaces = self.list_interfaces() interfaces = self.list_interfaces()
if device not in interfaces: if device not in interfaces:
@@ -258,15 +365,15 @@ def _parse_payload(payload: dict) -> dict:
if any(ch.isspace() for ch in device): if any(ch.isspace() for ch in device):
raise ValidationError("Field 'device' cannot contain whitespace") raise ValidationError("Field 'device' cannot contain whitespace")
raw_cidr = payload.get("cidr") cidr_raw = payload.get("cidr")
if raw_cidr is None: if cidr_raw is None:
raise ValidationError("Field 'cidr' is required") raise ValidationError("Field 'cidr' is required")
try: try:
cidr = int(raw_cidr) cidr = int(cidr_raw)
except (TypeError, ValueError) as exc: except (TypeError, ValueError) as exc:
raise ValidationError("Field 'cidr' must be an integer") from exc raise ValidationError("Field 'cidr' must be an integer") from exc
if cidr < 0 or cidr > 32: if cidr < 0 or cidr > 32:
raise ValidationError("Field 'cidr' must be in range 0..32") raise ValidationError("Field 'cidr' must be between 0 and 32")
return { return {
"name": name, "name": name,
@@ -1,2 +1,3 @@
fastapi==0.116.1 fastapi==0.116.1
uvicorn==0.35.0 uvicorn==0.35.0
dnspython==2.7.0
+67 -13
View File
@@ -2,6 +2,7 @@ name: docker-ip-addr-manager
services: services:
app: app:
image: joafri/docker-ip-addr-manager-app:main
build: build:
context: ./backend context: ./backend
dockerfile: Dockerfile dockerfile: Dockerfile
@@ -13,11 +14,25 @@ services:
security_opt: security_opt:
- no-new-privileges:true - no-new-privileges:true
environment: environment:
TZ: ${TZ} TZ: Europe/Stockholm
APP_PORT: ${APP_PORT:-31810} APP_PORT: "31810"
STATE_FILE: ${STATE_FILE:-/data/entries.json} STATE_FILE: /data/entries.json
DOCKER_API_URL: ${DOCKER_API_URL:-unix:///var/run/docker.sock} DOCKER_API_URL: unix:///var/run/docker.sock
DOCKER_TIMEOUT_SECONDS: ${DOCKER_TIMEOUT_SECONDS:-3} DOCKER_TIMEOUT_SECONDS: "3"
DNS_PROVIDER: none
DNS_BASE_DOMAIN: home.arpa
DNS_TTL_SECONDS: "120"
DNS_SYNC_INTERVAL_SECONDS: "15"
ADGUARD_URL: http://127.0.0.1:3000
ADGUARD_USERNAME: ""
ADGUARD_PASSWORD: ""
ADGUARD_API_TOKEN: ""
RFC2136_SERVER: ""
RFC2136_ZONE: ""
RFC2136_PORT: "53"
RFC2136_TSIG_KEY_NAME: ""
RFC2136_TSIG_SECRET: ""
RFC2136_TSIG_ALGORITHM: hmac-sha256
volumes: volumes:
- type: bind - type: bind
source: /DATA/AppData/$AppID/data source: /DATA/AppData/$AppID/data
@@ -37,17 +52,44 @@ services:
- container: DOCKER_TIMEOUT_SECONDS - container: DOCKER_TIMEOUT_SECONDS
description: description:
en_us: Timeout in seconds for Docker API requests en_us: Timeout in seconds for Docker API requests
- container: DNS_PROVIDER
description:
en_us: DNS backend (none, adguard, rfc2136)
- container: DNS_BASE_DOMAIN
description:
en_us: Base domain for generated hostnames like <name>.<domain>
- container: DNS_TTL_SECONDS
description:
en_us: TTL in seconds for DNS A records
- container: DNS_SYNC_INTERVAL_SECONDS
description:
en_us: Background DNS reconcile interval in seconds
- container: ADGUARD_URL
description:
en_us: AdGuard Home URL for DNS_PROVIDER=adguard
- container: ADGUARD_USERNAME
description:
en_us: AdGuard Home username for DNS_PROVIDER=adguard
- container: ADGUARD_PASSWORD
description:
en_us: AdGuard Home password for DNS_PROVIDER=adguard
- container: RFC2136_SERVER
description:
en_us: RFC2136 nameserver host or IP
- container: RFC2136_ZONE
description:
en_us: RFC2136 zone name (for example home.arpa)
volumes: volumes:
- container: /data - container: /data
description: description:
en_us: Persistent IP entry state en_us: Persistent IP entry state
socket-proxy: socket-proxy:
image: lscr.io/linuxserver/socket-proxy:version-24.02.26 image: lscr.io/linuxserver/socket-proxy:version-3.0.9-r0
container_name: docker-ip-addr-manager-socket-proxy container_name: docker-ip-addr-manager-proxy
restart: unless-stopped restart: unless-stopped
environment: environment:
TZ: ${TZ} TZ: Europe/Stockholm
CONTAINERS: 1 CONTAINERS: 1
INFO: 1 INFO: 1
PING: 1 PING: 1
@@ -77,10 +119,10 @@ x-casaos:
- arm64 - arm64
- arm - arm
main: app main: app
category: Network category: phirna
author: Zima Apps Team author: Joachim Friberg
developer: Zima Apps Team developer: Joachim Friberg
icon: https://www.svgrepo.com/show/49710/network.svg icon: https://cdn.simpleicons.org/docker
tagline: tagline:
en_us: Manage host LAN IP aliases for container port bindings en_us: Manage host LAN IP aliases for container port bindings
description: description:
@@ -88,8 +130,20 @@ x-casaos:
Adds/removes host interface IP aliases and shows whether an IP is used by Docker Adds/removes host interface IP aliases and shows whether an IP is used by Docker
container port bindings. Includes fail-closed disable/delete checks when usage cannot container port bindings. Includes fail-closed disable/delete checks when usage cannot
be validated. be validated.
Start by adding a new IP Entry in this app and connect it to the appropriate Device.
Then install, or update, a zima app and choose network: bridge.
* Enter a name for the app, and the app can create DNS records as <name>.<DNS_BASE_DOMAIN> when DNS sync is enabled
* Click add, and a new row appears under.
* Click "Enable" to have this app setup the host to listen to this IP Address
* To to the ZimaOS App Store, choose an app, and do a "Custom Install"
* Add your chosen ip to all fields related to a host port
Such as the ip to the WebUI, and under "Port", theres a "Host" column.
!!! DON'T add the ip-address to the Container column. !!!
Note:
Due to a builtin "convenience" checker in ZimaOS, even if each container gets their own IP, the UI might still complain if two applications use the same port
title: title:
en_us: Docker IP Addr Manager en_us: Docker IP Addr Manager
index: / index: /
port_map: ${APP_PORT:-31810} port_map: "31810"
scheme: http scheme: http
@@ -11,6 +11,7 @@ BACKEND_DIR = ROOT_DIR / "backend"
sys.path.insert(0, str(BACKEND_DIR)) sys.path.insert(0, str(BACKEND_DIR))
from app.docker_api import DockerApiError, DockerUsageResolver from app.docker_api import DockerApiError, DockerUsageResolver
from app.dns_sync import DnsSyncError
from app.models import IpEntry from app.models import IpEntry
from app.service import ConflictError, DependencyError, EntryService from app.service import ConflictError, DependencyError, EntryService
from app.storage import EntryStorage from app.storage import EntryStorage
@@ -75,12 +76,40 @@ class FakeIpManager:
self.present.discard((ip, cidr, device)) self.present.discard((ip, cidr, device))
class FakeDnsProvider:
def __init__(self, fail_upsert=False, fail_delete=False):
self.records = {}
self.upserts = []
self.deletes = []
self.fail_upsert = fail_upsert
self.fail_delete = fail_delete
def upsert_a_record(self, fqdn: str, ip: str, ttl: int):
if self.fail_upsert:
raise DnsSyncError("upsert failed")
self.records[fqdn] = ip
self.upserts.append((fqdn, ip, ttl))
def delete_a_record(self, fqdn: str):
if self.fail_delete:
raise DnsSyncError("delete failed")
self.records.pop(fqdn, None)
self.deletes.append(fqdn)
def assert_true(condition, message): def assert_true(condition, message):
if not condition: if not condition:
raise AssertionError(message) raise AssertionError(message)
def build_service(tmp_path: Path, entries=None, usage_resolver=None, ip_manager=None): def build_service(
tmp_path: Path,
entries=None,
usage_resolver=None,
ip_manager=None,
dns_provider=None,
dns_base_domain="home.arpa",
):
storage = EntryStorage(str(tmp_path / "entries.json")) storage = EntryStorage(str(tmp_path / "entries.json"))
if entries: if entries:
storage.save_entries(entries) storage.save_entries(entries)
@@ -93,6 +122,9 @@ def build_service(tmp_path: Path, entries=None, usage_resolver=None, ip_manager=
usage_resolver=resolver, usage_resolver=resolver,
ip_manager=ipm, ip_manager=ipm,
interface_provider=lambda: ["eth0", "eth1"], interface_provider=lambda: ["eth0", "eth1"],
dns_provider=dns_provider,
dns_base_domain=dns_base_domain,
dns_ttl_seconds=120,
) )
@@ -163,6 +195,79 @@ def test_reconcile_reapplies_enabled(tmp_path: Path):
assert_true(("10.0.4.10", 16, "eth0") in ip_manager.present, "enabled IP must be re-applied on startup reconcile") assert_true(("10.0.4.10", 16, "eth0") in ip_manager.present, "enabled IP must be re-applied on startup reconcile")
def test_dns_upsert_on_enable_when_used(tmp_path: Path):
entry = IpEntry(id="dns1", name="Lan App", ip="10.0.4.20", cidr=16, device="eth0", enabled=False)
resolver = FakeUsageResolver(mapping={"10.0.4.20": {"nginx"}})
ip_manager = FakeIpManager()
dns = FakeDnsProvider()
service = build_service(tmp_path, entries=[entry], usage_resolver=resolver, ip_manager=ip_manager, dns_provider=dns)
service.set_enabled("dns1", enabled=True)
assert_true(dns.records.get("lan-app.home.arpa") == "10.0.4.20", "DNS record should be created on enable+used")
def test_dns_no_upsert_on_enable_when_unused(tmp_path: Path):
entry = IpEntry(id="dns2", name="Lan App 2", ip="10.0.4.21", cidr=16, device="eth0", enabled=False)
resolver = FakeUsageResolver(mapping={})
ip_manager = FakeIpManager()
dns = FakeDnsProvider()
service = build_service(tmp_path, entries=[entry], usage_resolver=resolver, ip_manager=ip_manager, dns_provider=dns)
service.set_enabled("dns2", enabled=True)
assert_true("lan-app-2.home.arpa" not in dns.records, "Unused entries must not create DNS record")
def test_dns_reconcile_deletes_when_no_longer_used(tmp_path: Path):
entry = IpEntry(id="dns3", name="Lan App 3", ip="10.0.4.22", cidr=16, device="eth0", enabled=True)
resolver = FakeUsageResolver(mapping={"10.0.4.22": {"nginx"}})
dns = FakeDnsProvider()
service = build_service(tmp_path, entries=[entry], usage_resolver=resolver, dns_provider=dns)
service.reconcile_dns_records()
assert_true(dns.records.get("lan-app-3.home.arpa") == "10.0.4.22", "record should exist after used reconcile")
resolver._mapping = {}
service.reconcile_dns_records()
assert_true("lan-app-3.home.arpa" not in dns.records, "record should be removed when usage disappears")
def test_dns_fail_closed_rolls_back_enable(tmp_path: Path):
entry = IpEntry(id="dns4", name="Lan App 4", ip="10.0.4.23", cidr=16, device="eth0", enabled=False)
resolver = FakeUsageResolver(mapping={"10.0.4.23": {"nginx"}})
ip_manager = FakeIpManager()
dns = FakeDnsProvider(fail_upsert=True)
service = build_service(tmp_path, entries=[entry], usage_resolver=resolver, ip_manager=ip_manager, dns_provider=dns)
failed = False
try:
service.set_enabled("dns4", enabled=True)
except ConflictError:
failed = True
assert_true(failed, "enable must fail-closed when DNS upsert fails")
current = service.list_entries().items[0]
assert_true(not current.enabled, "entry must roll back to disabled on DNS failure")
assert_true(not ip_manager.is_present("10.0.4.23", 16, "eth0"), "IP presence must roll back on DNS failure")
def test_dns_fail_closed_blocks_delete(tmp_path: Path):
entry = IpEntry(id="dns5", name="Lan App 5", ip="10.0.4.24", cidr=16, device="eth0", enabled=False)
resolver = FakeUsageResolver(mapping={})
dns = FakeDnsProvider(fail_delete=True)
service = build_service(tmp_path, entries=[entry], usage_resolver=resolver, dns_provider=dns)
failed = False
try:
service.delete_entry("dns5")
except ConflictError:
failed = True
assert_true(failed, "delete must fail-closed when DNS cleanup fails")
assert_true(len(service.list_entries().items) == 1, "entry must remain when delete fails")
def main(): def main():
test_exact_hostip_match_only() test_exact_hostip_match_only()
@@ -172,6 +277,11 @@ def main():
test_disable_blocked_when_docker_check_fails(tmp_path) test_disable_blocked_when_docker_check_fails(tmp_path)
test_delete_blocked_when_enabled(tmp_path) test_delete_blocked_when_enabled(tmp_path)
test_reconcile_reapplies_enabled(tmp_path) test_reconcile_reapplies_enabled(tmp_path)
test_dns_upsert_on_enable_when_used(tmp_path)
test_dns_no_upsert_on_enable_when_unused(tmp_path)
test_dns_reconcile_deletes_when_no_longer_used(tmp_path)
test_dns_fail_closed_rolls_back_enable(tmp_path)
test_dns_fail_closed_blocks_delete(tmp_path)
print("Integration tests passed") print("Integration tests passed")
+88
View File
@@ -0,0 +1,88 @@
# Llama Server
Local LLM inference server using llama.cpp. Serves GGUF models via OpenAI-compatible REST API.
**Image**: `ghcr.io/ggml-org/llama.cpp:server-b8840` (CPU-only, AVX2/AVX512)
## Purpose
- **Port**: 8080 (TCP)
- **Memory**: 8G reservation (7B Q4 models fit in ~6-7GB RAM)
- **Category**: AI / LLM inference
CPU-only inference with AVX2/AVX512 auto-detection. No GPU needed.
## Model Setup
llama-server does not bundle models. You must download GGUF files manually.
SSH into your ZimaOS device and run:
```bash
# Create models directory
mkdir -p /DATA/AppData/llama-server/models
# Example: Download Llama 3.2 3B Q4_K_M (~1.8GB)
curl -L -o /DATA/AppData/llama-server/models/llama-3.2-3b-q4_k_m.gguf \
"https://huggingface.co/QuantFactory/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct.Q4_K_M.gguf"
```
## Recommended Models for 16GB RAM
| Model | Size | Quant | RAM Needed | Speed (est.) |
|-------|------|-------|------------|--------------|
| Llama 3.2 3B | 1.8GB | Q4_K_M | ~4GB | ~15-20 tok/s |
| Phi-3.5 Mini 3B | 1.8GB | Q4_K_M | ~4GB | ~15-20 tok/s |
| Mistral 7B | 4.1GB | Q4_K_M | ~6-7GB | ~8-12 tok/s |
| Qwen 2.5 7B | 4.4GB | Q4_K_M | ~6-7GB | ~8-12 tok/s |
For 7B models, close other apps to free RAM. 8G reservation leaves headroom.
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `MODEL` | `llama-3.2-3b-q4_k_m.gguf` | Model filename in `/models` |
| `CTX_SIZE` | `2048` | Context window size (tokens) |
| `N_THREADS` | `0` | CPU threads (0 = auto) |
| `HOST` | `0.0.0.0` | Listen address |
| `PORT` | `8080` | API port |
| `MAX_TOKENS` | `512` | Max tokens per response |
Change `MODEL` to match your downloaded file. Restart container after changing.
## API Testing
Once running, test the API:
```bash
# Check server info
curl http://localhost:8080/v1/models
# Chat completions (OpenAI-compatible)
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "llama-3.2-3b-q4_k_m.gguf",
"messages": [{"role": "user", "content": "Hello, who are you?"}],
"max_tokens": 128
}'
```
## Volumes
| Path | Description |
|------|-------------|
| `/models` | GGUF model files |
| `/logs` | Server log output |
## Architecture
- `amd64` (Intel/AMD x86_64)
- `arm64` (Apple Silicon, ARM servers)
## Security
- `security_opt: no-new-privileges:true`
- `cap_drop: ALL`
- CPU-only, no privileged access needed
+82
View File
@@ -0,0 +1,82 @@
name: llama-server
services:
llama-server:
image: ghcr.io/ggml-org/llama.cpp:server-b8840@sha256:99d2554c4c8d5339649dde530056cf10771823d7cd983dbd0441da9c419976b1
container_name: llama-server
restart: unless-stopped
environment:
TZ: Europe/Stockholm
MODEL: llama-3.2-3b-q4_k_m.gguf
CTX_SIZE: "2048"
N_THREADS: "0"
HOST: 0.0.0.0
PORT: "8080"
MAX_TOKENS: "512"
ports:
- target: 8080
published: "8080"
protocol: tcp
volumes:
- type: bind
source: /DATA/AppData/$AppID/models
target: /models
- type: bind
source: /DATA/AppData/$AppID/logs
target: /logs
deploy:
resources:
reservations:
memory: 8G
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
x-casaos:
envs:
- container: MODEL
description:
en_us: Model filename inside /models (e.g. llama-3.2-3b-q4_k_m.gguf). Download GGUF files manually into /models.
- container: CTX_SIZE
description:
en_us: Context window size in tokens
- container: N_THREADS
description:
en_us: CPU threads (0 = auto-detect all cores)
- container: MAX_TOKENS
description:
en_us: Maximum tokens to generate per response
- container: TZ
description:
en_us: Timezone, for example Europe/Stockholm
ports:
- container: "8080"
description:
en_us: llama.cpp REST API port
volumes:
- container: /models
description:
en_us: Model GGUF files directory
- container: /logs
description:
en_us: Server log output
x-casaos:
architectures:
- amd64
- arm64
main: llama-server
category: phirna
author: Joachim Friberg
developer: Joachim Friberg
icon: https://cdn.simpleicons.org/llama
tagline:
en_us: CPU-only LLM inference server with REST API
description:
en_us: >
Local LLM inference server using llama.cpp. Serves GGUF models via OpenAI-compatible REST API.
CPU-only with AVX2/AVX512 optimization. Requires manual model download.
title:
en_us: Llama Server
index: /
port_map: "8080"
+73
View File
@@ -0,0 +1,73 @@
# OpenWebUI
Modern chat web interface for local LLMs. Connects to llama-server via Docker internal networking.
## Purpose
- **Port**: 3000 (TCP)
- **Memory**: 2G reservation
- **Category**: AI / LLM UI
Requires the **llama-server** app to be running first. Connects to `http://llama-server:8080` internally.
## Prerequisites
1. Deploy and start **llama-server** app first
2. Download a GGUF model into llama-server's `/models` directory
3. Ensure llama-server container is healthy
## Access
Open in browser:
```
http://<your-zimaos-host>:3000
```
First run may take a moment to initialize.
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `OLLAMA_BASE_URL` | `http://llama-server:8080` | Internal URL to llama-server API |
| `WEBUI_PORT` | `3000` | Container listen port |
| `TZ` | `Europe/Stockholm` | Timezone |
## If Connection Fails
1. Verify llama-server is running: `docker ps | grep llama-server`
2. Check llama-server logs: `docker logs llama-server`
3. Ensure llama-server MODEL env matches your downloaded file
4. From ZimaOS shell, test connectivity:
```bash
curl http://llama-server:8080/v1/models
```
## Volumes
| Path | Description |
|------|-------------|
| `/app/backend/data` | OpenWebUI persistent data (chat history, settings) |
## Architecture
- `amd64` (Intel/AMD x86_64)
- `arm64` (Apple Silicon, ARM servers)
## Security
- `security_opt: no-new-privileges:true`
- `cap_drop: ALL`
## Troubleshooting
**"Cannot connect to LLM" error in UI**
- Verify llama-server is running before open-webui
- Check that `OLLAMA_BASE_URL` is set to `http://llama-server:8080`
- Verify model file exists in `/DATA/AppData/llama-server/models/`
**Slow responses**
- 7B models on CPU are limited by single-thread performance
- 3B models recommended for interactive speeds (~15+ tok/s)
- Close other apps to free RAM
+68
View File
@@ -0,0 +1,68 @@
name: open-webui
services:
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
restart: unless-stopped
environment:
TZ: Europe/Stockholm
OLLAMA_BASE_URL: http://llama-server:8080
WEBUI_PORT: "3000"
ports:
- target: 3000
published: "3000"
protocol: tcp
volumes:
- type: bind
source: /DATA/AppData/$AppID/data
target: /app/backend/data
deploy:
resources:
reservations:
memory: 2G
depends_on:
- llama-server
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
x-casaos:
envs:
- container: OLLAMA_BASE_URL
description:
en_us: Internal URL to llama-server API (http://llama-server:8080)
- container: WEBUI_PORT
description:
en_us: Web UI listen port inside container
- container: TZ
description:
en_us: Timezone, for example Europe/Stockholm
ports:
- container: "3000"
description:
en_us: OpenWebUI web interface port
volumes:
- container: /app/backend/data
description:
en_us: OpenWebUI persistent data (chat history, settings)
x-casaos:
architectures:
- amd64
- arm64
main: open-webui
category: phirna
author: Joachim Friberg
developer: Joachim Friberg
icon: https://cdn.simpleicons.org/webui
tagline:
en_us: Modern chat UI for local LLMs
description:
en_us: >
OpenWebUI provides a modern, feature-rich web interface for interacting with local LLMs.
Connect to llama-server or any OpenAI-compatible API. Requires llama-server app to be running first.
title:
en_us: OpenWebUI
index: /
port_map: "3000"
+143
View File
@@ -0,0 +1,143 @@
# Snacks
Automated video library encoder with hardware acceleration (NVENC, QSV, VAAPI, AMF).
## Purpose
Snacks batch-transcodes video libraries using FFmpeg with hardware acceleration.
It monitors directories, skips already-encoded files, retries with fallbacks, and supports distributed cluster encoding across multiple ZimaOS nodes.
## Port
- `6767/tcp` — Web UI at `http://localhost:6767`
## Volumes
| Host path | Container path | Description |
|---|---|---|
| `/DATA/AppData/$AppID/media` | `/app/work/uploads` | Media library — source files to encode |
| `/DATA/AppData/$AppID/logs` | `/app/work/logs` | Transcoding logs |
| `/DATA/AppData/$AppID/config` | `/app/work/config` | Settings and SQLite database |
## Hardware Acceleration
Snacks uses GPU encoding via `/dev/dri`:
| Driver | Codecs | Devices |
|---|---|---|
| VAAPI (Linux) | H.265, H.264 | Intel iHD/i965, AMD VAAPI |
| QSV (Intel) | H.265, H.264 | Intel Quick Sync Video |
| NVENC (NVIDIA) | H.265, H.264 | NVIDIA GPUs via CUDA |
| AMF (AMD) | H.265, H.264 | AMD GPUs |
Auto-detection runs on first encode and picks the best available encoder.
## Cluster Mode
Snacks supports distributed encoding across multiple ZimaOS nodes.
- Nodes discover each other via UDP broadcast on the LAN
- One instance acts as coordinator; others are workers
- Jobs are assigned automatically; failed nodes are re-assigned
- A shared secret authenticates intra-cluster communication
**UDP broadcast requirement**: Cluster mode requires `network_mode: host` — bridge mode blocks LAN broadcast discovery, making nodes invisible to each other.
## Health Check
`http://localhost:6767/Home/Health` — returns HTTP 200 when the backend is ready.
## Privilegier och säkerhet
Aktiva säkerhetsinställningar i denna app:
- `security_opt: ["no-new-privileges:true"]`
- `cap_drop: ["ALL"]`
- `privileged: true`
- `network_mode: host`
- Device mount: `/dev/dri:/dev/dri`
Motivering:
- `no-new-privileges:true` och `cap_drop: ["ALL"]` kompenserar med lägsta möjliga capability-yta.
- Isolerad data-path under `/DATA/AppData/$AppID/...`.
## Säkerhetsavvikelser
### 1. `network_mode: host`
**Varför det behövs:**
- Snacks cluster nodes discover each other via UDP broadcast on the local network.
- Bridge mode only forwards unicast traffic; broadcast packets never reach other nodes.
- Without host networking, cluster mode is non-functional.
**Alternativ som utvärderats:**
- Bridge mode with port exposure: broadcasts are not forwarded by the Docker bridge.
- Static IP configuration: requires manual node addressing and is error-prone.
- Multicast DNS (mDNS): not supported by Docker bridge in all deployments.
**Risker:**
- Container has full access to all host ports.
- No network isolation between Snacks and other services on the host.
- If the container is compromised, the attacker has host network access.
**Riskreducering:**
- `cap_drop: ["ALL"]` minimizes syscall surface.
- `no-new-privileges:true` prevents privilege escalation.
- No sensitive host directories are mounted beyond the app-specific volumes.
---
### 2. `privileged: true`
**Varför det behävs:**
- `/dev/dri` (Direct Rendering Infrastructure) is required for VAAPI/QSV hardware acceleration.
- On standard Linux, this device is accessible without privileged mode if the user is in the `video` or `render` group.
- ZimaOS does not reliably provide these groups in the container runtime context, making `privileged: true` the only reliable way to grant device access.
**Alternativ som utvärderats:**
- `security_opt: ["apparmor:..."]` with specific `/dev/dri` access: not reliably portable across ZimaOS kernel configurations.
- Pre-create device nodes with specific permissions: does not work dynamically when the device appears.
- Skip hardware acceleration (software encoding only): defeats the primary purpose of the app.
**Risker:**
- Container has full root capabilities on the host.
- If container is compromised, attacker has theoretical access to all host resources.
- Hardware acceleration devices can be accessed directly.
**Riskreducering:**
- `cap_drop: ["ALL"]` drops all capabilities even when privileged.
- Only the specific `/dev/dri` device is mounted; no other host devices.
- Data volumes are scoped to `/DATA/AppData/$AppID/...`.
---
### 3. Device mount: `/dev/dri:/dev/dri`
**Varför det behövs:**
- VAAPI and QSV hardware encoding require direct access to the GPU render nodes in `/dev/dri`.
- Without this mount, FFmpeg falls back to software encoding which is 1050x slower on 4K content.
**Alternativ som utvärderats:**
- Specific device nodes (e.g., `/dev/dri/renderD128`): device names can vary by driver version and host kernel.
- No hardware acceleration: software fallback is too slow for practical use.
**Risker:**
- The container can enumerate and use all graphics devices on the host.
- On multi-user systems, other users' GPU resources may be accessible.
**Riskreducering:**
- `privileged: true` combined with `cap_drop: ["ALL"]` ensures the container cannot load additional kernel modules or escalate privileges.
- Only the render nodes are exposed; no other host devices are passed through.
+103
View File
@@ -0,0 +1,103 @@
name: snacks
services:
snacks:
image: derekshreds/snacks-docker:2.3.1
container_name: snacks
restart: unless-stopped
deploy:
resources:
reservations:
memory: 1G
environment:
- TZ=Europe/Stockholm
- PUID=1000
- PGID=1000
- ASPNETCORE_ENVIRONMENT=Production
- SNACKS_WORK_DIR=/app/work
- FFMPEG_PATH=/usr/lib/jellyfin-ffmpeg/ffmpeg
- FFPROBE_PATH=/usr/lib/jellyfin-ffmpeg/ffprobe
network_mode: host
volumes:
- type: bind
source: /DATA/AppData/$AppID/media
target: /app/work/uploads
- type: bind
source: /DATA/AppData/$AppID/logs
target: /app/work/logs
- type: bind
source: /DATA/AppData/$AppID/config
target: /app/work/config
devices:
- /dev/dri:/dev/dri
privileged: true
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:6767/Home/Health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
x-casaos:
envs:
- container: TZ
description:
en_US: Timezone, for example Europe/Stockholm
- container: PUID
description:
en_US: User ID for filesystem permissions
- container: PGID
description:
en_US: Group ID for filesystem permissions
- container: FFMPEG_PATH
description:
en_US: "FFmpeg binary path (default: /usr/lib/jellyfin-ffmpeg/ffmpeg). Use /usr/bin/ffmpeg on systems without jellyfin-ffmpeg."
- container: FFPROBE_PATH
description:
en_US: "FFprobe binary path (default: /usr/lib/jellyfin-ffmpeg/ffprobe). Use /usr/bin/ffprobe on systems without jellyfin-ffmpeg."
ports:
- container: "6767"
description:
en_US: Web UI port
volumes:
- container: /app/work/uploads
description:
en_US: Media library — source files to be encoded
- container: /app/work/logs
description:
en_US: Transcoding logs directory
- container: /app/work/config
description:
en_US: Application configuration and SQLite database
x-casaos:
architectures:
- amd64
main: snacks
category: phirna
author: Joachim Friberg
developer: Joachim Friberg
icon: https://cdn.simpleicons.org/snacks
tagline:
en_US: Automated video library encoder with hardware acceleration
description:
en_US: >-
Batch transcode your video library with hardware acceleration (NVENC, QSV, VAAPI, AMF).
Monitors directories, skips already-encoded files, and supports distributed cluster encoding.
Web UI at http://localhost:6767
title:
en_US: Snacks
index: /
port_map: "6767"
+12 -12
View File
@@ -2,24 +2,24 @@ name: steam-headless
services: services:
steam: steam:
image: lscr.io/linuxserver/steam:version-f4f48542@sha256:d7b9fbf302e05ae79248d1171fe9751b354f8397eafa1e13a3df0aa6a75de0b4 image: lscr.io/linuxserver/steam:version-f4f48542
container_name: steam-headless container_name: steam-headless
restart: unless-stopped restart: unless-stopped
shm_size: "1gb" shm_size: "1gb"
environment: environment:
TZ: ${TZ} TZ: Europe/Stockholm
PUID: ${PUID} PUID: "1000"
PGID: ${PGID} PGID: "1000"
STEAM_HTTP_PORT: ${STEAM_HTTP_PORT:-3000} STEAM_HTTP_PORT: "3000"
STEAM_HTTPS_PORT: ${STEAM_HTTPS_PORT:-3001} STEAM_HTTPS_PORT: "3001"
ports: ports:
- target: 3000 - target: 3000
published: ${STEAM_HTTP_PORT:-3000} published: "3000"
protocol: tcp protocol: tcp
- target: 3001 - target: 3001
published: ${STEAM_HTTPS_PORT:-3001} published: "3001"
protocol: tcp protocol: tcp
volumes: volumes:
@@ -63,9 +63,9 @@ x-casaos:
architectures: architectures:
- amd64 - amd64
main: steam main: steam
category: Games category: phirna
author: Zima Apps Team author: Joachim Friberg
developer: linuxserver.io developer: Joachim Friberg
icon: https://cdn.simpleicons.org/steam icon: https://cdn.simpleicons.org/steam
tagline: tagline:
en_us: Browser-based Steam desktop container for ZimaOS en_us: Browser-based Steam desktop container for ZimaOS
@@ -77,5 +77,5 @@ x-casaos:
title: title:
en_us: Steam Headless en_us: Steam Headless
index: / index: /
port_map: ${STEAM_HTTPS_PORT:-3001} port_map: "3001"
scheme: https scheme: https
+35 -35
View File
@@ -1,23 +1,23 @@
name: steam-moonlight name: steam-moonlight
x-steam-common: &steam-common x-steam-common: &steam-common
image: josh5/steam-headless:debian-0.2.0@sha256:540366bee31297c5679a5006a84dbca039ca62aaab695852b51b5f62dffd2c14 image: josh5/steam-headless:debian-0.2.0
restart: unless-stopped restart: unless-stopped
shm_size: ${SHM_SIZE:-2G} shm_size: "2G"
environment: environment:
TZ: ${TZ} TZ: Europe/Stockholm
PUID: ${PUID} PUID: "1000"
PGID: ${PGID} PGID: "1000"
UMASK: ${UMASK:-000} UMASK: "000"
USER_PASSWORD: ${USER_PASSWORD:-change-me} USER_PASSWORD: change-me
MODE: ${MODE:-primary} MODE: primary
WEB_UI_MODE: ${WEB_UI_MODE:-vnc} WEB_UI_MODE: vnc
PORT_NOVNC_WEB: ${STEAM_WEB_PORT:-8083} PORT_NOVNC_WEB: "8083"
ENABLE_STEAM: ${ENABLE_STEAM:-true} ENABLE_STEAM: "true"
STEAM_ARGS: ${STEAM_ARGS:--silent} STEAM_ARGS: -silent
ENABLE_SUNSHINE: ${ENABLE_SUNSHINE:-false} ENABLE_SUNSHINE: "false"
SUNSHINE_USER: ${SUNSHINE_USER:-admin} SUNSHINE_USER: admin
SUNSHINE_PASS: ${SUNSHINE_PASS:-change-me} SUNSHINE_PASS: change-me
services: services:
steam: steam:
@@ -29,7 +29,7 @@ services:
- ALL - ALL
ports: ports:
- target: 8083 - target: 8083
published: ${STEAM_WEB_PORT:-8083} published: "8083"
protocol: tcp protocol: tcp
volumes: volumes:
- type: bind - type: bind
@@ -83,24 +83,24 @@ services:
devices: devices:
- /dev/fuse - /dev/fuse
- /dev/uinput - /dev/uinput
- ${GPU_CARD_DEVICE:-/dev/dri/card0} - /dev/dri/card0
- ${GPU_RENDER_DEVICE:-/dev/dri/renderD128} - /dev/dri/renderD128
device_cgroup_rules: device_cgroup_rules:
- 'c 13:* rmw' - 'c 13:* rmw'
environment: environment:
TZ: ${TZ} TZ: Europe/Stockholm
PUID: ${PUID} PUID: "1000"
PGID: ${PGID} PGID: "1000"
UMASK: ${UMASK:-000} UMASK: "000"
USER_PASSWORD: ${USER_PASSWORD:-change-me} USER_PASSWORD: change-me
MODE: ${MODE:-primary} MODE: primary
WEB_UI_MODE: ${WEB_UI_MODE:-vnc} WEB_UI_MODE: vnc
PORT_NOVNC_WEB: ${STEAM_WEB_PORT:-8083} PORT_NOVNC_WEB: "8083"
ENABLE_STEAM: ${ENABLE_STEAM:-true} ENABLE_STEAM: "true"
STEAM_ARGS: ${STEAM_ARGS:--silent} STEAM_ARGS: -silent
ENABLE_SUNSHINE: "true" ENABLE_SUNSHINE: "true"
SUNSHINE_USER: ${SUNSHINE_USER:-admin} SUNSHINE_USER: admin
SUNSHINE_PASS: ${SUNSHINE_PASS:-change-me} SUNSHINE_PASS: change-me
volumes: volumes:
- type: bind - type: bind
source: /DATA/AppData/$AppID/moonlight-home source: /DATA/AppData/$AppID/moonlight-home
@@ -113,10 +113,10 @@ x-casaos:
architectures: architectures:
- amd64 - amd64
main: steam main: steam
category: Games category: phirna
author: Zima Apps Team author: Joachim Friberg
developer: Steam-Headless community developer: Joachim Friberg
icon: https://cdn.simpleicons.org/steam icon: https://moonlight-stream.org/images/moonlight.svg
tagline: tagline:
en_us: Steam web desktop with optional Moonlight profile en_us: Steam web desktop with optional Moonlight profile
description: description:
@@ -127,5 +127,5 @@ x-casaos:
title: title:
en_us: Steam Moonlight (Scaffold) en_us: Steam Moonlight (Scaffold)
index: / index: /
port_map: ${STEAM_WEB_PORT:-8083} port_map: "8083"
scheme: http scheme: http
+132
View File
@@ -20,6 +20,10 @@ Skelett för att bygga och underhålla ZimaOS/CasaOS-appar i ett eget appstore-r
├── featured-apps.json ├── featured-apps.json
├── recommend-list.json ├── recommend-list.json
└── scripts/ └── scripts/
├── build-appstore-zip.sh
├── build-and-push-image.sh
├── discover-zima-repos-and-apps.sh
├── map-unraid-images-to-zima-apps.sh
└── validate-appstore.sh └── validate-appstore.sh
``` ```
@@ -46,6 +50,134 @@ Inför release/publicering, kör strikt validering för högrisk-inställningar:
./scripts/validate-appstore.sh --enforce-risk-docs ./scripts/validate-appstore.sh --enforce-risk-docs
``` ```
Bygg och publicera app-specifika custom images (Docker Hub namespace `joafri` som default):
```bash
./scripts/build-and-push-image.sh --app-id caddy-autogen --tag 0.1.0
```
Scriptet läser `Apps/<app-id>/docker-compose.yaml` och bygger alla services som har `build:` om `--component` inte anges.
`--component` kan användas för en enskild service (service-namn eller context-path).
Arkitekturer hämtas från `x-casaos.architectures`; builden kör fail-open per arkitektur och visar varningar i slutet för misslyckade arch-builds.
## Scripts
### `validate-appstore.sh`
Basvalidering:
```bash
./scripts/validate-appstore.sh
```
Strikt riskdokumentationskontroll:
```bash
./scripts/validate-appstore.sh --enforce-risk-docs
```
### `build-and-push-image.sh`
Bygger och pushar appens custom images till Docker Hub (`joafri` som default).
```bash
./scripts/build-and-push-image.sh --app-id caddy-autogen --tag 0.2.0
```
Viktiga flaggor:
- `--component <value>`: bygg endast en service (service-namn eller context-path).
- `--repo <namespace>`: byt namespace (default `joafri`).
- `--no-push`: bygg utan push.
Beteende:
- Läser `Apps/<app-id>/docker-compose.yaml`.
- Bygger alla services med `build:` om `--component` inte anges.
- Läser arkitekturer från `x-casaos.architectures`.
- Bygger per arkitektur och skapar manifest-tag.
- Fail-open per arkitektur (varning i slutet om någon arch misslyckas).
### `build-appstore-zip.sh`
Bygger appstore-zip (inklusive `.yaml` -> `.yml` export i staging).
```bash
./scripts/build-appstore-zip.sh
```
Bygg och publicera `dist/` till `main`:
```bash
./scripts/build-appstore-zip.sh --push
```
Viktiga flaggor:
- `--push`: byter till `main`, `git add dist/`, commit `Updated appstore`, push `origin main`.
- `--strict-images`: fail-closed om någon image i apparna saknas online.
Image-verifiering:
- Scriptet verifierar alla `image:` i `Apps/*/docker-compose.yaml` (skippar `_template`) via registry-manifest.
- Standardläge: fail-open (varningar skrivs ut, zip byggs ändå).
- CI/Gitea-runner (`CI=true` eller `GITEA_ACTIONS=true`): scriptet blir automatiskt strikt och returnerar felkod om någon image saknas.
### `discover-zima-repos-and-apps.sh`
Hämtar appstore-repon från en ZimaOS-host via SSH (auto scan av kända CasaOS/ZimaOS-paths), validerar repo-zippar och skriver en inventerings-YAML.
```bash
./scripts/discover-zima-repos-and-apps.sh \
--host 10.0.1.10 \
--user root \
--out artifacts/zima-repo-app-inventory.yaml
```
Med explicit extra repo-URL (om host-scan missar något):
```bash
./scripts/discover-zima-repos-and-apps.sh \
--host 10.0.1.10 \
--repo-url https://example.org/my-appstore.zip \
--out artifacts/zima-repo-app-inventory.yaml
```
Endast manuella repo-URL:er (utan SSH-scan):
```bash
./scripts/discover-zima-repos-and-apps.sh \
--host 10.0.1.10 \
--skip-host-scan \
--insecure-tls \
--repo-url https://example.org/my-appstore.zip \
--out artifacts/zima-repo-app-inventory.yaml
```
`--insecure-tls` ska bara användas när CA-kedja saknas i den miljö där scriptet körs.
Output innehåller:
- `repositories[]`: repo-URL, källa (host-scan/manual), status och felorsak.
- `apps[]`: `app_id`, `title`, `repo_url` och normaliserade `images`.
### `map-unraid-images-to-zima-apps.sh`
Mappar en lista av Unraid-image-referenser mot inventerings-YAML från discovery-scriptet.
```bash
./scripts/map-unraid-images-to-zima-apps.sh \
--unraid-images /tmp/unraid-images.txt \
--inventory artifacts/zima-repo-app-inventory.yaml \
--out artifacts/unraid-to-zima-map.yaml
```
Output innehåller:
- `mapping[]` per image med `match_status` = `found`, `missing` eller `ambiguous`.
- `matched_apps[]` med träfforsaker (`image_exact`, `image_basename`, `name_alias`).
- `summary` för total/found/missing/ambiguous.
## Säkerhetsriktlinjer ## Säkerhetsriktlinjer
- Undvik privilegierad container, host network och `docker.sock` om det inte är absolut nödvändigt. - Undvik privilegierad container, host network och `docker.sock` om det inte är absolut nödvändigt.
+15
View File
@@ -0,0 +1,15 @@
## Backlog
| # | Done | Name | Source | What | Agent instructions |
|---|---|---|---|---|---|
| 1 | [x] | Snacks | https://github.com/derekshreds/snacks | Automated video library encoder | Branch `snacks/initial/add-video-encoder`; implemented in `Apps/snacks/` |
## Adding a new app
1. Copy `Apps/_template/``Apps/<app-id>/`
2. Set `name` in compose (lowercase + hyphen only)
3. Pin image to explicit version/tag (no `:latest`); verify tag exists in registry
4. Add `x-casaos` metadata (title, description, icon, category, author, port_map)
5. Write `README.md` with purpose, ports, volumes, envs, and risk justifications
6. Validate: `./scripts/validate-appstore.sh`
7. Run final validation before release: `./scripts/validate-appstore.sh --enforce-risk-docs`
+35 -6
View File
@@ -1,8 +1,37 @@
[ [
"Utilities", {
"Developer", "name": "phirna",
"Media", "font": "apps",
"Productivity", "description": "Phirna App Store"
"Network", },
"System" {
"name": "Utilities",
"font": "tools",
"description": ""
},
{
"name": "Developer",
"font": "terminal",
"description": ""
},
{
"name": "Media",
"font": "video",
"description": ""
},
{
"name": "Productivity",
"font": "note",
"description": ""
},
{
"name": "Network",
"font": "globe",
"description": ""
},
{
"name": "System",
"font": "settings",
"description": ""
}
] ]
BIN
View File
Binary file not shown.
+370
View File
@@ -0,0 +1,370 @@
#!/usr/bin/env bash
set -euo pipefail
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
default_repo="joafri"
usage() {
cat <<'USAGE'
Usage:
./scripts/build-and-push-image.sh --app-id <app-id> --tag <tag> [options]
Required:
--app-id <app-id> App folder under Apps/, for example: caddy-autogen
--tag <tag> Docker tag base (must not be latest)
Options:
--component <value> Optional. Service name OR build context path under app.
If omitted, build all services with build contexts.
--repo <namespace> Docker namespace/org (default: joafri)
--skip-non-buildable Exit 0 (with warning) when app has no services with build context
--no-push Build only, no push or manifest creation
-h, --help Show this help
Behavior:
- Reads buildable services from Apps/<app-id>/docker-compose.yaml.
- Reads architectures from x-casaos.architectures in the same compose file.
- Builds each architecture separately and tags as <tag>-<arch>.
- When pushing, creates a manifest tag <tag> from successful arch tags.
- Fail-open per architecture: failed arch builds are reported as warnings.
Examples:
./scripts/build-and-push-image.sh --app-id caddy-autogen --tag 0.2.0
./scripts/build-and-push-image.sh --app-id caddy-autogen --component agent --tag 0.2.0
USAGE
}
app_id=""
component=""
tag=""
repo_ns="$default_repo"
push_image=1
skip_non_buildable=0
while [[ $# -gt 0 ]]; do
case "$1" in
--app-id)
app_id="${2:-}"
shift 2
;;
--component)
component="${2:-}"
shift 2
;;
--tag)
tag="${2:-}"
shift 2
;;
--repo)
repo_ns="${2:-}"
shift 2
;;
--no-push)
push_image=0
shift
;;
--skip-non-buildable)
skip_non_buildable=1
shift
;;
-h|--help)
usage
exit 0
;;
*)
echo "ERROR: unknown argument: $1" >&2
usage
exit 2
;;
esac
done
if [[ -z "$app_id" || -z "$tag" ]]; then
echo "ERROR: --app-id and --tag are required" >&2
usage
exit 2
fi
if [[ "$tag" == "latest" ]]; then
echo "ERROR: tag 'latest' is not allowed by repo policy" >&2
exit 1
fi
if [[ "$app_id" =~ [^a-z0-9-] ]]; then
echo "ERROR: app-id must match [a-z0-9-]+" >&2
exit 1
fi
if [[ "$repo_ns" =~ [^a-zA-Z0-9._-] ]]; then
echo "ERROR: repo namespace contains invalid characters" >&2
exit 1
fi
component="${component#./}"
if [[ -n "$component" && ( "$component" == /* || "$component" == *".."* ) ]]; then
echo "ERROR: --component must be service name or relative subpath under Apps/$app_id" >&2
exit 1
fi
app_dir="$repo_root/Apps/$app_id"
compose_file="$app_dir/docker-compose.yaml"
if [[ ! -d "$app_dir" ]]; then
echo "ERROR: app directory not found: $app_dir" >&2
exit 1
fi
if [[ ! -f "$compose_file" ]]; then
echo "ERROR: missing compose file: $compose_file" >&2
exit 1
fi
if ! command -v docker >/dev/null 2>&1; then
echo "ERROR: docker is required" >&2
exit 1
fi
if ! docker buildx version >/dev/null 2>&1; then
echo "ERROR: docker buildx is required" >&2
exit 1
fi
if ! command -v jq >/dev/null 2>&1; then
echo "ERROR: jq is required" >&2
exit 1
fi
map_arch_to_platform() {
case "$1" in
amd64) echo "linux/amd64" ;;
arm64) echo "linux/arm64" ;;
arm) echo "linux/arm/v7" ;;
*) return 1 ;;
esac
}
read_compose_architectures() {
awk '
/^x-casaos:[[:space:]]*$/ { in_casa=1; next }
in_casa && /^[^[:space:]]/ { in_casa=0 }
in_casa && /^[[:space:]]+architectures:[[:space:]]*$/ { in_arch=1; next }
in_arch {
if ($0 ~ /^[[:space:]]*-[[:space:]]*[A-Za-z0-9_-]+[[:space:]]*$/) {
line=$0
sub(/^[[:space:]]*-[[:space:]]*/, "", line)
sub(/[[:space:]]*$/, "", line)
print line
next
}
if ($0 ~ /^[[:space:]]*$/) next
in_arch=0
}
' "$compose_file"
}
compose_json="$(mktemp)"
cleanup() {
rm -f "$compose_json"
}
trap cleanup EXIT
if [[ -z "${AppID:-}" ]]; then
export AppID="$app_id"
fi
docker compose -f "$compose_file" config --format json > "$compose_json"
mapfile -t all_services < <(
jq -r '.services | to_entries[] | select(.value.build != null) | .key' "$compose_json"
)
if [[ ${#all_services[@]} -eq 0 ]]; then
if [[ "$skip_non_buildable" -eq 1 ]]; then
echo "WARN: no buildable services found in $compose_file (skipped)"
exit 0
fi
echo "ERROR: no buildable services found in $compose_file" >&2
echo "Hint: pass --skip-non-buildable for batch loops over mixed app types." >&2
exit 1
fi
mapfile -t compose_arches < <(read_compose_architectures)
if [[ ${#compose_arches[@]} -eq 0 ]]; then
compose_arches=(amd64)
echo "WARN: no x-casaos.architectures found, defaulting to amd64"
fi
selected_services=()
if [[ -n "$component" ]]; then
for svc in "${all_services[@]}"; do
context_abs="$(jq -r --arg svc "$svc" '.services[$svc].build.context // empty' "$compose_json")"
if [[ -z "$context_abs" ]]; then
continue
fi
context_rel="$context_abs"
if [[ "$context_abs" == "$app_dir" ]]; then
context_rel="."
elif [[ "$context_abs" == "$app_dir"/* ]]; then
context_rel="${context_abs#"$app_dir"/}"
fi
if [[ "$svc" == "$component" || "$context_rel" == "$component" ]]; then
selected_services+=("$svc")
fi
done
if [[ ${#selected_services[@]} -eq 0 ]]; then
echo "ERROR: --component '$component' did not match any build service name or context path" >&2
echo "Build services in this app:" >&2
for svc in "${all_services[@]}"; do
ctx="$(jq -r --arg svc "$svc" '.services[$svc].build.context // empty' "$compose_json")"
if [[ "$ctx" == "$app_dir" ]]; then
ctx_rel="."
elif [[ "$ctx" == "$app_dir"/* ]]; then
ctx_rel="${ctx#"$app_dir"/}"
else
ctx_rel="$ctx"
fi
echo " - $svc (context: $ctx_rel)" >&2
done
exit 1
fi
else
selected_services=("${all_services[@]}")
fi
git_sha="unknown"
if git -C "$repo_root" rev-parse --is-inside-work-tree >/dev/null 2>&1; then
git_sha="$(git -C "$repo_root" rev-parse --short HEAD)"
fi
source_url="unknown"
if git -C "$repo_root" remote get-url origin >/dev/null 2>&1; then
source_url="$(git -C "$repo_root" remote get-url origin)"
fi
echo "App: $app_id"
echo "Compose: $compose_file"
echo "Services: ${selected_services[*]}"
echo "Architectures: ${compose_arches[*]}"
if [[ "$push_image" -eq 1 ]]; then
echo "Mode: build + push + manifest"
else
echo "Mode: build only (--no-push)"
fi
warn_messages=()
total_success=0
for svc in "${selected_services[@]}"; do
context_abs="$(jq -r --arg svc "$svc" '.services[$svc].build.context // empty' "$compose_json")"
dockerfile_rel="$(jq -r --arg svc "$svc" '.services[$svc].build.dockerfile // "Dockerfile"' "$compose_json")"
if [[ -z "$context_abs" ]]; then
warn_messages+=("$svc: missing build context in compose config, skipped")
continue
fi
case "$context_abs" in
"$app_dir"|"$app_dir"/*) ;;
*)
warn_messages+=("$svc: build context outside app dir ($context_abs), skipped")
continue
;;
esac
dockerfile_abs="$context_abs/$dockerfile_rel"
if [[ ! -f "$dockerfile_abs" ]]; then
warn_messages+=("$svc: Dockerfile missing at $dockerfile_abs, skipped")
continue
fi
image_repo="${repo_ns}/${app_id}-${svc}"
success_arch_refs=()
echo
echo "=== Service: $svc ==="
echo "Context: $context_abs"
echo "Dockerfile: $dockerfile_abs"
echo "Image repo: $image_repo"
for arch in "${compose_arches[@]}"; do
if ! platform="$(map_arch_to_platform "$arch")"; then
warn_messages+=("$svc: unsupported architecture '$arch' in x-casaos.architectures")
continue
fi
arch_ref="${image_repo}:${tag}-${arch}"
echo "Building $svc for $arch ($platform) -> $arch_ref"
build_cmd=(
docker buildx build
--platform "$platform"
--file "$dockerfile_abs"
--tag "$arch_ref"
--label "org.opencontainers.image.source=$source_url"
--label "org.opencontainers.image.revision=$git_sha"
)
if [[ "$push_image" -eq 1 ]]; then
build_cmd+=(--push)
else
build_cmd+=(--load)
fi
build_cmd+=("$context_abs")
if "${build_cmd[@]}"; then
success_arch_refs+=("$arch_ref")
total_success=$((total_success + 1))
else
warn_messages+=("$svc: build failed for arch '$arch' ($platform)")
fi
done
if [[ ${#success_arch_refs[@]} -eq 0 ]]; then
warn_messages+=("$svc: no successful architecture builds")
continue
fi
if [[ "$push_image" -eq 1 ]]; then
manifest_ref="${image_repo}:${tag}"
echo "Creating manifest tag: $manifest_ref"
if docker buildx imagetools create -t "$manifest_ref" "${success_arch_refs[@]}"; then
echo "Manifest created: $manifest_ref"
else
warn_messages+=("$svc: failed to create manifest tag ${manifest_ref}")
fi
else
first_ref="${success_arch_refs[0]}"
local_ref="${image_repo}:${tag}"
if docker tag "$first_ref" "$local_ref"; then
echo "Tagged local alias: $local_ref -> $first_ref"
else
warn_messages+=("$svc: failed to tag local alias ${local_ref}")
fi
fi
done
echo
if [[ "$total_success" -eq 0 ]]; then
echo "ERROR: no successful builds were produced" >&2
if [[ ${#warn_messages[@]} -gt 0 ]]; then
echo "Warnings:" >&2
for w in "${warn_messages[@]}"; do
echo " - $w" >&2
done
fi
exit 1
fi
echo "Build completed with $total_success successful arch build(s)."
if [[ ${#warn_messages[@]} -gt 0 ]]; then
echo
echo "Warnings (fail-open):"
for w in "${warn_messages[@]}"; do
echo " - $w"
done
fi
+251
View File
@@ -0,0 +1,251 @@
#!/usr/bin/env bash
set -euo pipefail
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
push_mode=0
strict_image_check=0
positional=()
while [[ $# -gt 0 ]]; do
case "$1" in
--push)
push_mode=1
shift
;;
--strict-images)
strict_image_check=1
shift
;;
-h|--help)
# handled below via usage
positional+=("$1")
shift
;;
*)
positional+=("$1")
shift
;;
esac
done
if [[ ${#positional[@]} -gt 2 ]]; then
echo "ERROR: too many arguments"
exit 2
fi
output_dir="${positional[0]:-$repo_root/dist}"
#zip_name="${positional[1]:-zima-appstore-$(date +%Y%m%d-%H%M%S).zip}"
zip_name="${positional[1]:-phirna-appstore.zip}"
usage() {
cat <<USAGE
Usage: $0 [output_dir] [zip_name]
$0 --push [output_dir] [zip_name]
$0 --strict-images [output_dir] [zip_name]
$0 --push --strict-images [output_dir] [zip_name]
Examples:
$0
$0 ./dist zima-apps-main.zip
$0 --push
$0 --strict-images
USAGE
}
if [[ "${positional[0]:-}" == "-h" || "${positional[0]:-}" == "--help" ]]; then
usage
exit 0
fi
if ! command -v zip >/dev/null 2>&1; then
echo "ERROR: 'zip' command not found"
exit 1
fi
if [[ "${CI:-}" == "true" || "${GITEA_ACTIONS:-}" == "true" ]]; then
strict_image_check=1
fi
required_root_files=(
"category-list.json"
"recommend-list.json"
"featured-apps.json"
)
for file in "${required_root_files[@]}"; do
if [[ ! -f "$repo_root/$file" ]]; then
echo "ERROR: missing required file: $file"
exit 1
fi
done
if [[ ! -d "$repo_root/Apps" ]]; then
echo "ERROR: missing required directory: Apps"
exit 1
fi
extract_images_from_compose() {
local compose_file="$1"
awk '
/^[[:space:]]*image:[[:space:]]*/ {
line=$0
sub(/^[[:space:]]*image:[[:space:]]*/, "", line)
sub(/[[:space:]]+#.*/, "", line)
gsub(/^["'"'"']|["'"'"']$/, "", line)
if (line != "") print line
}
' "$compose_file"
}
verify_images_online() {
local -a images=()
local -a missing=()
local compose_file app_id
while IFS= read -r compose_file; do
app_id="$(basename "$(dirname "$compose_file")")"
if [[ "$app_id" == "_template" ]]; then
continue
fi
while IFS= read -r image_ref; do
[[ -z "$image_ref" ]] && continue
images+=("$image_ref")
done < <(extract_images_from_compose "$compose_file")
done < <(find "$repo_root/Apps" -type f -name 'docker-compose.yaml' | sort)
if [[ ${#images[@]} -eq 0 ]]; then
echo "WARN: no images found to verify"
return 0
fi
mapfile -t images < <(printf '%s\n' "${images[@]}" | sort -u)
if ! command -v docker >/dev/null 2>&1; then
echo "WARN: docker is not installed; skipping online image verification"
if [[ "$strict_image_check" -eq 1 ]]; then
echo "ERROR: strict image verification is enabled (CI/Gitea or --strict-images)"
return 1
fi
return 0
fi
echo "Verifying ${#images[@]} image(s) online..."
for image_ref in "${images[@]}"; do
if docker manifest inspect "$image_ref" >/dev/null 2>&1; then
echo "OK: $image_ref"
else
echo "WARN: image not found or inaccessible: $image_ref"
missing+=("$image_ref")
fi
done
if [[ ${#missing[@]} -gt 0 ]]; then
echo
echo "Image verification warnings:"
for image_ref in "${missing[@]}"; do
echo " - $image_ref"
done
if [[ "$strict_image_check" -eq 1 ]]; then
echo
echo "ERROR: strict image verification is enabled and one or more images are missing."
return 1
fi
fi
}
if ! verify_images_online; then
exit 1
fi
is_git_repo=0
if git -C "$repo_root" rev-parse --is-inside-work-tree >/dev/null 2>&1; then
is_git_repo=1
fi
tmp_dir="$(mktemp -d)"
staging_dir="$tmp_dir/appstore"
cleanup() {
rm -rf "$tmp_dir"
}
trap cleanup EXIT
mkdir -p "$staging_dir/Apps"
copy_file() {
local rel="$1"
mkdir -p "$staging_dir/$(dirname "$rel")"
cp "$repo_root/$rel" "$staging_dir/$rel"
}
if [[ "$is_git_repo" -eq 1 ]]; then
# Build from tracked files only so accidental local files are excluded.
while IFS= read -r rel; do
[[ -z "$rel" ]] && continue
case "$rel" in
Apps/_template/*)
;;
Apps/*|category-list.json|recommend-list.json|featured-apps.json)
if [[ -f "$repo_root/$rel" ]]; then
copy_file "$rel"
fi
;;
esac
done < <(git -C "$repo_root" ls-files)
else
cp -R "$repo_root/Apps" "$staging_dir/"
rm -rf "$staging_dir/Apps/_template"
for file in "${required_root_files[@]}"; do
copy_file "$file"
done
fi
# Export compatibility: source uses .yaml, AppStore tooling often expects .yml
while IFS= read -r compose_yaml; do
compose_yml="${compose_yaml%.yaml}.yml"
cp "$compose_yaml" "$compose_yml"
rm -f "$compose_yaml"
done < <(find "$staging_dir/Apps" -type f -name 'docker-compose.yaml' | sort)
if ! find "$staging_dir/Apps" -type f -name 'docker-compose.yml' | grep -q .; then
echo "ERROR: no docker-compose.yml files found in exported Apps"
exit 1
fi
mkdir -p "$output_dir"
zip_path="$(cd "$output_dir" && pwd)/$zip_name"
rm -f "$zip_path"
(
cd "$staging_dir"
zip -rq "$zip_path" . -x "*.DS_Store" "__MACOSX/*"
)
if command -v shasum >/dev/null 2>&1; then
checksum="$(shasum -a 256 "$zip_path" | awk '{print $1}')"
echo "ZIP created: $zip_path"
echo "SHA256: $checksum"
else
echo "ZIP created: $zip_path"
fi
if [[ "$push_mode" -eq 1 ]]; then
if [[ "$is_git_repo" -ne 1 ]]; then
echo "ERROR: --push requires a git repository"
exit 1
fi
echo "Push mode enabled: switching to main and publishing dist/"
git -C "$repo_root" checkout main
git -C "$repo_root" add dist/
if git -C "$repo_root" diff --cached --quiet; then
echo "No staged changes in dist/. Nothing to commit."
exit 0
fi
git -C "$repo_root" commit -m "Updated appstore"
git -C "$repo_root" push origin main
fi
+467
View File
@@ -0,0 +1,467 @@
#!/usr/bin/env bash
set -euo pipefail
HOST=""
USER_NAME="root"
PORT="22"
IDENTITY=""
OUT_PATH="artifacts/zima-repo-app-inventory.yaml"
SSH_TIMEOUT="12"
HTTP_TIMEOUT="20"
MAX_ZIP_BYTES="$((512 * 1024 * 1024))"
EXTRA_REPO_URLS=()
SKIP_HOST_SCAN=0
INSECURE_TLS=0
usage() {
cat <<USAGE
Usage: $0 --host <host> [options]
Options:
--host <host> ZimaOS host/IP (required)
--user <user> SSH user (default: root)
--port <port> SSH port (default: 22)
--identity <path> SSH private key path
--out <path> Output YAML (default: artifacts/zima-repo-app-inventory.yaml)
--ssh-timeout <seconds> SSH connect timeout (default: 12)
--http-timeout <seconds> HTTP timeout for repo ZIP fetch (default: 20)
--max-zip-bytes <bytes> Max ZIP size to download per repo (default: 536870912)
--repo-url <url> Additional appstore ZIP URL (repeatable)
--skip-host-scan Skip SSH host scan and only use --repo-url values
--insecure-tls Disable TLS cert verification for ZIP downloads (use only if needed)
-h, --help Show this help
Behavior:
1. SSH to host and scan likely CasaOS/ZimaOS config paths for appstore ZIP URLs.
2. Validate each URL fail-closed (reachable ZIP + parseable app structure).
3. Write repository and app inventory to YAML.
USAGE
}
while [[ $# -gt 0 ]]; do
case "$1" in
--host)
HOST="${2:-}"
shift 2
;;
--user)
USER_NAME="${2:-}"
shift 2
;;
--port)
PORT="${2:-}"
shift 2
;;
--identity)
IDENTITY="${2:-}"
shift 2
;;
--out)
OUT_PATH="${2:-}"
shift 2
;;
--ssh-timeout)
SSH_TIMEOUT="${2:-}"
shift 2
;;
--http-timeout)
HTTP_TIMEOUT="${2:-}"
shift 2
;;
--max-zip-bytes)
MAX_ZIP_BYTES="${2:-}"
shift 2
;;
--repo-url)
EXTRA_REPO_URLS+=("${2:-}")
shift 2
;;
--skip-host-scan)
SKIP_HOST_SCAN=1
shift
;;
--insecure-tls)
INSECURE_TLS=1
shift
;;
-h|--help)
usage
exit 0
;;
*)
echo "ERROR: Unknown argument: $1" >&2
usage
exit 2
;;
esac
done
if [[ -z "$HOST" ]]; then
echo "ERROR: --host is required" >&2
usage
exit 2
fi
if ! [[ "$PORT" =~ ^[0-9]+$ ]]; then
echo "ERROR: --port must be numeric" >&2
exit 2
fi
if ! [[ "$SSH_TIMEOUT" =~ ^[0-9]+$ && "$HTTP_TIMEOUT" =~ ^[0-9]+$ && "$MAX_ZIP_BYTES" =~ ^[0-9]+$ ]]; then
echo "ERROR: timeout/size values must be numeric" >&2
exit 2
fi
if [[ -n "$IDENTITY" && ! -f "$IDENTITY" ]]; then
echo "ERROR: identity file not found: $IDENTITY" >&2
exit 2
fi
tmp_dir="$(mktemp -d)"
cleanup() {
rm -rf "$tmp_dir"
}
trap cleanup EXIT
raw_scan_tsv="$tmp_dir/raw-scan.tsv"
candidate_tsv="$tmp_dir/candidates.tsv"
manual_tsv="$tmp_dir/manual.tsv"
run_remote_scan() {
local -a ssh_cmd
ssh_cmd=(ssh -o BatchMode=yes -o ConnectTimeout="$SSH_TIMEOUT" -p "$PORT")
if [[ -n "$IDENTITY" ]]; then
ssh_cmd+=(-i "$IDENTITY")
fi
ssh_cmd+=("${USER_NAME}@${HOST}" "bash -s")
"${ssh_cmd[@]}" <<'REMOTE'
set -euo pipefail
roots=(
/etc/casaos
/var/lib/casaos
/usr/local/etc/casaos
/etc/zimaos
/var/lib/zimaos
/DATA/AppData
/DATA/.casaos
)
scan_roots=()
for path in "${roots[@]}"; do
if [[ -d "$path" ]]; then
scan_roots+=("$path")
fi
done
if [[ "${#scan_roots[@]}" -eq 0 ]]; then
exit 0
fi
grep -ERHIno --binary-files=without-match \
--include='*.yaml' --include='*.yml' \
--include='*.json' --include='*.conf' --include='*.ini' --include='*.txt' \
'https?://[^"[:space:]]+\.zip(\?[^"[:space:]]*)?' "${scan_roots[@]}" 2>/dev/null \
| awk -F: '{
file=$1
line=$2
$1=""
$2=""
sub(/^::?/, "", $0)
sub(/^:/, "", $0)
printf "%s\t%s\t%s\n", file, line, $0
}'
REMOTE
}
echo "[1/3] Scanning ZimaOS host for configured appstore repositories..."
if [[ "$SKIP_HOST_SCAN" -eq 1 ]]; then
echo "INFO: --skip-host-scan enabled; skipping SSH scan." >&2
: > "$raw_scan_tsv"
else
if ! run_remote_scan > "$raw_scan_tsv"; then
if [[ "${#EXTRA_REPO_URLS[@]}" -gt 0 ]]; then
echo "WARN: SSH host scan failed; continuing with --repo-url entries only." >&2
: > "$raw_scan_tsv"
else
echo "ERROR: SSH host scan failed and no --repo-url fallback provided." >&2
exit 1
fi
fi
fi
if [[ -s "$raw_scan_tsv" ]]; then
awk -F $'\t' 'NF>=3 && !seen[$3]++ { print $1 "\t" $2 "\t" $3 }' "$raw_scan_tsv" > "$candidate_tsv"
fi
: > "$manual_tsv"
if [[ "${#EXTRA_REPO_URLS[@]}" -gt 0 ]]; then
for url in "${EXTRA_REPO_URLS[@]}"; do
[[ -z "$url" ]] && continue
printf "%s\t%s\t%s\n" "manual-input" "0" "$url" >> "$manual_tsv"
done
fi
candidate_count=0
manual_count=0
if [[ -s "$candidate_tsv" ]]; then
candidate_count="$(wc -l < "$candidate_tsv" | tr -d ' ')"
fi
if [[ -s "$manual_tsv" ]]; then
manual_count="$(wc -l < "$manual_tsv" | tr -d ' ')"
fi
if [[ "$candidate_count" -eq 0 && "$manual_count" -eq 0 ]]; then
echo "ERROR: No appstore ZIP URLs discovered on host and no --repo-url provided." >&2
echo "Hint: pass one or more --repo-url <https://...zip> if host scan misses your setup." >&2
exit 1
fi
if [[ "$INSECURE_TLS" -eq 1 ]]; then
echo "WARN: --insecure-tls enabled; certificate verification is disabled for ZIP downloads." >&2
fi
echo "[2/3] Validating repositories and extracting app inventory..."
mkdir -p "$(dirname "$OUT_PATH")"
CANDIDATE_TSV="$candidate_tsv" \
MANUAL_TSV="$manual_tsv" \
OUT_PATH="$OUT_PATH" \
HOST="$HOST" \
HTTP_TIMEOUT="$HTTP_TIMEOUT" \
MAX_ZIP_BYTES="$MAX_ZIP_BYTES" \
INSECURE_TLS="$INSECURE_TLS" \
python3 - <<'PY'
import datetime as dt
import io
import os
import re
import ssl
import sys
import urllib.request
import zipfile
from pathlib import PurePosixPath
import yaml
candidate_tsv = os.environ["CANDIDATE_TSV"]
manual_tsv = os.environ["MANUAL_TSV"]
out_path = os.environ["OUT_PATH"]
host = os.environ["HOST"]
http_timeout = int(os.environ["HTTP_TIMEOUT"])
max_zip_bytes = int(os.environ["MAX_ZIP_BYTES"])
insecure_tls = os.environ["INSECURE_TLS"] == "1"
def normalize_image_ref(ref: str) -> str:
value = (ref or "").strip().strip('"\'')
if not value:
return ""
if "@" in value:
value = value.split("@", 1)[0]
if "/" in value:
prefix, tail = value.rsplit("/", 1)
if ":" in tail:
tail = tail.split(":", 1)[0]
value = f"{prefix}/{tail}"
else:
if ":" in value:
value = value.split(":", 1)[0]
return value.lower()
def read_sources(path: str, source_type: str):
results = []
if not os.path.exists(path):
return results
with open(path, "r", encoding="utf-8") as handle:
for line in handle:
line = line.rstrip("\n")
if not line:
continue
parts = line.split("\t")
if len(parts) < 3:
continue
evidence_file, evidence_line, url = parts[0], parts[1], parts[2]
url = url.strip()
if not url:
continue
results.append(
{
"url": url,
"source": {
"type": source_type,
"evidence_file": evidence_file,
"evidence_line": int(evidence_line) if evidence_line.isdigit() else evidence_line,
},
}
)
return results
source_items = read_sources(candidate_tsv, "host_scan") + read_sources(manual_tsv, "manual_input")
if not source_items:
print("ERROR: no source items found", file=sys.stderr)
sys.exit(1)
# Keep first evidence for each URL.
repo_sources = {}
for item in source_items:
repo_sources.setdefault(item["url"], item["source"])
def fetch_zip(url: str) -> bytes:
request = urllib.request.Request(
url,
headers={"User-Agent": "zima-repo-discovery/1.0", "Accept": "application/zip,application/octet-stream,*/*"},
)
context = ssl._create_unverified_context() if insecure_tls else None
with urllib.request.urlopen(request, timeout=http_timeout, context=context) as response:
status = getattr(response, "status", 200)
if status >= 400:
raise RuntimeError(f"HTTP {status}")
payload = response.read(max_zip_bytes + 1)
if len(payload) > max_zip_bytes:
raise RuntimeError(f"ZIP larger than allowed limit ({max_zip_bytes} bytes)")
return payload
def extract_title(compose_data):
if not isinstance(compose_data, dict):
return ""
x_casaos = compose_data.get("x-casaos")
if not isinstance(x_casaos, dict):
return ""
title = x_casaos.get("title")
if isinstance(title, str):
return title.strip()
if isinstance(title, dict):
for key in ("en_US", "en_us", "en", "sv_SE", "sv_se"):
value = title.get(key)
if isinstance(value, str) and value.strip():
return value.strip()
for value in title.values():
if isinstance(value, str) and value.strip():
return value.strip()
return ""
def extract_images(compose_data):
images = []
if not isinstance(compose_data, dict):
return images
services = compose_data.get("services")
if not isinstance(services, dict):
return images
for svc in services.values():
if not isinstance(svc, dict):
continue
image = svc.get("image")
if isinstance(image, str) and image.strip():
images.append(image.strip())
dedup = []
seen = set()
for image in images:
norm = normalize_image_ref(image)
if norm and norm not in seen:
seen.add(norm)
dedup.append(norm)
return dedup
all_repo_entries = []
app_index = {}
for repo_url, source in sorted(repo_sources.items(), key=lambda t: t[0].lower()):
entry = {
"url": repo_url,
"source": source,
"status": "error",
"error": "",
"app_count": 0,
"app_ids": [],
}
try:
payload = fetch_zip(repo_url)
with zipfile.ZipFile(io.BytesIO(payload)) as archive:
app_compose = {}
for member in archive.namelist():
member_lower = member.lower()
if not (member_lower.endswith("docker-compose.yml") or member_lower.endswith("docker-compose.yaml")):
continue
path = PurePosixPath(member)
parts = path.parts
if "Apps" not in parts:
continue
idx = parts.index("Apps")
if idx + 2 >= len(parts):
continue
app_id = parts[idx + 1]
if not app_id or app_id == "_template":
continue
app_compose.setdefault(app_id, member)
if not app_compose:
raise RuntimeError("ZIP contains no Apps/*/docker-compose.yml|yaml")
app_ids = sorted(app_compose.keys())
entry["status"] = "ok"
entry["error"] = ""
entry["app_count"] = len(app_ids)
entry["app_ids"] = app_ids
for app_id, compose_member in sorted(app_compose.items(), key=lambda t: t[0].lower()):
title = ""
images = []
try:
raw = archive.read(compose_member)
parsed = yaml.safe_load(raw.decode("utf-8", errors="replace"))
title = extract_title(parsed)
images = extract_images(parsed)
except Exception:
# Keep inventory resilient; compose parse failures should not drop app id.
title = ""
images = []
app_key = (repo_url, app_id)
app_index[app_key] = {
"app_id": app_id,
"title": title,
"repo_url": repo_url,
"images": images,
}
except Exception as exc:
entry["status"] = "error"
entry["error"] = str(exc)
all_repo_entries.append(entry)
apps = [app_index[key] for key in sorted(app_index.keys(), key=lambda t: (t[0].lower(), t[1].lower()))]
inventory = {
"schema_version": 1,
"generated_at": dt.datetime.now(dt.timezone.utc).isoformat(),
"host": host,
"repositories": all_repo_entries,
"apps": apps,
}
with open(out_path, "w", encoding="utf-8") as handle:
yaml.safe_dump(inventory, handle, sort_keys=False, allow_unicode=False)
ok_count = sum(1 for repo in all_repo_entries if repo["status"] == "ok")
err_count = len(all_repo_entries) - ok_count
print(f"Wrote {out_path}")
print(f"Repositories: total={len(all_repo_entries)} ok={ok_count} error={err_count}")
print(f"Apps discovered: {len(apps)}")
if ok_count == 0:
sys.exit(1)
PY
echo "[3/3] Done. Inventory written to: $OUT_PATH"
+331
View File
@@ -0,0 +1,331 @@
#!/usr/bin/env bash
set -euo pipefail
UNRAID_IMAGES=""
INVENTORY_PATH=""
OUT_PATH="artifacts/unraid-to-zima-map.yaml"
usage() {
cat <<USAGE
Usage: $0 --unraid-images <file> --inventory <yaml> [--out <yaml>]
Options:
--unraid-images <file> Text file with one image reference per line (required)
--inventory <yaml> Inventory YAML from discover-zima-repos-and-apps.sh (required)
--out <yaml> Output YAML (default: artifacts/unraid-to-zima-map.yaml)
-h, --help Show this help
USAGE
}
while [[ $# -gt 0 ]]; do
case "$1" in
--unraid-images)
UNRAID_IMAGES="${2:-}"
shift 2
;;
--inventory)
INVENTORY_PATH="${2:-}"
shift 2
;;
--out)
OUT_PATH="${2:-}"
shift 2
;;
-h|--help)
usage
exit 0
;;
*)
echo "ERROR: Unknown argument: $1" >&2
usage
exit 2
;;
esac
done
if [[ -z "$UNRAID_IMAGES" || -z "$INVENTORY_PATH" ]]; then
echo "ERROR: --unraid-images and --inventory are required" >&2
usage
exit 2
fi
if [[ ! -f "$UNRAID_IMAGES" ]]; then
echo "ERROR: unraid image file not found: $UNRAID_IMAGES" >&2
exit 2
fi
if [[ ! -f "$INVENTORY_PATH" ]]; then
echo "ERROR: inventory file not found: $INVENTORY_PATH" >&2
exit 2
fi
mkdir -p "$(dirname "$OUT_PATH")"
UNRAID_IMAGES="$UNRAID_IMAGES" \
INVENTORY_PATH="$INVENTORY_PATH" \
OUT_PATH="$OUT_PATH" \
python3 - <<'PY'
import datetime as dt
import os
import re
import sys
import yaml
unraid_images_path = os.environ["UNRAID_IMAGES"]
inventory_path = os.environ["INVENTORY_PATH"]
out_path = os.environ["OUT_PATH"]
ALIASES = {
"arch-sonarr": "sonarr",
"arch-radarr": "radarr",
"arch-prowlarr": "prowlarr",
"arch-overseerr": "overseerr",
"arch-flaresolverr": "flaresolverr",
"arch-qbittorrentvpn": "qbittorrent",
"arch-plexpass": "plex",
"open-webui": "openwebui",
"open_webui": "openwebui",
"act_runner": "act-runner",
"socket-proxy": "socketproxy",
"postgres": "postgresql",
"postgresql": "postgres",
}
GENERIC_BASE_IMAGES = {
"redis",
"postgres",
"postgresql",
"mariadb",
"mysql",
"mongo",
"mongodb",
"nginx",
"memcached",
"rabbitmq",
"valkey",
}
def normalize_text(value: str) -> str:
value = (value or "").strip().lower()
return re.sub(r"[^a-z0-9]+", "", value)
def normalize_image_ref(ref: str) -> str:
value = (ref or "").strip().strip('"\'')
if not value:
return ""
if "@" in value:
value = value.split("@", 1)[0]
if "/" in value:
prefix, tail = value.rsplit("/", 1)
if ":" in tail:
tail = tail.split(":", 1)[0]
value = f"{prefix}/{tail}"
else:
if ":" in value:
value = value.split(":", 1)[0]
return value.lower()
def basename_image(normalized_ref: str) -> str:
if not normalized_ref:
return ""
return normalized_ref.rsplit("/", 1)[-1]
def candidate_names_from_unraid_image(image_ref: str):
normalized = normalize_image_ref(image_ref)
base = basename_image(normalized)
candidates = set()
if base:
candidates.add(base)
candidates.add(base.replace("_", "-"))
if base.startswith("arch-"):
candidates.add(base[5:])
if base in ALIASES:
candidates.add(ALIASES[base])
replaced = base.replace("_", "-")
if replaced in ALIASES:
candidates.add(ALIASES[replaced])
return {c for c in candidates if c}
with open(inventory_path, "r", encoding="utf-8") as handle:
inventory = yaml.safe_load(handle) or {}
apps = inventory.get("apps") or []
if not isinstance(apps, list):
print("ERROR: inventory apps must be a list", file=sys.stderr)
sys.exit(1)
app_records = []
for app in apps:
if not isinstance(app, dict):
continue
app_id = str(app.get("app_id") or "").strip()
if not app_id:
continue
title = str(app.get("title") or "").strip()
repo_url = str(app.get("repo_url") or "").strip()
images = app.get("images") if isinstance(app.get("images"), list) else []
normalized_images = []
seen_images = set()
for image in images:
if not isinstance(image, str):
continue
norm = normalize_image_ref(image)
if not norm or norm in seen_images:
continue
seen_images.add(norm)
normalized_images.append(norm)
app_records.append(
{
"app_id": app_id,
"title": title,
"repo_url": repo_url,
"images": normalized_images,
"id_key": normalize_text(app_id),
"title_key": normalize_text(title),
}
)
index_exact = {}
index_basename = {}
for app in app_records:
for image in app["images"]:
index_exact.setdefault(image, []).append(app)
bname = basename_image(image)
if bname:
index_basename.setdefault(bname, []).append(app)
def app_name_matches(app, candidate_base: str) -> bool:
key = normalize_text(candidate_base)
if not key:
return False
candidate_keys = {key}
alias = ALIASES.get(candidate_base)
if alias:
candidate_keys.add(normalize_text(alias))
for alias_source, alias_target in ALIASES.items():
if alias_target == candidate_base:
candidate_keys.add(normalize_text(alias_source))
return (app["id_key"] in candidate_keys) or (app["title_key"] and app["title_key"] in candidate_keys)
with open(unraid_images_path, "r", encoding="utf-8") as handle:
unraid_images = [line.strip() for line in handle if line.strip() and not line.strip().startswith("#")]
mapping = []
for raw_image in unraid_images:
normalized = normalize_image_ref(raw_image)
base = basename_image(normalized)
matched = {}
# Strong match: exact normalized image reference from app compose.
# For generic bases (redis/postgres/etc), only accept exact matches when app name matches.
for app in index_exact.get(normalized, []):
if base in GENERIC_BASE_IMAGES and not app_name_matches(app, base):
continue
key = (app["repo_url"], app["app_id"])
matched.setdefault(
key,
{
"app_id": app["app_id"],
"title": app["title"],
"repo_url": app["repo_url"],
"reasons": set(),
},
)["reasons"].add("image_exact")
# Medium match: basename image.
# Restrict generic/shared images to avoid false positives from sidecar dependencies.
if base:
for app in index_basename.get(base, []):
if base in GENERIC_BASE_IMAGES and not app_name_matches(app, base):
continue
if base not in GENERIC_BASE_IMAGES and not app_name_matches(app, base):
continue
key = (app["repo_url"], app["app_id"])
matched.setdefault(
key,
{
"app_id": app["app_id"],
"title": app["title"],
"repo_url": app["repo_url"],
"reasons": set(),
},
)["reasons"].add("image_basename")
# Fallback match: inferred app name from image aliases.
for candidate_name in candidate_names_from_unraid_image(raw_image):
candidate_key = normalize_text(candidate_name)
if not candidate_key:
continue
for app in app_records:
if candidate_key == app["id_key"] or (app["title_key"] and candidate_key == app["title_key"]):
key = (app["repo_url"], app["app_id"])
matched.setdefault(
key,
{
"app_id": app["app_id"],
"title": app["title"],
"repo_url": app["repo_url"],
"reasons": set(),
},
)["reasons"].add("name_alias")
matched_apps = []
for key in sorted(matched.keys(), key=lambda item: (item[0].lower(), item[1].lower())):
info = matched[key]
matched_apps.append(
{
"app_id": info["app_id"],
"title": info["title"],
"repo_url": info["repo_url"],
"reasons": sorted(info["reasons"]),
}
)
if not matched_apps:
status = "missing"
elif len(matched_apps) == 1:
status = "found"
else:
status = "ambiguous"
mapping.append(
{
"unraid_image": raw_image,
"normalized_image": normalized,
"match_status": status,
"matched_apps": matched_apps,
}
)
summary = {
"total": len(mapping),
"found": sum(1 for item in mapping if item["match_status"] == "found"),
"missing": sum(1 for item in mapping if item["match_status"] == "missing"),
"ambiguous": sum(1 for item in mapping if item["match_status"] == "ambiguous"),
}
output = {
"schema_version": 1,
"generated_at": dt.datetime.now(dt.timezone.utc).isoformat(),
"inventory_source": inventory_path,
"summary": summary,
"mapping": mapping,
}
with open(out_path, "w", encoding="utf-8") as handle:
yaml.safe_dump(output, handle, sort_keys=False, allow_unicode=False)
print(f"Wrote {out_path}")
print(f"Summary: total={summary['total']} found={summary['found']} missing={summary['missing']} ambiguous={summary['ambiguous']}")
PY