# Civic Server Workshop — Build Guide

## 1. Decide what must stay local

List the website, database, source records, camera streams, AI tasks, retention periods and who may administer each service. Separate public information from private credentials, raw camera footage and protected records.

## 2. Pick a build

**CivicNode Micro** is a compact primary or departmental node. Use the 128 GB memory configuration when 30B–70B-class quantized models are part of the plan. Add a second unit or another host for failover.

**PerfectWorld Pro Rack** separates fast NVMe work from high-capacity storage and adds ECC, IPMI and accelerator slots. It is still one failure domain; replication is part of the design.

## 3. Storage layout

Suggested Micro layout:

- `rpool`: mirrored 4 TB NVMe for operating system, containers, database, search and current models
- `archive`: mirrored 24 TB HDD for records and selected video
- encrypted off-site backup: a second physical location or removable rotation

Suggested Pro layout:

- `fast`: mirrored enterprise NVMe for VMs, databases and indexes
- `archive`: eight-disk RAIDZ2 for records/video
- separate backup target: second node or off-site repository

Mirrors and RAIDZ protect availability from selected disk failures; they do not protect against deletion, malware, fire or operator error.

## 4. Network zones

Create separate VLANs or firewall zones for public services, administration, cameras/IoT and backups. Cameras should not have unrestricted internet access. Keep BMC/IPMI and the ESP32 monitor off the public web.

## 5. Install the host

Use a supported Linux distribution. For the simplest AMD GPU path, use Ubuntu Server LTS on bare metal. For service separation, use Proxmox VE and pass the selected GPU(s) into a dedicated Linux VM after testing reset and recovery behavior.

## 6. Deploy services

Start with the included Compose templates. Replace every example password, pin image versions after testing, configure TLS, and add health checks/backups before exposing services publicly.

## 7. Local AI

Start with a small GGUF model in llama.cpp. Measure:

- cold-load time
- tokens per second at the desired context
- memory used per concurrent request
- retrieval quality on the real corpus
- failure behavior when the model service is unavailable

Use a larger model only after the smaller one fails a defined quality test. Two GPUs require an inference engine and model that support multi-GPU partitioning.

## 8. Video processing

Route streams through MediaMTX, sample only the frames needed for analytics, and separate live viewing from inference. Continuous 4 Mbps recording creates about 43.2 GB per feed per day; motion-only retention can be much lower. Validate legal authority, public notice and retention rules before collection.

## 9. Burn-in and acceptance

- 24–72 hour memory test
- full storage SMART test and ZFS scrub
- CPU/GPU thermal load while all disks are active
- pull one disk from each redundant pool in a controlled test
- UPS transfer and automated shutdown test
- restore database and a sample archive onto another host
- security scan from the public and administration networks

## 10. Operations

Document version updates, failed jobs, model changes, retention deletions, backups and restores. Quarterly, restore a copy and prove that an operator other than the original builder can follow the runbook.