Operations

Namespace Hilms\Ops. Everything that makes the application runnable rather than featureful: queues, the schedule, monitoring, health, backups, the audit trail, mail configuration, where media is stored, the installer and the load dataset.

ops sits at the top of the module direction: it may import every module, and nothing imports it. That is why an audited model declares getActivitylogOptions() beside its own concerns, in its own module, and never here.

Queues

The connection is failover (QUEUE_CONNECTION): Redis, and while Redis cannot take a job, the deferred connection, which runs it right after the response. The driver is ours, Hilms\Ops\Queue\CommittedFailoverQueue: the redis connection keeps after_commit on, and Laravel’s own failover takes that promise to push at the commit for a push that succeeded — so when Redis refused it at the commit, the job was simply gone. Ours waits for the transaction itself, then picks a connection. retry_after comes from REDIS_QUEUE_RETRY_AFTER (330), which must stay above the longest timeout of the supervisors on that connection.

A job that may only run on a worker names a connection that never falls back. worker (Redis, or sync where the queue is in-process) carries an assistant’s request — #[Connection('worker')] on RunGeneration, because it is a paid call of minutes, often held for the night, and must not run inside somebody’s page request — and the queue health check’s heartbeat, routed with Queue::route(), because a heartbeat that ran in-process would prove a dead queue alive. Media moves have media-moves, with retry_after 3720 above their hour, and follow sync when the installation runs its queue in-process. The callers turn a refused dispatch into a message: the generation row records why, the move starts nothing. Failed jobs go to failed_jobs with database uuids.

Horizon 5 supervises three queues in every environment:

Queue	Balance	Processes	Tries	Backoff	Timeout	Memory	Long wait
default	auto	1–4	3	10/60/300 s	90 s	256 MB	60 s
media	—	1 (2 in production)	2	30/300 s	300 s	512 MB	300 s
media-moves	—	exactly 1	3	—	3600 s	256 MB	none

What is queued: the three mail notifications (SetUpAccount, EnrolledInCourse, CourseCompleted) and Hilms\Ai\Jobs\RunGeneration on default; image conversions on media; and Hilms\Library\Jobs\RelocateMedium, one per file moved between disks, on media-moves, where one process takes them one at a time, so two moves never overlap, and a large video has an hour. Horizon starts only the supervisors an environment lists, so a new one goes into environments.production and environments.local as well as defaults.

The master supervisor may use 128 MB (horizon.memory_limit). It boots the whole application, panel included, which takes about 66 MB: at Horizon’s default of 64 it restarted itself every few seconds — some 540 times an hour on a developer’s machine — and took its workers down with it each time, so the queue barely ran while nothing looked wrong. The production image boots leaner, which is why it never showed there. Every job of ours carries #[WithoutRelations] and #[DeleteWhenMissingModels], so a worker never restores a stale graph and a job whose model disappeared is dropped rather than retried. The suite keeps QUEUE_CONNECTION=sync.

The dashboard is at /horizon, behind web, auth and the View:Horizon permission in every environment, not only outside local. Hilms\Ops\Providers\HorizonServiceProvider defines that gate and routes Horizon’s own notifications to OPS_NOTIFICATION_EMAIL.

When Redis is away

Redis holds the cache, the queue and Horizon, and nothing a visitor needs to be served waits on it. That was not always so: with Redis stopped, the application died while booting — the panel translates its labels as it registers, the phrase overrides were read through the cache, and only a database error was caught — so every page, /up and every console command, artisan down included, failed before it began. Each piece below closed one way of dying, and each was reproduced by a test that failed first.

• Sessions live in the database (SESSION_DRIVER=database), so nobody is signed out while Redis is away and a full Redis no longer refuses every session write. A session in a database backup expired long before anybody could read it, and its cookie cannot be forged without APP_KEY, which no backup holds. It costs two queries a request, a read and an update of a few milliseconds (Quality has the measurement).
• The cache fails over (CACHE_STORE=failover): Redis, then the database store, through Hilms\Ops\Cache\TrippingFailoverStore, registered before anything builds a store. A store that fails is passed over for thirty seconds by every process on the machine, through a file under cache.stores.failover.trips (StoreTrips), and asked again after that; a store is never passed over when all of them are tripped. Without the trip a page waited for some twenty failed connections, and — worse — a full Redis answers reads while refusing writes, so a rate limit counted its attempts in the database and read them back from Redis as zero: the login limit failed open. REDIS_TIMEOUT (2 seconds) bounds a connect that would otherwise wait PHP’s minute.
• The two stores are settled once Redis is back. A forget reaches only the first store that answers, so after an outage the database store holds what was cached during it, and Redis — persisted by its append-only file — what was cached before it and forgotten only in the database: a role revoked during the outage would have come back with Redis’s copy of the permissions. hilms:cache:settle (Actions\SettleCache, every minute) empties both stores and lifts Redis’s trip as soon as Redis takes a write again (a full Redis still answers reads) while the database store holds anything.
• What the boot reads asks the table when the cache throws anything at all — the phrase overrides and the languages — so neither a request nor a console command can die before it begins (Languages).
• The queue falls back as above; Horizon exits while Redis is away and its supervisor (restart: unless-stopped, or Lando’s loop) brings it back. The site needs no restart of its own when Redis returns.
• Health says so (below): Redis fails, the queue check fails rather than beat in-process, RedisMemory warns before Redis fills, and /health turns 503 when the scheduler has stopped checking. Every failover is logged as a warning by Listeners\LogFailover, and the log does not drown: Horizon’s supervisors report the same RedisException several times a second while Redis is away — some 3.6 GB a day, measured — so bootstrap/app.php throttles reporting one to once a minute per message.

Measured on the local site after the fixes: with Redis stopped and with Redis full, every page answered 200 at its usual speed, sign-in worked and the login limit still answered 429 on the sixth try, the password-reset mail arrived, /health said 503 naming Redis, the queue and Horizon, and when Redis came back the settle emptied the fallback within a minute and Horizon resumed. A test takes Redis away with Tests\Support\RedisOutage (a real client pointed at a refused port) or RefusingStore.

The schedule

Hilms\Ops\Support\Scheduler holds the operational entries; the other modules keep their own beside the code they belong to. A test pins every command to its cron expression.

Command	When	Owner
health:queue-check-heartbeat	every minute	ops
horizon:snapshot	every five minutes	ops
health:check	every five minutes	ops
hilms:cache:settle	every minute	ops
hilms:entitlements:expire	hourly	learning
model:prune --model=Generation	daily	ai
backup:clean	01:00	ops
backup:run	01:30	ops
backup:monitor	03:30	ops
activitylog:clean	04:00	ops
personal-data-export:clean	04:30	ops
queue:prune-failed --hours=720	daily	ops
passport:purge	daily	api
telescope:prune	daily, local only	ops
health:schedule-check-heartbeat	every minute, registered last	ops

The schedule heartbeat is registered last on purpose: it only beats once everything before it has run, so the Schedule health check proves the whole schedule and not merely the cron entry.

Monitoring

Horizon at /horizon, Pulse at /pulse, Telescope at /telescope. Each is opened by one permission (View:Horizon, View:Pulse, View:Telescope), and a guest is sent to the login page rather than refused.

• Pulse 1 stores and ingests straight into the database, without the Servers card and its recorder, which a container cannot answer for. Pulse::user() resolves a viewer to their name and email. The cache allowlist (cache.serializable_classes) admits exactly the classes Pulse stores for its cards.
• Telescope 5 is a require-dev package, excluded from package discovery, and registered by OpsServiceProvider only when the application runs locally, so it can never boot in production. Its migration runs everywhere so migrate is identical. It authorises through the viewTelescope gate in every environment; the package’s “local means everyone” bypass is gone.

OpsPlugin adds the Monitoring navigation group: the Activity log resource, the Health page, the Backups page and external links to Horizon, Pulse and — locally — Telescope. Every entry is hidden unless the signed-in user holds its ability, and the two package pages are excluded from shield:generate because their abilities are seeded by hand as Shield custom permissions (Hilms\Ops\Enums\Ability).

artisan about --only=hilms reports the version, the database driver and connection, the languages the installation speaks with the default and the disabled ones marked, the time zone it keeps, whether Horizon is running, when the schedule last beat, whether the activity log is on, and where backups go.

Health

spatie/laravel-health, with the checks registered by Hilms\Ops\Health\HealthChecks: Database, DatabaseConnectionCount, Redis, RedisMemory, Cache, Queue (both queues, failing when a heartbeat job takes longer than five minutes; the heartbeat rides the worker connection, so a dead queue cannot pass by running it inside the scheduler), Schedule (heartbeat at most two minutes old), Horizon, UsedDiskSpace, Backups, and — in production only — DebugMode, Environment and OptimizedApp. A skipped check is not a failure.

Ten checks are ours:

Check	Watches
RedisMemoryCheck	Redis against its own maxmemory: warns at 80 %, fails at 95 %, because a full Redis refuses every write; without a maxmemory, warns above 512 MB and fails above 900 MB. A maxmemory-policy other than noeviction is a warning: an evicted job is a lost job
PhpExtensionsCheck	Fails on a missing required extension, warns on a missing recommended one
BinariesCheck	Fails without the database dump binary, warns without the image optimisers
WritablePathsCheck	The directories the application writes to
ConfigurationCheck	Reads configuration, not the environment, so a cached config is checked as it is served
PassportKeysCheck	The signing key pair
MailTransportCheck	Whether mail can leave at all (below)
ThemeCheck	That the active theme and every parent resolve, that a standalone theme has a built manifest, that a theme publishing a public/ directory is linked, and that the page it draws is still a page — the same Contract\Shell verdict hilms:theme:check reports (Theming). An active style holding values the theme no longer takes is a warning, naming them: the site still draws, in the theme’s values where the style’s cannot be used (Styles)
MediaStorageCheck	Every bucket disk media is written to, and every bucket disk a media row still names after the storage went back to this server, asked about one file — which proves the address, the bucket and the credentials in one request. Skipped while no bucket is written to or holds a file (below)
BlockContentCheck	How many block fields on this installation an editor could not save, and the first record holding one

The first five read Hilms\Ops\Support\Requirements, which parses the ext-* entries of composer.json — so the install guide, the checks and Composer itself can never drift apart, and RequirementsTest fails if they do.

BlockContentCheck warns and never fails: the site serves such a page perfectly well, and only the editor who opens it ever finds out. It is the one check that reads every page and every lesson there is, so its verdict is cached for half an hour as two scalars. hilms:blocks:check is the answer on demand:

bin/artisan hilms:blocks:check [--json]

It names where, which block, which field and why — Grid › Cell 2 › Image — exits 1 when it finds any, reads and never writes. Both go through Hilms\Ops\Support\BlockProblems, which walks the records and asks Hilms\Blocks\Support\BlockValidator: the editor’s own schema, filled from the record the way the panel fills it (The block editor).

GET /health returns the last stored run as JSON behind throttle:60,1 and the X-Secret-Token header. Hilms\Ops\Http\Middleware\RequireHealthToken answers 403 while HEALTH_SECRET_TOKEN is empty, rather than letting everyone through as the package’s own middleware does, and health.secret_token is a required configuration key in production. It answers 503 only when a check failed or crashed, and 200 for a warning: a disk at 84 % is worth reading, not worth paging anyone. It also answers 503 before anything has ever been checked, and when the last run is more than fifteen minutes old — three missed runs — naming when it ran: the results live in the database, so a scheduler that stopped would otherwise keep the last good verdict on show for as long as the site stands. /up stays public for a liveness probe.

Results are stored five days and shown on the panel’s Health page. Mail alerts go to OPS_NOTIFICATION_EMAIL and are off when it is empty.

Backups

spatie/laravel-backup: the database dump plus storage/app/public (public media), storage/app/private/lessons (course material) and storage/app/keys — the three directories that cannot be rebuilt from git. Media kept in a bucket is not in the archive: a bucket’s contents are the provider’s to keep. Gzipped, verified after writing, optionally encrypted with BACKUP_ARCHIVE_PASSWORD; an empty value means no encryption, because the package would otherwise encrypt with an empty password and libzip refuses to close such an archive.

Destinations: the local backups disk, plus s3-backups when BACKUP_S3_ENABLED is true. Retention defaults to everything for 7 days, then daily for 16, weekly for 8 weeks, monthly for 4 months, and no yearly copies, so a deleted account does not live on in a dump nobody looks at. The monitor warns above 5000 MB or a day without a backup.

The dump binary follows the driver (mysqldump or mariadb-dump), honours DB_DUMP_BINARY_PATH, uses a single transaction and a 300-second timeout. DB_DUMP_SKIP_SSL skips TLS for the dump, which is right only when the database sits on a private network offering a certificate no client can verify.

The panel’s Backups page lists the archives, creates one and downloads or deletes them, behind View:Backups.

The audit trail

spatie/laravel-activitylog 5 writes activity_log. Audited: Course, Section, Lesson and Category (their publishing fields), Page (title, slug, language, translation group, template, status, audience), Menu (name, visibility), MenuItem (title, url, target, audience — never its order, since a drag would otherwise write a row per item), MenuLocation, Entitlement and Enrollment (everything fillable, except that a moving resume point is progress rather than an event), User (name, email, verification, profile), MediaAsset (title, alt, description, visibility), ApiClient (name, scopes, revoked), Language (all four columns) and TranslationOverride (a corrected phrase is somebody’s decision), Branding, Style (name, description, tokens and which theme it is active in — its compiled sheet is not an event) and Generation (its status only). Only dirty attributes are stored and an empty change writes nothing.

Role changes are not model attributes, so Hilms\Ops\Listeners\LogRoleChange records role_attached and role_detached from spatie/permission’s events. The listener is discovered by internachi/modular; registering it again in a provider would double every entry.

The causer is the panel user, the API client behind a machine token, the person behind an MCP token, or nobody for the console. Anything that provisions rather than changes — seeders, the load dataset, hilms:install, hilms:upgrade, SeedStarterPages, SeedDefaultMenus — runs inside Activity::withoutLogging().

The morph columns are strings rather than unsigned integers, because subjects and causers mix bigint keys with the API client’s uuid. Entries are kept ACTIVITYLOG_CLEAN_AFTER_DAYS days (365) and pruned nightly.

The panel’s Activity log resource (admin only) is an index page: when, log, event badge, subject with its type, and who; filters by subject type (built from the morph map), event and date range; a “Details” action opens a modal with the before-and-after diff. Sorted by id descending, so entries that share a second keep their order.

Mail

Six transports are chosen in the panel: SMTP, Brevo, Resend, Postmark, Mailgun and Amazon SES (Hilms\Ops\Enums\MailTransport). log, array and sendmail stay environment-only — two of them deliver nothing and the third depends on a binary no form can configure.

Hilms\Ops\Support\MailConfiguration::apply() is the one bridge from MailSettings to the mailer. It sets mail.default, the chosen transport’s own entry, the services.* keys Laravel’s transports read, mail.from, and mail.to for the redirect, then calls MailManager::forgetMailers(). It runs twice: on $app->resolving(MailManager::class), so a request that sends nothing pays nothing, and on Queue::before(), so a long-running Horizon worker picks a change up on its next job rather than on its next restart. spatie’s MissingSettings is caught, so the first boot of hilms:install still mails through the environment.

A mailer the panel does not offer is left exactly as the environment set it. transport() answers null for log, array, sendmail or a failover group, and only the sender and the redirect are applied, because those belong to no transport. Forcing an unknown mailer onto SMTP is how a test suite, a developer’s machine or a deliberate failover starts dialling out. The Mail section says which mailer is in force when it is not one of the six, and a test asserts that the suite’s own mailer survives apply() untouched.

The redirect (always_to, seeded from MAIL_ALWAYS_TO) is a pin for a staging installation: while it is filled, every message goes to that address instead of the person it names. It is applied through mail.to, which every mailer the manager builds afterwards carries — never Mail::alwaysTo(), which would not survive forgetMailers(). The Mail section shouts about it in red for as long as it is set.

MailTransportCheck fails when the transport in force has no credentials in the panel or the environment, and — in production only — when it is log or array: an installation that cannot send mail looks healthy until somebody asks for a password reset.

“Mail” (Hilms\Ops\Filament\Pages\ManageMailSettings, a section of Settings under Integrations rather than anything under Monitoring, because it is configuration and not something to watch) holds the sender, the redirect, the transport and one form section per transport that appears only when it is chosen; every secret on it is a SecretInput. Its header action “Send test message” and hilms:mail:test {address} share Hilms\Ops\Actions\SendTestMail, which applies the settings first and answers with the address the provider was really handed.

Media storage

Where uploads are kept is chosen on the “Media and storage” section of Settings (Hilms\Ops\Filament\Pages\ManageMediaSettings, under Integrations, administrators only, found by searching for a bucket, S3, R2, Bunny Stream or the watermark), over Hilms\Ops\Settings\MediaSettings:

Setting	Holds
storage	Hilms\Ops\Enums\MediaStorage: local (this server) or s3 (a bucket)
provider	Hilms\Ops\Storage\BucketProvider: r2, aws, b2, hetzner, spaces or other; null reads as other
s3_account_id, s3_location	What a named provider is found by: R2’s account id, and a jurisdiction, region or location
s3_endpoint, s3_region, s3_path_style	Other’s typed addressing; a named provider builds them
s3_bucket, s3_key, s3_secret	The course-material bucket and the key; s3_secret is encrypted
s3_public_bucket, s3_public_url	A second bucket for public media and the address anybody may read it at; never the lesson bucket
bunny_stream_token_key	The Bunny Stream library’s token key, encrypted
watermark_videos	Whether an uploaded video carries the viewer’s address (off)

provider is cast in casts() with an EnumCast: spatie reads a property’s type from its docblock whenever it has one, and a descriptive docblock without @var left the provider a string that threw on the next load. The first settings migration seeds what the environment configures (MEDIA_S3_*, BUNNY_STREAM_TOKEN_KEY; storage is s3 exactly when media-library.lessons_disk is remote-lessons), and Other keeps falling back to the environment for a blank value. Both secrets are SecretInputs, never rendered back (Settings).

Providers. BucketProvider knows, for each case, the endpoint it builds from an account id or a location (R2 https://<id>[.eu|.fedramp].r2.cloudflarestorage.com, B2 https://s3.<region>.backblazeb2.com, Hetzner https://<fsn1|nbg1|hel1>.your-objectstorage.com, Spaces https://<slug>.digitaloceanspaces.com, none for AWS), the client region, the addressing, whether it sends per-file access lists, the public address of a bucket where its shape is fixed, the locations it offers and the patterns its inputs are held to — everything it builds ends in a URL and in the content security policy, so what does not match is refused rather than escaped. Hilms\Ops\Storage\BucketConfiguration is one complete setup, built from the settings (fromSettings()) or from the guide’s form (fromForm(), where a blank secret means the stored one), and answers the two disks’ configuration; its builders throw InvalidBucketConfiguration naming the field, never the value. A named provider needs a key and a secret: with a blank one the AWS SDK would go looking for an instance role at 169.254.169.254.

Hilms\Ops\Support\MediaConfiguration::apply() is the one bridge from the settings to configuration, and writes nothing else: the bucket’s details on both bucket disks — whatever storage is chosen, because moving files back from a bucket needs them as much as moving them there — then media-library.lessons_disk and media-library.disk_name, then blocks.bunny_stream_token_key and blocks.watermark_videos. A bucket always takes course material (remote-lessons) and takes public media (remote-public) only when a public bucket and its public address are both named and the public bucket is not the lesson bucket. This server keeps whatever local disk the environment names and replaces only a remote one with the default. A bucket disk is forgotten (Storage::forgetDisk()) only when its configuration changed, so a worker keeps its client from job to job. A stored setup that can no longer be built is reported and leaves the disks as they were, so it cannot fail every job. withBucket($setup, $closure) applies a setup only while the closure runs and puts both disk arrays back exactly, which is how an unsaved setup is checked.

The settings are read by models, disks, the video block and both content security policies, with no one service to hook, so they are applied as the application boots ($app->booted) and before every job that does not run on the sync connection, so a long-running worker picks a change up on its next job. The boot hook is skipped in an application that boots after Artisan has started in the same process: that is the one config:cache builds to read the configuration from, and applying the panel there would bake its values — the secret included — into the cache as if the environment had said them. MissingSettings and a QueryException are caught, so a build without a database and the first boot of hilms:install store media where the environment says. And because every process reads the settings before it could migrate, MigrationsEnded clears spatie’s settings cache and forgets every settings instance, or the object read at boot would lack what the migration added and be refused on its next save.

The check (Hilms\Ops\Actions\CheckBucket::handle(BucketConfiguration): BucketReport) proves a setup with one 19-byte file per bucket, always deleted. On the course-material bucket it writes, reads back, signs a five-minute address, opens that address over HTTP with Origin: <APP_URL origin> as a player would — which proves the signature, the region and the clock from outside — reads Access-Control-Allow-Origin from the answer, and asks the unsigned address, which must not open. On the public bucket it writes and reads through the public address, CORS included. Each BucketCheck is blocking or confirmable (the two CORS checks and the public read), each failure carries a reason and a fix in the provider’s words, and a check whose prerequisite failed is skipped. A refusal is said plainly by Hilms\Ops\Storage\ProviderAnswer: from the provider’s error code, with the code beside it (the secret does not belong to the key, the key is unknown, the bucket lives elsewhere), or, for a connection that never opened, that the address did not answer; anything without a code keeps the client’s words through StorageUnreachable, every configured secret replaced by [secret] and every address’s query taken out, cut at 600 characters, carrying no previous exception. A named provider without a key or secret fails before any request is made.

The page has four sections. Where new uploads go summarises the setup in three states (this server; a bucket, with its last check from Storage\LastCheck; this server with a bucket kept after Stop) and holds Set up a bucket, Check again, Change…, Stop using the bucket, Use the bucket again (checked first) and Forget the bucket (only while no media row’s file lies on a bucket disk, because such a record would name a disk that could no longer be built). Where your files are is the embedded Hilms\Ops\Livewire\MediaLocations: files and size per location, how many wait, and the move labelled with what it does (“Move 16 files (8.6 MB) to Cloudflare R2”), started through MediaMoves, polled every three seconds while it runs, locked meanwhile, its failures listed with their reasons, and, while a move runs, a notice when no Horizon master is running (Support\QueueWorker). Hosted video and Protection keep the Bunny key and the watermark, and the page’s own Save writes only those two.

The guide (Concerns\GuidesBucketSetup) is one Filament action with six steps() — Provider, Connect, Course material, Public files, Allow your site, Check and save — used by Set up and Change…, in the provider’s own words. The CORS rule comes from Storage\CorsRule in the shape the provider takes it. The check runs as the last step opens and on Check again; a blocking failure disables Save, a confirmable one asks for a “Save anyway” tick naming what will break. Actions\SaveBucket checks again on the server (a wizard’s afterValidation never runs on submit, and a browser can skip a step), refuses a changed bucket identity while files live in it (Storage\BucketFiles), stores and applies the setup, records the check, and calls Actions\RefreshPublicAddresses — the cached branding forgotten, the styles that draw a font in the public bucket recompiled — when the public address changed. Actions\SwitchMediaStorage and Actions\ForgetBucket are the other three doors.

docs/install/storage.md is the operator’s guide to the providers and to checking the result, and Media is what happens to a file afterwards.

Installing and upgrading

hilms:install and hilms:upgrade are built from idempotent Hilms\Ops\Install\Step classes, and both must stay safe to run twice; hilms:deploy chooses between them (below, under Deployment). Every step reports what it did or why it had nothing to do, and --no-interaction never prompts — it fails instead, which is what the container entrypoint needs.

hilms:install runs sixteen steps:

CheckRequirements  EnsureAppKey  Migrate  SeedBase  SyncStyles  SeedLanguages
SetTimezone  SeedPages  SeedMenus  CreateAdmin  StorageLink  LinkThemes
PassportKeys  PublishAssets  Optimize  RunHealthChecks

then prints the site, panel and health URLs. CheckRequirements runs the custom checks plus a database and cache probe, and a failure stops the run unless --force. SyncStyles (labelled Styles) syncs every theme’s built-in style files into their rows, activates a theme’s built-in while the theme has no active style (one that meets the enforced accessibility, where it is enforced), and compiles every style again against its theme; a file it refuses is counted, and the step points at hilms:theme:styles --check, which names it — it never stops the run (Styles). SeedLanguages creates the one language an installation ships with, from APP_LOCALE, and never touches the table again — the Languages page is the list from then on (Languages). SetTimezone asks which clock the installation keeps: --timezone= first, then the terminal, and only while the installation is still on the configured zone, so one that has chosen is never asked again (Time and clocks). SeedPages writes the starter pages onto an installation with no pages at all and points the three destinations at them (Pages); SeedMenus then gives each language a header and a footer pointing at those pages (Navigation). The administrator comes from --admin-*, then HILMS_ADMIN_*, then the terminal. StorageLink writes public/storage as a relative symlink, so it resolves wherever the application is mounted — an absolute one names the path the command happened to run at, which the container serving the site cannot follow — and it leaves alone a real directory standing where the link belongs. LinkThemes links what every theme publishes at public/themes/<name>, replacing a stale link and skipping a correct one.

hilms:upgrade reuses the same steps for a deployment, drops EnsureAppKey, SetTimezone, SeedPages and CreateAdmin, and adds RelocateMedia, RestartWorkers and, last of all, CheckBlocks:

CheckRequirements  Migrate  SeedBase  SyncStyles  SeedLanguages  SeedMenus  StorageLink
LinkThemes  PassportKeys  PublishAssets  RelocateMedia  Optimize  RestartWorkers
RunHealthChecks  CheckBlocks

SyncStyles is in the upgrade because a deploy may change a theme’s tokens under every style: each is compiled again — its sheet, the fonts it declares, its preload and its mail palette — and what a style can no longer use is left out and named rather than breaking the site.

RelocateMedia moves nothing: it counts the files that are not where their owner names (MediaRelocator::misplaced()) and says so, because a file is served from wherever its record says and moving every byte inside a starting container would keep the site down until the last one had crossed. Moving is the administrator’s act on the settings page, or hilms:media:relocate on the command line. SeedPages belongs to the install alone: an upgrade never hands an installation content, and an installation’s clock is nobody’s to change on a deployment. CheckBlocks is informational — a block that gained a required field in this version is a block every older copy of it is now missing something for, so the upgrade counts them and names hilms:blocks:check. It reports and never fails: content is an editor’s to fix, and an upgrade that stopped over a page nobody can save would be an upgrade nobody could finish.

--down holds the maintenance page — pre-rendered from the theme’s errors::503 — for the duration. SeedBase also prunes permissions no policy consults any more, so a role that held a dead one loses a no-op. PublishAssets publishes Filament’s assets and Livewire’s, because the front-end and the panel need different Livewire builds and only published files have URLs of their own; it is also what puts the panel’s own stylesheets and scripts, and every module’s, into public/{css,js}/app where the browser can read them (The block editor explains how they are versioned).

Adding a step means adding a Step class and listing it in one or both commands. Keep it idempotent and keep it quiet about what it did not need to do.

docs/install/ is the operator’s side of all this: requirements.md, docker.md, manual.md, upgrade.md, storage.md, languages.md, retention.md.

Deployment

Dockerfile builds the front-end with node:24-alpine and the application on serversideup/php:8.5-fpm-nginx, adding bcmath, exif, gd and intl, the MariaDB client (whose mysqldump dumps both engines) and the image optimisers, with opcache validation off, a 256 MB memory limit and 100 MB upload and body sizes. HILMS_VERSION is a build argument the release workflow sets to the tag the image is published under.

docker/entrypoint.d/50-hilms-install.sh runs hilms:deploy --no-interaction --force on every start when HILMS_AUTO_INSTALL=true — the application container: compose.yaml sets the variable on each service, false on the two workers, because .env reaches every service and two workers starting together would otherwise migrate side by side — and 40-hilms-optimize.sh warms the caches in the others. hilms:deploy keeps the decision in PHP, where it is tested: Hilms\Ops\Install\Installation::exists() answers whether the users and role tables exist and hold an administrator, and the command says so and hands over to hilms:upgrade or hilms:install, passing --force, --no-optimize and --no-interaction through and answering with that command’s exit code. An administrator is the signal because every step only the install has runs before or as the administrator is created, so whatever an install left undone after that point is exactly what an upgrade does. A Docker update is therefore pull and up -d; a failing step fails the container’s start, and hilms:upgrade by hand is the fallback.

compose.yaml runs app (the only published port, on 127.0.0.1), horizon, scheduler, mysql:8.4 and redis:7-alpine — append-only, bounded at --maxmemory 256mb under noeviction, because an unbounded Redis grows until the kernel kills the largest process on the machine, often the database, and a full one is now ridden out — with named volumes and rotated logs. RequirementsTest fails when a variable compose.yaml interpolates is missing from .env.production.example, or when a service publishes a port on anything but 127.0.0.1.

The load dataset

hilms:seed:load --courses=500 --students=5000 fills the database with generated categories, instructors, courses, sections, lessons, blocks, students, entitlements, enrolments and completions, through chunked insertOrIgnore — so no model event fires, nothing is logged, and a rerun with the same numbers adds nothing. It refuses a production database without --force. It is the one writer of entitlements and enrollments outside the learning actions and their tests, which is why it says so in its own docblock. Every block it writes is one the editor could have saved — a test holds the generated lessons to BlockValidator, since its callouts once named their kind under a field the block does not have and drew a broken lesson (Quality has what the load measured).

Tests

app-modules/ops/tests/Feature: the requirements against the install guide and composer.json, every custom health check, the failover cache and its trip, the settle, the committed failover queue and the worker-only connections, a stale health run, the health endpoint’s token and status codes, the scheduler’s cron expressions, the mail bridge including the untouched unknown mailer and the redirect, the test message, the media bridge, the bucket disks and the “Media and storage” page with its two actions, the activity resource and the role-change listener, hilms:blocks:check and the walk behind it, the default menus and their idempotency, each install and upgrade step run twice, hilms:deploy choosing the install on a new database and the upgrade on an installed one, the load dataset’s idempotency, and the Compose and environment invariants.