The Game Panel specification

RFC-0004: TOML configuration loading

Abstract

Configuration is read from TOML files: a main file, drop-in overrides applied in order, and one file for each enabled module, merged into a single tree with environment variables interpolated. A registry hydrates configuration objects from that tree in two seals, core configuration first so that the module system can read which modules are enabled, then each module’s configuration, and produces the immutable configuration catalogue.

Motivation

The panel’s configuration is written in TOML files that sysadmins edit, per ADR-0010. The configuration component holds configuration objects in a catalogue, per RFC-0002, but has no way to read them from files.

Modules provide configuration of their own. To register it, the module system needs to know which modules are enabled, and which modules are enabled is itself configuration, so it has to be readable before any module registers anything.

Proposal

Concepts

Term Meaning
Main configuration file config.toml, the base of the configuration.
Drop-in A TOML file in config.d, applied over the main configuration file.
Module configuration file A TOML file in modules-enabled, holding one module’s configuration. Its presence enables the module.
Module identifier A module’s name, taken from its module configuration file’s name without the .toml extension.
Configuration tree The array produced by reading and merging every file.
Section The part of the configuration tree a configuration object is hydrated from.
Core configuration Configuration hydrated before the module system runs, because the module system needs it.
Module configuration Configuration a module registers, hydrated once every module has registered.

Components

Component Responsibility
ConfigPaths Value object holding the paths the loader reads from.
TomlLoader Reads, merges and interpolates the TOML files into a configuration tree.
ConfigRegistry Mutable, and exists only during bootstrap. Hydrates configuration objects from the tree in two seals, and produces the ConfigCatalogue.
CoreConfig Maps top-level keys of the tree to core configuration classes.
ModulesEnabled Core configuration object holding the enabled module identifiers.
ConfigObject Contract every configuration object implements, hydrated through fromArray().

Paths

The loader reads from three locations:

config.toml
config.d/
    10-database.toml
    20-secrets.toml
modules-enabled/
    admin.toml
    billing.toml

ConfigPaths holds the absolute path of each: the main configuration file as configFile, the drop-in directory as configDir, and the module configuration directory as modulesEnabledDir. Bootstrap constructs it, from the environment, command-line flags or compiled defaults. The configuration component never works out paths itself.

Modules that are installed but not enabled are a concern of the command line that enables and disables them. Only modules-enabled matters to loading.

Loading

TomlLoader::load(ConfigPaths $paths): array builds the configuration tree in these steps:

  1. The main configuration file is parsed. If it cannot be read or parsed, loading throws.
  2. The drop-ins are parsed in order of filename, and each is merged over the tree in turn. A missing drop-in directory is treated as empty.
  3. The module configuration files are parsed in order of filename, and each is placed in the tree under modules, keyed by its module identifier. A missing module configuration directory is treated as empty.
  4. The module identifiers, in the same order, are placed in the tree under __enabled_modules.
  5. Environment variables are interpolated throughout the tree, module configuration included.

Only files with the .toml extension are read. The loader knows nothing of configuration objects: it translates files into an array.

[
    // keys from config.toml, with config.d/*.toml merged over them
    'modules'           => ['admin' => [/* admin.toml */], 'billing' => [/* billing.toml */]],
    '__enabled_modules' => ['admin', 'billing'],
]

modules and __enabled_modules are reserved for the loader. The main configuration file or a drop-in declaring either at its top level throws, naming the file that declared it.

A module identifier cannot contain a dot, because dots separate the parts of a section’s path. A module configuration file whose name contains a dot throws.

Merging

A drop-in is merged over the tree by these rules, applied from the top level down:

An empty array counts as a table, so an empty drop-in file changes nothing, and an empty array replaces a list but leaves a table as it was.

# config.toml
[database]
primary = "main"
hosts   = ["a", "b"]

[database.connections.main]
host = "localhost"
port = 3306

# config.d/10-database.toml
[database]
hosts = ["c"]

[database.connections.main]
port = 3307

After merging, database.primary is main, database.hosts is ["c"], and the main connection has the host localhost and the port 3307.

Interpolation

A string in the tree may contain any number of references to environment variables, each replaced by the variable’s value read from Env, per ADR-0006:

Reference Replaced with
${NAME} The variable’s value. If the variable is not set, or is null, loading throws.
${NAME:-default} The variable’s value, or the default when the variable is not set or is null. 
title = "${APP_NAME}"
dsn   = "${DB_HOST}:${DB_PORT}"

[database]
password = "${DB_PASSWORD}"
region   = "${REGION:-eu-west}"
tags     = ["${TAG_ONE}", "literal"]

A variable name is made of uppercase letters, digits and underscores, and does not start with a digit. A reference using any other name, such as ${name}, is left as it was written. Only strings are interpolated, including strings inside lists, and the result is always a string. Numbers, booleans and other values are left as they are.

A variable that is not set and has no default throws, naming the variable and the path to the value, such as database.password or database.tags[0].

Interpolation keeps secrets, such as database passwords, out of configuration files, which are plain text. Because it happens while the tree is loaded, Env must be initialised before the loader runs.

Configuration objects

ConfigObject requires fromArray(array $data): static, which creates the configuration object from its section of the tree. It replaces the __set_state() of RFC-0002.

A configuration object validates what it is given, and validation happens as it is hydrated. Its typed readonly properties enforce the shape and types of its values by construction. Assertions in its constructor enforce ranges, formats and the relationships between values, so an object created any other way, such as through a make() factory, is validated the same way. fromArray() asserts only that the keys it needs exist and have the right shape. A failed assertion throws.

final readonly class MailConfig implements ConfigObject
{
    public static function fromArray(array $data): static
    {
        Assert::keyExists($data, 'host', 'Host is not defined.');
        Assert::keyExists($data, 'port', 'Port is not defined.');

        return new self($data['host'], $data['port']);
    }

    private function __construct(
        public string $host,
        public int $port,
    ) {
        Assert::stringNotEmpty($host, 'Host is not defined.');
        Assert::range($port, 1, 65535, 'Port is out of range.');
    }
}

Registry

ConfigRegistry is the mutable half of the configuration component, sealed into the immutable ConfigCatalogue, per ADR-0003. It is constructed with the configuration tree and the core mapping, and is discarded once it has been sealed.

$tree     = new TomlLoader()->load($paths);
$registry = new ConfigRegistry($tree, CoreConfig::MAPPING);

$registry->sealCore();

$enabled = $registry->for(ModulesEnabled::class);

// Each enabled module registers its configuration.
$registry->register('admin', 'main', AdminConfig::class);

$catalogue = $registry->seal();
Method Effect
sealCore(): void Hydrates every core configuration object.
for(string $class): ConfigObject Returns a hydrated core configuration object by its class.
register(string $module, string $name, string $class): void Registers a module’s configuration class under the module and a name, to be hydrated by seal().
seal(): ConfigCatalogue Hydrates every registered module configuration object, and returns the catalogue holding them and the core configuration.

The registry passes through three phases, and each method can only be called in some of them:

Phase Entered by Permits
Open Construction sealCore()
Core sealed sealCore() for(), register() and seal()
Sealed seal() Nothing. The catalogue is used instead.

A method called in a phase that does not permit it throws ConfigLifecycleException.

Core configuration

Core configuration is what must be hydrated and available before the module system runs. CoreConfig::MAPPING maps a top-level key of the tree to the class hydrated from it:

public const array MAPPING = [
    '__enabled_modules' => ModulesEnabled::class,
];

sealCore() hydrates each class from its key. A missing key hydrates the class from an empty array, and a key holding anything other than an array throws. In the catalogue, core configuration is placed under the module engine, named by its key.

ModulesEnabled holds the enabled module identifiers as a list of strings, and nothing else. Its has(string $module): bool returns whether a module is enabled. It is the only core configuration.

The module system reads ModulesEnabled through for(), between the two seals. It does not scan the filesystem, or repeat the logic that decides which modules are enabled: the configuration component is the single source of which modules are enabled.

Module configuration

A module registers each of its configuration classes under its module identifier and a name. seal() hydrates the class from modules.{module}.{name} in the tree, which is the table {name} in the module’s configuration file.

# modules-enabled/admin.toml
[main]
value = "${ADMIN_VALUE}"

$registry->register('admin', 'main', AdminConfig::class);

A missing table hydrates the class from an empty array, and anything other than an array on that path throws. The module name engine is reserved for core configuration, and registering under it throws. So does registering with a module identifier or name that contains a dot.

In the catalogue, each module configuration object is placed under its module and name, so both $catalogue->get('admin', 'main') and $catalogue->for(AdminConfig::class) return it.

Hydration errors

Any failure while a configuration object is hydrated is wrapped in InvalidConfigException, naming the file, section and key it came from, so that sysadmins can find the problem without guessing:

Invalid config in modules-enabled/admin.toml at [main.value]: Value is not defined.

Database configuration

DatabaseConfig and ConnectionConfig, from RFC-0003, are hydrated through fromArray(), with their validation in their constructors.

primary    = "main"
persistent = false

[connections.main]
host     = "127.0.0.1"
port     = 3306
database = "panel"
username = "panel"
password = "${DB_PASSWORD}"

[connections.local]
socket   = "/var/run/mysqld/mysqld.sock"
database = "panel"
username = "panel"
password = ""
DatabaseConfig key Rule
primary Required, and not empty. A connection must be configured under it.
connections Required. A table of connections, keyed by name, with at least one. Each is a non-empty table hydrated into a ConnectionConfig, and a failure names the connection.
persistent Optional, and a boolean. Defaults to false.
ConnectionConfig key Rule
database, username Required, and not empty.
password Required, and may be empty.
socket Optional. When given, it must not be empty, and the connection is made through it.
host Required when no socket is given, and not empty.
port Required when no socket is given, and an integer.
options Optional, and a table.

A connection through a host must configure its port, which no longer defaults to 3306.

Errors

Every exception implements ConfigException, per RFC-0002.

Exception Extends Thrown when
InvalidConfigException RuntimeException A file cannot be read or parsed, a reserved key is declared, a module configuration file’s name contains a dot, an environment variable is not set and has no default, a section is not an array, or a configuration object fails to hydrate.
ConfigLifecycleException LogicException A registry method is called in a phase that does not permit it, a module registers as engine, or a module identifier or configuration name contains a dot.
ConfigNotRegisteredException RuntimeException for() is asked for a class that is not core configuration.

Out of scope

Alternatives considered

The decisions this design rests on are recorded separately, with the alternatives each one rejected:

No other alternatives were weighed.

Backwards compatibility

Open questions

Changelog

Sources

This site is open source. Improve this page.