modules.services

All pyobs process and filesystem logic – module discovery, start/stop/restart, log reading, config read/write, ACL resolution and editing, and ejabberd identity resolution. See How modules are managed and ACL matrix and per-module ACL editing for the user-facing behaviour this implements.

exception LogSearchError

Bases: Exception

A log search (file scan or journalctl –grep) failed outright – distinct from a search that ran fine and found nothing, so callers can tell the two apart instead of both looking like an empty result.

build_acl_matrix() → dict

Builds the fleet-wide (target x caller) ACL matrix.

Rows are every module list_modules() returns; columns are that same full module list plus every caller name mentioned in any module’s resolved acl: block (“allow” keys or “deny” entries) that isn’t itself a managed module (e.g. a human/external caller like “scheduler” if it has no config of its own) – every module is always a column, whether or not it’s ever actually referenced as a caller anywhere, so “could A reach B” is answerable for any pair, not just pairs where B happens to already appear in some acl: block. A module whose config/acl can’t be resolved (bad YAML, broken {include}, …) is still included as a row, with its “error” set, rather than aborting the whole scan.

build_comm_user_map() → dict[str, list[str]]

Maps every local module’s resolved comm.user to the list of module names using it – the reverse direction of find_modules_sharing_comm_user, built once across all of list_modules() rather than queried one identity at a time.

Feeds the fleet-wide Users page (DEVELOPMENT.md’s Ideas -> promoted here): unlike the module page’s own XMPP row, that page needs “for every registered ejabberd account, which module(s) if any use it” – the reverse lookup, not “for this one identity, which modules share it.”

build_module_classes() → dict[str, str]

Maps every local module name to its configured class: (e.g. “pyobs.modules.camera.BaseCamera”) – feeds api/modules/classes/ (issue #65), which lets an external caller (e.g. pyobs-portal) filter modules by interface on its own side, using its own pyobs-core install, without this app importing pyobs.interfaces or the module’s actual class itself.

A module whose config can’t be resolved, or that has no top-level “class:” key, is simply omitted – not included with a None/error value – since the caller only cares about modules it can actually resolve to a class.

build_package_version_matrix(per_host: list[tuple[str, list[dict]]]) → dict

Turns get_package_overview()-shaped per-host package lists into a package x host matrix for the fleet Overview page – one row per pyobs-* package name (the union across every host), one cell per host in the same order as hosts, each either that host’s get_package_overview() entry for the package or None if that host doesn’t have it installed at all. Mirrors merge_acl_matrices’ row[“cells”]-is-a-dict-keyed-by-column shape turned into a positional list instead (row[“cells”][c] there vs. cells[i] here) – Django templates can’t do a dict lookup keyed by a {% for %} loop variable, only a literal attribute/key, so the per-host values need to already be in column order by the time they reach the template (see fleet_overview.html’s parallel {% for host in package_hosts %} / {% for cell in pkg.cells %} loops).

latest_version is taken from whichever host happened to report one – PyPI has no notion of “per host”, so any host’s non-None reading is as good as any other’s; a package only installed on an unreachable host reports None here, same as get_package_overview()’s own “latest lookup failed” case.

config_stale(name: str) → bool | None

Whether name’s on-disk config has changed since the running process last started.

None means unknown, not “fresh” – either the module was never snapshotted (started before this feature shipped, or its run dir was cleared) or its config file is currently missing.

create_module(name: str) → None

Creates a brand-new module config with minimal starter YAML – unlike save_config, which refuses to write a file that doesn’t exist yet, this is the one path that’s allowed to. Refuses if a config with this name already exists, same as it would if someone tried to hand-create a file that’s already there.

find_modules_sharing_comm_user(user: str) → list[str]

Every locally-configured module whose resolved comm.user equals user.

Needed because ejabberd-user-management.md’s write actions (register/change_password/ ban_account/unban_account/unregister) affect every module sharing an XMPP identity, not just whichever module’s page an action was triggered from – ejabberd-integration.md’s own “third bug” documents _test and camera sharing one comm.user for real, in this exact fleet, not a hypothetical edge case.

get_comm_user(name: str) → str | None

Resolves a module’s own XMPP identity – its comm.user, e.g. “camera” in comm: {user: camera, …}. Display-only convenience wrapper around get_resolved_comm, dropping the password/source – most callers (dashboard, module page) only ever show this value, they don’t edit it or need its credential. See get_resolved_comm for the fuller resolution ejabberd-user-management.md’s write actions need.

get_module_class(name: str) → str | None

Returns a module’s configured top-level class: (fully-qualified class name), resolved the same way get_resolved_acl resolves acl: – via pre_process_yaml + yaml.safe_load, since class: can equally arrive through a YAML anchor/merge key, not just be written directly. None if the config doesn’t exist, doesn’t parse, or has no top-level “class:” key.

get_module_status(name: str) → str

Returns ‘running’, ‘stopped’, or ‘unknown’.

get_module_versions(name: str) → dict[str, str] | None

{distribution: version} from the newest “Loaded pyobs packages:” line in name’s log, or None if the module isn’t running or its log has no such line yet (started before the pyobs-core change that logs it, or the line has rotated out of retention).

Cached by PID, mirroring _process_cache: the version set is fixed for the lifetime of a process, so a status poll must not re-shell-out to tac/journalctl every time. A cached entry is reused only while the PID still matches; a stopped module clears it.

get_package_overview() → list[dict]

list_pyobs_packages() plus each package’s latest available version, fetched in parallel since each lookup (a PyPI JSON round-trip, or a git ls-remote for a package installed via a PYOBS_MANAGED_PACKAGES git/URL spec – see _is_vcs_managed) is its own network call and this page’s whole point is showing every pyobs-* package at once.

get_package_update_status() → dict

Reports the current/last background update job. See the module comment above _pkg_update_lock_flock_file for the on-disk job-state design this reads.

get_resolved_acl(name: str) → tuple[dict | None, str | None]

Returns (acl_block, source) for a module’s effective acl: config, resolving any {include} the same way pyobs-core does.

acl_block is the raw “acl:” dict (with “allow”/”deny”/”mode” keys) or None if the module has no acl: key at all (fully open access). source is None if the block is defined directly in the module’s own config file, or the shared fragment’s name (as used by list_shared_configs()/get_shared_config()) if pulled in via {include}.

get_resolved_comm(name: str) → tuple[str | None, str | None, str | None]

Returns (comm_user, comm_password, source) for a module’s effective comm: block – the same resolution get_resolved_acl uses for acl:, via pre_process_yaml + yaml.safe_load, since comm: can equally arrive via {include} or a YAML anchor/merge key (a real config uses comm: {<<: *comm, user: camera, password: pyobs}).

comm_user/comm_password are None if the module has no comm: block at all (confirmed real example: HttpFileCache) or the respective sub-key is missing – not an error, just “this module was never expected to have an XMPP identity” (see ejabberd-integration.md, “Where it surfaces”). source is None if comm: is defined directly in the module’s own file, or the shared fragment’s name if pulled in via {include}.

The password is needed (not just user) for ejabberd-user-management.md’s register action: it registers a new XMPP account using whatever password the module’s config already declares, rather than prompting for a new one – the whole point is making an existing comm.user/comm.password config actually work, not choosing a fresh credential. source is needed for that same doc’s config write-back (change_password), which must refuse to edit comm.password: when it resolves to a shared fragment, exactly the guard save_local_acl already applies to acl:.

Also returns all-None if resolution itself fails – e.g. an {include}’d fragment that no longer exists – the same as “no comm: block”, rather than raising. Unlike get_resolved_acl (whose only caller, resolve_and_validate_acl, already catches this), get_resolved_comm is called directly from several views (dashboard status polling, module ejabberd endpoints, the Users page), so one module’s broken include must not crash the whole fleet view.

git_clone() → tuple[bool, str]

Clone the configured repository, set up sparse checkout, and create config symlink.

The clone target is the repository root (_git_repo_dir()), never PYOBS_CONFIG_DIR. Refuses to clone if that directory already contains a Git working tree or non-empty contents.

After a successful clone, creates a symlink from PYOBS_CONFIG_DIR to <repo_dir>/<subpath> so pyobs can read configs via its standard path.

Returns (success, message).

git_commit(message: str) → tuple[bool, str]

Commit all staged changes.

Passes an explicit author identity via -c rather than relying on a global git config user.name/user.email already being set for whichever system user runs pyobs-web-admin – without it, git refuses the commit outright with “Author identity unknown” (this is exactly what PYOBS_CONFIG_GIT_AUTHOR_NAME/EMAIL exist to avoid), and -c only overrides it for this one invocation rather than mutating the repo’s/user’s actual git config.

git_fetch() → tuple[bool, str]

Fetch remote updates without modifying the working tree.

Uses –force on tags: these are deployment checkouts that never create their own tags, so a local tag diverging from origin (e.g. a re-cut release) should always lose to origin rather than blocking the whole fetch with a “would clobber existing tag” rejection.

git_init_if_needed() → tuple[bool, str]

Lazy init: clone if the repository doesn’t exist yet.

git_pull() → tuple[bool, str]

Pull from origin.

Passes -c pull.rebase=false explicitly rather than relying on the managed repo’s/system’s git config already setting a reconcile strategy – without it, git refuses divergent branches outright (“Need to specify how to reconcile divergent branches”). A merge from diverged branches also creates a merge commit, so the author identity is passed the same way git_commit() does – without it, git refuses with “empty ident name” on a system where no global git identity is configured (e.g. the service user on iagvtsrv).

Auto-stashes uncommitted changes around the pull (seen live on astro159): git refuses to merge over ANY uncommitted change to a file the incoming commits also touch, even when the two edits are in non-overlapping regions of that file (“local changes … would be overwritten by merge”) – unlike the CONFLICT case below, that happens before a merge is even attempted, so there’s nothing to abort and no way to retry without first getting the working tree clean. Stashing first (a clean tree just stashes nothing – “No local changes to save”) and popping back after lets a normal 3-way merge reconcile non-overlapping edits automatically, same as it would for two independently committed changes.

git_push() → tuple[bool, str]

Push current branch to origin.

git_repo_exists() → bool

Check whether a Git working tree exists at the configured repository root.

git_reset() → tuple[bool, str]

Discard all uncommitted changes (reset working tree to HEAD).

git_stage_all() → tuple[bool, str]

Stage all changes.

git_status() → dict

Get repository status.

Includes repo_exists (git_repo_exists()) so a proxied call (see api_git_status) carries the target host’s repo presence along with its status in one round trip – git_config_page needs both and, when the active host is remote, has no other way to reach that host’s git_repo_exists() short of a second proxied endpoint.

list_pyobs_packages() → list[dict]

Installed pyobs-* packages (name + version), via pip list –format=json rather than importlib.metadata – pyobs-web-admin itself may run in a different environment than pyobs (see _pip_exec), so introspecting its own imports wouldn’t reflect what pyobs actually has installed. Also includes any package – pyobs-prefixed or not – listed in settings.PYOBS_MANAGED_PACKAGES, but only if it’s actually installed: that setting can extend which installed packages are shown/managed, never invent an entry for one that isn’t really there.

merge_acl_matrices(per_host: list[tuple[str, dict]]) → dict

Combines each host’s build_acl_matrix()-shaped result into one fleet-wide matrix – see acl-matrix.md, “Hub mode interaction”. per_host is a list of (host_name, matrix) pairs, e.g. [(“localhost”, build_acl_matrix()), (“MONETS”, <that host’s own matrix, fetched via the hub proxy>), …].

Each host only knows about the callers its own modules’ acl: blocks reference, so a row fetched from one host is missing cells for callers that only appear on some other host. Cells are therefore recomputed here against the union of every host’s callers, reusing _acl_cell (a pure function of a target’s acl: dict + a caller name – safe to call again outside the host that originally resolved that acl:) rather than trusting each host’s own host-local cells.

merge_log_lines(line_lists: list[list[str]], lines: int) → list[str]

Merges several already-formatted, already-oldest-first-ordered log line lists into one list ordered by each line’s own leading timestamp, trimmed to the overall last lines.

Used both for the file backend’s per-module tail merge (_get_all_logs_file) and, in views.py, for combining each hub host’s own already-merged fleet-wide result into one cross-host view – same “no shared time index, so merge-and-trim after the fact” shape either way, just one level up in the second case.

merge_module_classes(per_host: list[tuple[str, dict[str, str]]]) → list[dict[str, str]]

Combines each host’s build_module_classes()-shaped result (a flat {name: class} dict) into one fleet-wide list, tagged by host – see module-classes-fleet-aggregation.md. per_host is a list of (host_name, classes) pairs, e.g. [(“localhost”, build_module_classes()), (“MONETS”, <that host’s own dict, fetched via the hub proxy>), …].

No collision arbitration: a same-named module on two hosts becomes two distinct rows, disambiguated by host, rather than one overwriting the other – the same choice merge_acl_matrices makes for ACL rows.

pyobs_core_version() → Version | None

Installed pyobs-core version in the same environment PYOBS_EXEC runs pyobs in (see _pip_exec) – the version actually producing the logs/behavior this app has to interpret, which can differ from host to host and drift out of step with pyobs-web-admin’s own release. None if pyobs-core isn’t listed there or its version string doesn’t parse.

Cached for _PYOBS_CORE_VERSION_CACHE_TTL: this sits on the journald log-fetch hot path (polled every few seconds per open browser tab), and list_pyobs_packages() shells out to pip each call – far too slow to redo on every request. Tests that need a fresh read should reset the module-level _pyobs_core_version_cache global directly.

resolve_and_validate_acl(name: str) → tuple[dict | None, str | None, str | None]

Like get_resolved_acl, but also validates the acl:’s shape (allow must be a mapping, deny must be a list) and catches any resolution error (bad YAML, broken {include}, …) into a returned message instead of raising. Returns (acl, source, error) – acl and source are None whenever error is set. Shared by build_acl_matrix (one row’s error shouldn’t abort the whole fleet-wide scan) and the single-module ACL tab endpoint (api_acl’s GET), which need the identical error-handling contract.

save_comm_password(user: str, new_password: str) → list[str]

Writes new_password into comm.password: for every local module whose comm.user resolves to user, splicing just that sub-key (_replace_comm_password). Returns the list of module names updated.

All-or-nothing: if any matching module’s comm: resolves to a shared fragment, raises before writing to any of them – a partial write (some modules updated, others left with a now-stale password) would be a worse outcome than not writing at all, exactly the risk ejabberd-user-management.md’s Design section calls out for a shared comm.user. If verification fails partway through (some files written, a later one doesn’t check out), rolls back every file this call itself wrote, mirroring save_local_acl’s safety net but extended across the whole matching set.

save_local_acl(name: str, acl: dict | None) → None

Writes a structured acl: edit (from the matrix’s per-target edit form) into a module’s own raw config file.

Splices just the acl: block into the raw text (_replace_local_acl_block) rather than doing a full YAML round-trip of the whole file, since the raw file can contain bare {include …} lines that aren’t valid standalone YAML on their own (see pyobs_config.pre_process_yaml) – a generic YAML parser can’t load it directly.

Refuses to write if the module’s acl: currently comes from a shared fragment; callers must route that edit to the fragment’s own file instead (get_resolved_acl’s source). After writing, re-resolves the module’s acl: and rolls back to the original content if it doesn’t match what was requested – seeing acl-matrix.md’s note on the splice’s simplifying assumption, this is the safety net against a silent bad write rather than trying to make the splice logic exhaustively correct up front.

stale_packages(running: dict[str, str], installed: dict[str, str]) → list[str]

Sorted distribution names where running (get_module_versions’ result for one module) disagrees with installed (from list_pyobs_packages) – “outdated” means “upgraded but not restarted” (running vs. installed), not running vs. latest-PyPI, which the Packages page already covers.

Iterates over running only, never installed: running is pyobs-core’s loaded_pyobs_packages(), which only includes what this module actually imported (its top-level name present in sys.modules – pyobs/utils/versions.py), while installed is the full host-wide pip list. A camera module loading pyobs-core+pyobs-fli has no reason to import pyobs-telescope or pyobs-iagvt just because they happen to be installed on the same host – unioning the two would flag every such unrelated package as “outdated” and make Restart-outdated restart modules that aren’t actually stale. A name missing from installed (a driver the process still has loaded that’s since been uninstalled) still counts as differing via the .get(n) on the installed side.

update_package_start(name: str, installed_version: str) → tuple[bool, str]

Starts pip install –upgrade for name detached from this request and returns immediately. See the module comment above for the on-disk job-state design.