System Logs

Ghaf’s logging pipeline collects journald and auditd records, aggregates them in the Admin VM, and can forward them to a remote backend. Transport protection, storage protection, retention, and log sealing address different security properties and are configured independently.

Logging Architecture

System VMs and Application VMs produce system and security events through systemd-journald. Auditd events enter the same local journal. Two independent paths consume these records:

Host or VM journald ----> Local logseald producer ----> Admin VM sealer
        |
        +---------------> Journal forwarding --------> Admin VM aggregator
                                                       |
                                                       +----> Remote backend

The sealing path creates tamper evidence at the source. The forwarding path supports aggregation, monitoring, and remote retention. An interruption on one path does not change the data format or service behavior of the other path.

Log producers and local collection

The log sources in Ghaf are systemd-journald and the Linux auditing subsystem, auditd. Together, they form the per-component record of operational activity and security-relevant events.

On the host and every logging-enabled VM, system services, kernel messages, and application/service output are collected by systemd-journald. In parallel, auditd captures security audit events based on the configured rule set and routes those events into the journal. This provides one local stream for persistence, sealing, and forwarding.

Each logging-enabled component also runs a logseald producer. The producer builds a cryptographically chained record of its local journal before the normal forwarding path aggregates those records. The Admin VM runs its own producer and the central sealer. See Clock-Independent Log Sealing for details.

Audit Policy and Rule Coverage

Audit collection uses layered rules. Each VM has Nix-specific, STIG-aligned, OSPP-aligned, and role-specific rules. The Ghaf Host has additional rules for its responsibilities and distinct attack surface.

Intra-Platform Log Forwarding

The logging-enabled host and non-admin VMs use systemd-journal-upload to send journals over HTTPS with GIVC mutual TLS. In Admin VM, ghaf-journal-receiver.service stores them in the shared native journal at /var/log/ghaf-journal/shared/remote.journal. Alloy reads this journal and the Admin VM’s local journal for forwarding to the remote backend.

Ghaf’s receiver patch accepts known certificate identities and replaces any sender-supplied _GHAF_SOURCE with the authenticated source name. Alloy uses this field for host and drops entries without a known source. Use these identities for attribution; _HOSTNAME is untrusted. A compromised sender can fabricate its own logs, but impersonating another VM requires credentials valid for that VM.

Shared Journal Retention

When enabled, ghaf.logging.journalRetention supplies the same maxDiskUsage, maxRetention, and MaxFileSec settings to local journals and the shared receiver. Both use a 100 MiB maximum file size. The receiver rotates and removes expired files internally, including during idle periods, without interrupting uploads.

The shared disk budget covers all forwarded sources and is separate from local journal budgets. A noisy VM can shorten retention for others. Cleanup operates on whole files, and active files can temporarily exceed the budget. Deleted records are unavailable for Alloy replay. Logseald seals local journals independently.

Secure Logging

Ghaf applies separate controls while logs are stored, forwarded, and retained across isolation boundaries. Integrity, retention, and confidentiality are complementary properties.

Integrity and tamper evidence: A logseald producer on every logging-enabled host and VM creates chained blocks from the local journal. The central sealer in the Admin VM authenticates the producer, validates the complete block, and signs the accepted chain position. This operates without relying on synchronized wall-clock time.
Retention (Journal and Audit): Retention policies are treated as a first-class concern and are applied in a tiered manner: (i) per-VM retention for local journal/audit logs, (ii) retention at the Admin VM for aggregated logs, and (iii) retention at the remote backend for long-term storage. Explicit retention boundaries help balance forensic usefulness with storage constraints and reduce exposure from over-retaining sensitive telemetry.
Confidentiality at rest (Log security / Data-at-rest encryption): Local journal and audit storage can be protected via data-at-rest encryption, limiting the impact of offline access or storage-layer compromise.
Confidentiality in transit: Protected channels cover the VM-to-Admin-VM path and the Admin-VM-to-remote-server path. They protect logs from passive observation and modification while the records cross trust boundaries.

Clock-jump recovery

When logging and ghaf.logging.recovery.enable are enabled, ghaf-clock-jump-watcher.service detects changes in wall-clock time. ghaf-journal-alloy-recover.service restarts installed log forwarders with an uptime-based cooldown. It does not restart journald or Fail2Ban, rotate journals, or wait for clock synchronization. Logseald seals independently of this recovery path. Journald’s built-in sealing is explicitly disabled with Seal=no.

Encryption in Transit and at Rest

Why encrypt logs

Logs are sensitive by design. They routinely contain process IDs, usernames, device identifiers, configuration snapshots, kernel error messages, and occasionally secrets from misconfigured software. Exposing these provides reconnaissance value and can aid privilege escalation.
Ghaf separates functionality across MicroVMs and hardened domains. Log streams frequently cross trust boundaries (for example, from an AppVM to a SysVM or host). Anything that crosses a boundary must assume a hostile transport.
Logs are part of the forensic record. Confidentiality controls limit who can read them; log sealing makes changes to retained evidence detectable. Neither property replaces the other.

Encryption in transit

Implemented with TLS/SSL-protected channels (for example, HTTPS endpoints, syslog over TLS, gRPC with mTLS, or SSH tunnels) between hosts and MicroVMs.
Protects confidentiality when logs traverse networks, virtual busses, or shared host paths between VMs.
Prevents tampering and spoofing by authenticating endpoints and enabling integrity checks, so attackers cannot inject or modify events in flight.
Aligns with Ghaf’s compartmentalization model (modules/microvm/*) and least-privilege service design: even if an intermediate domain is compromised, encrypted channels limit what can be observed or altered.

Encryption at rest

When storage encryption is enabled, whole-disk encryption protects host and VM storage. Store exported archives on encrypted partitions or object storage with server-side encryption.
Defends against offline access: if a device, VM image, or storage medium is stolen or inspected, encrypted log files remain unreadable without keys.
Limits blast radius of host or service compromises by requiring access to both storage and keys.
Complements tamper-evidence controls by protecting stored log content from unauthorized offline reading.
Complements kernel/host hardening profiles (see modules/common/profiles/kernel-hardening.nix) and auditing (modules/common/security/audit/*), which increase the value and sensitivity of what is logged.

Threats mitigated

Eavesdropping on inter‑VM communications or host paths used to collect/forward logs.
On‑path modification or injection of events to hide activity or create false trails.
Data harvesting from powered‑off devices, snapshots, or backups.
Abuse of excess privileges by non‑logging components that can access raw transport or storage.

Operational benefits

Compliance alignment with common security baselines that require protecting security telemetry at rest and in transit.
Higher signal quality for detection and faster, more confident incident response.
Consistent behavior across environments thanks to declarative, reproducible configuration (Nix‑based builds) and hardened systemd/AppArmor service sandboxes.