Forward Secure Sealing

Forward Secure Sealing (FSS) provides cryptographic tamper-evidence for Ghaf audit logs. It uses HMAC-SHA256 chains to seal journal entries at regular intervals, ensuring any modification to sealed entries is detectable.

Overview

FSS addresses a critical security requirement: ensuring audit logs cannot be tampered with by attackers who gain root access. Traditional logging provides no cryptographic guarantees—an attacker can modify or delete entries without detection. FSS solves this by:

Cryptographic sealing: Journal entries are sealed with HMAC-SHA256 at configurable intervals
Forward security: The sealing key evolves forward; compromising the current key cannot forge past entries
Per-component isolation: Each Ghaf component (host and VMs) maintains independent FSS keys

Configuration

FSS is automatically enabled when ghaf.logging.enable = true. Configuration options:

ghaf.logging.fss = {
  enable = true;              # Enable FSS (default: follows ghaf.logging.enable)
  sealInterval = "15min";     # Interval between seals (default: 15min)
  verifyOnBoot = true;        # Run verification on boot (default: true)
  verifySchedule = "hourly";  # Periodic verification schedule (default: hourly)
};

Seal Interval Considerations

Interval	Tamper Granularity	Storage Overhead	Verification Time
5min	Fine	Higher (~2%)	Longer
15min	Moderate	Moderate (~0.5%)	Moderate
1h	Coarse	Lower (~0.1%)	Faster

Key Management

Key Locations

Component	Sealing Key	Verification Key
Host	`/var/log/journal/<id>/fss`	`/persist/common/journal-fss/ghaf-host/verification-key`
VMs	`/var/log/journal/<id>/fss`	`/etc/common/journal-fss/<vm-name>/verification-key`

On the host, /var/log/journal is bind-mounted from /persist/var/log/journal when FSS is enabled. In VMs, FSS is enabled only when StorageVM-backed journal preservation is available so the journal directory and journald sealing key survive reboot.

Verification Key Backup

Critical: Back up verification keys to secure offline storage immediately after first boot.

# On host, copy all verification keys
cp -r /persist/common/journal-fss/ /secure-backup/

# Or for a specific component
cat /persist/common/journal-fss/ghaf-host/verification-key

The verification key enables:

Independent verification of exported journal archives
Audit by external parties without access to the live system
Disaster recovery verification

Lifecycle Receipts

Ghaf FSS verification distinguishes active integrity failures from expected journald lifecycle artifacts using content-bound receipts. A receipt is a small TSV record written when FSS setup or recovery creates an archive that may later fail plain journalctl --verify even though it is not evidence of current active log corruption.

Receipt schema:

v1 <path> <inode> <size> <boot_id> <mtime> <sha256> <reason> <event_id>

Fields:

Field	Meaning
`v1`	Receipt schema version
`path`	Journal archive path recorded at the lifecycle event
`inode`	Inode observed when the receipt was created
`size`	Size observed when the receipt was created
`boot_id`	Boot ID that produced the receipt
`mtime`	Archive modification time observed at receipt creation
`sha256`	Content hash used for verification-time identity matching
`reason`	Lifecycle reason, such as `pre-activation-rotation`, recovery, or `unclean-shutdown`
`event_id`	Event identifier tying receipts from the same operation together

Receipt classes:

Class	Producer	Verification behavior
Pre-activation archive	`journal-fss-setup.service`	Current-boot receipts become verified exceptions; earlier boot receipts warn
Recovery archive	`ghaf-journal-alloy-recover`	Current-boot receipts become verified exceptions; earlier boot receipts warn
Unclean shutdown	journald recovery plus FSS setup	Matching `.journal~` archives become verified exceptions or warnings depending on boot

Validation is fail-closed:

A matching on-disk archive must have the recorded SHA256 content hash.
A present archive with different content is a receipt mismatch and fails verification.
A missing archive is tolerated because journald retention may legitimately delete old archives.
A receipt never exempts the live system.journal; active system journal verification failures remain critical.

Operations

Manual Verification

# Basic verification (uses local key)
journalctl --verify

# Verify with explicit key
journalctl --verify --verify-key=$(cat /persist/common/journal-fss/ghaf-host/verification-key)

Service Status

# Check FSS setup status
systemctl status journal-fss-setup

# Check verification timer
systemctl list-timers journal-fss-verify

# View FSS-related logs
journalctl -t journal-fss

Hardware Testing

A test script is available for deployed systems:

# Build the test
nix build .#checks.x86_64-linux.fss-test

# Deploy and run
scp result/bin/fss-test root@ghaf-host:/tmp/
ssh root@ghaf-host /tmp/fss-test

The test verifies:

FSS setup service completed
Sealing key exists
Verification key extracted
Initialization sentinel present
Journal integrity passes
Verification timer active
Audit rules configured

Key Rotation

Key rotation destroys the existing tamper-evidence chain. Only rotate when necessary:

# 1. Archive and verify existing journals
journalctl --verify
journalctl -o export > /backup/journal-archive-$(date +%Y%m%d).export

# 2. Remove initialization sentinel
rm /persist/common/journal-fss/ghaf-host/initialized

# 3. Clear sealing key (optional, regenerated on boot)
rm /var/log/journal/*/fss

# 4. Reboot to regenerate keys
reboot

# 5. Backup new verification key
cat /persist/common/journal-fss/ghaf-host/verification-key > /secure-backup/

Offline Verification

To verify journals on a separate system:

# On the Ghaf system: export journals
journalctl -o export > journal.export

# Transfer journal.export and verification-key to verification system

# On verification system: verify the export
journalctl --verify --verify-key=<verification-key> --file=journal.export

Troubleshooting

Verification Fails with “FAIL”

Indicates an integrity or corruption issue:

# Check which files failed
journalctl --verify 2>&1 | grep FAIL

# If only .journal~ files fail, these are temp files (not critical).
# If a recorded bootstrap/recovery system@*.journal archive fails while
# system.journal passes, fss-test and journal-fss-verify treat FSS as healthy.
# If user-*.journal files fail while system.journal passes, treat this as a
# warning and investigate journal rotation/recovery timing.
# Active system.journal failures remain critical.

“Failed to parse seed” Error

The verification key is malformed or unavailable:

# Check key format (should be ~35 bytes with '/' separator)
cat /persist/common/journal-fss/ghaf-host/verification-key | wc -c

# Regenerate if necessary (see Key Rotation above)

Service Not Starting

# Check conditions
systemctl show journal-fss-setup --property=ConditionResult

# Common issues:
# - /var/log/journal not writable (check permissions)
# - Already initialized (check for initialized sentinel)

Security Considerations

What FSS Protects Against

Modification of sealed log entries
Insertion of false entries into sealed ranges
Backdating of events

What FSS Does Not Protect Against

Deletion of entire journal files (mitigated by remote log forwarding)
Real-time tampering before sealing (mitigated by short seal intervals)
Denial of service against logging
Local-root forgery of the on-host verification verdict. The setup and verify services trust unauthenticated plaintext “receipt” and activation-state files under /var/log/journal/<machine-id>/ (root-owned, mode 0644). A local-root attacker who tampers a sealed archive can append a matching receipt line (its recorded sha256 equals the tampered content) to convert the on-host verdict from AUDIT_LOG_INTEGRITY_FAIL into a silent pass. Authoritative tamper detection therefore requires offline journalctl --verify --verify-key= against an exported archive using an off-host verification-key backup (see Offline Verification), which this attack cannot defeat.
The per-boot pre-activation window. Sealing is now activated at runtime, after the clock-readiness barrier, rather than statically at journald start, so entries logged before activation completes on each boot are collected but not yet FSS-trusted.

Audit Integration

FSS integrates with Ghaf’s audit subsystem. The following events are monitored:

journal_fss_keys: Write/attribute changes to FSS key directory
journal_sealed_logs: Write/attribute changes to journal files
machine_id_read: Reads of machine-id (used in journal path)