Preventing Infinite Loops in Observers

Reference Prompt: @[content/lost-in-public/prompts/workflow/Write-an-Issue-Resolution-Breadcrumb.md]

Preventing Infinite Loops in Observers

Context

While developing the FileSystemObserver for frontmatter consistency in Markdown files, we encountered a critical bug: the observer would enter an infinite loop when processing files with malformed frontmatter—specifically, when a file contained multiple frontmatter sections (multiple --- delimiters). This caused the observer to repeatedly append new frontmatter blocks instead of replacing the existing one, leading to file corruption, high CPU usage, and a breakdown in the observer’s intended function.

Problem

  • Symptom:Files processed by the observer would end up with multiple YAML frontmatter sections, causing the observer to repeatedly trigger on the same file.
  • Root Cause:The observer did not correctly detect and handle malformed frontmatter blocks. It would append a new block rather than replacing or repairing the existing one, creating a feedback loop.

Solution

Detection

  • Implemented logic to detect when a file contains more than one frontmatter block (multiple --- delimiters).
  • Added robust parsing to extract only the first valid frontmatter section and ignore or repair any subsequent malformed sections.

Repair

  • When malformed frontmatter is detected, the observer reconstructs the file with a single, correct frontmatter block at the top, followed by the intended content.
  • This prevents the creation of duplicate or corrupted frontmatter and ensures the observer can process files idempotently.

Steps Taken

  1. Detection:Scanned files for multiple frontmatter sections using regular expressions and line-by-line parsing.
  2. Extraction:Extracted the first valid frontmatter block and the actual Markdown content, discarding any additional/malformed sections.
  3. Reconstruction:Rewrote the file with only one frontmatter block at the top.
  4. Testing:Created test cases with intentionally malformed files to ensure the observer could repair them without entering a loop.
  5. Logging:Added detailed logging to report when a file is repaired due to malformed frontmatter.

Reasoning

  • Idempotency: The observer must be able to process the same file multiple times without causing further corruption or triggering unnecessary updates.
  • Resilience: By repairing malformed files, we reduce the risk of future bugs and make the system more robust for all contributors.
  • Transparency: Logging repairs provides an audit trail for developers, making it easier to debug and maintain the system.

Lessons Learned

  • Always validate file structure before making modifications, especially in automated observers.
  • Infinite loops are often caused by feedback between file changes and file watchers—idempotent operations are essential.
  • Comprehensive logging and test cases are critical for catching and fixing these issues early.

References

  • Implementation location: tidyverse/observers/fileSystemObserver.ts
  • Map of relevant paths: /content/lost-in-public/rag-input/Map-of-Relevant-Paths.md
  • Prompt followed: @[content/lost-in-public/prompts/workflow/Write-an-Issue-Resolution-Breadcrumb.md]

Updated Solution: Atomic Property Collector Pattern for Observer Idempotency

Context: Why Infinite Loops Occur

Traditional observer implementations can enter infinite loops when file writes by the observer are detected as new changes, especially if serialization or frontmatter structure is inconsistent. This is compounded if subsystems/services are not clearly separated or if the observer itself mutates files in multiple stages.

New Process: Key-Value Property Collector with Expectation Management

To guarantee idempotency and prevent infinite loops, we have adopted a rigorous observer–service orchestration pattern:
  1. Full Extraction & Delegation
    • The observer extracts the complete frontmatter from the file and sends this to each subsystem/service/utility.
  2. Subsystem/Service Evaluation
    • Each subsystem independently evaluates whether it needs to act (e.g., is a UUID missing? Is OpenGraph data stale?).
    • Each subsystem returns a temporary expectation object (e.g., { expectSiteUUID: true } or { expectOpenGraph: true }) to the observer’s propertyCollector.
  3. Expectation Management
    • The propertyCollector maintains two in-memory structures:
      • Expectations: What key-value pairs are pending (e.g., awaiting API responses or sync operations)?
      • Working Frontmatter: The current state of frontmatter, to be updated only when all expectations are fulfilled.
  4. Subsystem Execution
    • If a subsystem needs to act:
      • Sync tasks (e.g., UUID generation): Immediately generate and return the new key-value pair (e.g., { site_uuid: "uuid-value" }).
      • Async/API tasks: Initiate the call(s), wait for all responses, then return the merged key-value results (e.g., { og_image: "...", og_title: "..." }).
    • Subsystems log their actions/results based on user-configurable flags.
    • The propertyCollector updates its state, removing fulfilled expectations as results arrive.
  5. Atomic Merge & Write
    • Once all expectations are fulfilled:
      1. The propertyCollector updates the date_modified field FIRST.
      2. The observer writes the merged frontmatter back to disk in a single operation.
      3. The observer stores the file path and timestamp in a temporary audit memory/state for traceability.
      4. The entire output is logged if enabled.
  6. Idempotency & Logging
    • No write occurs unless there are actual changes.
    • All logic is aggressively commented and logged for transparency and debugging.
    • This ensures the observer can process the same file repeatedly without triggering itself again—eliminating infinite loops.

Benefits

  • No Redundant Writes: Only changed key-value pairs are written, and only once per operation.
  • Clear Separation of Concerns: Subsystems/services are responsible for their own evaluation and execution logic.
  • Auditability: Every change, expectation, and fulfilled result is logged and can be traced.
  • Idempotency: The observer’s operations are repeatable and safe, even if run multiple times on the same file.
{{ ... }}