Fire up the Observer

USER GOAL:

Reintroduce the tooling collection to the observer system by implementing a watcher.

STEPS:

  1. Read the rest of this file for the context window.
  2. Read previous implementations in site_archive/observers. These were archived because of infinite loops and frontmatter corruption that was too difficult to isolate and fix. At points, they were working for the tooling collection, but misbehavior was rampant for the other collections.
  3. Suggest the best way to implement a watcher for the tooling collection.
  4. Implement the plan.
  5. Test the implementation AND ONLY the "tooling" collection subsystem on a single subdirectory of the tooling collection, the path to be set by the User.
  6. Run all the subsystems at once to see if they work together.

The Observer System

This document provides instructions for augmenting, improving, or running the filesystem observer to process new files within specified directories that contain a unique content collection for rendering on the website.

Previous Mistakes:

  • Infinite Loops: The observer was triggering itself, leading to an infinite loop. This was resolved by adding a universal "propertyCollector" in the main observer, which then delegates all tasks to the appropriate watchers, handlers, services, and utils. The propertyCollector immediately receives an "expectation" object, which is a list of properties that will be created or modified by the watchers, handlers, services, and utils. The propertyCollector also includes a cooldown period and an in memory log of files that have been processed.
  • Random Frontmatter Corruption: When introducing new functionality to the observer system, functionality that had already been honed and was error free would be duplicated for convenience of Code Generation and reasoning within one file. But, the use of regular expressions and text manipulation to extract, evaluate, generate and update frontmatter created inconsistencies, and sometimes glaring errors that corrupted entire directories of content, which then needed to be fixed. We have MAINLY solved for this by using DRY principles and keeping extraction, evaluation, and update logic in a single-source-of-truth that is used by all watchers, handlers, services, and utils. IT IS CRITICAL TO FOLLOW THIS PRINCIPLE AND NOT UNNECESSARILY DUPLICATE FRONTMATTER TRANSFORMATION LOGIC ACROSS FILES.

Integrated Observer System

Independent Watchers for Collections

tidyverse/observers/watchers We decided to avoid repeating a monolithic FileSystemObserver that handles all collections. There were several key moments where one collection was being corrupted by the monolithic observer, and it was not possible to isolate the issue. We would then need to move the fileSystemObserver file to the archive, and start from scratch.
So, instead, we will have independent watchers for each collection. This involves repeating a number of functions across separate watcher files, but it is a necessary evil to ensure that each collection is processed correctly. And if there is a glitch for one collection, the remaining watchers can stay active in the observer.

List of Watchers

  • tidyverse/observers/watchers/conceptsWatcher.ts
  • tidyverse/observers/watchers/vocabularyWatcher.ts
  • tidyverse/observers/watchers/essaysWatcher.ts
  • tidyverse/observers/watchers/remindersWatcher.ts
  • tidyverse/observers/watchers/toolkitsWatcher.ts
  • tidyverse/observers/watchers/issueResolutionWatcher.ts

List of Templates

  • tidyverse/observers/templates/concepts.ts
  • tidyverse/observers/templates/vocabulary.ts
  • tidyverse/observers/templates/essays.ts
  • tidyverse/observers/templates/reminders.ts
  • tidyverse/observers/templates/prompts.ts
  • tidyverse/observers/templates/specifications.ts
  • tidyverse/observers/templates/issue-resolution.ts
  • tidyverse/observers/templates/tooling.ts

List of Handlers

  • tidyverse/observers/handlers/addSiteUUID.ts
  • tidyverse/observers/handlers/processOpenGraphMetadata.ts
  • tidyverse/observers/handlers/processScreenshotMetadata.ts

List of Services

  • tidyverse/observers/services/openGraphService.ts
  • tidyverse/observers/services/screenshotService.ts

List of Utils

  • tidyverse/observers/utils/extractStringValueForFrontmatter.ts
  • tidyverse/observers/utils/yamlFrontmatter.ts

List of User Options

  • tidyverse/observers/userOptionsConfig.ts

Unique Collections or Observers

Tooling and Toolkit

While other collections are rendered as articles in content layouts, the "toolCollection" or "toolingCollection" is rendered as a Card Crid, with visually rich cards displaying the tools and their metadata. Therefore, we use third party APIs to generate screenshots and OpenGraph images for the tools, which are accessed through our services and handlers.
Found in: content/tooling

Citation Processing

We have done one isolated run of a Citation Processor, and it works but the resulting format may not be optimal. We need to evaluate the output and make product decisions before we can integrate it into the core FileSystemObserver.

Running the Observer

To run the observer and process both citations and frontmatter:
bash 
cd /Users/mpstaton/code/lossless-monorepo/tidyverse/observers
pnpm start
This will run the start script defined in package.json, which executes ts-node index.ts.
You can also specify a custom content root directory as an argument:
bash 
pnpm start -- ../../content
This will:
  1. Start the FileSystemObserver
  2. Watch for file changes in configured directories
  3. Process citations in markdown files (convert numeric to hex)
  4. Validate and update frontmatter
  5. Generate reports periodically

How Citation Processing Works

When the observer detects a new or modified markdown file:
  1. First, it processes citations using the processCitationsInFile method
    • Converts numeric citations to hex format
    • Updates the citation registry
    • Adds footnotes section if needed
  2. Then, it processes frontmatter in a separate operation
    • Validates against templates
    • Adds missing required fields
    • Normalizes property names

Monitoring Results

The observer will output logs to the console showing:
  • Files being processed
  • Citations being converted
  • Frontmatter being updated
  • Any errors or warnings
Reports are generated in the content/reports directory every 5 minutes and when the observer is shut down.