Major Markdown Pipeline Overhaul—Micromark-First, Proprietary Extensions, and Deep Technical Documentation

Major Markdown Pipeline Overhaul—Micromark-First, Proprietary Extensions, and Deep Technical Documentation

Summary of Changes

1. Technical Documentation

  • Created:
    • How-Micromark-Handles-Markdown-AST.md — In-depth, step-by-step technical guide to micromark's pipeline, extension points, and tokenization model.
    • How-Remark-GFM-renders-Tables.md — Exhaustive breakdown of how remark-gfm leverages micromark and mdast-util-gfm to parse/render tables, with guidance for independent reimplementation.
  • Purpose:
    • To serve as a single source of truth for future markdown architecture, debugging, and extension development.

2. Prompt for Contractors/Developers

  • Created:
    • Our-Extended-Markdown-Requirements-as-a-Micromark-Extension.md — Ultra-explicit, actionable prompt for building our entire extended markdown flavor as a set of micromark extensions, bypassing remark/unified entirely.
  • Purpose:
    • To enable direct, low-level, maintainable extension development and onboarding for new contributors.

3. Pipeline Philosophy and Spec Alignment

  • Aligned:
    • All new documentation and prompts reference and enforce the requirements in Maintain-a-Proprietary-Extended-Markdown-Flavor-Rendering-Pipeline.md.
    • Emphasized strict processing order, modular extension design, aggressive commenting, and error handling.

4. Project Rules and Constraints

  • Reinforced:
    • No new files outside assigned locations or naming conventions.
    • All extension logic must be implemented as micromark constructs.
    • TypeScript is preferred for all extension logic.

Rationale

  • This overhaul is motivated by the need for full control, transparency, and extensibility in our markdown pipeline—removing unified/remark as a bottleneck and source of complexity.
  • The new documentation and prompts will drastically accelerate onboarding, debugging, and future feature development.

Next Steps

  • Begin implementation of proprietary micromark extensions per the new prompt and documentation.
  • Use the technical guides as the foundation for all future markdown-related work.