Major Markdown Pipeline Overhaul—Micromark-First, Proprietary Extensions, and Deep Technical Documentation
Major Markdown Pipeline Overhaul—Micromark-First, Proprietary Extensions, and Deep Technical Documentation
Summary of Changes
1. Technical Documentation
- Created:
How-Micromark-Handles-Markdown-AST.md
— In-depth, step-by-step technical guide to micromark's pipeline, extension points, and tokenization model.How-Remark-GFM-renders-Tables.md
— Exhaustive breakdown of how remark-gfm leverages micromark and mdast-util-gfm to parse/render tables, with guidance for independent reimplementation.
- Purpose:
- To serve as a single source of truth for future markdown architecture, debugging, and extension development.
2. Prompt for Contractors/Developers
- Created:
Our-Extended-Markdown-Requirements-as-a-Micromark-Extension.md
— Ultra-explicit, actionable prompt for building our entire extended markdown flavor as a set of micromark extensions, bypassing remark/unified entirely.
- Purpose:
- To enable direct, low-level, maintainable extension development and onboarding for new contributors.
3. Pipeline Philosophy and Spec Alignment
- Aligned:
- All new documentation and prompts reference and enforce the requirements in
Maintain-a-Proprietary-Extended-Markdown-Flavor-Rendering-Pipeline.md
. - Emphasized strict processing order, modular extension design, aggressive commenting, and error handling.
4. Project Rules and Constraints
- Reinforced:
- No new files outside assigned locations or naming conventions.
- All extension logic must be implemented as micromark constructs.
- TypeScript is preferred for all extension logic.
Rationale
- This overhaul is motivated by the need for full control, transparency, and extensibility in our markdown pipeline—removing unified/remark as a bottleneck and source of complexity.
- The new documentation and prompts will drastically accelerate onboarding, debugging, and future feature development.
Next Steps
- Begin implementation of proprietary micromark extensions per the new prompt and documentation.
- Use the technical guides as the foundation for all future markdown-related work.