yozorajs
Yozora
See Yozora document (or https://yozorajs.github.io) for more details.
🎉 Why named "yozora" ?
Yozora is the Roman sound of Japanese 「よぞら」, taken from the lyrics in『花鳥風月』 by the band 世界の終わり.
This project is a monorepo that aims to implement a highly extensible, pluggable Markdown parser. Based on the idea of middlewares, the core algorithm @yozora/core-parser will schedule tokenizers (such as @yozora/tokenizer-autolink) to complete the parsing tasks. More accurately, yozora is an algorithm to parse Markdown or its extended syntax contents into an abstract syntax tree (AST).
✨ Features
🔖 Fully support all the rules mentioned in the GFM specification, and has passed almost all test cases created based on the examples in the specification (except the one https://github.github.com/gfm/#example-653, as there is no plan to support native HTML tags in the React Renderer, for the Yozora AST, so I'm a little lazy to do the tag filtering. If you need it, you can do the filtering by yourself).
See @yozora/parser-gfm or @yozora/parser-gfm-ex for further information.
🚀 Robust.
All codes are written in Typescript, with the guarantee of strictly static type checking.
Eslint and Prettier to constrain coding styles to avoid error-prone problems such as hack syntax and shadow variables.
Tested with Jest, and passed a large number of test cases.
💚 Tidy: No third-party dependencies.
⚡️ Efficient.
The parsing complexity is the length of source contents multiplied by the number of tokenizers, which has reached the lower bound of theoretical complexity.
The parser API supports streaming read-in (using generators /iterators for input), and supports parsing while read-in (Only block-level data is supported yet).
Carefully handle the array creation / concat operations. To reused the array as much as possible during the entire matching phase, only use the array index to delineate the matching range. And a lot of strategies applied to reduce duplicated matching / parsing operations.
🩹 Compatibility, the parsed syntax tree is compatible with the one defined in Mdast.
Even if some data types are not compatible in the future, it is easy to traverse the AST for adaptation and modification through the API provided in @yozora/ast-util.
🎨 Extendibility, Yozora comes with a plug-in system, which allowed Yozora to schedule the tokenizers through an internal algorithms to complete the parsing tasks.
- It's easy to create and integrate custom tokenizers.
All tokenizers can be mounted or unmounted freely.
Some tokenizers of the data types that not mentioned in GFM have been implemented in this repository, such as @yozora/tokenizer-admonition, @yozora/tokenizer-footnote, etc. All of them are built into @yozora/parser in default, you can uninstall them at will, if you don't like it.
Usage
@yozora/parser: (Recommended) A Markdown parser with rich built-in tokenizers.
import YozoraParser from '@yozora/parser' const parser = new YozoraParser() parser.parse('source content')@yozora/parser-gfm: A Markdown parser that supports GFM specification. Built-in tokenizers that supports all grammars mentioned in GFM specification (excluding the extended grammar mentioned in the specification, such as table).
import GfmParser from '@yozora/parser-gfm' const parser = new GfmParser() parser.parse('github flavor markdown contents')@yozora/parser-gfm-ex: A Markdown parser that supports GFM specification. Built-in tokenizers that supports all grammars mentioned in GFM specification (including the extended grammar mentioned in the specification, such as table).
import GfmExParser from '@yozora/parser-gfm-ex' const parser = new GfmExParser() parser.parse('github flavor markdown contents (with gfm extensions enabled)')Content AST into markup content
import { DefaultMarkupWeaver } from '@yozora/markup-weaver' const weaver = new DefaultMarkupWeaver() weaver.weave({ "type": "root", "children": [ { "type": "paragraph", "children": [ { "type": "text", "value": "emphasis: " }, { "type": "strong", "children": [ { "type": "text", "value": "foo \"" }, { "type": "emphasis", "children": [ { "type": "text", "value": "bar" } ] }, { "type": "text", "value": "\" foo" } ] } ] } ] }) // => emphasis: **foo "*bar*" foo**
Overview
Parsers
| Parser | Description | | :------------------------ | :--------------------------------------------------------------------------------------- | | @yozora/parser | A markdown parser with rich built-in tokenizers | | @yozora/parser-gfm | A markdown parser with built-in tokenizers to fully support GFM (without GFM extensions) | | @yozora/parser-gfm-ex | A markdown parser with built-in tokenizers to fully support GFM and GFM extensions |
Weavers
| Weaver | Description | | :------------------------ | :----------------------------- | | @yozora/markup-weaver | Weave AST into markup content. |
Tokenizers
| Tokenizer | Description | | :---------------------------------------- | :----------------------------------------------------------------------------------------------------------------- | | @yozora/tokenizer-admonition | Resolve admonitions | | @yozora/tokenizer-autolink | Resolve GFM Autolinks | | @yozora/tokenizer-autolink-extension | Resolve GFM Autolinks (extension) | | @yozora/tokenizer-blockquote | Resolve GFM blockquotes | | @yozora/tokenizer-break | Resolve GFM hard line breaks and GFM soft line breaks | | @yozora/tokenizer-definition | Resolve GFM link reference definitions | | @yozora/tokenizer-delete | Resolve GFM strikethrough (extension) | | @yozora/tokenizer-ecma-import | Resolve ECMAScript
importstatements | | @yozora/tokenizer-emphasis | Resolve GFM emphasis and strong emphasis | | @yozora/tokenizer-fenced-code | Resolve GFM fenced code blocks | | @yozora/tokenizer-footnote | Resolve footnotes | | @yozora/tokenizer-footnote-definition | Resolve footnote definitions | | @yozora/tokenizer-footnote-reference | Resolve footnote references | | @yozora/tokenizer-heading | Resolve GFM ATX headings | | @yozora/tokenizer-html-block | Resolve GFM HTML blocks | | @yozora/tokenizer-html-inline | Resolve GFM raw HTML | | @yozora/tokenizer-image | Resolve GFM images | | @yozora/tokenizer-image-reference | Resolve GFM reference images | | @yozora/tokenizer-indented-code | Resolve GFM indented code blocks | | @yozora/tokenizer-inline-code | Resolve GFM code spans | | @yozora/tokenizer-inline-math | Resolve inline formulas | | @yozora/tokenizer-link | Resolve GFM links | | @yozora/tokenizer-link-reference | Resolve GFM reference links | | @yozora/tokenizer-list | Resolve GFM lists (with GFM list items and GFM task list items) | | @yozora/tokenizer-math | Resolve block formulas | | @yozora/tokenizer-paragraph | Resolve GFM paragraphs | | @yozora/tokenizer-setext-heading | Resolve GFM setext headings | | @yozora/tokenizer-table | Resolve GFM tables | | @yozora/tokenizer-text | Resolve GFM textual contents | | @yozora/tokenizer-thematic-break | Resolve GFM thematic breaks |Utils
| Package | Description | | :------------------------- | :------------------------------------------------------------------- | | @yozora/ast | Yozora markdown ast types and constants | | @yozora/ast-util | Utility functions to handle Yozora markdown ast | | @yozora/character | Utility functions to handle characters encoded in ascii and unicode. | | @yozora/core-parser | Types and utility functions for building a Yozora Parser. | | @yozora/core-tokenizer | Types and utility functions for building a Yozora Tokenizer. | | @yozora/invariant | A simple invariant function |
Scaffolds
| Package | Description | | :----------------------------- | :------------------------------------------------------------------------------- | | @yozora/eslint-config | Eslint configs for yozora tokenizers | | @yozora/jest-for-tokenizer | Jest util for testing yozora tokenizers | | @yozora/template-tokenizer | Templates for creating a Yozora tokenizer (
InlineTokenizer/BlockTokenizer) |
💡 FAQ
How to use yozora with gatsby?
- Try the @yozora/gatsby-transformer and @yozora/gatsby-images
How to implemented custom tokenizer?
Use @yozora/template-tokenizer to create a custom tokenizer with predefined boilerplates.
Check @yozora/core-tokenizer for implementation details of tokenizer.
Check @yozora/jest-for-tokenizer for information about testing the custom tokenizer.
Check @yozora/core-parser and @yozora/parser for information on how to integrate a custom tokenzier.
It's also recommended to refer to the existing tokenizers implementation to create a custom one.
💬 Contact
📄 License
Yozora is MIT licensed.
Related
- ✨光和尘一直想要一个清爽博客: Why this project was written.
- @yozora/react-markdown: A library that renders Yozora AST into React components.
- @yozora/html-markdown: A library that renders Yozora AST into html strings.