Skip to content

HTML: parse a page, select from it, read it and its styles, turn it into Markdown

shotcup:html reads HTML the way a browser does. parse(source) takes the HTML of a page or a piece of one and returns its tree as plain objects: the document, its doctype, elements with their attributes, text, comments. It is written in native code, follows the WHATWG HTML Standard's parsing rules to the letter (every tokenizer state and insertion mode, the repair of misnested tags, tables and templates, SVG and MathML inside HTML), and is fast enough to read a 430 KB page inside a Free run: that page costs about 0.4 million of the 5 million instructions a Free run has. toMarkdown turns a page, or any node of a tree, into CommonMark with GitHub tables, task lists and strikethrough (below). The module needs no engine switch.

JavaScript
import * as html from "shotcup:html";

import fs from "fs";

const doc = html.parse(fs.readFileSync("/page.html", "utf8"));
console.log(doc.mode, doc.children.length);

parse does not fetch, run scripts or load anything: the page's scripts are text, a noscript element's content is read as markup (scripting is off), and the tree has no links to anywhere. Markup that is not valid is repaired as browsers repair it and never throws: <p>One<p>Two is two paragraphs, <b><i>x</b>y</i> reopens the i after the b, a stray <td> outside a table is dropped. A source that is not a string is a TypeError; the limits below are RangeErrors you can catch.

The tree

Every node is an object with a type. The tree is read-only: its objects and arrays are frozen, so assigning, adding, deleting, push, splice and the like throw a TypeError (only Object.defineProperty on an element of an array is not stopped); build a new tree to change one. JSON.stringify of it is the document.

typeProperties
documentmode ("no-quirks", "limited-quirks" or "quirks", from the doctype), children
doctypename, publicId, systemId (empty strings when the doctype has none)
elementnamespace ("html", "svg" or "mathml"), name, attributes, children; a template also has content, { type: "fragment", children }, which holds what is inside it
textvalue
commentvalue
processing-instructiontarget, value

An attribute is { name, value }, in source order, with the first of two attributes of one name kept as browsers keep it. name is the name the element has: lowercase in HTML, viewBox and foreignObject in SVG as the standard cases them, and a name with a prefix on the attributes the standard puts in a namespace (xlink:href, xml:lang, xmlns), which also have namespace: "xlink", "xml" or "xmlns". Text is merged as the DOM merges it: the characters between two tags are one text node, and a document has no text children.

JavaScript
// every link of a page with its text
const links = [];
const walk = (node) => {
  if (node.type === "element" && node.namespace === "html" && node.name === "a") {
    const href = node.attributes.find((a) => a.name === "href")?.value;
    if (href) links.push({ href, node });
  }
  for (const child of node.children ?? []) walk(child);
  if (node.content) for (const child of node.content.children) walk(child);
};
walk(doc);

A node has no parent and no helper methods: a walk carries the path it needs. The tree is the same shape for every node kind that has children, so one recursive function reads all of it. Or select the nodes you want:

Selecting nodes

select(tree, selector) returns the first element below tree that a CSS selector matches (or null), selectAll(tree, selector) all of them as an array in document order, matches(node, selector) says whether an element matches, and closest(node, selector) returns the element itself or its nearest ancestor that does. The nodes you get are the objects of your tree, so selectAll(doc, "a")[0] === doc.children[...] is true and a node can be handed to text, attr or another select.

JavaScript
import * as html from "shotcup:html";

const doc = html.parse(page);
for (const row of html.selectAll(doc, "table.prices tbody tr:not(.total)")) {
  const [name, price] = html.selectAll(row, "td").map((cell) => html.text(cell));
  console.log(name, price);
}

tree is a document, an element or a fragment of a tree parse returned (or one a script builds of the same objects). A page is read into a native index once per run and kept, so a loop of selects over the rows of a page costs one read; and a row selected from a page is matched with the whole page around it, as a browser does: select(row, ".prices td") sees the table above the row. The index is kept (the three trees used last) for trees that are frozen and plain, as everything parse returns is: objects whose properties are their own values, with no getters and no prototype but Object.prototype. A tree you build and change, or one with getters, is read again for each call, and a row of a page that is no longer kept is matched with the tree it hangs from in your object, not with a page it is not in. A kept tree is a snapshot: changing a frozen parse tree through Object.defineProperty (the one change that freezing leaves open on an array element) is not seen by the selectors until the tree is read again.

Selectors are Selectors Level 4, parsed with the CSS Syntax Level 3 rules (so escapes, strings and comments are as in a style sheet):

FamilySupported
Simpletype, *, .class, #id, the namespace prefixes `* and (no other prefix is declared), & (the same as :scope`)
Attributes[a] [a=b] [a~=b] `[a=b] [a^=b] [a$=b] [a*=b], quoted or not, with the i and s flags; the values the HTML Standard matches without regard to case (type, lang, rel`...) do so
Combinatorsdescendant, >, +, ~
Logical:is(), :where() (an invalid or empty entry is dropped, as the standard says), :not(), :has() with relative selectors (:has(> img), :has(+ p); not inside itself)
Structure:root :empty :first-child :last-child :only-child :first-of-type :last-of-type :only-of-type :nth-child() :nth-last-child() (also 2n+1 of S) :nth-of-type() :nth-last-of-type() :scope :heading :heading(1, 2)
Language and direction:lang() (extended filtering, quoted ranges too; the language is the nearest lang of an HTML element or xml:lang of an SVG or MathML one, else the page's <meta http-equiv="content-language">; :lang("") matches what has none), :dir(ltr) and :dir(rtl) (an HTML element's direction is the one its dir attribute gives: ltr, rtl, or auto, the first strong character of the text under it; else its parent's, left to right at the root)
Links and forms:link :any-link :checked :default :disabled :enabled :required :optional :read-only :read-write :placeholder-shown :indeterminate :open :defined, as the markup states them: :checked follows the standard's checkedness and selectedness (the last checked radio button of a group, the option a select keeps selected), :disabled counts a disabled fieldset and an option or optgroup of a disabled select, :required and :optional only where required applies
Never match:hover :active :focus :focus-visible :focus-within :visited :target :target-within :local-link and the media, fullscreen and validity states: nothing interacts with a page here, so they parse and match nothing; a pseudo-element (::before, ::slotted(.a), ::part(label)) parses and matches no element, and so do :host, :host(.a) and :host-context(.a), which match only from a shadow tree's own style sheet; the argument of :host(), :host-context() and ::slotted() is one compound selector and that of ::part() one or more names, anything else a SyntaxError

:empty is as in browsers: an element with a text node of any length, white space included, is not empty (the editor's draft of the standard would let white space through; no browser does). Anything else is a SyntaxError that names the selector and the position. Matching follows the standard: HTML names are matched without regard to case, SVG and MathML names with it; class names and ids ignore case in a quirks-mode document; a template's content is not looked into; :root is the document's element.

Reading a page

CallGives
text(node)what a reader sees, as a browser's innerText gives it: white space collapsed, blocks and <br> on their own lines, paragraphs a blank line apart, table cells separated by a tab and rows by a line, <pre> kept; script, style, template, head, iframe, noembed, a closed dialog, an SVG title, hidden elements and hidden inputs left out. Only the user agent's style sheet decides: the page's own styles are not read, a tfoot is not moved below the body, and a closed details shows its content. The text of an element inside a pre keeps its white space, and that of one inside a hidden element is its textContent
textContent(node)every text node below the node, as the DOM gives them
attr(node, name)an attribute's value (the name in any case for an HTML element), or null
links(tree, { baseUrl }?)[{ href, text, rel, title, node }] for each a and area with an href, the address resolved against the page's <base href> (the first one) and the baseUrl you give for the page's own address
images(tree, { baseUrl }?)[{ src, alt, srcset: [{ src, descriptor }], width, height, node }] for each img, addresses resolved the same way
tables(tree)[{ caption, header, rows, node }] for each table, nested ones too: colspan and rowspan expanded (each slot holds its cell's text), short rows padded, header the last row of a thead or a first row of th cells (else null), rows the others
jsonLd(tree)the parsed JSON of each <script type="application/ld+json"> (an array is spread; a block that is not JSON is skipped)
meta(tree, { baseUrl }?){ title, description, canonical, lang, charset, keywords, author, robots, viewport, themeColor, generator, openGraph, twitter, other }: the first of each tag, Open Graph and Twitter tags by their names without the prefix (a repeated key is an array), every other meta by name
forms(tree, { baseUrl }?)[{ id, name, action, method, enctype, fields, node }], each field { tag, type, name, value, checked, disabled, required, node } (a select also options); read-only: nothing is submitted

An address that is already absolute is read as the URL Standard reads it, with or without a base: the scheme in lower case (the host too for http, https, ftp, ws, wss and file), the scheme's own port and leading zeros in a port dropped (HTTP://Example.COM:0080/z is http://example.com/z). A relative address that cannot be made absolute (no baseUrl and a relative <base href>) comes back as the page wrote it. Scraping puts these together: fetch a page, select from it and write CSV.

Styles: getComputedStyle

getComputedStyle(tree, node, pseudoElement?, options?) answers what a browser's getComputedStyle answers for an element of the tree: the value of every CSS property after the page's style sheets, the user agent's and the element's own style attribute have been applied, inherited and computed. The tree comes first, as in the other calls that read a page, because the page's style sheets are the tree's and a node of the tree has no link to its parents.

JavaScript
import * as html from "shotcup:html";

const doc = html.parse(page);
const button = html.select(doc, "button.primary");
const style = html.getComputedStyle(doc, button);
console.log(style.color, style.backgroundColor, style.getPropertyValue("--brand"));
console.log(html.getComputedStyle(doc, button, "::before").content);

The answer is a frozen object shaped like a CSSStyleDeclaration:

MemberGives
style.color, style.backgroundColor, style["background-color"]every property by its CSS name and its camel-cased name (webkitTextStroke and WebkitTextStroke for the -webkit- ones, cssFloat for float), shorthands included
style["--name"]the element's custom properties, its own and inherited
style.length, style[0], style.item(i)the names a browser enumerates: every longhand, in alphabetical order with the -webkit- ones last, then the custom properties
style.getPropertyValue(name)a value by its CSS name (any case; a custom property as written); "" for a name that is not a property
style.getPropertyPriority(name), style.cssText, style.parentRule"", "" and null, as for any computed style

Object.keys and JSON.stringify see the names and values; a write is a TypeError.

What applies: the HTML Standard's user-agent style sheet, the presentational attributes (bgcolor, width, align...), the page's <style> elements and style attributes, and declarative shadow roots (<template shadowrootmode>, with :host and ::slotted(); reach the elements inside one through the template's content). A <link rel="stylesheet"> and an @import are not fetched: pass the CSS as options.css, which is added after the page's own sheets. In a tree without a document (a fragment, or what fromElement returns) each top-level element is styled as the root.

Options: width and height (the viewport media queries and vw and vh see, in CSS pixels; 800 and 600 when not given), media ("screen" or "print"), colorScheme ("light" or "dark", for prefers-color-scheme and light-dark()) and css.

Pseudo-elements: ::before, ::after, ::marker, ::first-line, ::first-letter (also with one colon), ::placeholder, ::selection, ::file-selector-button, ::backdrop, ::target-text, ::spelling-error, ::grammar-error, ::checkmark and ::picker-icon. Another one gives an empty declaration, and a string that does not start with a colon is ignored, as in a browser.

Values are a browser's for an element that is not laid out: colours as rgb() (or the colour's own space, oklch(), color(display-p3 ...)), lengths in pixels, line-height as pixels, positions as percentages, and min-width: auto as 0px except for a flex or grid item. Nothing is laid out yet, so width, height, the margins, the paddings and the insets are their computed values (auto, 50%), not the pixel sizes a browser gives for an element on the screen, and @container size queries do not match. A shorthand is written from its longhands as Chromium writes it (margin: 16px 2px, border-top: 2px solid and the colour, font: italic 12px / 24px serif), and is "" when the longhands cannot be written as one (four different border sides).

Errors and limits: a tree, node or option of the wrong kind, a node that is not in the tree or not an element are TypeErrors. The page's style sheets, the user agent's and options.css count against the plan's CSS limits (the table below); a sheet past them is a RangeError. The styles of the whole tree are computed at the first call and kept with the tree for the same options, so the calls after it cost little: on the 478 KB shotcup home page (2,638 elements), parse costs about 360,000 of a Free run's 5 million instructions, the first getComputedStyle about 250,000 more and each one after it about 3,200.

React components: fromElement

A card, a slide or a report is easiest to write as components. fromElement reads an element tree, the objects React, Preact and tools such as Satori build ({ type, props }), and returns the same tree parse returns, so toMarkdown, select, selectAll, text, links and tables work on it, and so will the renderer. No React is needed: write the objects by hand, with a three-line h, or bring JSX (the engine compiles it). The Social card from components, no React template is a JSX card read this way.

JavaScript
import * as html from "shotcup:html";

const h = (type, props, ...children) => ({ type, props: children.length ? { ...props, children } : { ...props } });
const Stat = ({ label, value }) =>
  h("div", { style: { display: "flex", padding: 8, fontSize: 14 } }, h("b", null, label), " ", value);
const tree = html.fromElement(
  h("section", { className: "card" }, h("h1", null, "Report"), h(Stat, { label: "Runs", value: 12 })),
);
// { type: "fragment", children: [ <section class="card">, ... ] }
console.log(html.toMarkdown(tree));              // "# Report

**Runs** 12"
console.log(html.select(tree, "b").children[0]); // { type: "text", value: "Runs" }
console.log(html.toMarkdown(h(Stat, { label: "A", value: 1 }))); // an element works without the step

fromElement(element) takes anything a component may return: an element, a string, a number, an array of them, a fragment, true, false, null and undefined (the last four are nothing). It returns { type: "fragment", children }, frozen like parse's trees; adjacent texts are one text node. toMarkdown also takes an element directly. The other functions take a tree: passing them an element is a TypeError that names fromElement, because select returns the very objects of the tree it was given, and an element has no tree until it is read.

Elements. The element's type is a string (an HTML, SVG or MathML element), a function component (called with its props object, children included, and what it returns is read in its place; a component may return an element, an array, a string or null), a fragment, or React's memo and forwardRef wrapping one. Real React and Preact elements are read as they are ($$typeof is recognised, the key and ref of an element are not attributes). Hooks, class components, context, lazy components and async components are refused with a TypeError whose message says so and tells you to render the component on the server: they need React itself. Render them on the server and read the HTML (react-dom/server's renderToStaticMarkup or preact-render-to-string, as the React server-rendered document and Preact document templates do) with html.parse(markup).

Attributes are the ones react-dom/server prints, and the tree is the one parse makes of that markup:

PropResult
className, htmlFor, tabIndex, acceptCharset, httpEquiv, crossOriginclass, for, tabindex, accept-charset, http-equiv, crossorigin
style (an object)one style attribute, name:value;name:value: camelCase to kebab-case (WebkitTransition is -webkit-transition, msFlex is -ms-flex), a custom property (--accent) and its value kept, a number is px except 0 and the unitless names (opacity, lineHeight, zIndex, flexGrow, fontWeight, ...), a string is trimmed, null, undefined, booleans and "" are skipped; a string is kept as written, as Preact prints one; anything else is a TypeError
SVG: strokeWidth, fillOpacity, xlinkHref, viewBox, gradientUnitsstroke-width, fill-opacity, xlink:href (in the xlink namespace), viewBox and the other names the standard cases keep their case
disabled, hidden, checked, open, required, ..."" when true, nothing when false; download and capture accept true or a string
data-*, aria-*a boolean prints "true" or "false"; contentEditable, draggable, spellCheck and value print booleans the same way
onClick and every name starting on, ref, functions, Symbols, null, undefined, a boolean for any other nameno attribute
href, src, action, formAction, xlinkHref starting javascript:React's blocked-URL text, as in react-dom
dangerouslySetInnerHTML: { __html }the string parsed as HTML in the element's own context; its nodes are the element's children
a name that is not an attribute name (a space, a quote, <)no attribute: a prop can never end the tag

An HTML element's attribute names are lowercased and an SVG element's are spelled as the standard spells them, as the parser does. <input> prints value and checked (or defaultValue and defaultChecked) after its other attributes, a <textarea>'s value, defaultValue or child text is its text, <option selected> prints selected. A void element (br, img, input, ...) with children, an element name that is not a letter followed by letters, digits and :_.-, an object, a function or a Symbol as a child are a TypeError with the fix. The children of foreignObject are HTML again, those of svg and math elements their namespace.

What is not the same as react-dom/server: it moves title, meta, link, style and script into the head and adds preload links for images, select takes a value (put selected on the option), and the parser repairs markup that is not valid (a p inside a p, table rows without a tbody) where fromElement builds what the objects say. Escaping needs no care: a text or an attribute value is a value of the tree, not markup, and what a prop holds can never become a tag or another attribute.

Limits are parse's, from the table below: the elements, texts and components called count as nodes, the nesting of elements, arrays and components together is the nesting limit, the props of an element or a style the attribute limit, and the bytes of names, texts, attribute values and style strings the source limit. A cap is a RangeError. A card of 1,000 elements, written as 250 components, costs about 150,000 of a Free run's 5 million instructions as a tree and 74,000 when toMarkdown reads it directly.

Limits

A source or a tree past a limit is a RangeError before anything is returned. The limits belong to your plan:

PlanSourceNodesNestingAttributes on one tag
Free1 MiB30,000512256
Pro8 MiB250,0005121,024
Scale8 MiB250,0005121,024
Platform8 MiB250,0005121,024

A selector has its own limits, and a tree you build is limited like one parse returns (the nodes, the nesting and the attributes above):

PlanSelector sizeSelector textNesting
Free5121 MiB128
Pro4,0968 MiB128
Scale4,0968 MiB128
Platform4,0968 MiB128

The style sheets getComputedStyle reads (the user agent's, the page's, the style attributes and options.css) have their own limits:

PlanOne sheetTokensRulesDeclarationsOne var()All var() of an element
Free1 MiB150,00010,00030,0000.0625 MiB1 MiB
Pro8 MiB1,500,000100,000300,0001 MiB8 MiB
Scale8 MiB1,500,000100,000300,0001 MiB8 MiB
Platform8 MiB1,500,000100,000300,0001 MiB8 MiB

Past one of them is a RangeError, except a single var() past its limit, which makes that value invalid, as a cycle does. The cascade's memory comes out of the run's and its work is charged to the run's instructions.

The size of a selector counts its compound selectors, simple selectors, combinators and pseudo-class arguments together (a b costs 5: two compounds, two type selectors and one combinator); the text counts in UTF-8 bytes. The nesting here is maxCssDepth for brackets in the text, and pseudo-classes whose arguments are selectors (:is, :where, :not, :has, :nth-child(... of S)) nest at most 24 deep on every plan.

The source counts in UTF-8 bytes. Nodes are elements, text, comments, the doctype and processing instructions (text that browsers merge is one node). Nesting is how many elements can be inside one another, html and body included, 512 as in browsers (text and comments inside the deepest element do not count); a document that nests deeper is refused, not flattened. The source is limited before it is read, the nodes while it is parsed. The tree itself costs the run's memory: about 230 bytes of the heap for each node and about a kilobyte of the run's resident memory, so a Free run's memory allows less than its node limit when the nodes carry many attributes, and the run then ends as resource-limit.

Markdown: toMarkdown

html.toMarkdown(input, options?) returns the Markdown of a page, as a string. input is a string of HTML (read exactly as parse reads it) or a node of a tree: a document, an element, a fragment (such as a template's content) or a text node, as parse returns them or as your script builds them (names in any case, attributes and children optional; comments, doctypes and processing instructions are skipped, a node of another shape is a TypeError). Convert a part of a page by passing that part.

JavaScript
import * as html from "shotcup:html";

const page = await (await fetch("https://example.com/docs/intro")).text();
console.log(html.toMarkdown(page, { baseUrl: "https://example.com/docs/intro", content: "main" }));
OptionValuesMeaning
baseUrlan absolute URL with a host (https://example.com/page)Relative links and image addresses are made absolute against it (../x, /x, ?q, #f, //host/x, \ as / for http, https, ftp, ws, wss and file, the way a browser resolves them). Without it relative ones are written as they are; an address that is already absolute is read as the URL Standard says either way. A port is read as the URL Standard reads it: leading zeros go (:08 is :8), the scheme's own port goes (:0080 on http), and a port that is not digits or is above 65535 makes the address invalid, so a baseUrl with one is a TypeError and a link with one is written as it stands in the page. A <base> element is ignored: pass the page's address.
content"all" (default), "main""main" converts only the content of the page, by the rule below.
links"inline" (default), "reference"[text](url "title"), or [text][1] with the definitions at the end, one for each address and title.

A value of the wrong type or an option value that is not listed is a TypeError, which quotes at most 64 characters of what you passed; an option the function does not know is ignored.

What it writes

HTMLMarkdown
h1 to h6ATX headings (# ); a heading with no text is left out; a line break inside is a space
p, div, section, article, main, header, footer, nav, aside, figure, fieldset, form, details, address, center and the likeparagraphs: the text and inline elements between block elements form one paragraph each
em, i, cite, dfn, var*emphasis*
strong, b**strong**
del, s, strike~~strike~~
code, kbd, samp, tta code span, with a backtick string no run of backticks inside can end; code that holds a link is written as code spans with the link around its own part
qthe text in straight double quotes
a with an href[text](address "title"); no link for an empty or unusable address (javascript:, vbscript:, data:, or over 8 KiB) or for a link inside a link; the address is wrapped in <> where it needs it
img![alt](address "title"); data-src or the first srcset candidate stand in for a missing src; a data: image, a one-pixel tracker and an image with no address (its alt stays as text) are not written
bra hard break (a backslash at the end of the line; <br> in a table cell, a space in a heading); two in a row end the paragraph; none at the start or the end of a block
hr--- (*** or ___ where a list marker or a setext heading would read the dashes otherwise)
ul, ol (also menu, dir), lilists, nested to any depth, - or 1. (numbers as below); the list is loose, with a blank line between its items, when an item has a p of its own; two bullet lists that touch use different bullets, and two numbered lists that touch have a blank HTML comment between them (see below), so that they stay two; text that is not in an item goes into an item of its own (an empty element makes none; in an ol it is a bullet item, as HTML gives it no number), and an li the parser left inside a wrapper (a link, a div) is still an item
li that begins with <input type=checkbox>a task list item, - [x] or - [ ]
blockquote> before every line, nested
prea fenced block, longer fence than any run of backticks inside, the language from class="language-x" or "lang-x" on the pre or its first code; line breaks as in the source, br as a line feed; an empty one is left out; links inside are lost
tablea GitHub table (below)
dl, dt, ddCommonMark has no definition list: a dt is a paragraph in bold, a dd an ordinary block under it; summary is bold the same way
span, sub, sup, u, ins, mark, small, abbr, label, button, ruby, time, font, picture, any unknown element, MathMLnothing of their own: their content is kept (H<sub>2</sub>O is H2O)
script, style, template, noscript, head (title, meta, link, base), svg, iframe, object, embed, canvas, video, audio, select, option, textarea, input (but see task lists), progress, meter, commentsdropped with everything inside them
an element with hidden, aria-hidden="true", or a style containing display:none or visibility:hiddendropped with everything inside it

White space is collapsed as a browser collapses it: a run of spaces, tabs, line feeds and form feeds is one space, and there is none at the start or end of a block, around a break or inside the edge of a mark (<b> x </b> is **x** between its spaces). A non-breaking space is kept in the text and trimmed at the edges of a block; at the edge of a mark it moves outside the mark as a space does, and stays what it was (a<b>x&nbsp;</b>y is a**x** and one no-break space and y). The invisible characters (the soft hyphen, the zero width space and joiners, the left-to-right and right-to-left marks, the word joiner and the byte order mark; not the embeddings, overrides and isolates, which change how the text around them reads) are treated like white space at those edges, so a mark that holds nothing visible is not written (<b>&#8203;</b> is no mark at all, and the character stays where it was in the text), and they are trimmed at the edges of a block. An inline element that spans blocks is applied to each paragraph in it: a link around a heading and a paragraph is a link in the heading and a link in the paragraph. Control characters are dropped.

Text is escaped so that the Markdown reads back as the same text: \, *, [, ], a backtick and ~ always; _ except between two letters or digits; <, & and ! where they could start markup, a character reference or an image; | in a table cell; and at the start of a line what would make a block (#, >, -, +, =, 1., a line of hyphens, colons and pipes). A mark whose delimiters cannot work where it stands (CommonMark's flanking rules, say a<b>"x"</b>) is written as HTML, <strong>, <em> or <del>, which every CommonMark reader accepts. A bare address in text stays text; renderers such as GitHub's make it a link.

Numbered lists

CommonMark numbers a list from its first item and counts up by one, so it cannot say <ol reversed>, an li with a value, or any numbering that is not a run of consecutive numbers. The converter numbers the items as HTML does (the first item has start, else the number of items for reversed, else 1; each next item is one more, or one less when reversed; an li with a value has that number and the next ones go on from it; text that is not in an li has no number) and writes each run of items that are one apart as one list. An item that breaks the run begins another list, which starts at that item's own number, so every number you read is the number HTML shows: <ol reversed><li>a<li>b<li>c</ol> is three lists, 3. a, 2. b and 1. c. Every numbered list uses ., and two numbered lists that touch have a blank HTML comment, <!-- -->, on a line of its own between them, the separator CommonMark itself names for lists that would otherwise run together. Renderers that pass HTML through, GitHub's among them, show nothing for it; one that escapes HTML shows it as a line of text, and the numbers stay right either way:

text
3. a

<!-- -->

2. b

<!-- -->

1. c

A number that a marker cannot show, below 0 or above 999,999,999, is a bullet item that begins with it as text, - -1. a, in the item's first paragraph (a paragraph of its own when the item begins with another block); a number beyond two billion in either direction is read as two billion. A value on an item of a ul is ignored, and in a table cell each item of an ol is written with its own number (2. a<br>1. b).

The items that count are the ones a browser counts, with the exceptions at the end of this section: every li under the ol with no list of its own in between, in the order of the markup, wherever it stands and whether or not the converter writes it. One in a wrapper, a quote or a table cell that the converter cannot make an item of is written as the block it is; one in code written as text is text; one in a part of a table that is not written (a second caption, an li straight in a table or a row) or in an element that is dropped though a browser shows it (aria-hidden, visibility:hidden, a map, an object without data, the parts around the main content) is not written; each still uses its number, so the items around it have the numbers the page shows. An li a browser does not render is not counted: one with hidden or display:none in its style, one inside an element that is dropped because a browser does not show its content (script, template, a form control, media, an iframe, an object with data) and one inside an img, an input or an SVG element other than foreignObject. Only the markup decides, never the page's style sheets (an li the page styles display:block still counts), and a closed details, a closed dialog and a popover are written with their content and their items count, where a browser hides them.

Tables

A table whose cells hold inline content, paragraphs, lists and line breaks is a GitHub table. The header is the first row of the thead or, without one, the first row; the rows follow in the order a browser shows them (thead, tbody, tfoot); a caption is a paragraph before the table. The columns are the widest row (at most 1,000, and a table of more than a million cells is a RangeError). A cell with colspan or rowspan is written in its first position and the positions it covers are empty. align or text-align on a header cell gives the column its alignment. A paragraph, list item or line break in a cell becomes a <br> (a list item starts with - or its number). A body row with nothing in it is left out.

A table with a heading, a code block, a quote, a rule or another table in a cell is a layout table: it is not written as a table, its cells are read as blocks one after another.

The main content

content: "main" converts the page's content and leaves out what surrounds it, by a fixed rule (no scoring model, no readability library):

  1. The first main element, or an element with role="main"; if the page has several, those that hold at least a quarter of the text of the longest.
  2. Without one, the article elements the same way.
  3. Without either, the element reached from the body by going down into the child that holds at least 60% of the text (links not counted), as long as it is a container (div, section, a table cell and the like), and blocks in it that are nearly all links (five links or more, at least four fifths of the text) are dropped.

Whatever the rule picks, these are left out: nav, aside, form and dialog elements; header and footer outside a main or an article; elements whose role is navigation, complementary, search, dialog or (outside the main content) banner or contentinfo; and elements whose class or id holds a word such as cookie, consent, newsletter, popup, modal, sidebar, breadcrumb, advert, ad, share, social, related, comments, menu, nav or banner, unless the element holds a main or an article or half of the page's text. The English Wikipedia article on TypeScript is 116,000 characters of Markdown as a whole and 57,000 as main content.

Limits and cost

The input is limited as for parse (the source, nodes, nesting and attributes below; a tree you pass counts its nodes, its depth, the attributes of each element and the UTF-8 bytes of its names and texts), and the Markdown may be at most four times the plan's source limit in bytes: past it is a RangeError, as is a cycle in a tree (it runs into the nesting limit). The conversion is native code like parse: its memory is reserved before it is allocated and its work is charged to the run's instructions, with the timeout and a cancel checked as it goes. The 428 KB English Wikipedia article on TypeScript, parsed and converted in a Free run, costs 170 thousand of the 5 million instructions, 23 MiB of resident memory and 1.2 MiB of the heap, and gives 116,000 characters; turndown with domino, run as JavaScript, took 20.4 million instructions and 265 MB for the same page.

What is not here yet

Pages built by JavaScript are not what parse sees: it reads the HTML it is given, and nothing runs. Drawing a page is not here yet, and with it the pixel sizes getComputedStyle gives for an element on the screen, container size queries and linked style sheets.

This page as Markdown: /docs/html.md