Scraping: fetch a page, select from it, write the result
A run can fetch a page, read it as a browser does, pick what it wants with CSS selectors and write the result as JSON or CSV, all in one script. The pieces are fetch (Network: GET and HEAD to the hosts the run allows; Pro, Scale, Platform allow hosts, Free has no network) and shotcup:html (HTML: parse, select, selectAll, text, attr, links, tables, jsonLd, meta). Nothing runs on the page: you get the HTML the server sent, not what its scripts would build.
Your duty
Scraping a site is your decision and your responsibility. Shotcup does not read a site's robots.txt or its terms of use for you and does not enforce them: check both before you fetch, honour what they say, keep your request rate polite, and do not take what you are not allowed to take or use it in a way the site forbids. The acceptable use page applies to every run.
An example
Fetch a page of a catalogue, read each product and write a JSON file and a CSV file. allowedHosts goes in the run request.
import * as html from "shotcup:html";
import fs from "fs";
const url = "https://shop.example.com/catalogue/page-1.html";
const response = await fetch(url);
if (!response.ok) throw new Error("fetch failed: " + response.status);
const doc = html.parse(await response.text());
const rows = html.selectAll(doc, "article.product").map((card) => ({
title: html.text(html.select(card, "h3 a")),
price: html.text(html.select(card, ".price")),
inStock: html.select(card, ".availability.in-stock") !== null,
link: html.links(card, { baseUrl: url })[0]?.href ?? null,
image: html.images(card, { baseUrl: url })[0]?.src ?? null,
}));
const quote = (value) => '"' + String(value ?? "").replaceAll('"', '""') + '"';
const columns = ["title", "price", "inStock", "link", "image"];
const csv = [columns.join(","), ...rows.map((row) => columns.map((c) => quote(row[c])).join(","))].join("\n");
fs.writeFileSync("/out/products.json", JSON.stringify(rows, null, 2));
fs.writeFileSync("/out/products.csv", csv + "\n");
console.log(rows.length + " products");curl https://shotcup.dev/api/v1/runs \
-H "Authorization: Bearer $SHOTCUP_API_KEY" \
-H "Content-Type: application/json" \
-d @- <<'JSON'
{
"code": "<the script above as a JSON string>",
"allowedHosts": ["shop.example.com"]
}
JSONThe files under /out come back as artifacts of the run. A page that redirects to another host needs that host in allowedHosts too.
Ways to read a page
- Tables.
html.tables(doc)gives every table as{ caption, header, rows }withcolspanandrowspanexpanded, so a table of figures isrowsand a header row away from a CSV. - Structured data. Many pages carry their data as JSON-LD (
html.jsonLd(doc)) or as Open Graph and Twitter tags (html.meta(doc).openGraph): read those before you scrape the layout; they change less often. - Links.
html.links(doc, { baseUrl: url })resolves every address against the page's<base href>and the address you fetched, so a list of next pages islinksfiltered byrelor by a selector (html.selectAll(doc, "a[rel=next]")). - Text.
html.text(node)returns what a reader sees (white space collapsed, blocks on their own lines);html.textContent(node)every text node as it is. - Forms.
html.forms(doc)lists a form's fields and their values to read. Nothing is submitted:fetchis GET and HEAD only.
Limits that apply
A page is limited by the plan's HTML limits (bytes, nodes, depth, attributes), a selector by its selector limits, and the run by its time, instruction and memory limits, all on the HTML page. A script that walks many pages starts a fetch for each, one after the other, within the run's time limit.
This page as Markdown: /docs/scraping.md