Skip to content

Scraping: fetch a page, select from it, write the result

A run can fetch a page, read it as a browser does, pick what it wants with CSS selectors and write the result as JSON or CSV, all in one script. The pieces are fetch (Network: GET and HEAD to the hosts the run allows; Pro, Scale, Platform allow hosts, Free has no network) and shotcup:html (HTML: parse, select, selectAll, text, attr, links, tables, jsonLd, meta). Nothing runs on the page: you get the HTML the server sent, not what its scripts would build.

Your duty

Scraping a site is your decision and your responsibility. Shotcup does not read a site's robots.txt or its terms of use for you and does not enforce them: check both before you fetch, honour what they say, keep your request rate polite, and do not take what you are not allowed to take or use it in a way the site forbids. The acceptable use page applies to every run.

An example

Fetch a page of a catalogue, read each product and write a JSON file and a CSV file. allowedHosts goes in the run request.

JavaScript
import * as html from "shotcup:html";
import fs from "fs";

const url = "https://shop.example.com/catalogue/page-1.html";
const response = await fetch(url);
if (!response.ok) throw new Error("fetch failed: " + response.status);
const doc = html.parse(await response.text());

const rows = html.selectAll(doc, "article.product").map((card) => ({
  title: html.text(html.select(card, "h3 a")),
  price: html.text(html.select(card, ".price")),
  inStock: html.select(card, ".availability.in-stock") !== null,
  link: html.links(card, { baseUrl: url })[0]?.href ?? null,
  image: html.images(card, { baseUrl: url })[0]?.src ?? null,
}));

const quote = (value) => '"' + String(value ?? "").replaceAll('"', '""') + '"';
const columns = ["title", "price", "inStock", "link", "image"];
const csv = [columns.join(","), ...rows.map((row) => columns.map((c) => quote(row[c])).join(","))].join("\n");

fs.writeFileSync("/out/products.json", JSON.stringify(rows, null, 2));
fs.writeFileSync("/out/products.csv", csv + "\n");
console.log(rows.length + " products");
Shell
curl https://shotcup.dev/api/v1/runs \
  -H "Authorization: Bearer $SHOTCUP_API_KEY" \
  -H "Content-Type: application/json" \
  -d @- <<'JSON'
{
  "code": "<the script above as a JSON string>",
  "allowedHosts": ["shop.example.com"]
}
JSON

The files under /out come back as artifacts of the run. A page that redirects to another host needs that host in allowedHosts too.

Ways to read a page

  • Tables. html.tables(doc) gives every table as { caption, header, rows } with colspan and rowspan expanded, so a table of figures is rows and a header row away from a CSV.
  • Structured data. Many pages carry their data as JSON-LD (html.jsonLd(doc)) or as Open Graph and Twitter tags (html.meta(doc).openGraph): read those before you scrape the layout; they change less often.
  • Links. html.links(doc, { baseUrl: url }) resolves every address against the page's <base href> and the address you fetched, so a list of next pages is links filtered by rel or by a selector (html.selectAll(doc, "a[rel=next]")).
  • Text. html.text(node) returns what a reader sees (white space collapsed, blocks on their own lines); html.textContent(node) every text node as it is.
  • Forms. html.forms(doc) lists a form's fields and their values to read. Nothing is submitted: fetch is GET and HEAD only.

Limits that apply

A page is limited by the plan's HTML limits (bytes, nodes, depth, attributes), a selector by its selector limits, and the run by its time, instruction and memory limits, all on the HTML page. A script that walks many pages starts a fetch for each, one after the other, within the run's time limit.

This page as Markdown: /docs/scraping.md