---
title: "Scraping: fetch a page, select from it, write JSON or CSV"
description: "Fetch a page under allowedHosts, read it with shotcup:html and CSS selectors, and write JSON or CSV to /out; robots.txt and the site's terms are your responsibility."
canonical: "https://shotcup.dev/docs/scraping"
last_updated: "2026-10-18"
---
# Scraping: fetch a page, select from it, write the result

A run can fetch a page, read it as a browser does, pick what it wants with CSS
selectors and write the result as JSON or CSV, all in one script. The pieces
are `fetch` ([Network](https://shotcup.dev/docs/quickstart#network): GET and HEAD to the hosts
the run allows; Pro, Scale, Platform allow hosts, Free has no network) and
`shotcup:html` ([HTML](https://shotcup.dev/docs/html): `parse`, `select`, `selectAll`,
`text`, `attr`, `links`, `tables`, `jsonLd`, `meta`). Nothing runs on the
page: you get the HTML the server sent, not what its scripts would build.

## Your duty

Scraping a site is your decision and your responsibility. Shotcup does not
read a site's `robots.txt` or its terms of use for you and does not enforce
them: check both before you fetch, honour what they say, keep your request
rate polite, and do not take what you are not allowed to take or use it in
a way the site forbids. The [acceptable use](https://shotcup.dev/acceptable-use) page applies to
every run.

## An example

Fetch a page of a catalogue, read each product and write a JSON file and a
CSV file. `allowedHosts` goes in the run request.

```js
import * as html from "shotcup:html";
import fs from "fs";

const url = "https://shop.example.com/catalogue/page-1.html";
const response = await fetch(url);
if (!response.ok) throw new Error("fetch failed: " + response.status);
const doc = html.parse(await response.text());

const rows = html.selectAll(doc, "article.product").map((card) => ({
  title: html.text(html.select(card, "h3 a")),
  price: html.text(html.select(card, ".price")),
  inStock: html.select(card, ".availability.in-stock") !== null,
  link: html.links(card, { baseUrl: url })[0]?.href ?? null,
  image: html.images(card, { baseUrl: url })[0]?.src ?? null,
}));

const quote = (value) => '"' + String(value ?? "").replaceAll('"', '""') + '"';
const columns = ["title", "price", "inStock", "link", "image"];
const csv = [columns.join(","), ...rows.map((row) => columns.map((c) => quote(row[c])).join(","))].join("\n");

fs.writeFileSync("/out/products.json", JSON.stringify(rows, null, 2));
fs.writeFileSync("/out/products.csv", csv + "\n");
console.log(rows.length + " products");
```

```sh
curl https://shotcup.dev/api/v1/runs \
  -H "Authorization: Bearer $SHOTCUP_API_KEY" \
  -H "Content-Type: application/json" \
  -d @- <<'JSON'
{
  "code": "<the script above as a JSON string>",
  "allowedHosts": ["shop.example.com"]
}
JSON
```

The files under `/out` come back as artifacts of the run. A page that
redirects to another host needs that host in `allowedHosts` too.

## Ways to read a page

- **Tables.** `html.tables(doc)` gives every table as `{ caption, header,
  rows }` with `colspan` and `rowspan` expanded, so a table of figures is
  `rows` and a header row away from a CSV.
- **Structured data.** Many pages carry their data as JSON-LD (`html.jsonLd(doc)`)
  or as Open Graph and Twitter tags (`html.meta(doc).openGraph`): read those
  before you scrape the layout; they change less often.
- **Links.** `html.links(doc, { baseUrl: url })` resolves every address against
  the page's `<base href>` and the address you fetched, so a list of next pages
  is `links` filtered by `rel` or by a selector (`html.selectAll(doc, "a[rel=next]")`).
- **Text.** `html.text(node)` returns what a reader sees (white space collapsed,
  blocks on their own lines); `html.textContent(node)` every text node as it is.
- **Forms.** `html.forms(doc)` lists a form's fields and their values to
  read. Nothing is submitted: `fetch` is GET and HEAD only.

## Limits that apply

A page is limited by the plan's HTML limits (bytes, nodes, depth,
attributes), a selector by its selector limits, and the run by its time,
instruction and memory limits, all on the [HTML](https://shotcup.dev/docs/html#limits) page. A
script that walks many pages starts a fetch for each, one after the other,
within the run's time limit.
