
One agent collects prices, listings and job posts from several services on a schedule, cleans the mess into one table and pings you only when something actually changed.
You describe a source once: its URL, how pages are paginated and which fields you need. The agent fetches the page, lets the model extract fields into a fixed schema, deduplicates by a stable key, saves a snapshot and writes a diff against the previous run.
- Tables: sources, runs, items, item_changes
- Edge function with fetch plus HTML to text cleanup
- Structured extraction with a strict field schema
- Scheduled trigger every few hours
- Alerts by email or webhook plus CSV export
Sell monitoring as a subscription per source: resellers pay for competitor prices, recruiters for job feeds, agencies for lead lists. Charge more for shorter intervals and API access.
A weekend
- Check the rules first: read robots.txt and the terms of the service, scrape only public pages, and prefer an official API when the site offers one.
- Create the sources table with url, pagination pattern, css hint, field list, interval and active flag. One row equals one monitored service.
- Write an edge function fetch_source: it loads the HTML with a normal user agent, strips scripts and styles, and trims the text to the part that holds the listings.
- Pass that text to the model with a strict JSON schema (title, price, currency, url, published_at, raw_key) and temperature zero. Never let the model invent fields, missing values must be null.
- Deduplicate on raw_key: upsert into items, and when a tracked field differs from the stored row insert a record into item_changes with the old and new value.
- Add retries with backoff, a per-source rate limit and a runs table that logs status, item count and error text so a broken selector is visible immediately.
- Schedule the function per source interval, then send a digest email or webhook that lists only rows from item_changes since the last digest.
- Build the UI: source list with health badge, run history, a filterable items table, a change feed and a CSV export button.
Build a scraping and monitoring agent. Data: table "sources" (name, url, pagination_pattern, selector_hint, fields jsonb, interval_minutes, active). Table "runs" (source_id, status, items_found, error, started_at, finished_at). Table "items" (source_id, raw_key unique per source, title, price numeric, currency, url, published_at, payload jsonb, first_seen_at, last_seen_at). Table "item_changes" (item_id, field, old_value, new_value, created_at). Server: an edge function that takes a source id, fetches the page with a normal user agent, strips scripts, styles and navigation, trims the text to the listing area, then asks the model to extract rows into a strict JSON schema with temperature zero and null for missing values. Upsert rows into items on raw_key and insert item_changes whenever a tracked field differs. Log every attempt into runs, retry twice with backoff on network errors and respect a per source rate limit. Follow pagination up to a configurable page limit. Automation: a scheduled job that picks active sources whose interval has elapsed, runs them one by one, then sends a digest email or webhook containing only item_changes created since the previous digest. UI: sources page with add and edit form, health badge from the last run, run history with error text, items table with search and filters, a change feed and a CSV export button. Rules: only public pages, honour robots.txt, keep all keys and model calls on the server, enable row level security so each user sees only their own sources and items.



















