Crawling as accountable infrastructure

Crawl API

Fetch and extract web material with explicit limits, source records, and respect for the rules at the other end.

For developers and researchers who need repeatable collection without operating an unaccountable scraping fleet.

Current availability

Service contracts and crawler integrations exist. There is no deployed public endpoint, quota system, or durable job service.

Incubating

Why use it

A smaller path from intent to outcome.

  1. 01

    Set the boundary first

    Requests should carry scope, policy, and resource limits before work begins.

  2. 02

    Respect site rules

    Robots and network-safety controls belong in the path, not in a README footnote.

  3. 03

    Retain the source trail

    Results should preserve where bytes came from and what failed.

Development record

What is real now—and what still has to be earned.

The badge names this product’s place in the portfolio, not public availability. This record separates working parts from missing product proof.

In the code today

  • A Rust service and protobuf contract
  • Connections to Argand’s robots, network-safety, and crawler layers
  • Policy-oriented request and result types

Before broader release

  • A deployable public API with quotas and durable jobs
  • Abuse prevention, billing boundaries, and source-native receipts

Place in Argand

Crawl API productizes a narrow part of the crawler; it is not unrestricted proxy access.

Continue exploring

For site owners Submit pages, inspect crawl status, and help Argand reach the original source. Archive Self-hostable research archives with evidence capture, timelines, maps, and citations. Alerts Monitor topics and sources without revealing what you care about to an ad network.