> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gately.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Crawling API

> Extract web content programmatically

# Web Crawling API

Gately AI offers a powerful web crawling and scraping API that allows you to extract data from websites programmatically. This suite of tools helps you gather information, analyze content, and build data-driven applications.

## Available Endpoints

<CardGroup cols={2}>
  <Card title="Scrape" icon="screwdriver-wrench" href="/api-reference/endpoint/scrape">
    Extract content from a single URL with advanced options
  </Card>

  <Card title="Crawl" icon="spider" href="/api-reference/endpoint/crawl">
    Recursively crawl multiple pages starting from a base URL
  </Card>

  <Card title="Crawl Status" icon="magnifying-glass-chart" href="/api-reference/endpoint/crawl-status">
    Check the status of an ongoing or completed crawl
  </Card>

  <Card title="Site Map" icon="sitemap" href="/api-reference/endpoint/map">
    Generate a map of all pages on a website
  </Card>
</CardGroup>

## Features

* **Content Extraction** - Extract clean, structured content from web pages
* **Recursive Crawling** - Crawl entire websites with depth control
* **Rendering Support** - Handles JavaScript rendering for SPAs
* **Custom Selectors** - Target specific elements on a page
* **Rich Media** - Extract images, videos, and other media
* **Screenshots** - Capture full-page or element screenshots
* **Headless Browser** - Perform automated browser actions

## Common Use Cases

<AccordionGroup>
  <Accordion title="Content Aggregation">
    Build news aggregators, product comparison sites, or research tools by extracting content from multiple sources.
  </Accordion>

  <Accordion title="Market Research">
    Monitor competitor pricing, product offerings, or content strategies through automated data collection.
  </Accordion>

  <Accordion title="Lead Generation">
    Extract contact information from business directories, professional networks, or company websites.
  </Accordion>

  <Accordion title="SEO Analysis">
    Analyze website structure, content, and metadata for optimization opportunities.
  </Accordion>
</AccordionGroup>

## Important Considerations

<Warning>
  Always respect website terms of service, robots.txt files, and maintain reasonable request rates to avoid being blocked.
</Warning>

<Note>
  Some websites implement anti-scraping measures. Our API uses advanced techniques to work around common limitations, but cannot guarantee success on all sites.
</Note>