> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gately.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Scraping

> Extract content from web pages with advanced customization options

<Note>
  The API reference for this endpoint is not yet available in OpenAPI format. Use the examples below to make requests.
</Note>

## Overview

The Web Scraping service allows you to extract content from any web page with rich customization options. Use this endpoint to scrape content, take screenshots, and process web pages.

## Base URL

```bash theme={null}
https://api.gately.ai/v1/web
```

## Request Format

```bash theme={null}
curl --location 'https://api.gately.ai/v1/web' \
--header 'Authorization: Bearer YOUR_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "model": "scrape",
    "params": {
        "url": "https://example.com",
        "formats": ["markdown", "links"],
        "onlyMainContent": true
    }
}'
```

### Parameters

<ParamField body="model" type="string" required default="scrape">
  Model identifier (must be set to `scrape`)
</ParamField>

<ParamField body="params" type="object" required>
  <Expandable title="Scrape Parameters">
    <ParamField body="url" type="string" required>
      The URL of the webpage to scrape
    </ParamField>

    <ParamField body="formats" type="array">
      Types of content to extract:

      * `markdown`
      * `html`
      * `rawHtml`
      * `links`
      * `screenshot`
      * `screenshot@fullPage`
      * `json`
    </ParamField>

    <ParamField body="onlyMainContent" type="boolean" default={true}>
      Extract only main content, excluding headers/footers
    </ParamField>
  </Expandable>
</ParamField>

## Advanced Options

<AccordionGroup>
  <Accordion title="Content Filtering" icon="filter">
    <ParamField body="params.includeTags" type="array">
      Array of HTML tags to include in output
    </ParamField>

    <ParamField body="params.excludeTags" type="array">
      Array of HTML tags to exclude from output
    </ParamField>
  </Accordion>

  <Accordion title="Page Actions" icon="play">
    <ParamField body="params.actions" type="array">
      Actions to perform before scraping
    </ParamField>

    <ParamField body="params.waitFor" type="integer" default="0">
      Delay in milliseconds before scraping
    </ParamField>
  </Accordion>

  <Accordion title="Request Settings" icon="gear">
    <ParamField body="params.headers" type="object">
      Custom headers for the request
    </ParamField>

    <ParamField body="params.timeout" type="integer" default="30000">
      Request timeout in milliseconds
    </ParamField>
  </Accordion>
</AccordionGroup>

## Response Format

<ResponseField name="id" type="string">
  Unique identifier for the request
</ResponseField>

<ResponseField name="object" type="string">
  Object type (e.g., "scrape.completion")
</ResponseField>

<ResponseField name="created" type="integer">
  Unix timestamp when the request was created
</ResponseField>

<ResponseField name="model" type="string">
  Model used for the request (scrape)
</ResponseField>

<ResponseField name="data" type="object">
  <Expandable title="Data fields">
    <ResponseField name="success" type="boolean">
      Indicates if the scraping was successful
    </ResponseField>

    <ResponseField name="data" type="object">
      <Expandable title="Scraped data">
        <ResponseField name="markdown" type="string">
          Markdown version of the content (if requested)
        </ResponseField>

        <ResponseField name="html" type="string">
          Clean HTML version (if requested)
        </ResponseField>

        <ResponseField name="links" type="array">
          List of extracted links (if requested)
        </ResponseField>

        <ResponseField name="metadata" type="object">
          Page metadata (title, description, etc.)
        </ResponseField>
      </Expandable>
    </ResponseField>
  </Expandable>
</ResponseField>

## Example Response

```json theme={null}
{
  "id": "scrape-aec35e29703d4eafb1c45cda9fd66951",
  "object": "scrape.completion",
  "created": 1732654897,
  "model": "scrape",
  "data": {
    "success": true,
    "data": {
      "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples in documents...",
      "links": [
        "https://www.iana.org/domains/example"
      ],
      "metadata": {
        "title": "Example Domain",
        "description": "",
        "language": "en",
        "sourceURL": "https://example.com",
        "statusCode": 200
      }
    }
  }
}
```

## Example: Screenshot Capture

```bash theme={null}
curl --location 'https://api.gately.ai/v1/web' \
--header 'Authorization: Bearer YOUR_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "model": "scrape",
    "params": {
        "url": "https://example.com",
        "formats": ["screenshot@fullPage"],
        "waitFor": 1000
    }
}'
```

## Example: Advanced Page Interaction

```bash theme={null}
curl --location 'https://api.gately.ai/v1/web' \
--header 'Authorization: Bearer YOUR_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "model": "scrape",
    "params": {
        "url": "https://example.com",
        "formats": ["markdown"],
        "actions": [
            {"type": "click", "selector": "#consent-button"},
            {"type": "wait", "milliseconds": 1000},
            {"type": "click", "selector": "#more-content"}
        ]
    }
}'
```