361 lines
13 KiB
Markdown
361 lines
13 KiB
Markdown
# Seed
|
||
|
||
Stealth web crawler with a live admin panel.
|
||
Bypasses Cloudflare / ArvanCloud bot-detection, executes configurable user flows, and tracks every run in a SQLite-backed dashboard.
|
||
|
||
---
|
||
|
||
## Table of Contents
|
||
|
||
1. [Overview](#overview)
|
||
2. [Architecture](#architecture)
|
||
3. [Project Structure](#project-structure)
|
||
4. [Prerequisites](#prerequisites)
|
||
5. [Quick Start — Local](#quick-start--local)
|
||
6. [Quick Start — Docker](#quick-start--docker)
|
||
7. [Configuration](#configuration)
|
||
8. [Scenarios](#scenarios)
|
||
9. [Dynamic Flow System](#dynamic-flow-system)
|
||
10. [Admin Panel](#admin-panel)
|
||
11. [Makefile Reference](#makefile-reference)
|
||
12. [Extending the Crawler](#extending-the-crawler)
|
||
|
||
---
|
||
|
||
## Overview
|
||
|
||
Seed runs two independent entry scenarios on a target website and then executes a fully configurable multi-step flow after each login:
|
||
|
||
| Scenario | Description |
|
||
|---|---|
|
||
| **OTP Login** | Types a phone number into the site's login form, waits for an SMS-OTP, submits it, then runs the flow |
|
||
| **Fresh Session** | Opens the browser with a clean cookie jar / localStorage so the site treats the visitor as a brand-new user, then runs the flow |
|
||
|
||
The **flow** consists of three automatic steps configured live from the admin panel:
|
||
|
||
1. **Search** — types one or more search texts into a configurable search box
|
||
2. **Infinite Scroll** — scrolls down (window or a specific container) until a target item appears or a scroll limit is hit
|
||
3. **Click** — clicks each target item by its ID
|
||
|
||
Every run — success or failure — is recorded in SQLite and visible on the dashboard.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```
|
||
Browser (undetected-chromedriver + selenium-stealth)
|
||
│
|
||
├── Scenario 1: OTP Login ──┐
|
||
└── Scenario 2: Fresh Session ┘
|
||
│
|
||
DynamicFlow
|
||
(reads active config
|
||
from SQLite at runtime)
|
||
│
|
||
┌──────────────┼──────────────┐
|
||
search_step scroll_step click_step
|
||
│
|
||
SQLite (data/tracker.db)
|
||
│
|
||
FastAPI Admin Panel
|
||
(dashboard + flow config CRUD)
|
||
```
|
||
|
||
- **Anti-detection**: `undetected-chromedriver` patches Chrome's automation flags; `selenium-stealth` masks `navigator.webdriver`, plugins, and language headers. A random user-agent is chosen per session.
|
||
- **Flow config is hot**: the crawler reads the active config from the DB on every run — no restart needed after editing a config in the admin panel.
|
||
- **SQLite**: zero-infrastructure database. In Docker it lives on a named volume (`seed-data`) so it survives container rebuilds.
|
||
|
||
---
|
||
|
||
## Project Structure
|
||
|
||
```
|
||
seed/
|
||
├── crawler/
|
||
│ ├── driver.py # stealth Chrome factory + human_delay helper
|
||
│ ├── dynamic_flow.py # search → scroll → click executor
|
||
│ ├── flow_runner.py # legacy step-registry runner (extendable)
|
||
│ └── scenarios/
|
||
│ ├── otp_login.py # Scenario 1: SMS-OTP login
|
||
│ └── fresh_session.py # Scenario 2: fresh cookie session
|
||
├── admin/
|
||
│ ├── main.py # FastAPI app wiring
|
||
│ ├── models.py # SQLite DDL + async CRUD helpers
|
||
│ ├── routes.py # REST endpoints (stats + flow config CRUD)
|
||
│ └── static/
|
||
│ └── dashboard.html # Single-file SPA (no build step)
|
||
├── config.py # Typed config from environment / .env
|
||
├── main.py # CLI entry point
|
||
├── pyproject.toml # deps + Pyright config
|
||
├── uv.toml # uv index override (forces PyPI)
|
||
├── Makefile # developer shortcuts
|
||
├── Dockerfile # multi-stage: builder + runtime (Chrome included)
|
||
├── docker-compose.yml # admin service + on-demand crawler service
|
||
└── .env.example # all configurable variables with defaults
|
||
```
|
||
|
||
---
|
||
|
||
## Prerequisites
|
||
|
||
### Local
|
||
|
||
| Tool | Version | Notes |
|
||
|---|---|---|
|
||
| Python | ≥ 3.11 | |
|
||
| [uv](https://docs.astral.sh/uv/) | latest | `curl -LsSf https://astral.sh/uv/install.sh \| sh` |
|
||
| Google Chrome | any recent | must be installed on the host |
|
||
|
||
### Docker
|
||
|
||
| Tool | Notes |
|
||
|---|---|
|
||
| Docker ≥ 24 | |
|
||
| Docker Compose v2 | bundled with Docker Desktop |
|
||
|
||
Chrome is installed inside the Docker image — no host Chrome needed.
|
||
|
||
---
|
||
|
||
## Quick Start — Local
|
||
|
||
```bash
|
||
# 1. Clone and enter the project
|
||
git clone <repo-url> seed && cd seed
|
||
|
||
# 2. Install dependencies (creates .venv automatically)
|
||
make install
|
||
|
||
# 3. Configure
|
||
cp .env.example .env
|
||
$EDITOR .env # set TARGET_URL, PHONE_NUMBERS, ADMIN_PASSWORD, …
|
||
|
||
# 4. Start the admin panel
|
||
make admin
|
||
# → http://localhost:8000
|
||
|
||
# 5. In another terminal — run a scenario
|
||
make fresh # fresh-session scenario
|
||
make otp # all phones from PHONE_NUMBERS in .env
|
||
make otp PHONE=+98910... # single phone
|
||
```
|
||
|
||
---
|
||
|
||
## Quick Start — Docker
|
||
|
||
```bash
|
||
# 1. Copy and fill in the env file
|
||
cp .env.example .env
|
||
$EDITOR .env
|
||
|
||
# 2. Build the image
|
||
make build
|
||
|
||
# 3. Start the admin panel (detached)
|
||
make up
|
||
# → http://localhost:8000
|
||
|
||
# 4. Run scenarios inside Docker
|
||
make docker-fresh
|
||
make docker-otp PHONE=+98910...
|
||
|
||
# 5. Watch logs
|
||
make logs
|
||
|
||
# 6. Stop everything
|
||
make down
|
||
```
|
||
|
||
The SQLite database is stored in the `seed-data` Docker volume and persists across rebuilds.
|
||
Worker browser sessions are recorded at 12 FPS into the shared `seed-recordings`
|
||
volume. The admin panel's **Recordings** tab lists and downloads the newest three
|
||
completed sessions.
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
Copy `.env.example` to `.env` and set values before running anything.
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `TARGET_URL` | `https://example.com` | The website to crawl |
|
||
| `ADMIN_USERNAME` | `admin` | Admin panel username |
|
||
| `ADMIN_PASSWORD` | `changeme` | Admin panel password — **change this** |
|
||
| `ADMIN_HOST` | `127.0.0.1` | Bind address for FastAPI (Docker overrides to `0.0.0.0`) |
|
||
| `ADMIN_PORT` | `8000` | Port for the admin panel |
|
||
| `DB_PATH` | `data/tracker.db` | Path to the SQLite file |
|
||
| `HEADLESS` | `true` | Set to `false` to watch the browser (local only) |
|
||
| `CHROME_BINARY` | _(auto)_ | Explicit path to Chrome, e.g. `/usr/bin/google-chrome-stable` |
|
||
| `RECORD_SESSIONS` | `false` | Record browser sessions; Docker workers override this to `true` |
|
||
| `RECORDING_FPS` | `12` | Recording frame rate |
|
||
| `RECORDING_MAX_FILES` | `3` | Number of completed recordings retained across all workers |
|
||
| `RECORDINGS_DIR` | `recordings` | Recording directory; Docker uses the shared `/app/recordings` volume |
|
||
| `PHONE_NUMBERS` | _(empty)_ | Comma-separated list of phone numbers for Scenario 1 |
|
||
| `FLOW_STEPS` | `step_home` | Comma-separated step names for the legacy step-registry runner |
|
||
|
||
---
|
||
|
||
## Scenarios
|
||
|
||
### Scenario 1 — SMS-OTP Login (`crawler/scenarios/otp_login.py`)
|
||
|
||
1. Navigates to `TARGET_URL`
|
||
2. Finds the phone-number input (CSS / XPath selector, configurable at top of file)
|
||
3. Types the phone number character-by-character (human pacing)
|
||
4. Clicks the "Send OTP" button
|
||
5. Calls `otp_resolver(phone)` in a polling loop (up to 12 × 5–8 s) — **you must implement this** to fetch the OTP from your SMS gateway / webhook
|
||
6. Types the OTP and confirms login
|
||
7. Waits for a success indicator on the page
|
||
|
||
**To wire up your SMS gateway**, edit `otp_resolver` inside `main.py → run_otp()`:
|
||
|
||
```python
|
||
def otp_resolver(phone: str) -> str:
|
||
# Example: poll your own API
|
||
resp = httpx.get(f"https://sms-api.example.com/latest?phone={phone}", timeout=5)
|
||
return resp.json().get("code", "")
|
||
```
|
||
|
||
**To update selectors** to match the target site, edit the class-level constants at the top of `OTPLoginScenario`:
|
||
|
||
```python
|
||
_PHONE_FIELD_SELECTOR = (By.CSS_SELECTOR, "input[name='mobile']")
|
||
_OTP_FIELD_SELECTOR = (By.CSS_SELECTOR, "input[name='code']")
|
||
_SUCCESS_INDICATOR = (By.CSS_SELECTOR, ".home-feed")
|
||
```
|
||
|
||
### Scenario 2 — Fresh Session (`crawler/scenarios/fresh_session.py`)
|
||
|
||
1. Calls `driver.delete_all_cookies()`
|
||
2. Clears `localStorage` and `sessionStorage` via JavaScript
|
||
3. Navigates to `TARGET_URL`
|
||
4. Waits for `document.readyState === 'complete'`
|
||
|
||
Each invocation of this scenario gets a brand-new Chrome profile via `undetected-chromedriver`, so no state leaks between runs even without explicit clearing.
|
||
|
||
---
|
||
|
||
## Dynamic Flow System
|
||
|
||
After either scenario completes login / session setup, `DynamicFlow` runs the configured steps.
|
||
**All settings are edited live from the admin panel** — no code changes or restarts needed.
|
||
|
||
### How it works
|
||
|
||
```
|
||
For each search_text in config:
|
||
1. search_step → find search box → type text → submit → wait for results
|
||
2. For each target_item_id in config:
|
||
scroll_step → scroll until item selector matches (or max_scrolls hit)
|
||
click_step → scroll element into view → click it
|
||
```
|
||
|
||
### Selector templates
|
||
|
||
The **Item Selector Template** field supports a `{item_id}` placeholder:
|
||
|
||
| Template | Item ID | Resolved selector |
|
||
|---|---|---|
|
||
| `[data-id='{item_id}']` | `prod-123` | `[data-id='prod-123']` |
|
||
| `.result[data-sku="{item_id}"]` | `SKU-99` | `.result[data-sku="SKU-99"]` |
|
||
| `.product-card` | _(any)_ | `.product-card` (first visible match) |
|
||
|
||
Both CSS and XPath are supported for every selector field.
|
||
|
||
### Infinite scroll behaviour
|
||
|
||
| Config field | Effect |
|
||
|---|---|
|
||
| `Scroll Container Selector` | Scrolls this element; blank → scrolls `window` |
|
||
| `Max Scrolls` | Hard limit — aborts after this many scroll iterations |
|
||
| `Scroll Pause (ms)` | Sleep between each scroll to let new content load |
|
||
| `No New Content Timeout (ms)` | If the DOM height hasn't grown for this long → stop and mark as failed |
|
||
|
||
---
|
||
|
||
## Admin Panel
|
||
|
||
Accessible at `http://localhost:8000` (credentials from `.env`).
|
||
|
||
### Dashboard tab
|
||
|
||
| Widget | Shows |
|
||
|---|---|
|
||
| Stat cards | Total runs · Unique identifiers · Successes · Failures · Success rate |
|
||
| Scenario Breakdown | Bar chart: success rate per scenario (`otp` / `fresh`) |
|
||
| Top Failure Reasons | Most frequent error strings with counts |
|
||
| Recent Runs table | Last 50 runs with ID, time, scenario, identifier, status, duration, failure reason |
|
||
|
||
Auto-refreshes every 15 seconds.
|
||
|
||
### Flow Configs tab
|
||
|
||
Create, edit, activate, and delete flow configurations.
|
||
Only one config is **active** at a time; the crawler reads it on every run.
|
||
|
||
Each config stores:
|
||
- Search texts (one per line)
|
||
- Search box selector + type (CSS / XPath)
|
||
- Optional submit button selector (falls back to Enter key)
|
||
- Results container selector (waited for after search)
|
||
- Scroll container selector, max scrolls, pause, no-new-content timeout
|
||
- Item selector template + type
|
||
- Target item IDs (one per line)
|
||
|
||
---
|
||
|
||
## Makefile Reference
|
||
|
||
```
|
||
make install Bootstrap: create venv, install all deps
|
||
make sync Sync venv with lockfile (fast)
|
||
make lock Re-resolve and update uv.lock
|
||
make typecheck Run Pyright
|
||
make lint Alias for typecheck
|
||
|
||
make admin Start admin panel locally → http://localhost:8000
|
||
make otp Run OTP scenario (all phones from .env)
|
||
make otp PHONE=+98... Run OTP scenario for one phone
|
||
make fresh Run fresh-session scenario
|
||
|
||
make build Build Docker image
|
||
make up Start admin panel in Docker (detached)
|
||
make down Stop containers
|
||
make logs Follow container logs
|
||
make shell Shell into the running admin container
|
||
make docker-otp Run OTP scenario inside Docker
|
||
make docker-fresh Run fresh-session scenario inside Docker
|
||
|
||
make clean Remove .venv, __pycache__, .pyc files
|
||
```
|
||
|
||
---
|
||
|
||
## Extending the Crawler
|
||
|
||
### Add a new flow step (legacy registry)
|
||
|
||
```python
|
||
# crawler/flow_runner.py
|
||
@register_step("step_add_to_cart")
|
||
def step_add_to_cart(driver: uc.Chrome) -> dict[str, object]:
|
||
btn = driver.find_element(By.CSS_SELECTOR, "button.add-to-cart")
|
||
btn.click()
|
||
return {"added": True}
|
||
```
|
||
|
||
Then set `FLOW_STEPS=step_home,step_add_to_cart` in `.env`.
|
||
These static steps run in addition to (after) the dynamic flow.
|
||
|
||
### Add a new scenario
|
||
|
||
1. Create `crawler/scenarios/my_scenario.py` with a class following the same pattern as `OTPLoginScenario`
|
||
2. Add a branch in `main.py` and `Makefile`
|
||
|
||
### Use a remote WebDriver (Selenium Grid)
|
||
|
||
Replace `make_driver()` in `crawler/driver.py` with a `webdriver.Remote(...)` call pointing at your Grid hub.
|