DiscovAI crawl

DiscovAI Crawl API ?️?

One API to scrape everything you need from URLs for your AI tool and vector database.

? Work in Progress ?

? Features

Our API provides a comprehensive suite of data extraction and processing capabilities:

• ? Clean HTML (JavaScript and CSS removed)
• LLM-friendly Markdown conversion
• Ad-free, cookie banner-free, and dialog-free content
• ? Website screenshots (auto-saved to AWS S3 or Cloudflare R2)
• ? LLM-generated SEO-friendly content
• ? LLM-extracted key information (summary, features, FAQs, etc.)
• ? Ready-to-use embeddings for vector database integration (auto-saved to db)

? Installation

pnpm i
cd apps/api && pnpm exec playwright install

Usage

pnpm dev
open http://localhost:3000

? API Response Structure

{
  "clean_html": "...",
  "LLM_friendly_markdown": "...",
  "clean_text": "...",
  "screenshot_url": "...",
  "llm_extracts_key_info": {
    "what": "...",
    "summary": "...",
    "features": ["...", "..."],
    "faqs": [{"q": "...", "a": "..."}]
  },
  "llm_summarized_detail": "...",
  "embeddings": [...]
}

Documentation

TODO

? Contributing

TODO