|
59 | 59 | ) |
60 | 60 |
|
61 | 61 | _CATALOG_DESCRIPTION_MD = ( |
62 | | - "# wiki\n\n" |
63 | | - "Wikipedia / MediaWiki full-text search and page retrieval for SQL and RAG, over the free " |
64 | | - "official MediaWiki Action API + REST summary endpoint (no key, no scraping).\n\n" |
65 | | - "**Functions**\n\n" |
66 | | - "- `wiki_search(query, lang := 'en', count := 10, max_pages := 1, api_url := '')` — table " |
67 | | - "function: ranked full-text results.\n" |
68 | | - "- `wiki_page(title[, lang])` — scalar: a page's plain-text summary extract.\n" |
69 | | - "- `wiki_page_summary(title, lang := 'en', api_url := '')` — table function: the rich " |
70 | | - "single-row page summary.\n\n" |
| 62 | + "# Wikipedia & MediaWiki Search in SQL\n\n" |
| 63 | + "Run Wikipedia full-text search and page-summary retrieval directly from DuckDB SQL — " |
| 64 | + "powered by the free, official MediaWiki API, with no API key, no scraping, and no setup.\n\n" |
| 65 | + "This VGI extension turns any DuckDB session into a live Wikipedia and MediaWiki client. " |
| 66 | + "It is built for engineers and data teams who want to ground large language models with " |
| 67 | + "encyclopedic facts, enrich rows with authoritative context, or run ad-hoc knowledge lookups " |
| 68 | + "without leaving SQL. Because it speaks plain MediaWiki, it works against English Wikipedia by " |
| 69 | + "default, any of the 300+ Wikipedia language editions via a single `lang` argument, and any " |
| 70 | + "third-party MediaWiki wiki via `api_url`. It is a thin, polite **egress connector**: queries " |
| 71 | + "leave the engine for the wiki, results come back as Arrow rows, and the worker process never " |
| 72 | + "crashes on a missing page or a flaky network.\n\n" |
| 73 | + "Under the hood the extension calls the [MediaWiki Action API](https://www.mediawiki.org/wiki/API:Main_page) " |
| 74 | + "for ranked search and the [MediaWiki REST API](https://www.mediawiki.org/wiki/API:REST_API) " |
| 75 | + "page-summary endpoint for rich lead extracts and thumbnails — the same free, open interfaces " |
| 76 | + "that power [Wikipedia](https://en.wikipedia.org/) itself. The networked client sends a " |
| 77 | + "descriptive User-Agent, applies per-call timeouts, and retries with backoff on rate limits and " |
| 78 | + "server errors, so it stays well-behaved against the public Wikimedia endpoints. The software is " |
| 79 | + "open source and mirrors the upstream [MediaWiki source on GitHub](https://github.com/wikimedia/mediawiki); " |
| 80 | + "see the [MediaWiki search API documentation](https://www.mediawiki.org/wiki/API:Search) for the " |
| 81 | + "underlying query semantics.\n\n" |
| 82 | + "**SQL use cases and function surface.** Use `wiki_search(query, lang := 'en', count := 10, " |
| 83 | + "max_pages := 1, api_url := '')` — a table function — to run full-text search and get ranked " |
| 84 | + "rows of article title, plain-text snippet, page id, word count and canonical URL, ideal for " |
| 85 | + "retrieval-augmented generation (RAG) candidate selection. Use the `wiki_page(title)` / " |
| 86 | + "`wiki_page(title, lang)` scalar to fetch a single page's plain-text summary extract inline in a " |
| 87 | + "`SELECT` (returns NULL on a missing page rather than erroring). Use `wiki_page_summary(title, " |
| 88 | + "lang := 'en', api_url := '')` — a table function — for the rich single-row summary including " |
| 89 | + "the extract, canonical URL, thumbnail and page id. Pair it with vgi-embed and vgi-rerank to " |
| 90 | + "build knowledge-grounding pipelines entirely in SQL.\n\n" |
71 | 91 | "Retrieved article **content is CC-BY-SA** — attribution and share-alike are the caller's " |
72 | | - "responsibility." |
| 92 | + "responsibility; the `url` column is provided so you can link back and attribute." |
73 | 93 | ) |
74 | 94 |
|
75 | 95 | _SCHEMA_DESCRIPTION_LLM = ( |
|
0 commit comments