libercrawl

A local-first, open-source web extraction studio. Scrape, crawl, map and archive websites with a refined dark interface and a self-hosted…

A new way for AI to search for data
View on GitHub
1GitHub stars

Python, shell

What XERJ indexed

Snapshot

Whole repository

The whole repository at commit d45bd3cb2a (2026-10-23).

Selected work

crsalass
@crsalass

“stabilize RabbitMQ startup health checks”. 6 files, 229 lines in scripts, tests.

Licence

The repository

AGPL-3.0

From GitHub's licence detection. A person still checks the LICENSE file at the pinned commit.

The XERJ corpus

Approach only

The author agreed by email on 2026-10-23. Agents read it for how it works, not to copy it. Answers name the repo, the commit and the file:line.

in licence reviewnot in the hub yetsuggested use: approach only

data-extractiondocker-composefastapilocal-firstosintscrapingweb-crawler

Example questions

The prompt

Paste it into your AI agent. It clones crsalass/libercrawl at the pinned commit and indexes it on your machine.

Search the GitHub repository crsalass/libercrawl with XERJ (https://hub.xerj.org/corpus/crsalass--libercrawl).
Run these in a shell, in order, and check each result before the next:
1. xerj --help  (success: it prints usage). If xerj is missing: curl -fsSL https://xerj.org/get | sh
2. git clone https://github.com/crsalass/libercrawl libercrawl && git -C libercrawl checkout d45bd3cb2a638e64829e05dceaf45ff152729484
3. Start a local node and leave it running: xerj --insecure --data-dir ./xerj-data
4. xerj autoindex ./libercrawl --url http://127.0.0.1:9200 --prefix libercrawl --progress plain
5. xerj autoindex map --url http://127.0.0.1:9200
Then answer my question from the indexed files and cite the ax_path of every hit. Query syntax: https://xerj.org/llms.txt
Licence: AGPL-3.0. Read it for the approach only, do not copy code.

Related

Selected work and consent: the XERJ registry.