CrawlLit

An intelligent, multi-threaded web crawler with smart priority scoring for efficient content discovery. CrawlLit uses text-based relevance scoring to prioritize pages based on your search query, making it perfect for targeted web scraping and content discovery

A new way for AI to search for data

View on GitHub

1GitHub stars

1commits on main

0forks

Python 81.6%, Batchfile 18.4% · created 2025-10-29 · last push 2025-10-29

What XERJ indexed

Snapshot

Whole repository

The whole repository at commit 2ffe3e080a. It is the newest commit on main.

Selected work

Tuhin Chakrabarty
@Tuhin-SnapD

Licence

The repository

NONE

LICENSE file checked at the pinned commit; the author agreed to inclusion.

The XERJ corpus

NONE, with credit

The owner added it on hub.xerj.org after signing in with GitHub, on 2026-10-29. Same terms as the repository. Answers name the repo, the commit and the file:line.

selected for the XERJ Corpus Hublicence clearedsuggested use: adapt with attribution

The prompt

Paste it into your AI agent. It clones Tuhin-SnapD/CrawlLit at the pinned commit and indexes it on your machine.

Search the GitHub repository Tuhin-SnapD/CrawlLit with XERJ (https://hub.xerj.org/corpus/tuhin-snapd--crawllit).
Run these in a shell, in order, and check each result before the next:
1. xerj --help  (success: it prints usage). If xerj is missing: curl -fsSL https://xerj.org/get | sh
2. git clone https://github.com/Tuhin-SnapD/CrawlLit CrawlLit && git -C CrawlLit checkout 2ffe3e080a62ff5ac07a6e0aef6ca7eecd324dc7
3. Start a local node and leave it running: xerj --insecure --data-dir ./xerj-data
4. xerj autoindex ./CrawlLit --url http://127.0.0.1:9200 --prefix crawllit --progress plain
5. xerj autoindex map --url http://127.0.0.1:9200
Then answer my question from the indexed files and cite the ax_path of every hit. Query syntax: https://xerj.org/llms.txt
Licence: check the LICENSE file. Keep the copyright notice and credit the authors.

Tuhin Chakrabarty Web and frontend platform corpora

Stats: GitHub API, fetched 2026-10-29. Selected work and consent: the XERJ registry.