Uses HTTPX to request targets, follow redirects, classify responses, and recurse into HTML-like content without turning the crawler into a full site downloader.
SYSTEM OVERVIEW
See the site behind the site.
HTTPXRay is a focused HTTP spider and link-intelligence utility designed to catalog what a site exposes through crawlable HTML. It follows pages, discovers linked resources, records where links came from, classifies internal and external targets, tracks crawl state, stores request results, and gives the collected dataset a permanent SQLite home.
Version 2.0 turns the original crawler into a more complete analysis environment. The PySide6 desktop interface provides visual exploration and live telemetry, while the companion cli.py exposes the same crawl/archive concept for headless work, scripting, repeatable jobs, remote terminals, and automation.
2INTERFACES
SQLiteCRAWL ARCHIVE
6EXPORT TYPES
100MAX GUI WORKERS
999MAX GUI DEPTH
2.0CURRENT VERSION
CAPABILITIES
Built for crawl visibility and control.
Catalogs anchors, stylesheets, scripts, images, iframes, media sources, video, audio, and form actions discovered in HTML source.
Run strictly inside the target domain or deliberately follow external links when a broader crawl map is required.
Stores visit state, HTTP status, content type, errors, depth, source URL, domain, path, query information, and discovery timestamps.
Search across URLs, sources, link types, content types, domains, and errors to cut large crawl datasets down to the intelligence you need.
Export a complete crawl as an SQLite database, clear the working dataset, begin a fresh crawl, and later import archived crawl data back into HTTPXRay.
Dark terminal-inspired desktop interface with target controls, worker/depth tuning, status metrics, link table, browser actions, live telemetry, and export tools.
The headless CLI provides crawl, stats, list, search, export, database archive, database restore, clear, view, help, and interactive shell operation.
Move crawl intelligence into human-readable reports, structured data, portable text, PDF output, or complete SQLite archives.
CRAWL / ARCHIVE / RESET
A clean database workflow is now part of the design.
HTTPXRay 2.0 treats crawl data as something worth preserving. Instead of leaving every run trapped inside one continuously growing database, you can archive a completed crawl as a native SQLite database and move immediately into a clean working dataset.
1. Crawl
Run HTTPXRay against the target and build the working dataset.
Run HTTPXRay against the target and build the working dataset.
2. Analyze
Filter, inspect, open, copy, search, and review the discovered links and HTTP metadata.
Filter, inspect, open, copy, search, and review the discovered links and HTTP metadata.
3. Export
Generate reports or archive the complete SQLite database.
Generate reports or archive the complete SQLite database.
4. Clear
Reset the working link dataset while leaving the schema ready for another crawl.
Reset the working link dataset while leaving the schema ready for another crawl.
5. Restore
Import a saved HTTPXRay database whenever an earlier crawl needs to be examined again.
Import a saved HTTPXRay database whenever an earlier crawl needs to be examined again.
# archive current crawl
python cli.py db-export target-before-clear.db
# begin clean
python cli.py crawl https://example.com --clear-first
# inspect current database
python cli.py stats
python cli.py list --limit 100
# restore an archive later
python cli.py db-import target-before-clear.dbDatabase portability: the GUI and CLI are designed around the same HTTPXRay 2.0 crawl database concept, making archived datasets useful beyond a single interface.
HEADLESS OPERATION
Full CLI, not a stripped-down fallback.
The HTTPXRay 2.0 command-line interface is intended to be useful on its own. It supports direct subcommands for scripting and automation as well as an interactive httpxray> shell for exploratory terminal work.
| Command | Purpose |
|---|---|
crawl | Crawl a target with optional external following, worker count, maximum depth, and clean-database start. |
stats | Display totals for visited, internal, external, successful, and error records. |
list | Print database records with optional filtering and output limits. |
search | Search crawl data from the interactive shell. |
export | Generate CSV, JSON, TXT, HTML, or PDF output. |
db-export | Archive the complete working SQLite database. |
db-import | Restore an HTTPXRay database archive. |
clear | Clear stored links and prepare the database for another crawl. |
view | Create a browser-readable HTML view of database records. |
shell | Enter the interactive HTTPXRay command console. |
OUTPUT & PORTABILITY
Export everything.
Spreadsheet-friendly structured rows for sorting, filtering, comparison, and downstream processing.
Machine-readable crawl records for APIs, scripts, data pipelines, and application integration.
Simple portable output for terminal review, logs, archival notes, and quick text processing.
Standalone human-readable crawl report with clickable URLs and organized HTTP/link metadata.
Portable document output for sharing, records, reporting, and offline review.
Native full-dataset archive preserving the crawl in a form HTTPXRay can later import and continue analyzing.
PYSIDE6 DESKTOP APPLICATION
Explore the crawl visually.
The HTTPXRay desktop interface provides a dense but readable operations console built around the live crawl. Target URL, external-link behavior, workers, maximum depth, browser integration, database controls, filters, open/copy actions, live telemetry, status metrics, and every export format are kept in one view.
The design intentionally stays close to terminal ergonomics: dark surfaces, phosphor-green telemetry, monospace status language, compact control density, and minimal visual distraction. It is a graphical interface without disguising the fact that HTTPXRay is a technical tool.
WHAT HTTPXRAY RECORDS
More than a list of URLs.
| Field / Category | What it tells you |
|---|---|
| URL / Normalized URL | The discovered address and normalized crawl identity used to prevent duplicate processing. |
| Source URL | The page or resource where the link was discovered. |
| Root URL | The crawl seed associated with the record. |
| Domain / Scheme / Path / Query | Parsed URL components useful for filtering and analysis. |
| Link Type | Whether the record originated from an anchor, script, image, stylesheet/link, iframe, media source, form, or another supported HTML element. |
| Internal / External | Whether the target belongs to the crawl seed domain. |
| Queued / Visited / Visit OK | The crawler state associated with the target. |
| Status Code | The HTTP response status returned during the crawl. |
| Content Type | The response MIME/content classification used during crawl decisions and analysis. |
| Error Text | Request or processing errors captured for failed visits. |
| Depth | How far the target was discovered from the crawl seed. |
| Discovery / Visit Time | Timestamps that provide crawl chronology and historical context. |
DOWNLOADS
Choose your build.
HTTPXRay 2.0
Current release
PySide6 graphical interface, full HTTPXRay CLI, SQLite archive workflow, HTML/PDF/CSV/JSON/TXT export, database import/export, filtering, telemetry, and the current crawler engine.
HTTPXRay 1.0
Legacy release
The original generation remains available for anyone who specifically needs or wants the earlier HTTPXRay build.
RESPONSIBLE USE
Built for visibility, not stealth.
HTTPXRay is intended for development, administration, testing, auditing, research, and analysis of systems you own or are authorized to examine. It is a crawler and dataset tool, not an evasion framework. Crawl responsibly, respect target infrastructure, and tune workers/depth appropriately for the environment being examined.