bhoot found that download_cdfer 416-loops when a complete cdfer.sqlite3 already
exists in the cache (e.g. a prior run downloaded it but died before
conversion/cleanup): resuming a complete file sends Range: bytes=<size>- which
the server answers with HTTP 416, and raise_for_status treated that as a
retryable error, spinning until max_retries then failing.
Fix: HEAD once for Content-Length up front and short-circuit when the local file
already matches the full size; and in the loop, treat a 416 as "complete" when
the size matches (else drop the stale partial and restart fresh). Remove the now
-unused _head_last_modified helper (HEAD is done inline). Add a no-network test.
Verified live: happy path still downloads+converts 616k parts (~50s); a
pre-existing complete file now short-circuits in ~0.4s instead of looping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
People want to be able to pull the whole catalog, and prior downloads stalled or
repeated the same parts. Two fixes:
- Full catalog correctness: yaqwsx's cache.sqlite3 (the full ~10GB set) stores
category/manufacturer as IDs with no v_components view, so the convert left
those fields blank. Build an equivalent v_components join for yaqwsx-style
sources so the full catalog converts with category/subcategory/manufacturer
populated. Clarify in the tool schema that source="yaqwsx" = full catalog
(needs 7z) vs cdfer = in-stock subset.
- Resumable downloads: CDFER stream download now uses a read timeout and resumes
from the partial file via HTTP Range on interruption (up to 5 retries); the
yaqwsx curl calls use -C - / --retry. Addresses the stall/partial-download
complaints.
Adds a yaqwsx-schema conversion test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per review feedback: CDFER's upstream scraper pipeline has stalled for weeks
(broken cart API + a bug in their scraper), so a "fresh download" can still be
old data. Compute the catalog age from the source Last-Modified header, expose
catalog_age_days, and emit a stale=True + warning when older than 14 days that
points users to source='yaqwsx' (fresh, needs 7z) or source='official'. Surface
the warning in the MCP tool output.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The download_jlcpcb_database tool paged the community JLCSearch API with an
offset parameter, but that endpoint is a search front-end that ignores offset
and returns the same first 100 parts on every page, so a full catalog download
was impossible.
Add commands/jlcpcb_downloader.py with a layered strategy that reuses prebuilt
catalogs the whole ecosystem already trusts:
- CDFER single-file SQLite (primary; no 7z/zip, reliable on Windows)
- yaqwsx/jlcparts split 7z (fallback; only if a 7z CLI is present)
- official JLCPCB API (optional; cursor pagination, if credentials set)
Conversion reads CDFER's v_components view (or sniffs the largest table for
yaqwsx), C-prefixes integer lcsc, derives library_type from basic/preferred,
maps mfr->mfr_part, and normalizes price JSON to the manager's [{qty,price}]
shape. Rewire _handle_download_jlcpcb_database to use it (closing/reopening the
manager connection so the on-disk db can be rewritten on Windows). Remove the
broken offset loop from jlcsearch.py (client kept for interactive lookups).
Reduce download_jlcpcb.py to a thin CLI wrapper and update the TS tool schema.
Verified end-to-end against live CDFER: 616k parts downloaded + converted in
~40s, FTS search and price-break parsing correct. New unit tests cover the
conversion and source fall-through; no network in tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
