nearxos/kicad-mcp-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(jlcpcb): download prebuilt catalog instead of broken JLCSearch offset loop (#199)

Browse Source 
The download_jlcpcb_database tool paged the community JLCSearch API with an
offset parameter, but that endpoint is a search front-end that ignores offset
and returns the same first 100 parts on every page, so a full catalog download
was impossible.

Add commands/jlcpcb_downloader.py with a layered strategy that reuses prebuilt
catalogs the whole ecosystem already trusts:
  - CDFER single-file SQLite (primary; no 7z/zip, reliable on Windows)
  - yaqwsx/jlcparts split 7z (fallback; only if a 7z CLI is present)
  - official JLCPCB API (optional; cursor pagination, if credentials set)

Conversion reads CDFER's v_components view (or sniffs the largest table for
yaqwsx), C-prefixes integer lcsc, derives library_type from basic/preferred,
maps mfr->mfr_part, and normalizes price JSON to the manager's [{qty,price}]
shape. Rewire _handle_download_jlcpcb_database to use it (closing/reopening the
manager connection so the on-disk db can be rewritten on Windows). Remove the
broken offset loop from jlcsearch.py (client kept for interactive lookups).
Reduce download_jlcpcb.py to a thin CLI wrapper and update the TS tool schema.

Verified end-to-end against live CDFER: 616k parts downloaded + converted in
~40s, FTS search and price-break parsing correct. New unit tests cover the
conversion and source fall-through; no network in tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
mixelpixx
2026-05-24 11:53:12 -04:00
parent d765bfec78
commit b21b1410eb
6 changed files with 773 additions and 376 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
python/commands/jlcsearch.py
View File

@@ -137,59 +137,12 @@ class JLCSearchClient:
logger.error(f"Failed to get part C{lcsc_number}: {e}")
return None
def download_all_components(
self, callback: Optional[Callable[[int, str], None]] = None, batch_size: int = 100
) -> List[Dict]:
"""
Download all components from jlcsearch database
Note: tscircuit API has a hard-coded 100 result limit per request.
Full catalog download requires ~25,000 paginated requests (~40-60 minutes).
Args:
callback: Optional progress callback function(parts_count, status_msg)
batch_size: Number of parts per batch (max 100 due to API limit)
Returns:
List of all parts
"""
all_parts = []
offset = 0
logger.info("Starting full jlcsearch parts database download...")
while True:
try:
batch = self.search_components("components", limit=batch_size, offset=offset)
# Stop if no results returned (end of catalog)
if not batch or len(batch) == 0:
break
all_parts.extend(batch)
offset += len(batch)
if callback:
callback(len(all_parts), f"Downloaded {len(all_parts)} parts...")
else:
logger.info(f"Downloaded {len(all_parts)} parts so far...")
# Continue pagination - API returns exactly 100 results per page until exhausted
# Only stop when we get 0 results (handled above)
# Rate limiting - be nice to the API
time.sleep(0.1)
except Exception as e:
logger.error(f"Error downloading parts at offset {offset}: {e}")
if len(all_parts) > 0:
logger.warning(f"Partial download available: {len(all_parts)} parts")
return all_parts
else:
raise
logger.info(f"Download complete: {len(all_parts)} parts retrieved")
return all_parts
# NOTE: bulk catalog download was removed (issue #199). The JLCSearch
# endpoint is a *search front-end* that ignores the ``offset`` parameter,
# so offset-paged "download everything" loops returned the same first 100
# parts forever. Full-catalog download now uses a prebuilt source via
# ``commands.jlcpcb_downloader.download_database()``. This client remains
# for interactive/parametric lookups only (search_components, etc.).
def test_jlcsearch_connection() -> bool: