Junkyard Python Scraper

I wanted a faster way to see what was showing up at the local junkyard without clicking through their website every day. So I built a Python scraper that pulls their inventory, saves it to a CSV, and updates whenever I run it. The code is on GitHub at https://github.com/grafman56/CarPartScraper if you want the real thing; here’s the shape of it and the parts that matter.

The core loop

The script grabs car listings (make, model, year, yard location, arrival date) and appends anything new to a CSV. The skeleton is nothing exotic:

import csv, time, requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "Mozilla/5.0 (inventory checker; personal use)"}

def scrape_page(url):
    r = requests.get(url, headers=HEADERS, timeout=15)
    r.raise_for_status()
    soup = BeautifulSoup(r.text, "html.parser")
    for row in soup.select(".vehicle-row"):   # selector depends on the site
        yield {
            "year":    row.select_one(".year").get_text(strip=True),
            "make":    row.select_one(".make").get_text(strip=True),
            "model":   row.select_one(".model").get_text(strip=True),
            "yard":    row.select_one(".yard").get_text(strip=True),
            "arrived": row.select_one(".date").get_text(strip=True),
        }
    time.sleep(2)   # be polite, it's one small business's web server

Incremental saves are the whole point

A scraper that overwrites its output every run only tells you what’s there today. Appending new rows and keeping the old ones turns it into a history, and the history is where the useful questions get answered: how long do cars actually stick around before they’re crushed, does a certain model show up often enough that I can wait for a better one, which yard is worth the drive this weekend.

Dedupe is what makes incremental mode work. Load the existing CSV into a set of keys first, then only write rows you haven’t seen:

seen = set()
try:
    with open("inventory.csv", newline="") as f:
        seen = {(r["year"], r["make"], r["model"], r["yard"], r["arrived"])
                for r in csv.DictReader(f)}
except FileNotFoundError:
    pass

new_rows = [r for r in all_rows if tuple(r.values()) not in seen]

Once it’s in a CSV, pandas answers the fun questions in a couple of lines:

import pandas as pd
df = pd.read_csv("inventory.csv")
df[df.model.str.contains("325|330")].sort_values("arrived")
df.groupby("model").size().sort_values(ascending=False).head(20)

Caveats worth knowing

Sites change their markup without asking you, so keep the CSS selectors in one place at the top of the script instead of scattered through the code. Check whether the site has an actual JSON endpoint behind the search page before parsing HTML, because half the time the page is just rendering an API response you could request directly. And rate-limit yourself. A two-second sleep between pages costs you nothing on a daily run and keeps you from being the reason they add a captcha.

The scraper doesn’t pull parts for me, but it makes the hunt a lot smarter. Instead of driving to the yard on a hunch, I know the 330 showed up Tuesday.