Can we replace a paid scraping service for daily Instagram and YouTube intake with a free route run from a home connection, and what do we lose?

Hypothesis

Our daily media intake watches a list of creator accounts on Instagram and YouTube, fetches new posts and transcribes them. It ran on a paid scraping platform. The suspicion was that the cost came from backfills, not from the daily run, and that most of what we pay for could be done for free from a residential connection, where platforms are far less hostile than they are to datacenter addresses. If so, the paid service only needs to be the fallback.

What we built

• A cost baseline. The paid plan (39 USD a month including 39 USD of usage) hit its hard cap on 2026-10-09. A normal daily run of the weekly scrape chain costs about 0.73 USD. The 30-day and 90-day backfills on October 5 to 8 cost about 95 USD. We raised the cap to 120 USD as a stopgap.
• A hand-tested free route. Before building anything we probed each source by hand:
  • YouTube channel feeds never blocked us.
  • youtube-transcript-api from a home IP was blocked after about 19 fast requests. At one request per 90 seconds, 28 of 29 ran clean.
  • Anonymous Instagram access was rate limited at once. A logged-in instagrapi session fetched 15 of 15 reels with video URLs at 20 second pacing.
  • One reel went to a local Whisper server and came back as 532 words in 55 seconds.
• A full-size run. 54 Instagram accounts over a four day window, run October 12 and 13, then compared against the paid service's output.
• A service. An intake-fetch container on the DGX Spark, on the home line. The server-side syncs call it first ("primary": "free" in each config), and the paid scraper is the per-account and per-run backup.

Learnings

• Where you run it matters more than what you run. The same libraries that get blocked from a datacenter worked from the home connection. The service has to live there, not on the server fleet.
• Pacing is the whole trick. 19 fast transcript requests got blocked; one every 90 seconds did not. Instagram at 20 seconds ran 52 of 54 accounts without a single block.
• Free covers most, not all. Over four days the free route found 141 reels and the paid service 229, with 138 in common. Whisper transcribed 140 of the 144 reels the free route fetched, in 32 minutes. The gap is mostly reels that do not show on the profile listing (likely trial reels or duplicate variants), which the library never lists. Two accounts no longer resolved at all.
• YouTube feeds undercount. They cap at 15 videos per channel, so one channel lost 25 videos, and they include Shorts (32 extra). Listing through the YouTube Data API is the fix. Transcripts: 86 of 92 free at the 90 second pace.
• Library versions matter. An older instagrapi release was refused by the platform as out of date, while the current one worked.
• A caveat worth stating. The Instagram route uses a real, logged-in account and an unofficial library. That is fragile and sits in a grey area of the platform's terms, which is why we kept the paid backup instead of deleting it.
• Not yet measured. We have not published a before and after bill for the new setup, so we make no savings claim yet.

Log

• 2026-10-15: Free route built and live, with the paid scraper as backup. YouTube listing still runs on the paid service because the YouTube Data API was not enabled on the project we tried.
• 2026-10-12: Full-size run over 54 Instagram accounts: 52 fetched, 144 reels, 32 minutes, 140 transcribed, no blocks. YouTube transcripts 86 of 92.
• 2026-10-09: Paid plan hits its hard cap. Backfills identified as the cost. Cap raised to 120 USD. Free route hand-tested.