All micro-internships
Micro-internship brief ≈14h across 2 daysbuilder

Build a respectful web scraper and publish the dataset

Data engineering starts with getting data nobody hands you. Build a scraper with AI-drafted code for genuinely public data — checking robots.txt first, rate-limiting like a good citizen — and publish the resulting dataset with your code.

AI Application DevelopmentPython for DataWeb DevelopmentWeb Scraping

Posted by The AI Internships

The work brief

  1. 01Pick a public, non-personal data target: government notice boards, college listings, book catalogues, sports stats pages. Check its robots.txt and terms — document what they allow, and pick a different site if they say no.
  2. 02Build the scraper with AI help (Python + requests/BeautifulSoup or Playwright): fetch politely (delays between requests), parse, save to CSV.
  3. 03Expect the site to fight back: inconsistent HTML, pagination, encoding weirdness. Log what broke and how you handled it.
  4. 04Publish code + the collected CSV (100+ rows) in a public repo with a README stating the source, date, and robots.txt status.

What you’ll produce

4 deliverables

Submission standard

Submit the target + ethics check, the repo, the dataset as CSV, and the obstacles log. Scraping personal data or ignoring robots.txt is an automatic fail. Include your AI workflow: tools, best prompts, and what you changed from the AI’s output. Your repo must show the scraper and dataset committed across more than one commit, with the robots.txt status quoted in the README.

  • The target site, what its robots.txt/terms say (quote it), and why this data is fair game

    Written responseRequired

  • Public repo: scraper code + dataset + README

    Public repo: scraper code + dataset + README — commit history should show the scraper and dataset landing across multiple commits (not one paste), and the README must state source, date, and robots.txt status. A single dump commit fails the authenticity gate.

    Public linkRequired

  • A sample of your scraped dataset as CSV (100+ rows in the repo; paste 30+ here)

    Paste comma-separated data with one header row and one record per line. Use consistent column names and remove private information.

    CSV dataRequired

  • What the site threw at you and how you handled it (messy HTML, pagination, encoding…)

    Written responseRequired

You’ll complete these inside your private workspace.

How your work is evaluated

The passing benchmark is 70/100.

Respectful practice

29%

robots.txt genuinely checked and quoted; rate limiting in the code; no personal data.

Working scraper

29%

The code produces the dataset; the data is clean and structured.

Obstacle honesty

14%

The log shows real site-fighting and real solutions.

Real dataset, real build

29%

Fails if the repo is dead/single-dump, the CSV is under 100 rows or looks invented (too-clean, no real-world messiness), or robots.txt isn't genuinely quoted.

How we grade your AI usage

30% of your score

Using AI is the point — it’s the skill this certificate proves. You’ll answer three short questions about how you used it: what you asked, what was wrong with its first answer, and what you changed. Specific, honest answers score high. “I pasted the brief and submitted the answer” scores near zero.