All internships

AI Micro-Internship · ~240 min · builder

Build a respectful web scraper and publish the dataset

Data engineering starts with getting data nobody hands you. Build a scraper for genuinely public data — checking robots.txt first, rate-limiting like a good citizen — and publish the resulting dataset with your code.

Steps

  1. 01Pick a public, non-personal data target: government notice boards, college listings, book catalogues, sports stats pages. Check its robots.txt and terms — document what they allow, and pick a different site if they say no.
  2. 02Build the scraper with AI help (Python + requests/BeautifulSoup or Playwright): fetch politely (delays between requests), parse, save to CSV.
  3. 03Expect the site to fight back: inconsistent HTML, pagination, encoding weirdness. Log what broke and how you handled it.
  4. 04Publish code + the collected CSV (100+ rows) in a public repo with a README stating the source, date, and robots.txt status.

Tools: Python + an AI assistant, A public data source, GitHub

Your submission

Submit the target + ethics check, the repo, the dataset as CSV, and the obstacles log. Scraping personal data or ignoring robots.txt is an automatic fail.