# Scraping data

#### Outline
2. Scraping data
  - OpenAI ruined it for everybody
    - reddit
    - GitHub (GitLab has always been tricky)
    - StackOverflow
    - news sites
    - Twitter
  - LLMs have poluted the infosphere (internet search with "before:2023")
  - `wget` recursively
  - `requests.get`
  - `pandas` and `bz2` dataframe for DB 
  - `pandas.read_html`
  - `bs4` to extract title, subtitle, hrefs (show notes, transcripts) 
  - `re`
  - new T.W.A.T. episodes (anyone want to help)
  - robustness ( `getattr`, `dict.get`, `if obj or` )
