2020/02/09
## virtualenv: pycas2020
Good Way to Learn Python!
But, Who Actually Reads These A to Z?
(spoiler: not me)
me and my programming books
The Main Things to Know in a Web Scraping Project:
- Is it worth the trouble?
- Is it ethical?
- Tools available in
BeautifulSoupandrequests - What to look for in html code
- Parsing json objects with
json - Rudimentary
pandasskills <pro-tip> All you need to know about html is how tags work </pro-tip>
What to Look for in a Scraping Project:
- A sizeable amount of structured data with a regular repeatable format.
- Identical formating is not required, but the more edge cases present, the more complicated the scraping will be.
Ethics in Scraping
Accessing vast troves of information can be intoxicating.
Just because it’s possible doesn’t mean it should be done
Legal Considerations
(note: I have zero legal training - this is not legal advice!)
- Are you scraping copyrighted material?
- Will your scraping activity compromise individual privacy?
- Are you making a large number of request that may overload or damage a server?
- Is it possible the scraping will expose intellectual property you do not own?
- Are there terms of service governing use of the website and are you following those?
- Will your scraping activities diminish the value of the original data?