6 lines
411 B
Markdown
6 lines
411 B
Markdown
|
|
Robots.txt is an increasingly important file found on websites that determine whether you permit a website crawler to index your page for search engine optimization. As webscraping is entirely legal in the US, this is the wild west of scraping and thus I want to keep mu brain and information safe from scraping.
|
|
|
|
|
|
*Resources*:
|
|
- [Robots.txt file examples](https://blog.hubspot.com/marketing/robots-txt-file) |