shway
/
Notepad
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Notepad/enter/Coding Tips (Classical)/Terminal Tips/GUIs/Internet/Websites/Robots.txt Files.md

411 B
Raw Blame History

Robots.txt is an increasingly important file found on websites that determine whether you permit a website crawler to index your page for search engine optimization. As webscraping is entirely legal in the US, this is the wild west of scraping and thus I want to keep mu brain and information safe from scraping.

Resources: