shway
/
Notepad
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Updated: Sunday, November 05,2023-11-05 01:02:59

main
Shwetha Jayaraj 2023-11-05 01:03:01 -04:00
parent 4b96dbd58c
commit 03ff2da472
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
enter/Coding Tips (Classical)/Terminal Tips/GUIs/Internet/Websites/Robots.txt Files.md
View File

@ -1,7 +1,7 @@
Robots.txt is an increasingly important file found on websites that determine whether you permit a website crawler to index your page for search engine optimization. As web-scraping is entirely legal in the US, this is the wild west of scraping and thus I want to keep mu brain and information safe from scraping.
Fun Fact: Google [open-sourced](https://opensource.googleblog.com/2019/07/googles-robotstxt-parser-is-now-open.html) their [robots.txt parser](https://github.com/google/robotstxt) in 2019 f you want to see an example of reverse engineering the robots.txt file for search indexing.
Fun Fact: Google [open-sourced](https://opensource.googleblog.com/2019/07/googles-robotstxt-parser-is-now-open.html) their [robots.txt parser](https://github.com/google/robotstxt) in 2019 if you want to see an example of reverse engineering the robots.txt file for search indexing.
*Resources*:
- [Robots.txt file examples](https://blog.hubspot.com/marketing/robots-txt-file)