Skip to main content

Browse the archive

Staging
v0.5.1
https://github.com/python/cpython
09 December 2020, 11:06:31 UTC
Revision 122541beceeccce4ef8a9bf739c727ccdcbf2f28 authored by Raymond Hettinger on 13 May 2014, 04:56:33 UTC, committed by Raymond Hettinger on 13 May 2014, 04:56:33 UTC 
Issue 21469:  Mitigate risk of false positives with robotparser.
 
* Repair the broken link to norobots-rfc.txt.

* HTTP response codes >= 500 treated as a failed read rather than as a not
found.  Not found means that we can assume the entire site is allowed.  A 5xx
server error tells us nothing.

* A successful read() or parse() updates the mtime (which is defined to be "the
  time the robots.txt file was last fetched").

* The can_fetch() method returns False unless we've had a read() with a 2xx or
4xx response.  This avoids false positives in the case where a user calls
can_fetch() before calling read().

* I don't see any easy way to test this patch without hitting internet
resources that might change or without use of mock objects that wouldn't
provide must reassurance.
1 parent 73308d6
Raw File
Tip revision: 122541beceeccce4ef8a9bf739c727ccdcbf2f28 authored by Raymond Hettinger on 13 May 2014, 04:56:33 UTC
Issue 21469: Mitigate risk of false positives with robotparser.
Tip revision: 122541b
.gitignore 
*.cover
*.o
*.orig
*.pyc
*.pyd
*.pyo
*.rej
*.swp
*~
.gdb_history
Doc/build/
Doc/tools/docutils/
Doc/tools/jinja/
Doc/tools/jinja2/
Doc/tools/pygments/
Doc/tools/sphinx/
Lib/lib2to3/*.pickle
Lib/test/data/*
Lib/_sysconfigdata.py
Lib/plat-mac/errors.rsrc.df.rsrc
Makefile
Makefile.pre
Misc/python.pc
Misc/python-config.sh
Modules/Setup
Modules/Setup.config
Modules/Setup.local
Modules/config.c
Modules/ld_so_aix
Modules/_freeze_importlib
Modules/_testembed
PCbuild/*.bsc
PCbuild/*.dll
PCbuild/*.exe
PCbuild/*.exp
PCbuild/*.lib
PCbuild/*.ncb
PCbuild/*.o
PCbuild/*.pdb
PCbuild/Win32-temp-*
PCbuild/amd64/
.purify
Parser/pgen
__pycache__
autom4te.cache
build/
buildno
config.cache
config.log
config.status
config.status.lineno
core
db_home
config.log
config.status
libpython*.a
libpython*.so*
platform
pybuilddir.txt
pyconfig.h
python
python-config
python-config.py
python.exe
python-gdb.py
python.exe-gdb.py
reflog.txt
.svn/
tags
TAGS
.coverage
coverage/
htmlcov/
back to top