Staging
v0.5.1
v0.5.1
https://github.com/python/cpython
Revision 122541beceeccce4ef8a9bf739c727ccdcbf2f28 authored by Raymond Hettinger on 13 May 2014, 04:56:33 UTC, committed by Raymond Hettinger on 13 May 2014, 04:56:33 UTC
* Repair the broken link to norobots-rfc.txt. * HTTP response codes >= 500 treated as a failed read rather than as a not found. Not found means that we can assume the entire site is allowed. A 5xx server error tells us nothing. * A successful read() or parse() updates the mtime (which is defined to be "the time the robots.txt file was last fetched"). * The can_fetch() method returns False unless we've had a read() with a 2xx or 4xx response. This avoids false positives in the case where a user calls can_fetch() before calling read(). * I don't see any easy way to test this patch without hitting internet resources that might change or without use of mock objects that wouldn't provide must reassurance.
1 parent 73308d6
Tip revision: 122541beceeccce4ef8a9bf739c727ccdcbf2f28 authored by Raymond Hettinger on 13 May 2014, 04:56:33 UTC
Issue 21469: Mitigate risk of false positives with robotparser.
Issue 21469: Mitigate risk of false positives with robotparser.
Tip revision: 122541b
.hgignore
.gdb_history
.purify
.svn/
^.idea/
.DS_Store
Makefile$
Makefile.pre$
TAGS$
autom4te.cache$
^build/
^Doc/build/
buildno$
config.cache
config.log
config.status
config.status.lineno
db_home
platform$
pyconfig.h$
python$
python.exe$
python-config$
python-config.py$
reflog.txt$
tags$
Lib/plat-mac/errors.rsrc.df.rsrc
Misc/python.pc
Misc/python-config.sh$
Modules/Setup$
Modules/Setup.config
Modules/Setup.local
Modules/config.c
Modules/ld_so_aix$
Parser/pgen$
^lcov-report/
^core
^python-gdb.py
^python.exe-gdb.py
^pybuilddir.txt
syntax: glob
libpython*.a
libpython*.so*
*.swp
*.o
*.pyc
*.pyo
*.pyd
*.cover
*~
Lib/lib2to3/*.pickle
Lib/test/data/*
Misc/*.wpu
PC/python_nt*.h
PC/pythonnt_rc*.h
PC/*.obj
PC/*.exe
PC/*/*.exe
PC/*/*.exp
PC/*/*.lib
PC/*/*.bsc
PC/*/*.dll
PC/*/*.pdb
PC/*/*.user
PC/*/*.ncb
PC/*/*.suo
PC/*/Win32-temp-*
PC/*/x64-temp-*
PC/*/amd64
PCbuild/*.exe
PCbuild/*.dll
PCbuild/*.pdb
PCbuild/*.lib
PCbuild/*.exp
PCbuild/*.o
PCbuild/*.ncb
PCbuild/*.bsc
PCbuild/*.user
PCbuild/*.suo
PCbuild/*.*sdf
PCbuild/Win32-temp-*
PCbuild/x64-temp-*
PCbuild/*-pgi
PCbuild/*-pgo
PCbuild/amd64
PCbuild/ipch
Tools/unicode/build/
Tools/unicode/MAPPINGS/
BuildLog.htm
__pycache__
Modules/_freeze_importlib
Modules/_testembed
.coverage
coverage/
htmlcov/
*.gcda
*.gcno
*.gcov
coverage.info
Computing file changes ...
