Staging
v0.8.1
v0.8.1
https://github.com/python/cpython
Revision 122541beceeccce4ef8a9bf739c727ccdcbf2f28 authored by Raymond Hettinger on 13 May 2014, 04:56:33 UTC, committed by Raymond Hettinger on 13 May 2014, 04:56:33 UTC
* Repair the broken link to norobots-rfc.txt. * HTTP response codes >= 500 treated as a failed read rather than as a not found. Not found means that we can assume the entire site is allowed. A 5xx server error tells us nothing. * A successful read() or parse() updates the mtime (which is defined to be "the time the robots.txt file was last fetched"). * The can_fetch() method returns False unless we've had a read() with a 2xx or 4xx response. This avoids false positives in the case where a user calls can_fetch() before calling read(). * I don't see any easy way to test this patch without hitting internet resources that might change or without use of mock objects that wouldn't provide must reassurance.
1 parent 73308d6
Tip revision: 122541beceeccce4ef8a9bf739c727ccdcbf2f28 authored by Raymond Hettinger on 13 May 2014, 04:56:33 UTC
Issue 21469: Mitigate risk of false positives with robotparser.
Issue 21469: Mitigate risk of false positives with robotparser.
Tip revision: 122541b
File | Mode | Size |
---|---|---|
Python-ast.c | -rw-r--r-- | 223.6 KB |
_warnings.c | -rw-r--r-- | 29.7 KB |
asdl.c | -rw-r--r-- | 1.4 KB |
ast.c | -rw-r--r-- | 117.4 KB |
bltinmodule.c | -rw-r--r-- | 74.3 KB |
ceval.c | -rw-r--r-- | 147.0 KB |
ceval_gil.h | -rw-r--r-- | 8.9 KB |
codecs.c | -rw-r--r-- | 38.7 KB |
compile.c | -rw-r--r-- | 124.6 KB |
condvar.h | -rw-r--r-- | 11.0 KB |
dtoa.c | -rw-r--r-- | 81.3 KB |
dup2.c | -rw-r--r-- | 705 bytes |
dynamic_annotations.c | -rw-r--r-- | 6.6 KB |
dynload_aix.c | -rw-r--r-- | 5.6 KB |
dynload_dl.c | -rw-r--r-- | 509 bytes |
dynload_hpux.c | -rw-r--r-- | 1.8 KB |
dynload_next.c | -rw-r--r-- | 3.8 KB |
dynload_shlib.c | -rw-r--r-- | 3.3 KB |
dynload_stub.c | -rw-r--r-- | 186 bytes |
dynload_win.c | -rw-r--r-- | 9.5 KB |
errors.c | -rw-r--r-- | 29.9 KB |
fileutils.c | -rw-r--r-- | 26.8 KB |
formatter_unicode.c | -rw-r--r-- | 48.7 KB |
frozen.c | -rw-r--r-- | 1.6 KB |
frozenmain.c | -rw-r--r-- | 2.6 KB |
future.c | -rw-r--r-- | 4.6 KB |
getargs.c | -rw-r--r-- | 53.7 KB |
getcompiler.c | -rw-r--r-- | 366 bytes |
getcopyright.c | -rw-r--r-- | 511 bytes |
getopt.c | -rw-r--r-- | 3.4 KB |
getplatform.c | -rw-r--r-- | 130 bytes |
getversion.c | -rw-r--r-- | 289 bytes |
graminit.c | -rw-r--r-- | 41.3 KB |
import.c | -rw-r--r-- | 64.7 KB |
importdl.c | -rw-r--r-- | 3.6 KB |
importdl.h | -rw-r--r-- | 544 bytes |
importlib.h | -rw-r--r-- | 226.4 KB |
makeopcodetargets.py | -rwxr-xr-x | 1.2 KB |
marshal.c | -rw-r--r-- | 47.7 KB |
modsupport.c | -rw-r--r-- | 14.1 KB |
mysnprintf.c | -rw-r--r-- | 3.3 KB |
mystrtoul.c | -rw-r--r-- | 8.1 KB |
opcode_targets.h | -rw-r--r-- | 6.2 KB |
peephole.c | -rw-r--r-- | 24.8 KB |
pyarena.c | -rw-r--r-- | 5.5 KB |
pyctype.c | -rw-r--r-- | 7.7 KB |
pyfpe.c | -rw-r--r-- | 623 bytes |
pyhash.c | -rw-r--r-- | 13.0 KB |
pymath.c | -rw-r--r-- | 1.5 KB |
pystate.c | -rw-r--r-- | 25.2 KB |
pystrcmp.c | -rw-r--r-- | 613 bytes |
pystrtod.c | -rw-r--r-- | 38.2 KB |
pythonrun.c | -rw-r--r-- | 81.4 KB |
pytime.c | -rw-r--r-- | 6.5 KB |
random.c | -rw-r--r-- | 7.5 KB |
sigcheck.c | -rw-r--r-- | 529 bytes |
strdup.c | -rw-r--r-- | 247 bytes |
structmember.c | -rw-r--r-- | 8.7 KB |
symtable.c | -rw-r--r-- | 55.5 KB |
sysmodule.c | -rw-r--r-- | 62.3 KB |
thread.c | -rw-r--r-- | 12.4 KB |
thread_foobar.h | -rw-r--r-- | 1.4 KB |
thread_nt.h | -rw-r--r-- | 10.3 KB |
thread_pthread.h | -rw-r--r-- | 17.2 KB |
traceback.c | -rw-r--r-- | 17.3 KB |
Computing file changes ...