Staging
v0.5.1
v0.5.1
https://github.com/python/cpython
Revision 122541beceeccce4ef8a9bf739c727ccdcbf2f28 authored by Raymond Hettinger on 13 May 2014, 04:56:33 UTC, committed by Raymond Hettinger on 13 May 2014, 04:56:33 UTC
* Repair the broken link to norobots-rfc.txt. * HTTP response codes >= 500 treated as a failed read rather than as a not found. Not found means that we can assume the entire site is allowed. A 5xx server error tells us nothing. * A successful read() or parse() updates the mtime (which is defined to be "the time the robots.txt file was last fetched"). * The can_fetch() method returns False unless we've had a read() with a 2xx or 4xx response. This avoids false positives in the case where a user calls can_fetch() before calling read(). * I don't see any easy way to test this patch without hitting internet resources that might change or without use of mock objects that wouldn't provide must reassurance.
1 parent 73308d6
Tip revision: 122541beceeccce4ef8a9bf739c727ccdcbf2f28 authored by Raymond Hettinger on 13 May 2014, 04:56:33 UTC
Issue 21469: Mitigate risk of false positives with robotparser.
Issue 21469: Mitigate risk of false positives with robotparser.
Tip revision: 122541b
| File | Mode | Size |
|---|---|---|
| VS9.0 | ||
| bdist_wininst | ||
| example_nt | ||
| icons | ||
| WinMain.c | -rw-r--r-- | 449 bytes |
| _msi.c | -rw-r--r-- | 30.7 KB |
| config.c | -rw-r--r-- | 5.0 KB |
| dl_nt.c | -rw-r--r-- | 4.3 KB |
| dllbase_nt.txt | -rw-r--r-- | 3.5 KB |
| empty.c | -rw-r--r-- | 137 bytes |
| errmap.h | -rw-r--r-- | 2.1 KB |
| errmap.mak | -rw-r--r-- | 95 bytes |
| frozen_dllmain.c | -rw-r--r-- | 3.9 KB |
| generrmap.c | -rw-r--r-- | 849 bytes |
| getpathp.c | -rw-r--r-- | 24.5 KB |
| icons.mak | -rw-r--r-- | 213 bytes |
| icons.rc | -rw-r--r-- | 59 bytes |
| launcher.c | -rw-r--r-- | 47.8 KB |
| launcher.ico | -rw-r--r-- | 19.3 KB |
| make_versioninfo.c | -rw-r--r-- | 1.2 KB |
| msvcrtmodule.c | -rw-r--r-- | 14.0 KB |
| py.ico | -rw-r--r-- | 19.3 KB |
| pyc.ico | -rw-r--r-- | 19.3 KB |
| pycon.ico | -rw-r--r-- | 19.3 KB |
| pyconfig.h | -rw-r--r-- | 20.5 KB |
| pylauncher.rc | -rw-r--r-- | 1.3 KB |
| python3.def | -rw-r--r-- | 35.0 KB |
| python3.mak | -rw-r--r-- | 611 bytes |
| python34gen.py | -rw-r--r-- | 663 bytes |
| python34stub.def | -rw-r--r-- | 13.4 KB |
| python3dll.c | -rw-r--r-- | 134 bytes |
| python_exe.rc | -rw-r--r-- | 49 bytes |
| python_nt.rc | -rw-r--r-- | 1.8 KB |
| readme.txt | -rw-r--r-- | 3.8 KB |
| testpy.py | -rw-r--r-- | 836 bytes |
| winreg.c | -rw-r--r-- | 59.7 KB |
| winsound.c | -rw-r--r-- | 5.5 KB |
Computing file changes ...
