Skip to main content

Browse the archive

Staging
v0.5.1
https://github.com/python/cpython
09 December 2020, 11:06:31 UTC
Revision 122541beceeccce4ef8a9bf739c727ccdcbf2f28 authored by Raymond Hettinger on 13 May 2014, 04:56:33 UTC, committed by Raymond Hettinger on 13 May 2014, 04:56:33 UTC 
Issue 21469:  Mitigate risk of false positives with robotparser.
 
* Repair the broken link to norobots-rfc.txt.

* HTTP response codes >= 500 treated as a failed read rather than as a not
found.  Not found means that we can assume the entire site is allowed.  A 5xx
server error tells us nothing.

* A successful read() or parse() updates the mtime (which is defined to be "the
  time the robots.txt file was last fetched").

* The can_fetch() method returns False unless we've had a read() with a 2xx or
4xx response.  This avoids false positives in the case where a user calls
can_fetch() before calling read().

* I don't see any easy way to test this patch without hitting internet
resources that might change or without use of mock objects that wouldn't
provide must reassurance.
1 parent 73308d6
Raw File
Tip revision: 122541beceeccce4ef8a9bf739c727ccdcbf2f28 authored by Raymond Hettinger on 13 May 2014, 04:56:33 UTC
Issue 21469: Mitigate risk of false positives with robotparser.
Tip revision: 122541b
pythonw.vcxproj.filters 
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup>
    <Filter Include="Resource Files">
      <UniqueIdentifier>{0434cf11-a311-4a92-8a6c-4164aa79a7f2}</UniqueIdentifier>
    </Filter>
    <Filter Include="Source Files">
      <UniqueIdentifier>{e1d8ea6b-c65d-42f4-9eed-6010846ed378}</UniqueIdentifier>
    </Filter>
  </ItemGroup>
  <ItemGroup>
    <ResourceCompile Include="..\PC\python_exe.rc">
      <Filter>Resource Files</Filter>
    </ResourceCompile>
  </ItemGroup>
  <ItemGroup>
    <ClCompile Include="..\PC\WinMain.c">
      <Filter>Source Files</Filter>
    </ClCompile>
  </ItemGroup>
</Project>
back to top