Staging
v0.5.1
v0.5.1
https://github.com/python/cpython
Tip revision: 09a7df8301bef5a5ac957371ae9e19c68acdca0a authored by Georg Brandl on 19 December 2010, 12:33:52 UTC
#3243 follow-up: remove debugging print and fix docs; data is a bytes object.
#3243 follow-up: remove debugging print and fix docs; data is a bytes object.
Tip revision: 09a7df8
3.2.rst
****************************
What's New In Python 3.2
****************************
:Author: Raymond Hettinger
:Release: |release|
:Date: |today|
.. $Id$
Rules for maintenance:
* Anyone can add text to this document. Do not spend very much time
on the wording of your changes, because your text will probably
get rewritten.
* The maintainer will go through Misc/NEWS periodically and add
changes; it's therefore more important to add your changes to
Misc/NEWS than to this file.
* This is not a complete list of every single change; completeness
is the purpose of Misc/NEWS. Some changes I consider too small
or esoteric to include. If such a change is added to the text,
I'll just remove it. (This is another reason you shouldn't spend
too much time on writing your addition.)
* If you want to draw your new text to the attention of the
maintainer, add 'XXX' to the beginning of the paragraph or
section.
* It's OK to just add a fragmentary note about a change. For
example: "XXX Describe the transmogrify() function added to the
socket module." The maintainer will research the change and
write the necessary text.
* You can comment out your additions if you like, but it's not
necessary (especially when a final release is some months away).
* Credit the author of a patch or bugfix. Just the name is
sufficient; the e-mail address isn't necessary. It's helpful to
add the issue number:
XXX Describe the transmogrify() function added to the socket
module.
(Contributed by P.Y. Developer; :issue:`12345`.)
This saves the maintainer the effort of going through the SVN log
when researching a change.
This article explains the new features in Python 3.2 as compared to 3.1. It
focuses on a few highlights and gives a few examples. For full details, see the
:source:`Misc/NEWS <Misc/NEWS>` file.
PEP 384: Defining a Stable ABI
==============================
In the past, extension modules built for one Python version were often
not usable with other Python versions. Particularly on Windows, every
feature release of Python required rebuilding all extension modules that
one wanted to use. This requirement was the result of the free access to
Python interpreter internals that extension modules could use.
With Python 3.2, an alternative approach becomes available: extension
modules which restrict themselves to a limited API (by defining
Py_LIMITED_API) cannot use many of the internals, but are constrained
to a set of API functions that are promised to be stable for several
releases. As a consequence, extension modules built for 3.2 in that
mode will also work with 3.3, 3.4, and so on. Extension modules that
make use of details of memory structures can still be built, but will
need to be recompiled for every feature release.
.. seealso::
:pep:`384` - Defining a Stable ABI
PEP written by Martin von Löwis.
PEP 389: Argparse Command Line Parsing Module
=============================================
A new module for command line parsing, :mod:`argparse`, was introduced to
overcome the limitations of :mod:`optparse` which did not provide support for
positional arguments (not just options), subcommands, required options and other
common patterns of specifying and validating options.
This module has already has wide-spread success in the community as a
third-party module. Being more fully featured than its predecessor, the
:mod:`argparse` module is now the preferred module for command-line processing.
The older module is still being kept available because of the substantial amount
of legacy code that depends on it.
Here's an annotated example parser showing features like limiting results to a
set of choices, specifying a *metavar* in the help screen, validating that one
or more positional arguments is present, and making a required option::
import argparse
parser = argparse.ArgumentParser(
description = 'Manage servers', # main description for help
epilog = 'Tested on Solaris and Linux') # displayed after help
parser.add_argument('action', # argument name
choices = ['deploy', 'start', 'stop'], # one of four allowed values
help = 'action on each target') # help msg
parser.add_argument('targets',
metavar = 'HOSTNAME', # var name used in help msg
nargs = '+', # require 1 or more targets
help = 'url for target machines') # help msg explanation
parser.add_argument('-u', '--user', # -u or --user option
required = True, # make this a required argument
help = 'login as user')
Example of calling the parser on a command string::
>>> cmd = 'deploy sneezy.example.com sleepy.example.com -u skycaptain'
>>> result = parser.parse_args(cmd.split())
>>> result.action
'deploy'
>>> result.targets
['sneezy.example.com', 'sleepy.example.com']
>>> result.user
'skycaptain'
Example of the parser's automatically generated help::
>>> parser.parse_args('-h'.split())
usage: manage_cloud.py [-h] -u USER
{deploy,start,stop} HOSTNAME [HOSTNAME ...]
Manage servers
positional arguments:
{deploy,start,stop} action on each target
HOSTNAME url for target machines
optional arguments:
-h, --help show this help message and exit
-u USER, --user USER login as user
Tested on Solaris and Linux
An especially nice :mod:`argparse` feature is the ability to define subparsers,
each with their own argument patterns and help displays::
import argparse
parser = argparse.ArgumentParser(prog='HELM')
subparsers = parser.add_subparsers()
parser_l = subparsers.add_parser('launch', help='Launch Control') # first subgroup
parser_l.add_argument('-m', '--missiles', action='store_true')
parser_l.add_argument('-t', '--torpedos', action='store_true')
parser_m = subparsers.add_parser('move', help='Move Vessel', # second subgroup
aliases=('steer', 'turn')) # equivalent names
parser_m.add_argument('-c', '--course', type=int, required=True)
parser_m.add_argument('-s', '--speed', type=int, default=0)
$ ./helm.py --help # top level help (launch and move)
$ ./helm.py launch --help # help for launch options
$ ./helm.py launch --missiles # set missiles=True and torpedos=False
$ ./helm.py steer --course 180 --speed 5 # set movement parameters
.. seealso::
:pep:`389` - New Command Line Parsing Module
PEP written by Steven Bethard.
:ref:`upgrading-optparse-code` for details on the differences from
:mod:`optparse`.
PEP 391: Dictionary Based Configuration for Logging
====================================================
The :mod:`logging` module provided two kinds of configuration, one style with
function calls for each option or another style driven by an external file saved
in a :mod:`ConfigParser` format. Those options did not provide the flexibility
to create configurations from JSON or YAML files, nor did they support
incremental configuration, which is needed for specifying logger options from a
command line.
To support a more flexible style, the module now offers
:func:`logging.config.dictConfig` for specifying logging configuration with
plain Python dictionaries. The configuration options include formatters,
handlers, filters, and loggers. Here's a working example of a configuration
dictionary::
{"version": 1,
"formatters": {"brief": {"format": "%(levelname)-8s: %(name)-15s: %(message)s"},
"full": {"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"},
},
"handlers": {"console": {
"class": "logging.StreamHandler",
"formatter": "brief",
"level": "INFO",
"stream": "ext://sys.stdout"},
"console_priority": {
"class": "logging.StreamHandler",
"formatter": "full",
"level": "ERROR",
"stream": "ext://sys.stderr"},
},
"root": {"level": "DEBUG", "handlers": ["console", "console_priority"]}}
If that dictionary is stored in a file called :file:`conf.json`, it can loaded
and called with code like this::
>>> import logging.config
>>> logging.config.dictConfig(json.load(open('conf.json', 'rb')))
>>> logging.info("Transaction completed normally")
>>> logging.critical("Abnormal termination")
.. seealso::
:pep:`391` - Dictionary Based Configuration for Logging
PEP written by Vinay Sajip.
PEP 3148: The ``concurrent.futures`` module
============================================
Code for creating and managing concurrency is being collected in a new toplevel
namespace, *concurrent*. Its first member is a *futures* package which provides
a uniform high level interface for managing threads and processes.
The design for :mod:`concurrent.futures` was inspired by
*java.util.concurrent.package*. In that model, a running call and its result
are represented by a :class:`~concurrent.futures.Future` object which abstracts
features common to threads, processes, and remote procedure calls. That object
supports status checks (running or done), timeouts, cancellations, adding
callbacks, and access to results or exceptions.
The primary offering of the new module is a pair of executor classes for
launching and managing calls. The goal of the executors is to make it easier to
use existing tools for making parallel calls. They save the effort needed to
setup a pool of resources, launch the calls, create a results queue, add
time-out handling, and limit the total number of threads, processes, or remote
procedure calls.
Ideally, each application should share a single executor across multiple
components so that process and thread limits can be centrally managed. This
solves the design challenge that arises when each component has its own
competing strategy for resource management.
Both classes share a common interface with three methods:
:meth:`~concurrent.futures.Executor.submit` for scheduling a callable and
returning a :class:`~concurrent.futures.Future` object;
:meth:`~concurrent.futures.Executor.map` for scheduling many asynchronous calls
at a time, and :meth:`~concurrent.futures.Executor.shutdown` for freeing
resources. The class is a :term:`context manager` and can be used within a
:keyword:`with` statement to assure that resources are automatically released
when currently pending futures are done executing.
A simple of example of :class:`~concurrent.futures.ThreadPoolExecutor` is a
launch of four parallel threads for copying files::
import shutil
with ThreadPoolExecutor(max_workers=4) as e:
e.submit(shutil.copy, 'src1.txt', 'dest1.txt')
e.submit(shutil.copy, 'src2.txt', 'dest2.txt')
e.submit(shutil.copy, 'src3.txt', 'dest3.txt')
e.submit(shutil.copy, 'src3.txt', 'dest4.txt')
.. seealso::
:pep:`3148` - Futures -- Execute Computations Asynchronously
PEP written by Brian Quinlan.
:ref:`Code for Threaded Parallel URL reads<threadpoolexecutor-example>`, an
example using threads to fetch multiple web pages in parallel.
:ref:`Code for computing prime numbers in
parallel<processpoolexecutor-example>`, an example demonstrating
:class:`~concurrent.futures.ProcessPoolExecutor`.
PEP 3147: PYC Repository Directories
=====================================
Python's scheme for caching bytecode in *.pyc* files did not work well in
environments with multiple python interpreters. If one interpreter encountered
a cached file created by another interpreter, it would recompile the source and
overwrite the cached file, thus losing the benefits of caching.
The issue of "pyc fights" has become more pronounced as it has become
commonplace for Linux distributions to ship with multiple versions of Python.
These conflicts also arise with CPython alternatives such as Unladen Swallow.
To solve this problem, Python's import machinery has been extended to use
distinct filenames for each interpreter. Instead of Python 3.2 and Python 3.3 and
Unladen Swallow each competing for a file called "mymodule.pyc", they will now
look for "mymodule.cpython-32.pyc", "mymodule.cpython-33.pyc", and
"mymodule.unladen10.pyc". And to prevent all of these new files from
cluttering source directories, the *pyc* files are now collected in a
"__pycache__" directory stored under the package directory.
Aside from the filenames and target directories, the new scheme has a few
aspects that are visible to the programmer:
* Imported modules now have a :attr:`__cached__` attribute which stores the name
of the actual file that was imported:
>>> import collections
>>> collections.__cached__
'c:/py32/lib/__pycache__/collections.cpython-32.pyc'
* The tag that is unique to each interpreter is accessible from the :mod:`imp`
module:
>>> import imp
>>> imp.get_tag()
'cpython-32'
* Scripts that try to deduce source filename from the imported file now need to
be smarter. It is no longer sufficient to simply strip the "c" from a ".pyc"
filename. Instead, use the new functions in the :mod:`imp` module:
>>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc')
'c:/py32/lib/collections.py'
>>> imp.cache_from_source('c:/py32/lib/collections.py')
'c:/py32/lib/__pycache__/collections.cpython-32.pyc'
* The :mod:`py_compile` and :mod:`compileall` modules have been updated to
reflect the new naming convention and target directory.
.. seealso::
:pep:`3147` - PYC Repository Directories
PEP written by Barry Warsaw.
PEP 3149: ABI Version Tagged .so Files
======================================
The PYC repository directory allows multiple bytecode cache files to be
co-located. This PEP implements a similar mechanism for shared object files by
giving them a common directory and distinct names for each version.
The common directory is "pyshared" and the file names are made distinct by
identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the
major and minor version numbers, and optional build flags (such as "d" for
debug, "m" for pymalloc, "u" for wide-unicode). For an arbitrary package "foo",
you may see these files when the distribution package is installed::
/usr/share/pyshared/foo.cpython-32m.so
/usr/share/pyshared/foo.cpython-33md.so
In Python itself, the tags are accessible from functions in the :mod:`sysconfig`
module::
>>> import sysconfig
>>> sysconfig.get_config_var('SOABI') # find the version tag
'cpython-32mu'
>>> sysconfig.get_config_var('SO') # find the full filename extension
'cpython-32mu.so'
.. seealso::
:pep:`3149` - ABI Version Tagged .so Files
PEP written by Barry Warsaw.
Other Language Changes
======================
Some smaller changes made to the core Python language are:
* String formatting for :func:`format` and :meth:`str.format` gained new
capabilities for the format character **#**. Previously, for integers in
binary, octal, or hexadecimal, it caused the output to be prefixed with '0b',
'0o', or '0x' respectively. Now it can also handle floats, complex, and
Decimal, causing the output to always have a decimal point even when no digits
follow it.
>>> format(20, '#o')
'0o24'
>>> format(12.34, '#5.0f')
' 12.'
(Suggested by Mark Dickinson and implemented by Eric Smith in :issue:`7094`.)
* The interpreter can now be started with a quiet option, ``-q``, to suppress
the copyright and version information in an interactive mode.
(Contributed by Marcin Wojdyr in issue:`1772833`).
* The :func:`hasattr` function used to catch and suppress any Exception. Now,
it only catches :exc:`AttributeError`. Under the hood, :func:`hasattr` works
by calling :func:`getattr` and throwing away the results. This is necessary
because dynamic attribute creation is possible using :meth:`__getattribute__`
or :meth:`__getattr__`. If :func:`hasattr` were to just scan instance and class
dictionaries it would miss the dynamic methods and make it difficult to
implement proxy objects.
(Discovered by Yury Selivanov and fixed by Benjamin Peterson; :issue:`9666`.)
* The :func:`str` of a float or complex number is now the same as its
:func:`repr`. Previously, the :func:`str` form was shorter but that just
caused confusion and is no longer needed now that the shortest possible
:func:`repr` is displayed by default:
>>> repr(math.pi)
'3.141592653589793'
>>> str(math.pi)
'3.141592653589793'
(Proposed and implemented by Mark Dickinson; :issue:`9337`.)
* :class:`memoryview` objects now have a :meth:`~memoryview.release()` method
and they also now support the context manager protocol. This allows timely
release of any resources that were acquired when requesting a buffer from the
original object.
>>> with memoryview(b'abcdefgh') as v:
... print(v.tolist())
...
[97, 98, 99, 100, 101, 102, 103, 104]
(Added by Antoine Pitrou; :issue:`9757`.)
* Previously it was illegal to delete a name from the local namespace if it
occurs as a free variable in a nested block::
>>> def outer(x):
... def inner():
... return x
... inner()
... del x
This is now allowed. Remember that the target of an :keyword:`except` clause
is cleared, so this code which used to work with Python 2.6, raised a
:exc:`SyntaxError` with Python 3.1 and now works again::
>>> def f():
... def print_error():
... print(e)
... try:
... something
... except Exception as e:
... print_error()
... # implicit "del e" here
(See :issue:`4617`.)
* The internal :c:type:`structsequence` tool now creates subclasses of tuple.
This means that C generated structures like those returned by :func:`os.stat`,
:func:`time.gmtime`, and :func:`sys.version_info` now work like a
:term:`named tuple` and are more interoperable with functions and methods that
expect a tuple as an argument. The is a big step forward in making the C
structures as flexible as their pure Python counterparts.
(Suggested by Arfrever Frehtes Taifersar Arahesis and implemented
by Benjamin Peterson in :issue:`8413`.)
* Warnings are now easier control. An :envvar:`PYTHONWARNINGS` environment
variable is now available as an alternative to using ``-W`` at the command
line.
(Suggested by Barry Warsaw and implemented by Philip Jenvey in :issue:`7301`.)
* A new warning category, :exc:`ResourceWarning`, has been added. It is
emitted when potential issues with resource consumption or cleanup
are detected. It is silenced by default in normal release builds, but
can be enabled through the means provided by the :mod:`warnings`
module, or on the command line.
A :exc:`ResourceWarning` is issued at interpreter shutdown if the
:data:`gc.garbage` list isn't empty. This is meant to make the programmer
aware that their code contains object finalization issues.
A :exc:`ResourceWarning` is also issued when a :term:`file object` is destroyed
without having been explicitly closed. While the deallocator for such
object ensures it closes the underlying operating system resource
(usually, a file descriptor), the delay in deallocating the object could
produce various issues, especially under Windows. Here is an example
of enabling the warning from the command line::
$ ./python -q -Wdefault
>>> f = open("foo", "wb")
>>> del f
__main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'>
(Added by Antoine Pitrou and Georg Brandl in :issue:`10093` and :issue:`477863`.)
* :class:`range` objects now support *index* and *count* methods. This is part
of an effort to make more objects fully implement the
:class:`collections.Sequence` :term:`abstract base class`. As a result, the
language will have a more uniform API. In addition, :class:`range` objects
now support slicing and negative indices. This makes *range* more
interoperable with lists::
>>> range(0, 100, 2).count(10)
1
>>> range(0, 100, 2).index(10)
5
>>> range(0, 100, 2)[5]
10
>>> range(0, 100, 2)[0:5]
range(0, 10, 2)
(Contributed by Daniel Stuzback in :issue:`9213` and by Alexander Belopolsky
in :issue:`2690`.)
* The :func:`callable` builtin function from Py2.x was resurrected. It provides
a concise, readable alternative to using an :term:`abstract base class` in an
expression like ``isinstance(x, collections.Callable)``:
>>> callable(max)
True
>>> callable(20)
False
(See :issue:`10518`.)
* Python's import mechanism can now load module installed in directories with
non-ASCII characters in the path name.
(Required extensive work by Victor Stinner in :issue:`9425`.)
New, Improved, and Deprecated Modules
=====================================
Python's standard library has undergone significant maintenance efforts and
quality improvements.
The biggest news for Python 3.2 is that the :mod:`email` package and
:mod:`nntplib` modules now work correctly with the bytes/text model in Python 3.
For the first time, there is correct handling of inputs with mixed encodings.
Throughout the standard library, there has been more careful attention to
encodings and text versus bytes issues. In particular, interactions with the
operating system are now better able to pass non-ASCII data using the Windows
mcbs encoding, locale aware encodings, or UTF-8.
Another significant win is the addition of substantially better support for
*SSL* connections and security certificates.
In addition, more functions and classes now have a :term:`context manager` to
support convenient and reliable resource clean-up using the
:keyword:`with`-statement.
email
-----
The usability of the :mod:`email` package in Python 3 has been mostly fixed by
the extensive efforts of R. David Murray. The problem was that emails are
typically read and stored in the form of :class:`bytes` rather than :class:`str`
text, and they may contain multiple encodings within a single email. So, the
email package had to be extended to parse and generate email messages in bytes
format.
* New functions :func:`~email.message_from_bytes` and
:func:`~email.message_from_binary_file`, and new classes
:class:`~email.parser.BytesFeedParser` and :class:`~email.parser.BytesParser`
allow binary message data to be parsed into model objects.
* Given bytes input to the model, :meth:`~email.message.Message.get_payload`
will by default decode a message body that has a
:mailheader:`Content-Transfer-Encoding` of *8bit* using the charset
specified in the MIME headers and return the resulting string.
* Given bytes input to the model, :class:`~email.generator.Generator` will
convert message bodies that have a :mailheader:`Content-Transfer-Encoding` of
*8bit* to instead have a *7bit* :mailheader:`Content-Transfer-Encoding`.
* A new class :class:`~email.generator.BytesGenerator` produces bytes as output,
preserving any unchanged non-ASCII data that was present in the input used to
build the model, including message bodies with a
:mailheader:`Content-Transfer-Encoding` of *8bit*.
* The :mod:`smtplib` :class:`~smtplib.SMTP` class now accepts a byte string
for the *msg* argument to the :meth:`~smtplib.SMTP.sendmail` method,
and a new method, :meth:`~smtplib.SMTP.send_message` accepts a
:class:`~email.message.Message` object and can optionally obtain the
*from_addr* and *to_addrs* addresses directly from the object.
.. XXX Update before 3.2rc1 to reflect all of the latest work and add examples.
(Proposed and implemented by R. David Murray, :issue:`4661` and :issue:`10321`.)
elementtree
-----------
The :mod:`xml.etree.ElementTree` package and its :mod:`xml.etree.cElementTree`
counterpart have been updated to version 1.3.
Several new and useful functions and methods have been added:
* :func:`xml.etree.ElementTree.fromstringlist` which builds an XML document
from a sequence of fragments
* :func:`xml.etree.ElementTree.register_namespace` for registering a global
namespace prefix
* :func:`xml.etree.ElementTree.tostringlist` for string representation
including all sublists
* :meth:`xml.etree.ElementTree.Element.extend` for appending a sequence of zero
or more elements
* :meth:`xml.etree.ElementTree.Element.iterfind` searches an element and
subelements
* :meth:`xml.etree.ElementTree.Element.itertext` creates a text iterator over
an element and its sub-elements
* :meth:`xml.etree.ElementTree.TreeBuilder.end` closes the current element
* :meth:`xml.etree.ElementTree.TreeBuilder.doctype` handles a doctype
declaration
Two methods have been deprecated:
* :meth:`xml.etree.ElementTree.getchildren` use ``list(elem)`` instead.
* :meth:`xml.etree.ElementTree.getiterator` use ``Element.iter`` instead.
For details of the update, see `Introducing ElementTree
<http://effbot.org/zone/elementtree-13-intro.htm>`_ on Fredrik Lundh's website.
(Contributed by Florent Xicluna and Fredrik Lundh, :issue:`6472`.)
functools
---------
* The :mod:`functools` module includes a new decorator for caching function
calls. :func:`functools.lru_cache` can save repeated queries to an external
resource whenever the results are expected to be the same.
For example, adding a caching decorator to a database query function can save
database accesses for popular searches::
@functools.lru_cache(maxsize=300)
def get_phone_number(name):
c = conn.cursor()
c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,))
return c.fetchone()[0]
>>> for name in user_requests:
... get_phone_number(name) # cached lookup
To help with choosing an effective cache size, the wrapped function is
instrumented for tracking cache statistics:
>>> get_phone_number.cache_info()
CacheInfo(hits=4805, misses=980, maxsize=300, currsize=300)
If the phonelist table gets updated, the outdated contents of the cache can be
cleared with:
>>> get_phone_number.cache_clear()
(Contributed by Raymond Hettinger and incorporating design ideas from
Jim Baker, Miki Tebeka, and Nick Coghlan.)
* The :func:`functools.wraps` decorator now adds a :attr:`__wrapped__` attribute
pointing to the original callable function. This allows wrapped functions to
be introspected. It also copies :attr:`__annotations__` if defined. And now
it also gracefully skips over missing attributes such as :attr:`__doc__` which
might not be defined for the wrapped callable.
(By Nick Coghlan and Terrence Cole; :issue:`9567`, :issue:`3445`, and
:issue:`8814`.)
* To help write classes with rich comparison methods, a new decorator
:func:`functools.total_ordering` will use a existing equality and inequality
methods to fill-in the remaining methods.
For example, supplying *__eq__* and *__lt__* will enable
:func:`~functools.total_ordering` to fill-in *__le__*, *__gt__* and *__ge__*::
@total_ordering
class Student:
def __eq__(self, other):
return ((self.lastname.lower(), self.firstname.lower()) ==
(other.lastname.lower(), other.firstname.lower()))
def __lt__(self, other):
return ((self.lastname.lower(), self.firstname.lower()) <
(other.lastname.lower(), other.firstname.lower()))
(Contributed by Raymond Hettinger.)
* To aid in porting programs from Python 2, the :func:`~functools.cmp_to_key`
function converts an old-style comparison function to
modern :term:`key function`:
>>> # locale-aware sort order
>>> sorted(iterable, key=cmp_to_key(locale.strcoll))
For sorting examples and a brief sorting tutorial, see the `Sorting HowTo
<http://wiki.python.org/moin/HowTo/Sorting/>`_ tutorial.
(Contributed by Raymond Hettinger.)
itertools
---------
* The :mod:`itertools` module has a new :func:`~itertools.accumulate` function
modeled on APL's *scan* operator and on Numpy's *accumulate* function:
>>> list(accumulate(8, 2, 50))
[8, 10, 60]
>>> prob_dist = [0.1, 0.4, 0.2, 0.3]
>>> list(accumulate(prob_dist)) # cumulative probability distribution
[0.1, 0.5, 0.7, 1.0]
For an example using :func:`~itertools.accumulate`, see the :ref:`examples for
the random module <random-examples>`.
(Contributed by Raymond Hettinger and incorporating design suggestions
from Mark Dickinson.)
collections
-----------
* The :class:`collections.Counter` class now has two forms of in-place
subtraction, the existing *-=* operator for `saturating subtraction
<http://en.wikipedia.org/wiki/Saturation_arithmetic>`_ and the new
:meth:`~collections.Counter.subtract` method for regular subtraction. The
former is suitable for `multisets <http://en.wikipedia.org/wiki/Multiset>`_
which only have positive counts, and the latter is more suitable for use cases
that allow negative counts:
>>> tally = Counter(dogs=5, cat=3)
>>> tally -= Counter(dogs=2, cats=8) # saturating subtraction
>>> tally
Counter({'dogs': 3})
>>> tally = Counter(dogs=5, cats=3)
>>> tally.subtract(dogs=2, cats=8) # regular subtraction
>>> tally
Counter({'dogs': 3, 'cats': -5})
(Contributed by Raymond Hettinger.)
* The :class:`collections.OrderedDict` class has a new method
:meth:`~collections.OrderedDict.move_to_end` which takes an existing key and
moves it to either the beginning or end of an ordered sequence. When the
dictionary sequence is being used as a queue, these operations correspond to
"move to the front of the line" or "move to the back of the line":
>>> d = OrderedDict.fromkeys(['a', 'b', 'X', 'd', 'e'])
>>> list(d)
['a', 'b', 'X', 'd', 'e']
>>> d.move_to_end('X', last=True)
>>> list(d)
['a', 'b', 'd', 'e', 'X']
>>> d.move_to_end('X', last=False)
>>> list(d)
['X', 'a', 'b', 'd', 'e']
(Contributed by Raymond Hettinger.)
* The :class:`collections.deque` grew two new methods :meth:`~collections.deque.count`
and :meth:`collections.deque.reverse` that make them more substitutable for
:class:`list` when needed:
>>> d = deque('simsalabim')
>>> d.count('s')
2
>>> d.reverse()
>>> d
deque(['m', 'i', 'b', 'a', 'l', 'a', 's', 'm', 'i', 's'])
(Contributed by Raymond Hettinger.)
datetime
--------
* The :mod:`datetime` module has a new type :class:`~datetime.timezone` that
implements the :class:`~datetime.tzinfo` interface by returning a fixed UTC
offset and timezone name. This makes it easier to create timezone aware
datetime objects:
>>> datetime.now(timezone.utc)
datetime.datetime(2010, 12, 8, 21, 4, 2, 923754, tzinfo=datetime.timezone.utc)
>>> datetime.strptime("01/01/2000 12:00 +0000", "%m/%d/%Y %H:%M %z")
datetime.datetime(2000, 1, 1, 12, 0, tzinfo=datetime.timezone.utc)
* Also, :class:`~datetime.timedelta` objects can now be multiplied by
:class:`float` and divided by :class:`float` and :class:`int` objects.
(Contributed by Alexander Belopolsky in :issue:`1289118`, :issue:`5094` and
:issue:`6641`.)
abc
---
The :mod:`abc` module now supports :func:`~abc.abstractclassmethod` and
:func:`~abc.abstractstaticmethod`.
These tools make it possible to define an :term:`Abstract Base Class` that
requires a particular :func:`classmethod` or :func:`staticmethod` to be
implemented.
(Patch submitted by Daniel Urban; :issue:`5867`.)
contextlib
----------
There is a new and slightly mind-blowing tool
:class:`~contextlib.ContextDecorator` that is helpful for creating a
:term:`context manager` that does double-duty as a function decorator.
As a convenience, this new functionality is used by
:func:`~contextlib.contextmanager` so that no extra effort is needed to support
both roles.
The basic idea is that both context managers and function decorators can be used
for pre-action and post-action wrappers. Context managers wrap a group of
statements using the :keyword:`with`-statement, and function decorators wrap a
group of statements enclosed in a function. So, occasionally there is a need to
write a pre/post action wrapper that can be used in either role.
For example, it is sometimes useful to wrap functions or groups of statements
with a logger that can track the time of entry and time of exit. Rather than
writing both a function decorator and a context manager for the task, the
:func:`~contextlib.contextmanager` provides both capabilities in a single
definition:
>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> @contextmanager
... def track_entry_and_exit(name):
... logging.info('Entering: {}'.format(name))
... yield
... logging.info('Exiting: {}'.format(name))
Formerly, this would have only been usable as a context manager:
>>> with track_entry_and_exit('widget loader'):
... print('Some time consuming activity goes here')
... load_widget()
Now, it can be used as a decorator as well:
>>> @track_entry_and_exit('widget loader')
... def activity():
... print('Some time consuming activity goes here')
... load_widget()
Trying to fulfill two roles at once places some limitations on the technique.
Context managers normally have the flexibility to return an argument usable by
the :keyword:`with`-statement, but there is no parallel for function decorators.
In the above example, there is not a clean way for the *track_entry_and_exit*
context manager does not have a way to return a logging instance for use in the
body of enclosed statements.
(Contributed by Michael Foord in :issue:`9110`.)
decimal and fractions
---------------------
Mark Dickinson crafted an elegant and efficient scheme for assuring that
different numeric datatypes will have the same hash value whenever their actual
values are equal (:issue:`8188`)::
>>> assert hash(Fraction(3, 2)) == hash(1.5) == \
hash(Decimal("1.5")) == hash(complex(1.5, 0))
An early decision to limit the inter-operability of various numeric types has
been relaxed. It is still unsupported (and ill-advised) to to have implicit
mixing in arithmetic expressions such as ``Decimal('1.1') + float('1.1')``
because the latter loses information in the process of constructing the binary
float. However, since existing floating point value can be converted losslessly
to either a decimal or rational representation, it makes sense to add them to
the constructor and to support mixed-type comparisons.
* The :class:`decimal.Decimal` constructor now accepts :class:`float` objects
directly so there in no longer a need to use the :meth:`~decimal.Decimal.from_float`
method (:issue:`8257`).
* Mixed type comparisons are now fully supported so that
:class:`~decimal.Decimal` objects can be directly compared with :class:`float`
and :class:`fractions.Fraction` (:issue:`2531` and :issue:`8188`).
Similar changes were made to :class:`fractions.Fraction` so that the
:meth:`~fractions.Fraction.from_float()` and :meth:`~fractions.Fraction.from_decimal`
methods are no longer needed (:issue:`8294`):
>>> Decimal(1.1)
Decimal('1.100000000000000088817841970012523233890533447265625')
>>> Fraction(1.1)
Fraction(2476979795053773, 2251799813685248)
Another useful change for the :mod:`decimal` module is that the
:attr:`Context.clamp` attribute is now public. This is useful in creating
contexts that correspond to the decimal interchange formats specified in IEEE
754 (see :issue:`8540`).
(Contributed by Mark Dickinson and Raymond Hettinger.)
ftp
---
The :class:`ftplib.FTP` class now supports the context manager protocol to
unconditionally consume :exc:`socket.error` exceptions and to close the FTP
connection when done::
>>> from ftplib import FTP
>>> with FTP("ftp1.at.proftpd.org") as ftp:
... ftp.login()
... ftp.dir()
...
'230 Anonymous login ok, restrictions apply.'
dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 .
dr-xr-xr-x 9 ftp ftp 154 May 6 10:43 ..
dr-xr-xr-x 5 ftp ftp 4096 May 6 10:43 CentOS
dr-xr-xr-x 3 ftp ftp 18 Jul 10 2008 Fedora
Other file-like objects such as :class:`mmap.mmap` and :func:`fileinput.input`
also grew auto-closing context managers::
with fileinput.input(files=('log1.txt', 'log2.txt')) as f:
for line in f:
process(line)
(Contributed by Tarek Ziadé and Giampaolo Rodolà in :issue:`4972`, and
by Georg Brandl in :issue:`8046` and :issue:`1286`.)
.. XXX mention os.popen and subprocess.Popen auto-closing of fds
gzip and zipfile
----------------
:class:`gzip.GzipFile` now implements the :class:`io.BufferedIOBase`
:term:`abstract base class` (except for ``truncate()``). It also has a
:meth:`~gzip.GzipFile.peek` method and supports unseekable as well as
zero-padded file objects.
The :mod:`gzip` module also gains the :func:`~gzip.compress` and
:func:`~gzip.decompress` functions for easier in-memory compression and
decompression. Keep in mind that text needs to be encoded in to :class:`bytes`
before compressing and decompressing:
>>> s = 'Three shall be the number thou shalt count, '
>>> s += 'and the number of the counting shall be three'
>>> b = s.encode() # convert to utf-8
>>> len(b)
89
>>> c = gzip.compress(b)
>>> len(c)
77
>>> gzip.decompress(c).decode()[:42] # decompress and convert to text
'Three shall be the number thou shalt count,'
(Contributed by Anand B. Pillai in :issue:`3488`; and by Antoine Pitrou, Nir
Aides and Brian Curtin in :issue:`9962`, :issue:`1675951`, :issue:`7471` and
:issue:`2846`.)
Also, the :class:`zipfile.ZipExtFile` class was reworked internally to represent
files stored inside an archive. The new implementation is significantly faster
and can be wrapped in a :class:`io.BufferedReader` object for more speedups. It
also solves an issue where interleaved calls to *read* and *readline* gave the
wrong results.
(Patch submitted by by Nir Aides in :issue:`7610`.)
shutil
------
The :func:`shutil.copytree` function has two new options:
* *ignore_dangling_symlinks*: when ``symlinks=False`` so that the function
copies the file pointed to by the symlink, not the symlink itself. This
option will silence the error raised if the file doesn't exist.
* *copy_function*: is a callable that will be used to copy files.
:func:`shutil.copy2` is used by default.
(Contributed by Tarek Ziadé.)
sqlite3
-------
The :mod:`sqlite3` module was updated to version 2.6.0. It has two new capabilities.
* The :attr:`sqlite3.Connection.in_transit` attribute is true if there is an
active transaction for uncommitted changes.
* The :meth:`sqlite3.Connection.enable_load_extension` and
:meth:`sqlite3.Connection.load_extension` methods allows you to load SQLite
extensions from ".so" files. One well-known extension is the fulltext-search
extension distributed with SQLite.
(Contributed by R. David Murray and Shashwat Anand; :issue:`8845`.)
socket
------
The :mod:`socket` module has two new improvements.
* Socket objects now have a :meth:`~socket.socket.detach()` method which puts
the socket into closed state without actually closing the underlying file
descriptor. The latter can then be reused for other purposes.
(Added by Antoine Pitrou; :issue:`8524`.)
* :func:`socket.create_connection` now supports the context manager protocol
to unconditionally consume :exc:`socket.error` exceptions and to close the
socket when done.
(Contributed by Giampaolo Rodolà; :issue:`9794`.)
ssl
---
* The :mod:`ssl` module has a new class, :class:`~ssl.SSLContext` which serves
as a container for various persistent SSL data, such as protocol settings,
certificates, private keys, and various other options. The
:meth:`~ssl.SSLContext.wrap_socket` method allows to create an SSL socket from
such an SSL context. (Added by Antoine Pitrou; :issue:`8550`.)
* A new function, :func:`ssl.match_hostname`, helps implement server identity
verification for higher-level protocols by implementing the rules of
HTTPS (from :rfc:`2818`), which are also suitable for other protocols.
(Added by Antoine Pitrou, :issue:`1589`).
* The :func:`ssl.wrap_socket` constructor function now takes a *ciphers*
argument that's a string listing the encryption algorithms to be allowed; the
format of the string is described `in the OpenSSL documentation
<http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`__. (Added
by Antoine Pitrou; :issue:`8322`.)
* When linked against a recent enough version of OpenSSL, the :mod:`ssl`
module now supports the Server Name Indication extension to the TLS
protocol, allowing for several "virtual hosts" using different certificates
on a single IP/port. This extension is only supported in client mode,
and is activated by passing the *server_hostname* argument to
:meth:`SSLContext.wrap_socket`.
(Added by Antoine Pitrou, :issue:`5639`.)
* Various options have been added to the :mod:`ssl` module, such as
:data:`~ssl.OP_NO_SSLv2` which allows to force disabling of the insecure and
obsolete SSLv2 protocol. (Added by Antoine Pitrou; :issue:`4870`.)
* Another change makes the extension load all of OpenSSL's ciphers and digest
algorithms so that they're all available. Some SSL certificates couldn't be
verified, reporting an "unknown algorithm" error. (Reported by Beda Kosata,
and fixed by Antoine Pitrou; :issue:`8484`.)
* The version of OpenSSL being used is now available as the module attributes
:data:`ssl.OPENSSL_VERSION` (a string), :data:`ssl.OPENSSL_VERSION_INFO` (a
5-tuple), and :data:`ssl.OPENSSL_VERSION_NUMBER` (an integer). (Added by
Antoine Pitrou; :issue:`8321`.)
nntp
----
The :mod:`nntplib` module has a revamped implementation with better bytes and
unicode semantics as well as more practical APIs. These improvements break
compatibility with the nntplib version in Python 3.1, which was partly
dysfunctional in itself.
(Contributed by Antoine Pitrou in :issue:`9360`)
certificates
------------
:class:`http.client.HTTPSConnection`, :class:`urllib.request.HTTPSHandler`
and :func:`urllib.request.urlopen` now take optional arguments to allow for
server certificate checking against a set of Certificate Authorities,
as recommended in public uses of HTTPS.
(Added by Antoine Pitrou, :issue:`9003`.)
unittest
--------
* The command-line call, ``python -m unittest`` can now accept file paths
instead of module names for running specific tests (:issue:`10620`). The new
test discovery can find tests within packages, locating any test importable
from the top level directory. The top level directory can be specified with
the `-t` option, a pattern for matching files with ``-p``, and a directory to
start discovery with ``-s``::
$ python -m unittest discover -s my_proj_dir -p '_test.py'
(Contributed by Michael Foord.)
* The :mod:`unittest` module has two new methods,
:meth:`~unittest.TestCase.assertWarns` and
:meth:`~unittest.TestCase.assertWarnsRegex` to check that a given warning type
is triggered by the code under test:
>>> with self.assertWarns(DeprecationWarning):
... legacy_function('XYZ')
Another new method, :meth:`~unittest.TestCase.assertCountEqual` is used to
compare two iterables to determine if their element counts are equal (whether
the same elements are present with the same number of occurrences regardless
of order)::
def test_anagram(self):
self.assertCountEqual('algorithm', 'logarithm')
A principal feature of the unittest module is an effort to produce meaningful
diagnostics when a test fails. When possible the failure is recorded along
with a diff of the output. This is especially helpful for analyzing log files
of failed test runs. However, since diffs can sometime be voluminous, there is
a new :attr:`~unittest.TestCase.maxDiff` attribute which sets maximum length of
diffs.
In addition the naming in the module has undergone a number of clean-ups. For
example, :meth:`~unittest.TestCase.assertRegex` is the new name for
:meth:`~unittest.TestCase.assertRegexpMatches` which was misnamed because the
test uses :func:`re.search`, not :func:`re.match`. Other methods using
regular expressions are now named using short form "Regex" in preference
to "Regexp" -- this matches the names used in other unittest implementations,
matches Python's old name for the :mod:`re` module, and it has unambiguous
camel-casing.
To improve consistency, some of long-standing method aliases are being
deprecated in favor of the preferred names:
- replace :meth:`assert_` with :meth:`.assertTrue`
- replace :meth:`assertEquals` with :meth:`.assertEqual`
- replace :meth:`assertNotEquals` with :meth:`.assertNotEqual`
- replace :meth:`assertAlmostEquals` with :meth:`.assertAlmostEqual`
- replace :meth:`assertNotAlmostEquals` with :meth:`.assertNotAlmostEqual`
Likewise, the ``TestCase.fail*`` methods deprecated in Python 3.1 are expected
to be removed in Python 3.3. See also the :ref:`deprecated-aliases` section in
the :mod:`unittest` documentation.
(Contributed by Ezio Melotti; :issue:`9424`.)
random
------
The integer methods in the :mod:`random` module now do a better job of producing
uniform distributions. Previously, they computed selections with
``int(n*random())`` which had a slight bias whenever *n* was not a power of two.
Now, multiple selections are made from a range upto the next power of two and a
selection is kept only when it falls within the range ``0 <= x < n``. The
functions and methods affected are :func:`~random.randrange`,
:func:`~random.randint`, :func:`~random.choice`, :func:`~random.shuffle` and
:func:`~random.sample`.
(Contributed by Raymond Hettinger; :issue:`9025`.)
poplib
------
* :class:`~poplib.POP3_SSL` class now accepts a *context* parameter, which is a
:class:`ssl.SSLContext` object allowing bundling SSL configuration options,
certificates and private keys into a single (potentially long-lived)
structure.
(Contributed by Giampaolo Rodolà; :issue:`8807`.)
* :class:`asyncore.dispatcher` now provides a
:meth:`~asyncore.dispatcher.handle_accepted()` method
returning a `(sock, addr)` pair which is called when a connection has actually
been established with a new remote endpoint. This is supposed to be used as a
replacement for old :meth:`~asyncore.dispatcher.handle_accept()` and avoids
the user to call :meth:`~asyncore.dispatcher.accept()` directly.
(Contributed by Giampaolo Rodolà; :issue:`6706`.)
tempfile
--------
The :mod:`tempfile` module has a new context manager,
:class:`~tempfile.TemporaryDirectory` which provides easy deterministic
cleanup of temporary directories:
>>> with tempfile.TemporaryDirectory() as tmpdirname:
... print 'created temporary directory', tmpdirname
(Contributed by Neil Schemenauer and Nick Coghlan; :issue:`5178`.)
inspect
-------
* The :mod:`inspect` module has a new function
:func:`~inspect.getgeneratorstate` to easily identify the current state of a
generator as one of ``GEN_CREATED``, ``GEN_RUNNING``, ``GEN_SUSPENDED`` or
``GEN_CLOSED``. (Contributed by Rodolpho Eckhardt and Nick Coghlan,
:issue:`10220`.)
* To support lookups without the possibility of activating a dynamic attribute,
the :mod:`inspect` module has a new function, :func:`~inspect.getattr_static`.
Unlike, :func:`hasattr`, this is a true read-only search, guaranteed not to
change state while it is searching. (Contributed by Michael Foord.)
pydoc
-----
The :mod:`pydoc` module now provides a much improved Web server interface,
as well as a new command-line option to automatically open a browser
window to display that server.
(Contributed by Ron Adam; :issue:`2001`.)
sysconfig
---------
The new :mod:`sysconfig` module makes it straight-forward to discover
installation paths and configuration variables which vary across platforms and
installations.
The module offers access simple access functions for platform and version
information:
* :func:`~sysconfig.get_platform` returning values like *linux-i586* or
*macosx-10.6-ppc*.
* :func:`~sysconfig.get_python_version` returns a Python version string in
the form, "3.2".
It also provides access to the paths and variables corresponding to one of
seven named schemes used by :mod:`distutils`. Those include *posix_prefix*,
*posix_home*, *posix_user*, *nt*, *nt_user*, *os2*, *os2_home*:
* :func:`~sysconfig.get_paths` makes a dictionary containing installation paths
for the current installation scheme.
* :func:`~sysconfig.get_config_vars` returns a dictionary of platform specific
variables.
There is also a convenient command-line interface::
C:\Python32>python -m sysconfig
Platform: "win32"
Python version: "3.2"
Current installation scheme: "nt"
Paths:
data = "C:\Python32"
include = "C:\Python32\Include"
platinclude = "C:\Python32\Include"
platlib = "C:\Python32\Lib\site-packages"
platstdlib = "C:\Python32\Lib"
purelib = "C:\Python32\Lib\site-packages"
scripts = "C:\Python32\Scripts"
stdlib = "C:\Python32\Lib"
Variables:
BINDIR = "C:\Python32"
BINLIBDEST = "C:\Python32\Lib"
EXE = ".exe"
INCLUDEPY = "C:\Python32\Include"
LIBDEST = "C:\Python32\Lib"
SO = ".pyd"
VERSION = "32"
abiflags = ""
base = "C:\Python32"
exec_prefix = "C:\Python32"
platbase = "C:\Python32"
prefix = "C:\Python32"
projectbase = "C:\Python32"
py_version = "3.2"
py_version_nodot = "32"
py_version_short = "3.2"
srcdir = "C:\Python32"
userbase = "C:\Documents and Settings\Raymond\Application Data\Python"
pdb
---
The :mod:`pdb` debugger module gained a number of usability improvements:
* :file:`pdb.py` now has a ``-c`` option that executes commands as given in a
:file:`.pdbrc` script file.
* A :file:`.pdbrc` script file can contain ``continue`` and ``next`` commands
that continue debugging.
* The :class:`Pdb` class constructor now accepts a *nosigint* argument.
* new commands: ``l(list)``, ``ll(long list`` and ``source`` for
listing source code.
* new commands: ``display`` and ``undisplay`` for showing or hiding
the value of an expression if it has changed.
* new command: ``interact`` for starting an interactive interpreter containing
the global and local names found in the current scope.
* breakpoints can be cleared by breakpoint number
(Contributed by Georg Brandl, Antonio Cuni and Ilya Sandler.)
configparser
------------
The :mod:`configparser` module was modified to improve usability and
predictability of the default parser and its supported INI syntax. The old
:class:`ConfigParser` class was removed in favor of :class:`SafeConfigParser`
which has in turn been renamed to :class:`~configparser.ConfigParser`. Support
for inline comments is now turned off by default and section or option
duplicates are not allowed in a single configuration source.
Config parsers gained a new API based on the mapping protocol::
>>> parser = ConfigParser()
>>> parser.read_string("""
... [DEFAULT]
... monty = python
...
... [phrases]
... the = who
... full = metal jacket
... """)
>>> parser['phrases']['full']
'metal jacket'
>>> section = parser['phrases']
>>> section['the']
'who'
>>> section['british'] = '%(the)s %(full)s %(monty)s!'
>>> parser['phrases']['british']
'who metal jacket python!'
>>> 'british' in section
True
The new API is implemented on top of the classical API so custom parser
subclasses should be able to use it without modifications.
The INI file structure accepted by config parsers can now be customized. Users
can specify alternative option/value delimiters and comment prefixes, change the
name of the *DEFAULT* section or switch the interpolation syntax. Along with
support for pluggable interpolation, an additional interpolation handler
:class:`~configparser.ExtendedInterpolation` was introduced::
>>> parser = ConfigParser(interpolation=ExtendedInterpolation())
>>> parser.read_dict({'buildout': {'directory': '/home/ambv/zope9'},
... 'custom': {'prefix': '/usr/local'}})
>>> parser.read_string("""
... [buildout]
... parts =
... zope9
... instance
... find-links =
... ${buildout:directory}/downloads/dist
...
... [zope9]
... recipe = plone.recipe.zope9install
... location = /opt/zope
...
... [instance]
... recipe = plone.recipe.zope9instance
... zope9-location = ${zope9:location}
... zope-conf = ${custom:prefix}/etc/zope.conf
... """)
>>> parser['buildout']['find-links']
'\n/home/ambv/zope9/downloads/dist'
>>> parser['instance']['zope-conf']
'/usr/local/etc/zope.conf'
>>> instance = parser['instance']
>>> instance['zope-conf']
'/usr/local/etc/zope.conf'
>>> instance['zope9-location']
'/opt/zope'
A number of smaller features were also introduced, like support for specifying
encoding in read operations, specifying fallback values for get-functions, or
reading directly from dictionaries and strings.
(All changes contributed by Łukasz Langa.)
.. XXX: Mention urllib.parse changes
Issue 9873 (Nick Coghlan):
- ASCII byte sequence support in URL parsing
- named tuple for urldefrag return value
Issue 5468 (Dan Mahn) for urlencode:
- bytes input support
- non-UTF8 percent encoding of non-ASCII characters
Issue 2987 for IPv6 (RFC2732) support in urlparse
.. XXX: Any updates to the WSGI bytes versus text problem?
Multi-threading
===============
* The mechanism for serializing execution of concurrently running Python threads
(generally known as the GIL or Global Interpreter Lock) has been rewritten.
Among the objectives were more predictable switching intervals and reduced
overhead due to lock contention and the number of ensuing system calls. The
notion of a "check interval" to allow thread switches has been abandoned and
replaced by an absolute duration expressed in seconds. This parameter is
tunable through :func:`sys.setswitchinterval()`. It currently defaults to 5
milliseconds.
Additional details about the implementation can be read from a `python-dev
mailing-list message
<http://mail.python.org/pipermail/python-dev/2009-October/093321.html>`_
(however, "priority requests" as exposed in this message have not been kept
for inclusion).
(Contributed by Antoine Pitrou.)
* Regular and recursive locks now accept an optional *timeout* argument to their
:meth:`acquire` method. (Contributed by Antoine Pitrou; :issue:`7316`.)
* Similarly, :meth:`threading.Semaphore.acquire` also gained a *timeout*
argument. (Contributed by Torsten Landschoff; :issue:`850728`.)
* Regular and recursive lock acquisitions can now be interrupted by signals on
platforms using pthreads. This means that Python programs that deadlock while
acquiring locks can be successfully killed by repeatedly sending SIGINT to the
process (by pressing :kbd:`Ctrl+C` in most shells).
(Contributed by Reid Kleckner; :issue:`8844`.)
Optimizations
=============
A number of small performance enhancements have been added:
* Python's peephole optimizer now recognizes patterns such ``x in {1, 2, 3}`` as
being a test for membership in a set of constants. The optimizer recasts the
:class:`set` as a :class:`frozenset` and stores the pre-built constant.
Now that the speed penalty is gone, it is practical to start writing
membership tests using set-notation. This style is both semantically clear
and operationally fast::
extension = name.rpartition('.')[2]
if extension in {'xml', 'html', 'xhtml', 'css'}:
handle(name)
(Patch and additional tests by Dave Malcolm; :issue:`6690`).
* Serializing and unserializing data using the :mod:`pickle` module is now
several times faster.
(Contributed by Alexandre Vassalotti, Antoine Pitrou
and the Unladen Swallow team in :issue:`9410` and :issue:`3873`.)
* The `Timsort algorithm <http://en.wikipedia.org/wiki/Timsort>`_ used in
:meth:`list.sort` and :func:`sorted` now runs faster and uses less memory
when called with a :term:`key function`. Previously, every element of
a list was wrapped with a temporary object that remembered the key value
associated with each element. Now, an array of keys and values are
sorted in parallel. This save the memory consumed by the sort wrappers,
and it saves time lost from during comparisons which where delegated
by the sort wrappers.
(Patch by Daniel Stuzback in :issue:`9915`.)
* JSON decoding performance is improved and memory consumption is reduced
whenever the same string is repeated for multiple keys. Also, JSON encoding
now uses the C speedups when the ``sort_keys`` argument is true.
(Contributed by Antoine Pitrou in :issue:`7451` and by Raymond Hettinger and
Antoine Pitrou in :issue:`10314`.)
* Recursive locks (created with the :func:`threading.RLock` API) now benefit
from a C implementation which makes them as fast as regular locks, and between
10x and 15x faster than their previous pure Python implementation.
(Contributed by Antoine Pitrou; :issue:`3001`.)
* The fast-search algorithm in stringlib is now used by the :meth:`split`,
:meth:`rsplit`, :meth:`splitlines` and :meth:`replace` methods on
:class:`bytes`, :class:`bytearray` and :class:`str` objects. Likewise, the
algorithm is also used by :meth:`rfind`, :meth:`rindex`, :meth:`rsplit` and
:meth:`rpartition`.
(Patch by Florent Xicluna in :issue:`7622` and :issue:`7462`.)
* String to integer conversions now work two "digits" at a time, reducing the
number of division and modulo operations.
(:issue:`6713` by Gawain Bolton, Mark Dickinson, and Victor Stinner.)
There were several other minor optimizations. Set differencing now runs faster
when one operand is much larger than the other (Patch by Andress Bennetts in
:issue:`8685`). The :meth:`array.repeat` method has a faster implementation
(:issue:`1569291` by Alexander Belopolsky). The :class:`BaseHTTPRequestHandler`
has more efficient buffering (:issue:`3709` by Andrew Schaaf). The
multi-argument form of :func:`operator.attrgetter` now function runs slightly
faster (:issue:`10160` by Christos Georgiou). And :class:`ConfigParser` loads
multi-line arguments a bit faster (:issue:`7113` by Łukasz Langa).
Unicode
=======
Python has been updated to Unicode 6.0.0. The new features of the
Unicode Standard that will affect Python users include:
* addition of 2,088 characters, including over 1,000 additional
symbols—chief among them the additional emoji symbols, which are
especially important for mobile phones;
* changes to character properties for existing characters including
- a general category change to two Kannada characters (U+0CF1,
U+0CF2), which has the effect of making them newly eligible for
inclusion in identifiers;
- a general category change to one New Tai Lue numeric character
(U+19DA), which has the effect of disqualifying it from
inclusion in identifiers.
For more information, see `Unicode Character Database Changes
<http://www.unicode.org/versions/Unicode6.0.0/#Database_Changes>`_
at the `Unicode Consortium <http://www.unicode.org/>`_ web site.
The :mod:`os` module has two new functions: :func:`~os.fsencode` and
:func:`~os.fsdecode`. Add :data:`os.environb`: bytes version of
:data:`os.environ`, :func:`os.getenvb` function and
:data:`os.supports_bytes_environ` constant.
``'mbcs'`` encoding doesn't ignore the error handler argument any more. By
default (strict mode), it raises an UnicodeDecodeError on undecodable byte
sequence and UnicodeEncodeError on unencodable character. To get the ``'mbcs'``
encoding of Python 3.1, use ``'ignore'`` error handler to decode and
``'replace'`` error handler to encode. ``'mbcs'`` supports ``'strict'`` and
``'ignore'`` error handlers for decoding, and ``'strict'`` and ``'replace'``
for encoding.
On Mac OS X, Python uses ``'utf-8'`` to decode the command line arguments,
instead of the locale encoding (which is ISO-8859-1 if the ``LANG`` environment
variable is not set).
By default, tarfile uses ``'utf-8'`` encoding on Windows (instead of
``'mbcs'``), and the ``'surrogateescape'`` error handler on all operating
systems.
* Added the *cp720* Arabic DOS encoding (:issue:`1616979`).
Documentation
=============
The documentation continues to be improved.
A table of quick links has been added to the top of lengthy sections such as
:ref:`built-in-funcs`. In the case of :mod:`itertools`, the links are
accompanied by tables of cheatsheet-style summaries to provide an overview and
memory jog without having to read all of the docs.
In some cases, the pure python source code can be helpful adjunct to the docs,
so now some modules feature quick links to the latest version of the source
code. For example, the :mod:`functools` module documentation has a quick link
at the top labeled :source:`functools Python source code <Lib/functools.py>`.
The docs now contain more examples and recipes. In particular, :mod:`re` module
has an extensive section, :ref:`re-examples`. Likewise, the :mod:`itertools`
module continues to be updated with new :ref:`itertools-recipes`.
The :mod:`datetime` module now has an auxiliary implementation in pure Python.
No functionality was changed. This just provides an easier-to-read
alternate implementation. (Contributed by Alexander Belopolsky.)
IDLE
====
* The format menu now has an option to clean-up source files by stripping
trailing whitespace (:issue:`5150`).
Build and C API Changes
=======================
Changes to Python's build process and to the C API include:
* The C functions that access the Unicode Database now accept and return
characters from the full Unicode range, even on narrow unicode builds
(Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and others). A visible difference
in Python is that :func:`unicodedata.numeric` now returns the correct value
for large code points, and :func:`repr` may consider more characters as
printable.
(Reported by Bupjoe Lee and fixed by Amaury Forgeot D'Arc; :issue:`5127`.)
* Computed gotos are now enabled by default on supported compilers (which are
detected by the configure script). They can still be disabled selectively by
specifying ``--without-computed-gotos``.
(Contributed by Antoine Pitrou; :issue:`9203`.)
* The option ``--with-wctype-functions`` was removed. The built-in unicode
database is now used for all functions.
(Contributed by Amaury Forgeot D'Arc; :issue:`9210`.)
* Hash values are now values of a new type, :c:type:`Py_hash_t`, which is
defined to be the same size as a pointer. Previously they were of type long,
which on some 64-bit operating systems is still only 32 bits long. As a
result of this fix, :class:`set` and :class:`dict` can now hold more than
``2**32`` entries on builds with 64-bit pointers (previously, they could grow
to that size but their performance degraded catastrophically).
(Suggested by Raymond Hettinger and implemented by Benjamin Peterson;
:issue:`9778`.)
* A new macro :c:macro:`Py_VA_COPY` copies the state of the variable argument
list. It is equivalent to C99 *va_copy* but available on all python platforms
(:issue:`2443`).
* A new C API function :c:func:`PySys_SetArgvEx` allows an embedded
interpreter to set sys.argv without also modifying :attr:`sys.path`
(:issue:`5753`).
* :c:macro:`PyEval_CallObject` is now only available in macro form. The
function declaration, which was kept for backwards compatibility reasons, is
now removed -- the macro was introduced in 1997 (:issue:`8276`).
* The is a new function :c:func:`PyLong_AsLongLongAndOverflow` which
is analogous to :c:func:`PyLong_AsLongAndOverflow`. The both serve to
convert Python :class:`int` into a native fixed-width type while providing
detection of cases where the conversion won't fit (:issue:`7767`).
* The :c:func:`PyUnicode_CompareWithASCIIString` now returns *not equal*
if the Python string in *NUL* terminated.
* There is a new function :c:func:`PyErr_NewExceptionWithDoc` that is
like :c:func:`PyErr_NewException` but allows a docstring to be specified.
This lets C exceptions have the same self-documenting capabilities as
their pure Python counterparts (:issue:`7033`).
* When compiled with the ``--with-valgrind`` option, the pymalloc
allocator will be automatically disabled when running under Valgrind. This
gives improved memory leak detection when running under Valgrind, while taking
advantage of pymalloc at other times (:issue:`2422`).
* Removed the "O?" format from the *PyArg_Parse* functions. The format is no
longer used and it had never been documented (:issue:`8837`).
There were a number of other small changes to the C-API. See the
:file:`Misc/NEWS` file for a complete list.
Porting to Python 3.2
=====================
This section lists previously described changes and other bugfixes that may
require changes to your code:
* The :mod:`configparser` module has a number of clean-ups. The major change is
to replace the old :class:`ConfigParser` class with long-standing preferred
alternative :class:`SafeConfigParser`. In addition there are a number of
smaller incompatibilites:
* The interpolation syntax is now validated on
:meth:`~configparser.ConfigParser.get` and
:meth:`~configparser.ConfigParser.set` operations. In the default
interpolation scheme, only two tokens with percent signs are valid: ``%(name)s``
and ``%%``, the latter being an escaped percent sign.
* The :meth:`~configparser.ConfigParser.set` and
:meth:`~configparser.ConfigParser.add_section` methods now verify that
values are actual strings. Formerly, unsupported types could be introduced
unintentionally.
* Duplicate sections or options from a single source now raise either
:exc:`~configparser.DuplicateSectionError` or
:exc:`~configparser.DuplicateOptionError`. Formerly, duplicates would
silently overwrite a previous entry.
* Inline comments are now disabled by default so now the **;** character
can be safely used in values.
* Comments now can be indented. Consequently, for **;** or **#** to appear at
the start of a line in multiline values, it has to be interpolated. This
keeps comment prefix characters in values from being mistaken as comments.
* ``""`` is now a valid value and is no longer automatically converted to an
empty string. For empty strings, use ``"option ="`` in a line.
* The :mod:`nntplib` module was reworked extensively, meaning that its APIs
are often incompatible with the 3.1 APIs.
* :class:`bytearray` objects can no longer be used as filenames; instead,
they should be converted to :class:`bytes`.
* PyArg_Parse*() functions:
* "t#" format has been removed: use "s#" or "s*" instead
* "w" and "w#" formats has been removed: use "w*" instead
* The :c:type:`PyCObject` type, deprecated in 3.1, has been removed. To wrap
opaque C pointers in Python objects, the :c:type:`PyCapsule` API should be used
instead; the new type has a well-defined interface for passing typing safety
information and a less complicated signature for calling a destructor.
* The :func:`sys.setfilesystemencoding` function was removed because
it had a flawed design.
* The :func:`random.seed` function and method now salt string seeds with an
sha512 hash function. To access the previous version of *seed* in order to
reproduce Python 3.1 sequences, set the *version* argument to *1*,
``random.seed(s, version=1)``.
* The previously deprecated :func:`string.maketrans` function has been removed
in favor of the static methods, :meth:`bytes.maketrans` and
:meth:`bytearray.maketrans`. This change solves the confusion around which
types were supported by the :mod:`string` module. Now, :class:`str`,
:class:`bytes`, and :class:`bytearray` each have their own **maketrans** and
**translate** methods with intermediate translation tables of the appropriate
type.
(Contributed by Georg Brandl; :issue:`5675`.)
* The previously deprecated :func:`contextlib.nested` function has been removed
in favor of a plain :keyword:`with` statement which can accept multiple
context managers. The latter technique is faster (because it is built-in),
and it does a better job finalizing multiple context managers when one of them
raises an exception::
>>> with open('mylog.txt') as infile, open('a.out', 'w') as outfile:
... for line in infile:
... if '<critical>' in line:
... outfile.write(line)
(Contributed by Georg Brandl and Mattias Brändström;
`appspot issue 53094 <http://codereview.appspot.com/53094>`_.)
