What’s New In Python 3.3¶

Summary – Release highlights¶

New syntax features:

• New yield from expression for generator delegation.

• The u'unicode' syntax is accepted again for str objects.

New library modules:

• faulthandler (helps debugging low-level crashes)

• ipaddress (high-level objects representing IP addresses and masks)

• lzma (compress data using the XZ / LZMA algorithm)

• unittest.mock (replace parts of your system under test with mock objects)

• venv (Python virtual environments, as in the popular virtualenv package)

New built-in features:

• Reworked I/O exception hierarchy.

Implementation improvements:

• Rewritten import machinery based on importlib.

• More compact unicode strings.

• More compact attribute dictionaries.

Significantly Improved Library Modules:

• C Accelerator for the decimal module.

• Better unicode handling in the email module (provisional).

Security improvements:

• Hash randomization is switched on by default.

Please read on for a comprehensive list of user-facing changes.

PEP 405: Virtual Environments¶

Virtual environments help create separate Python setups while sharing a system-wide base install, for ease of maintenance. Virtual environments have their own set of private site packages (i.e. locally installed libraries), and are optionally segregated from the system-wide site packages. Their concept and implementation are inspired by the popular virtualenv third-party package, but benefit from tighter integration with the interpreter core.

This PEP adds the venv module for programmatic access, and the pyvenv script for command-line access and administration. The Python interpreter checks for a pyvenv.cfg, file whose existence signals the base of a virtual environment’s directory tree.

PEP 420: Implicit Namespace Packages¶

Native support for package directories that don’t require __init__.py marker files and can automatically span multiple path segments (inspired by various third party approaches to namespace packages, as described in PEP 420)

PEP 3118: New memoryview implementation and buffer protocol documentation¶

The implementation of PEP 3118 has been significantly improved.

The new memoryview implementation comprehensively fixes all ownership and lifetime issues of dynamically allocated fields in the Py_buffer struct that led to multiple crash reports. Additionally, several functions that crashed or returned incorrect results for non-contiguous or multi-dimensional input have been fixed.

The memoryview object now has a PEP-3118 compliant getbufferproc() that checks the consumer’s request type. Many new features have been added, most of them work in full generality for non-contiguous arrays and arrays with suboffsets.

The documentation has been updated, clearly spelling out responsibilities for both exporters and consumers. Buffer request flags are grouped into basic and compound flags. The memory layout of non-contiguous and multi-dimensional NumPy-style arrays is explained.

PEP 393: Flexible String Representation¶

The Unicode string type is changed to support multiple internal representations, depending on the character with the largest Unicode ordinal (1, 2, or 4 bytes) in the represented string. This allows a space-efficient representation in common cases, but gives access to full UCS-4 on all systems. For compatibility with existing APIs, several representations may exist in parallel; over time, this compatibility should be phased out.

On the Python side, there should be no downside to this change.

On the C API side, PEP 393 is fully backward compatible. The legacy API should remain available at least five years. Applications using the legacy API will not fully benefit of the memory reduction, or - worse - may use a bit more memory, because Python may have to maintain two versions of each string (in the legacy format and in the new efficient storage).

PEP 397: Python Launcher for Windows¶

The Python 3.3 Windows installer now includes a py launcher application that can be used to launch Python applications in a version independent fashion.

This launcher is invoked implicitly when double-clicking *.py files. If only a single Python version is installed on the system, that version will be used to run the file. If multiple versions are installed, the most recent version is used by default, but this can be overridden by including a Unix-style “shebang line” in the Python script.

The launcher can also be used explicitly from the command line as the py application. Running py follows the same version selection rules as implicitly launching scripts, but a more specific version can be selected by passing appropriate arguments (such as -3 to request Python 3 when Python 2 is also installed, or -2.6 to specifically request an earlier Python version when a more recent version is installed).

In addition to the launcher, the Windows installer now includes an option to add the newly installed Python to the system PATH. (Contributed by Brian Curtin in bpo-3561.)

PEP 3151: Reworking the OS and IO exception hierarchy¶

The hierarchy of exceptions raised by operating system errors is now both simplified and finer-grained.

You don’t have to worry anymore about choosing the appropriate exception type between OSError, IOError, EnvironmentError, WindowsError, mmap.error, socket.error or select.error. All these exception types are now only one: OSError. The other names are kept as aliases for compatibility reasons.

Also, it is now easier to catch a specific error condition. Instead of inspecting the errno attribute (or args[0]) for a particular constant from the errno module, you can catch the adequate OSError subclass. The available subclasses are the following:

• BlockingIOError

• ChildProcessError

• ConnectionError

• FileExistsError

• FileNotFoundError

• InterruptedError

• IsADirectoryError

• NotADirectoryError

• PermissionError

• ProcessLookupError

• TimeoutError

And the ConnectionError itself has finer-grained subclasses:

• BrokenPipeError

• ConnectionAbortedError

• ConnectionRefusedError

• ConnectionResetError

Thanks to the new exceptions, common usages of the errno can now be avoided. For example, the following code written for Python 3.2:

from errno import ENOENT, EACCES, EPERM

try:
    with open("document.txt") as f:
        content = f.read()
except IOError as err:
    if err.errno == ENOENT:
        print("document.txt file is missing")
    elif err.errno in (EACCES, EPERM):
        print("You are not allowed to read document.txt")
    else:
        raise

can now be written without the errno import and without manual inspection of exception attributes:

try:
    with open("document.txt") as f:
        content = f.read()
except FileNotFoundError:
    print("document.txt file is missing")
except PermissionError:
    print("You are not allowed to read document.txt")

PEP 380: Syntax for Delegating to a Subgenerator¶

PEP 380 adds the yield from expression, allowing a generator to delegate part of its operations to another generator. This allows a section of code containing yield to be factored out and placed in another generator. Additionally, the subgenerator is allowed to return with a value, and the value is made available to the delegating generator.

While designed primarily for use in delegating to a subgenerator, the yield from expression actually allows delegation to arbitrary subiterators.

For simple iterators, yield from iterable is essentially just a shortened form of for item in iterable: yield item:

>>> def g(x):
...     yield from range(x, 0, -1)
...     yield from range(x)
...
>>> list(g(5))
[5, 4, 3, 2, 1, 0, 1, 2, 3, 4]

However, unlike an ordinary loop, yield from allows subgenerators to receive sent and thrown values directly from the calling scope, and return a final value to the outer generator:

>>> def accumulate():
...     tally = 0
...     while 1:
...         next = yield
...         if next is None:
...             return tally
...         tally += next
...
>>> def gather_tallies(tallies):
...     while 1:
...         tally = yield from accumulate()
...         tallies.append(tally)
...
>>> tallies = []
>>> acc = gather_tallies(tallies)
>>> next(acc)  # Ensure the accumulator is ready to accept values
>>> for i in range(4):
...     acc.send(i)
...
>>> acc.send(None)  # Finish the first tally
>>> for i in range(5):
...     acc.send(i)
...
>>> acc.send(None)  # Finish the second tally
>>> tallies
[6, 10]

The main principle driving this change is to allow even generators that are designed to be used with the send and throw methods to be split into multiple subgenerators as easily as a single large function can be split into multiple subfunctions.

PEP 409: Suppressing exception context¶

PEP 409 introduces new syntax that allows the display of the chained exception context to be disabled. This allows cleaner error messages in applications that convert between exception types:

>>> class D:
...     def __init__(self, extra):
...         self._extra_attributes = extra
...     def __getattr__(self, attr):
...         try:
...             return self._extra_attributes[attr]
...         except KeyError:
...             raise AttributeError(attr) from None
...
>>> D({}).x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 8, in __getattr__
AttributeError: x

Without the from None suffix to suppress the cause, the original exception would be displayed by default:

>>> class C:
...     def __init__(self, extra):
...         self._extra_attributes = extra
...     def __getattr__(self, attr):
...         try:
...             return self._extra_attributes[attr]
...         except KeyError:
...             raise AttributeError(attr)
...
>>> C({}).x
Traceback (most recent call last):
  File "<stdin>", line 6, in __getattr__
KeyError: 'x'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 8, in __getattr__
AttributeError: x

No debugging capability is lost, as the original exception context remains available if needed (for example, if an intervening library has incorrectly suppressed valuable underlying details):

>>> try:
...     D({}).x
... except AttributeError as exc:
...     print(repr(exc.__context__))
...
KeyError('x',)

PEP 414: Explicit Unicode literals¶

To ease the transition from Python 2 for Unicode aware Python applications that make heavy use of Unicode literals, Python 3.3 once again supports the “u” prefix for string literals. This prefix has no semantic significance in Python 3, it is provided solely to reduce the number of purely mechanical changes in migrating to Python 3, making it easier for developers to focus on the more significant semantic changes (such as the stricter default separation of binary and text data).

PEP 3155: Qualified name for classes and functions¶

Functions and class objects have a new __qualname__ attribute representing the “path” from the module top-level to their definition. For global functions and classes, this is the same as __name__. For other functions and classes, it provides better information about where they were actually defined, and how they might be accessible from the global scope.

Example with (non-bound) methods:

>>> class C:
...     def meth(self):
...         pass
>>> C.meth.__name__
'meth'
>>> C.meth.__qualname__
'C.meth'

Example with nested classes:

>>> class C:
...     class D:
...         def meth(self):
...             pass
...
>>> C.D.__name__
'D'
>>> C.D.__qualname__
'C.D'
>>> C.D.meth.__name__
'meth'
>>> C.D.meth.__qualname__
'C.D.meth'

Example with nested functions:

>>> def outer():
...     def inner():
...         pass
...     return inner
...
>>> outer().__name__
'inner'
>>> outer().__qualname__
'outer.<locals>.inner'

The string representation of those objects is also changed to include the new, more precise information:

>>> str(C.D)
"<class '__main__.C.D'>"
>>> str(C.D.meth)
'<function C.D.meth at 0x7f46b9fe31e0>'

PEP 412: Key-Sharing Dictionary¶

Dictionaries used for the storage of objects’ attributes are now able to share part of their internal storage between each other (namely, the part which stores the keys and their respective hashes). This reduces the memory consumption of programs creating many instances of non-builtin types.

PEP 362: Function Signature Object¶

A new function inspect.signature() makes introspection of python callables easy and straightforward. A broad range of callables is supported: python functions, decorated or not, classes, and functools.partial() objects. New classes inspect.Signature, inspect.Parameter and inspect.BoundArguments hold information about the call signatures, such as, annotations, default values, parameters kinds, and bound arguments, which considerably simplifies writing decorators and any code that validates or amends calling signatures or arguments.

PEP 421: Adding sys.implementation¶

A new attribute on the sys module exposes details specific to the implementation of the currently running interpreter. The initial set of attributes on sys.implementation are name, version, hexversion, and cache_tag.

The intention of sys.implementation is to consolidate into one namespace the implementation-specific data used by the standard library. This allows different Python implementations to share a single standard library code base much more easily. In its initial state, sys.implementation holds only a small portion of the implementation-specific data. Over time that ratio will shift in order to make the standard library more portable.

One example of improved standard library portability is cache_tag. As of Python 3.3, sys.implementation.cache_tag is used by importlib to support PEP 3147 compliance. Any Python implementation that uses importlib for its built-in import system may use cache_tag to control the caching behavior for modules.

Using importlib as the Implementation of Import¶

bpo-2377 - Replace __import__ w/ importlib.__import__ bpo-13959 - Re-implement parts of imp in pure Python bpo-14605 - Make import machinery explicit bpo-14646 - Require loaders set __loader__ and __package__

The __import__() function is now powered by importlib.__import__(). This work leads to the completion of “phase 2” of PEP 302. There are multiple benefits to this change. First, it has allowed for more of the machinery powering import to be exposed instead of being implicit and hidden within the C code. It also provides a single implementation for all Python VMs supporting Python 3.3 to use, helping to end any VM-specific deviations in import semantics. And finally it eases the maintenance of import, allowing for future growth to occur.

For the common user, there should be no visible change in semantics. For those whose code currently manipulates import or calls import programmatically, the code changes that might possibly be required are covered in the Porting Python code section of this document.

Other Language Changes¶

Some smaller changes made to the core Python language are:

• Added support for Unicode name aliases and named sequences. Both unicodedata.lookup() and '\N{...}' now resolve name aliases, and unicodedata.lookup() resolves named sequences too.

  (Contributed by Ezio Melotti in bpo-12753.)

• Unicode database updated to UCD version 6.1.0

• Equality comparisons on range() objects now return a result reflecting the equality of the underlying sequences generated by those range objects. (bpo-13201)

• The count(), find(), rfind(), index() and rindex() methods of bytes and bytearray objects now accept an integer between 0 and 255 as their first argument.

  (Contributed by Petri Lehtinen in bpo-12170.)

• The rjust(), ljust(), and center() methods of bytes and bytearray now accept a bytearray for the fill argument. (Contributed by Petri Lehtinen in bpo-12380.)

• New methods have been added to list and bytearray: copy() and clear() (bpo-10516). Consequently, MutableSequence now also defines a clear() method (bpo-11388).

• Raw bytes literals can now be written rb"..." as well as br"...".

  (Contributed by Antoine Pitrou in bpo-13748.)

• dict.setdefault() now does only one lookup for the given key, making it atomic when used with built-in types.

  (Contributed by Filip Gruszczyński in bpo-13521.)

• The error messages produced when a function call does not match the function signature have been significantly improved.

  (Contributed by Benjamin Peterson.)

A Finer-Grained Import Lock¶

Previous versions of CPython have always relied on a global import lock. This led to unexpected annoyances, such as deadlocks when importing a module would trigger code execution in a different thread as a side-effect. Clumsy workarounds were sometimes employed, such as the PyImport_ImportModuleNoBlock() C API function.

In Python 3.3, importing a module takes a per-module lock. This correctly serializes importation of a given module from multiple threads (preventing the exposure of incompletely initialized modules), while eliminating the aforementioned annoyances.

(Contributed by Antoine Pitrou in bpo-9260.)

Builtin functions and types¶

• open() gets a new opener parameter: the underlying file descriptor for the file object is then obtained by calling opener with (file, flags). It can be used to use custom flags like os.O_CLOEXEC for example. The 'x' mode was added: open for exclusive creation, failing if the file already exists.

• print(): added the flush keyword argument. If the flush keyword argument is true, the stream is forcibly flushed.

• hash(): hash randomization is enabled by default, see object.__hash__() and PYTHONHASHSEED.

• The str type gets a new casefold() method: return a casefolded copy of the string, casefolded strings may be used for caseless matching. For example, 'ß'.casefold() returns 'ss'.

• The sequence documentation has been substantially rewritten to better explain the binary/text sequence distinction and to provide specific documentation sections for the individual builtin sequence types (bpo-4966).

New Modules¶

$ python -q -X faulthandler
>>> import ctypes
>>> ctypes.string_at(0)
Fatal Python error: Segmentation fault

Current thread 0x00007fb899f39700:
  File "/home/python/cpython/Lib/ctypes/__init__.py", line 486 in string_at
  File "<stdin>", line 1 in <module>
Segmentation fault

Improved Modules¶

>>> import codecs
>>> encoder = codecs.getincrementalencoder('hz')('strict')
>>> b''.join(encoder.encode(x) for x in '\u52ff\u65bd\u65bc\u4eba\u3002 Bye.')
b'~{NpJ)l6HK!#~} Bye.'

mypolicy = compat32.clone(linesep='\r\n')

>>> m = Message(policy=SMTP)
>>> m['To'] = 'Éric <[email protected]>'
>>> m['to']
'Éric <[email protected]>'
>>> m['to'].addresses
(Address(display_name='Éric', username='foo', domain='example.com'),)
>>> m['to'].addresses[0].username
'foo'
>>> m['to'].addresses[0].display_name
'Éric'
>>> m['Date'] = email.utils.localtime()
>>> m['Date'].datetime
datetime.datetime(2012, 5, 25, 21, 39, 24, 465484, tzinfo=datetime.timezone(datetime.timedelta(-1, 72000), 'EDT'))
>>> m['Date']
'Fri, 25 May 2012 21:44:27 -0400'
>>> print(m)
To: =?utf-8?q?=C3=89ric?= <[email protected]>
Date: Fri, 25 May 2012 21:44:27 -0400

>>> m['cc'] = [Group('pals', [Address('Bob', 'bob', 'example.com'),
...                           Address('Sally', 'sally', 'example.com')]),
...            Address('Bonzo', addr_spec='[email protected]')]
>>> print(m)
To: =?utf-8?q?=C3=89ric?= <[email protected]>
Date: Fri, 25 May 2012 21:44:27 -0400
cc: pals: Bob <[email protected]>, Sally <[email protected]>;, Bonzo <[email protected]>

>>> m2 = message_from_string(str(m))
>>> m2['to']
'Éric <[email protected]>'

>>> m2['cc'].addresses
(Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com'), Address(display_name='Bonzo', username='bonz', domain='laugh.com'))
>>> m2['cc'].groups
(Group(display_name='pals', addresses=(Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com')), Group(display_name=None, addresses=(Address(display_name='Bonzo', username='bonz', domain='laugh.com'),))