Ansible and Python 3

Control node string strategy: the Unicode Sandwich

Until recently ansible-core supported Python 2.x and followed this strategy, known as the Unicode Sandwich (named after Python 2’s unicode text type). For Unicode Sandwich we know that at the border of our code and the outside world (for example, file and network IO, environment variables, and some library calls) we are going to receive bytes. We need to transform these bytes into text and use that throughout the internal portions of our code. When we have to send those strings back out to the outside world we first convert the text back into bytes. To visualize this, imagine a ‘sandwich’ consisting of a top and bottom layer of bytes, a layer of conversion between, and all text type in the center.

For compatibility reasons you will see a bunch of custom functions we developed (to_text/to_bytes/to_native) and while Python 2 is not a concern anymore we will continue to use them as they apply for other cases that make dealing with unicode problematic.

While we will not be using it most of it anymore, the documentation below is still useful for those developing modules that still need to support both Python 2 and 3 simultaneously.

Reading and writing to files

In Python 2, reading from files yields bytes. In Python 3, it can yield text. To make code that’s portable to both we don’t make use of Python 3’s ability to yield text but instead do the conversion explicitly ourselves. For example:

from ansible.module_utils.common.text.converters import to_text

with open('filename-with-utf8-data.txt', 'rb') as my_file:
    b_data = my_file.read()
    try:
        data = to_text(b_data, errors='surrogate_or_strict')
    except UnicodeError:
        # Handle the exception gracefully -- usually by displaying a good
        # user-centric error message that can be traced back to this piece
        # of code.
        pass

Writing to files is the opposite process:

from ansible.module_utils.common.text.converters import to_bytes

with open('filename.txt', 'wb') as my_file:
    my_file.write(to_bytes(some_text_string))

Note that we don’t have to catch UnicodeError here because we’re transforming to UTF-8 and all text strings in Python can be transformed back to UTF-8.

Filesystem interaction

Dealing with file names often involves dropping back to bytes because on UNIX-like systems file names are bytes. On Python 2, if we pass a text string to these functions, the text string will be converted to a byte string inside of the function and a traceback will occur if non-ASCII characters are present. In Python 3, a traceback will only occur if the text string can’t be decoded in the current locale, but it is still good to be explicit and have code which works on both versions:

import os.path

from ansible.module_utils.common.text.converters import to_bytes

filename = u'/var/tmp/くらとみ.txt'
f = open(to_bytes(filename), 'wb')
mtime = os.path.getmtime(to_bytes(filename))
b_filename = os.path.expandvars(to_bytes(filename))
if os.path.exists(to_bytes(filename)):
    pass

When you are only manipulating a filename as a string without talking to the filesystem (or a C library which talks to the filesystem) you can often get away without converting to bytes:

import os.path

os.path.join(u'/var/tmp/café', u'くらとみ')
os.path.split(u'/var/tmp/café/くらとみ')

On the other hand, if the code needs to manipulate the file name and also talk to the filesystem, it can be more convenient to transform to bytes right away and manipulate in bytes.

Interacting with other programs

Interacting with other programs goes through the operating system and C libraries and operates on things that the UNIX kernel defines. These interfaces are all byte-oriented so the Python interface is byte oriented as well. On both Python 2 and Python 3, byte strings should be given to Python’s subprocess library and byte strings should be expected back from it.

One of the main places in Ansible’s control node code that we interact with other programs is the connection plugins’ exec_command methods. These methods transform any text strings they receive in the command (and arguments to the command) to execute into bytes and return stdout and stderr as byte strings Higher level functions (like action plugins’ _low_level_execute_command) transform the output into text strings.

Module string strategy: Native String

In modules we use a strategy known as Native Strings. This makes things easier on the community members who maintain so many of Ansible’s modules, by not breaking backwards compatibility by mandating that all strings inside of modules are text and converting between text and bytes at the borders.

Native strings refer to the type that Python uses when you specify a bare string literal:

"This is a native string"

In Python 2, these are byte strings. In Python 3 these are text strings. Modules should be coded to expect bytes on Python 2 and text on Python 3.

Module_utils string strategy: hybrid

In module_utils code we use a hybrid string strategy. Although Ansible’s module_utils code is largely like module code, some pieces of it are used by the control node as well. So it needs to be compatible with modules and with the control node’s assumptions, particularly the string strategy. The module_utils code attempts to accept native strings as input to its functions and emit native strings as their output.

In module_utils code:

• Functions must accept string parameters as either text strings or byte strings.

• Functions may return either the same type of string as they were given or the native string type for the Python version they are run on.

• Functions that return strings must document whether they return strings of the same type as they were given or native strings.

Module-utils functions are therefore often very defensive in nature. They convert their string parameters into text (using ansible.module_utils.common.text.converters.to_text) at the beginning of the function, do their work, and then convert the return values into the native string type (using ansible.module_utils.common.text.converters.to_native) or back to the string type that their parameters received.

Use forward-compatibility boilerplate

Use the following boilerplate code at the top of all python files to make certain constructs act the same way on Python 2 and Python 3:

# Make coding more python3-ish
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

__metaclass__ = type makes all classes defined in the file into new-style classes without explicitly inheriting from object.

The __future__ imports do the following:

absolute_import:

Makes imports look in sys.path for the modules being imported, skipping the directory in which the module doing the importing lives. If the code wants to use the directory in which the module doing the importing, there’s a new dot notation to do so.

division:

Makes division of integers always return a float. If you need to find the quotient use x // y instead of x / y.

print_function:

Changes print from a keyword into a function.

Prefix byte strings with b_

Since mixing text and bytes types leads to tracebacks we want to be clear about what variables hold text and what variables hold bytes. We do this by prefixing any variable holding bytes with b_. For example:

filename = u'/var/tmp/café.txt'
b_filename = to_bytes(filename)
with open(b_filename) as f:
    data = f.read()

We do not prefix the text strings instead because we only operate on byte strings at the borders, so there are fewer variables that need bytes than text.

Import Ansible’s bundled Python six library

The third-party Python six library exists to help projects create code that runs on both Python 2 and Python 3. Ansible includes a version of the library in module_utils so that other modules can use it without requiring that it is installed on the remote system. To make use of it, import it like this:

from ansible.module_utils import six

Handle exceptions with as

In order for code to function on Python 2.6+ and Python 3, use the new exception-catching syntax which uses the as keyword:

try:
    a = 2/0
except ValueError as e:
    module.fail_json(msg="Tried to divide by zero: %s" % e)

Do not use the following syntax as it will fail on every version of Python 3:

try:
    a = 2/0
except ValueError, e:
    module.fail_json(msg="Tried to divide by zero: %s" % e)

Update octal numbers

In Python 2.x, octal literals could be specified as 0755. In Python 3, octals must be specified as 0o755.

Use str.format() for Python 2.6 compatibility

Starting in Python 2.6, strings gained a method called format() to put strings together. However, one commonly used feature of format() wasn’t added until Python 2.7, so you need to remember not to use it in Ansible code:

# Does not work in Python 2.6!
new_string = "Dear {}, Welcome to {}".format(username, location)

# Use this instead
new_string = "Dear {0}, Welcome to {1}".format(username, location)

Both of the format strings above map positional arguments of the format() method into the string. However, the first version doesn’t work in Python 2.6. Always remember to put numbers into the placeholders so the code is compatible with Python 2.6.

Use percent format with byte strings

In Python 3.x, byte strings do not have a format() method. However, it does have support for the older, percent-formatting.

b_command_line = b'ansible-playbook --become-user %s -K %s' % (user, playbook_file)