Remote, in-memory Python package/module importing through HTTP/S

A feature that Python misses and has become popular in other languages is the remote loading of packages/modules.

httpimport lets Python packages and modules to be installed and imported directly in Python interpreter's process memory, through remote URIs, and more...

with httpimport.remote_repo('http://my-codes.example.com/python_packages'):
  import package1

with httpimport.pypi_repo():
  import distlib # https://pypi.org/project/distlib/

print(distlib.__version__)
# '0.3.6' <-- currently latest (https://github.com/pypa/distlib/blob/0.3.6/distlib/__init__.py#L9)

with httpimport.github_repo('operatorequals', 'httpimport', ref='master'):
  import httpimport as httpimport_upstream
  # Also works with 'bitbucket_repo' and 'gitlab_repo'

url = "https://gist.githubusercontent.com/operatorequals/ee5049677e7bbc97af2941d1d3f04ace/raw/e55fa867d3fb350f70b2897bb415f410027dd7e4"

with httpimport.remote_repo(url):
  import hello

hello.hello()
# Hello world

# From HTTP/S URL
http_module = httpimport.load('package1', 'https://my-codes.example.com/python_packages')
print(http_module)
<module 'package1' from 'https://my-codes.example.com/python_packages/package1/__init__.py'>

# From PyPI
pypi_module = httpimport.load('distlib', importer_class=httpimport.PyPIImporter)
print(pypi_module)
<module 'distlib' from 'https://files.pythonhosted.org/packages/76/cb/6bbd2b10170ed991cf64e8c8b85e01f2fb38f95d1bc77617569e0b0b26ac/distlib-0.3.6-py2.py3-none-any.whl#distlib/__init__.py'>

No file is touching the disk in the process

# with httpimport.remote_repo('https://example.com/packages.tar'):
# with httpimport.remote_repo('https://example.com/packages.tar.bz2'):
# with httpimport.remote_repo('https://example.com/packages.tar.gz'):
# with httpimport.remote_repo('https://example.com/packages.tar.xz'):
with httpimport.remote_repo('https://example.com/packages.zip'):
  import test_package

Any package can be served for httpimport using a simple HTTP/S Server:

echo 'print("Hello httpimport!")' > module.py
python -m http.server
Serving HTTP on 0.0.0.0 port 8000 ...

>>> import httpimport
>>> with httpimport.remote_repo("http://127.0.0.1:8000"):
...   import module
...
Hello httpimport!

After v1.0.0 it is possible to set HTTP Authentication, Custom Headers, Proxies and several other things using URL and Named Profiles!

URL Profiles are INI configurations, setting specific per-URL options, as below:

[http://127.0.0.1:8000]
allow-plaintext: yes ; also 'true' and '1' evaluate to True

[https://example.com]
proxy-url: https://127.0.0.1:8080 ; values must not be in quotes (')

Now, requests to http://127.0.0.1:8000 will be allowed (HTTP URLs do not work by default) and requests to https://example.com will be sent to an HTTP Proxy.

with httpimport.remote_repo("https://example.com"): # URL matches the URL profile
  import module_accessed_through_proxy

Named Profiles are like URL profiles but do not specify a URL and need to be explicitly used:

[github]
headers:
  Authorization: token <Github-Token>

And the above can be used as follows:

with httpimport.github_repo('operatorequals','httpimport-private-test', profile='github'):
  import secret_module

When importing from PyPI extra options can be used, as described in the profile below:

[pypi]

# The location of a 'requirements.txt' file
# to use for PyPI project versions
requirements-file: requirements-dev.txt

# Inline 'requirements.txt' syntax appended
requirements:
  distlib==0.3.5
  sampleproject==3.0.0

# Only version pinning notation is supported ('==')
# with 'requirements' and 'requirements-file' options

# A map that contains 'module': 'PyPI project' tuples
# i.e: 'import sample' --> search 'sample' module at 'sampleproject' PyPI Project:
# https://pypi.org/project/sampleproject/
project-names:
  sample: sampleproject

The PyPI Profiles can be used exactly like all Named Profiles:

with httpimport.pypi_repo(profile='pypi'):
  import distlib
  import sample
distlib.__version__
# '0.3.5' <-- pinned in the profile 'requirements' option
sample.__url__
# 'https://files.pythonhosted.org/packages/ec/a8/5ec62d18adde798d33a170e7f72930357aa69a60839194c93eb0fb05e59c/sampleproject-3.0.0-py3-none-any.whl#sample/__init__.py' <-- loaded from 'sampleproject'

Additionally, all other options cascade to PyPI profiles, such as HTTPS Proxy (HTTP proxies won't work, as PyPI is hosted with HTTPS), headers, etc.

Profiles can be provided as INI strings to the set_profile function and used in all httpimport functions:

httpimport.set_profile("""
[profile1]

proxy-url: https://my-proxy.example.com
headers:
  Authorization: Basic ...
  X-Hello-From: httpimport
  X-Some-Other: HTTP header
""")
with httpimport.remote_repo("https://code.example.com", profile='profile1'):
  import module_accessed_through_proxy

Profiles are INI configuration strings parsed using Python configparser module.

The ConfigParser object for httpimport is the global variable httpimport.CONFIG and can be used freely:

import httpimport
httpimport.CONFIG.read('github.ini') # Read profiles from a file

with httpimport.github_repo('operatorequals','httpimport-private-test', profile='github'):
  import secret_module

The httpimport module automatically loads Profiles found in $HOME/.httpimport.ini and under the $HOME/.httpimport/ directory. Profiles under $HOME/.httpimport/ override ones found in $HOME/.httpimport.ini.

HTTP options

• zip-password - v1.0.0
• proxy-url - v1.0.0
• headers - v1.0.0
• allow-plaintext - v1.0.0
• ca-verify - v1.3.0
• ca-file - v1.3.0

PyPI-only options

• project-names - v1.2.0
• requirements - v1.2.0
• requirements-file - v1.2.0

• allow-compiled

• auth

• auth-type

• tls-cert

• tls-key

• tls-passphrase

import httpimport
import logging

logging.getLogger('httpimport').setLevel(logging.DEBUG)

Using the httpimport with plain HTTP URLs is highly discouraged

As HTTP traffic can be read and changed from all intermediate hosts (unlike HTTPS), it is possible for a remote adversary to alter the HTTP responses consumed by httpimport and add arbitrary Python code to the downloaded packages/modules. This directly results in arbitrary Remote Code Execution on your current user's context of your host!

In other words, using plain HTTP through the Internet can compromise your host without a way to notice it.

• ldsink - The RELOAD flag and Bug Fixes
• lavvy - the load() function
• superloach - Deprecation of imp module in Python3 in favour of importlib
• yanliakos - Bug Fix
• rkbennett - Relative Imports fix, Proxy support

In case my work helped you, you can always buy me a beer or a liter of gas through the Internet or in case you meet me personally.

In the second case we can talk about any video of Internet Historian or Ordinary Things, while listening to a Lofi Girl Playlist, like the citizens of the Internet that we are.