Nothing Special   »   [go: up one dir, main page]
Skip to content

Latest commit

 

History

History
198 lines (146 loc) · 9.19 KB

README.md

File metadata and controls

198 lines (146 loc) · 9.19 KB
Raw

Lithops

Lithops is a Python multi-cloud distributed computing framework. It allows you to run unmodified local python code at massive scale in the main serverless computing platforms. Lithops delivers the user’s code into the cloud without requiring knowledge of how it is deployed and run. Moreover, its multicloud-agnostic architecture ensures portability across cloud providers.

Lithops is specially suited for highly-parallel programs with little or no need for communication between processes, but it also supports parallel applications that need to share state among processes. Examples of applications that run with Lithops include Monte Carlo simulations, deep learning and machine learning processes, metabolomics computations, and geospatial analytics, to name a few.

Installation

  1. Install Lithops from the PyPi repository:

    pip install lithops

  2. Execute a Hello World function:

    lithops hello

Configuration

Lithops provides an extensible backend architecture (compute, storage) that is designed to work with different Cloud providers and on-premise backends. In this sense, you can code in python and run it unmodified in IBM Cloud, AWS, Azure, Google Cloud, Aliyun and Kubernetes or OpenShift.

Follow these instructions to configure your compute and storage backends

Multicloud Lithops

High-level API

Lithops is shipped with 2 different high-level Compute APIs, and 2 high-level Storage APIs

Futures API

Multiprocessing API

 
from lithops import FunctionExecutor

def hello(name):
    return f'Hello {name}!'

with FunctionExecutor() as fexec:
    f = fexec.call_async(hello, 'World')
    print(f.result())
 
from lithops.multiprocessing import Pool

def double(i):
    return i * 2

with Pool() as pool:
    result = pool.map(double, [1, 2, 3, 4])
    print(result)

Storage API

Storage OS API

 
from lithops import Storage

if __name__ == "__main__":
    st = Storage()
    st.put_object(bucket='mybucket',
                  key='test.txt',
                  body='Hello World')

    print(st.get_object(bucket='lithops',
                        key='test.txt'))
 
from lithops.storage.cloud_proxy import os

if __name__ == "__main__":
    filepath = 'bar/foo.txt'
    with os.open(filepath, 'w') as f:
        f.write('Hello world!')

    dirname = os.path.dirname(filepath)
    print(os.listdir(dirname))
    os.remove(filepath)

You can find more usage examples in the examples folder.

Execution Modes

Lithops is shipped with 3 different modes of execution. The execution mode allows you to decide where and how the functions are executed.

  • Localhost Mode

    This mode allows you to execute functions on the local machine using processes, providing a convenient way to leverage Lithops' distributed computing capabilities without relying on cloud resources. This mode is particularly useful for development, testing, and debugging purposes. This is the default mode of execution if no configuration is provided.

  • Serverless Mode

    This mode allows you to execute functions on popular serverless compute services, leveraging the scalability, isolation, and automatic resource provisioning provided by these platforms. With serverless mode, you can easily parallelize task execution, harness the elastic nature of serverless environments, and simplify the development and deployment of scalable data processing workloads and parallel applications.

  • Standalone Mode

    This mode provides the capability to execute functions on one or multiple virtual machines (VMs) simultaneously, in a serverless-like fashion, without requiring manual provisioning as everything is automatically created. This mode can be used in a private cluster or in the cloud, where functions within each VM are executed using parallel processes.

Documentation

For documentation on using Lithops, see latest release documentation or current github docs.

If you are interested in contributing, see CONTRIBUTING.md.

Additional resources

Blogs and Talks

Papers

Acknowledgements

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825184 (CloudButton).