LDAF is a NodeJS-based framework designed to simplify the offloading of software, mainly the parts that connect to third-party services, to servers, optimizing system resource and data usage.

Communication protocols set the frequency and size of protocol data units (PDU) exchanged between the communicating entities. These determine the amount of communication traffic between the entities.

A network interface card, for example, builds an Ethernet frame that is comprised of a payload, and control data in the frame header and tail. The size of the frame varies according to the size of the payload. The number of frames depends on communication requirements of upper layer protocols that rely on the Ethernet connection. RS-232 link settings define numbers of start and data bits in a RS-232 frame. Along with a given transmission bit rate (set by the serial port hardware) this defines the maximum effective transmission rate. Two TCP agents exchange several segments to set up a TCP session, in which they then bi directionally transport application payload and manage the session.

In application layer protocols various modes of operation can be found. A simple stateless client-server operation, e.g. in HTTP, is defined by a request sent to a server and a corresponding response received by a client. WebSocket provides full-duplex communication channels over a single TCP connection. The channel establishment starts with the HTTP, which is during the handshake upgraded to the WebSocket. A typical WebSocket session would therefore comprise of handshake signaling messages, frames to close the session and data frames that exchange the application data.

These examples result in communication bitrate requirements imposed to the underlying layers and finally to the physical capabilities of the communication channel. Sometimes the communication traffic can be reduced by optimizing the application logic or configuring application layer protocol settings. However, if we would like to control the bitrate to the maximum possible level, we need to further address both, the number and the size of protocol data units exchanged. This requires modifications of both the PDU semantics, as well as the operation of the application protocol itself.

In the Ethereum network for example, an application can access a remote Ethereum node over HTTP or WebSocket. The traffic passed between the client and the application arises from event notifications and transaction exchange. Some of these events are periodic (e.g. last block number notification) and the others are application specific (e.g. an event generated by a smart contract, when a predefined condition in the blockchain is met). If HTTP is applied, the periodic traffic results in about 2400 B/s traffic between the application and the node. If WebSocket is employed instead, this is reduced to about 170 B/s. But to reduce this amount even further, an optimized application layer protocol needs to be applied.

LBE offers such a solution. It runs over TCP and can be used to replace any application layer protocol to provide a very low bit-rate point-to-point connection. This is usually a connection between a communication constrained end-device and a LBE proxy. The device runs a LBE client, which connects to the proxy. The LBE proxy assures the appropriate mapping between the LBE connection and the application protocol required by the target application servers.

The LBE defines proprietary message encodings and formats, which can have a minimal size. For each intended use case a new service can be set in the proxy. The service maps the messages between the LBE and the application protocol, where it can apply application specific logic and rules. This makes it possible to reduce the number of messages, passed between the client and the proxy. Such a rule could forward from the proxy to the client only every tenth new block notification, if for example the LBE client application does not have to be informed by every single event. This only would reduce the periodic traffic in the Ethereum example from 170 B/s to about 12 B/s.

Today, many devices are using mobile networks that either charge per data transferred, like the networks our smartphones use, or are intended to be used by low power devices and are simply not capable of fast data speeds, like most IoT networks.

In both of these cases it is crucial that we minimize the amount of data that our apps need to transfer.

First, we can lower the frequency of the data packets. This means either changes in the functionality (for example: retrieve data less often) or enabling the server to “push” data to the device (remove the need for polling).

However, there is only so much we can gain from these approaches without negatively impacting the performance of the app.

The second, and probably main, area of focus should be the size of the packets. We often use inefficient encoding (JSON) to get data that we don’t (non-custom API) need over a protocol with very high overhead (HTTP).

You’ll notice that what has been described here is actually your average REST API.

So all we have to do is maintain a connection with the server (to enable the server to “push” data), then efficiently encode only the data that we need and send it using a protocol with low overhead (for example WebSocket)

This is not a problem if we built our own server software from the ground up, but more and more apps are designed to use third party services to function. Everything from the Google API to Thingspeak to the Ethereum network. These services are not designed to save data, and most are accessed by a simple API (which, as we’ve discovered, is very inefficient).

To still use all these third-party services while optimizing the connection to your app you need to create a custom API proxy that maintains a connection with the device and makes API calls on its behalf. This way you have full control over the encoding, payload, and protocols used, as well as the opportunity to move some of the load from the device to the server (for example the server can poll an API, and notify the device if changes occur)

In our tests the average data rate of our device went from 2400B/s to ~12B/s without any loss in functionality. This case was a very good use case, but even if your device saved 1/10 as much data as ours, that would still be a 20x reduction in data usage.

Obviously, writing software to do this can be very time consuming. You have to set up a server accept and track connections, use (and probably in part develop) the correct encoding, handle all kinds of errors, and worry about all those request-response mechanics

This is where LBE comes in. Using this software, all you have to do to make this data-saving proxy is handle the messages. Everything else is done for you. More information about how it does this and how you can use it below.

1. Download the git repository
2. Install dependencies with npm i
3. Define the environment variables (preferably in the .env file)
4. Run using node index.js

LDAF on is good out of the box if you just want to see how it works and how to use it, but you really should should create a service if you want to use it properly.

Environment variables control a number of things throughout LDAF. The intended way to set them is through a .env file, but you can do this differently if you want to.

WS - set to false to disable WS server (defauly: true)

WS_PORT - the port the WS server will use (default: 8547)

The WebSocket server is enabled by default, and is very useful for testing and monitoring, but it can be disabled

Note that some Extensions may use other environment variables. Check their documentation for more.

LDAF is comprised of services. These are modules, that devices are assigned to, that then handle their requests and feed them appropriate information.

Your service should be in a folder in the services folder that is named after the service. LDAF will only access the main.js file in that folder, but you can add others if needed (schemas, smart contract definitions, ...)

The intended way to create a service is to extend the "Service" object (./services/Service.js). Note that when using the js-binary extension this object expects a "schemas.js" file in the service's folder with all the schema definitions it needs to encode and decode its messages.

To access the Service class, execute the function exported by the Service module with an array of required extensions as the parameter. It should look something like: const Service = (require('./Service'))(['web3'])

This object includes:

• ee - EventEmitter - the global event emitter

• clg - function - console.log that also prints the name of the service

• cle - function - console.error that also prints the name of the service

• serviceName - String - the name of the service

• connections - Array of connection objects - the connections currently assigned to this service

• newConnection - function(connection) - The parameter is an object with 2 properties. The first, conn, is the connection object, and the second, offset, is the type offset that the service should use (the sum of the number of types services before it use). This is the function that will be called to assign a new device to this service.

• init - function - This runs when the service starts. Use this function to any APIs you want to use; listen for events that devices should (eventually) be notified about.

• stopSubs - function - called when the current subscriptions should be cancelled - the last device disconnects, the connection to Geth is lost

• sendToAllConnections - function(messageType, messageObj) - encodes messageObj into a message of the specified type and sends it to all connections that are currently using this service. This is used with for push messages.

• sendToConnection - function(connection, seq, messageType, messageObj) - encodes messageObj into a message of the specified type and sends it to the connection. This is used with for push messages. Set seq to 0 for a push message.

When a message arrives at the service, it gets decoded and is emitted as an event with the name of the

For example services check the examples and services folders in the main directory

When a device connects to LDAF, it must supply an array of services that it will use. The device, along with the required type offset, is then added to each service. This way devices can mix and match services as much as they want to as long as the total number of types does not exceed 256^typeLen. For most cases typeLen=1 should be plenty, but you can use a higher typeLen at the expense of higher overhead. Just make sure that the device and the Connection are configured the same.

A service definition is a JS Object that tells LDAF what kinds of messages will be transferred as well as how to encode/decode them. It contains the name of the service, and an array of messageTypes.

Each messageType must have the followint properties:

• name - String - the name of the message type. You can use this name to refference the messageType
• encode - function(Object) - returns a Buffer or String instance containing the data from Object
• decode - function(Buffer) - returns a JS Object containing the data from the Buffer instance
• type- String - when will this messageType be used
  • "psh" - pushed from the server
  • "req" - request from the client
  • "res" - response to a request

LDAF will find the appropriate messageType based on its type and name. No two messageTypes can have the same type AND name (they can share a name if the type is different).

Here is an example of a file containing a service definition. In this case js-binary is used.

const Type = require('js-binary').Type;
const emptyType = new Type({});
const exportObj = {
    name: 'exampleServiceDef',
    messageTypes:
        [
            {
                name: 'newBlock',
                type: 'psh',
                schema: new Type({
                    blockNumber: 'uint'
                })
            },


            {
                name: 'isSyncing',
                type: 'req',
                schema: emptyType
            },
            {
                name: 'isSyncing',
                type: 'psh',
                schema: new Type({
                    isSyncing: 'uint'
                })
            },
            {
                name: 'isSyncing',
                type: 'res',
                schema: new Type({
                    isSyncing: 'uint'
                })
            },


            {
                name: 'getTransaction',
                type: 'req',
                schema: new Type({
                    txHash: 'Buffer'
                })
            },
            {
                name: 'getTransaction',
                type: 'res',
                schema: new Type({
                    from: 'Buffer',
                    to: 'Buffer',
                    value: 'float',
                    blockNumber: 'uint'
                })
            },
        ]
};

//This makes it simpler to define encode and decode functions
for(let messageType of exportObj.messageTypes){
    messageType.encode = messageType.schema.encode.bind(messageType.schema);
    messageType.decode = messageType.schema.decode.bind(messageType.schema);
}

module.exports = exportObj

The schema objects are never used directly. They are there only so the for loop can correctly assign the encode and decode functions. Also keep in mind that this is just my implementation of a serviceDef and yours can be different.

It is important to keep track of the number of messageTypes you're using. All the services a device is using can have a maximum of 256^typeLen types. However, this is bound to the device, not the server. A server could have thousands of messageTypes spread across its services, but devices may still work with a typeLen of 1.

LDAF is designed so that users can easily add ways for devices can connect to it. To do this, all connection types must look the same to the rest of the code. This is the specification any new connection types you add must follow in order to be compatible.

When assigning a Connection to a Service, the Service's newConnection function should be used. The parameter of this functuon is an object containing the current offset (TODO: more about this... link) and a connection object.

This object must include the following properties:

• services - Array of strings - The names of the services that should accept the connection.
• send - function(type, seq, encodedMessage) - this sends the message of the specified type and sequence number.
• close - function - close the connection

It must also emit the following events:

• error - with an error object as the parameter - when there's an error in the connection
• close - when the connection gets closed so that services can clean up accordingly
• message - with a message object as the parameter {type:int, seq:int, payload:hex string} (the payload is still encoded) - when a new message arrives

Of course, these objects can contain other, custom properties and events that your services may need, depending on the application, but this can lead to problems and should be avoided if possible.

Try to support environment variables where possible.

To start using a new connection type, simply "require" it in "index.js".

LDAF has two levels of encoding. The first is tied to the connection type, and the second is tied to the service. The encoding described on this page is the default. Connection types and services may use their own encoding.

The first encoding that happens is to get the type and sequence numbers from the packet. Each of these can be 1, 2 or 4 Bytes long, but the default is 1 Byte. You can change these values by setting typeLen and seqLen in the transcoder options object.

When encoding, the sequence number is omitted when a message is not a response to a request and does not need to be tracked in this way. The server can never receive a request encoded like this and can therefore assume the sequence number will always be present.

The encoding for this level is provided by the encode and decode functions defined in the Service definition. These can be all kinds of libraries, custom functions, or even JSON.parse/stringify if wrapped correctly. The important thing is that they transcode data between a Buffer and a regular JS Object. You can use different encodings for each service and even each specific messageType.

Here are my recommendations for this layer:

JS-Binary is a JS library that is simple to use, fast, and efficient. It is recommended that you use this if your device will be running JS (NodeJS).

Google Protocol Buffers accomplishes the same task as js-binary, but it has implementations in various languages including C++, Java, Python, and JS. It also has a lot of very useful features. Look at the documentation for more. The downsides are that the encoded data tends to be slightly larger and it's not as simple to use as js-binary.

The Service.js file exports a function that returns a custom Service object for you to extend. This object comes pre-loaded with extensions that you'll need.

These extensions work with the Service's embedded event emitter that runs the extension's functions at the same time as it runs the Service's functions. In practice this means that whenever the init() function on a service is called, any listener to the 'init' event will also run.

Note that these functions are called in the order of the elements in the extension selection array.

Let's say you want your service to print out the number of active connections it has whenever a new device connects. To do this you have to add an entry into the extension selection array, but instead of the name of the extension (for example 'web3'), we have to add a function that will set up all the necessary event listeners. For our example, this is the function:

function () {
        this.on('newConnection', () => {
            this.clg('Someone new connected! :) Now there are ' + this.connections.length + ' devices connected.')
        });
    }

These are the built-in Extensions at the moment.

You can enable the web3 extension by adding 'web3' (as a string) to the array of extensions before initializing your service object.

When this extension is enabled the following will change:

• stopSubs() and init() will be called when the connection with Geth is lost/regained

• a web3 object will be accessible from the service via this.web3. This object is refreshed automatically when needed (Geth connection problems), but working with it when it isn't connected will cause problems. Use the 'gethConnectionLost' event on the global event emitter to handle this

• a web3Subs object will be added to the main object. All subscriptions (created in the init() function) should be assigned to it. When needed, these subscriptions will be automatically shut down and restarted.

The Web3 Extension uses the following environment variables:

WEB3 - (true/false) - set to false to disable web3. If Web3 is disabled the other environment variables in this section are not used. If it is enabled you will have to install Web3 (default: true)

WEB3_CONNECTION_TYPE - (ws/ipc) - set the way web3 will connect to the Ethereum node (Geth) (REQUIRED)

WEB3_WS_ADDR - (URL) - of the Ethereum node's WS server. Only used if the connection type is set to ws (default: ws://localhost:8546)

WEB3_IPC_PATH - (path) - path to the Ethereum node's IPC file. Only used if the connection type is set to ipc (default: ~/.ethereum/testnet/geth.ipc)

Check the examples for more.

LbeClient.js is a class that programs can use to connect to the LDAF server. Simply provide it with the necessary options (described below), and it will allow you to send requests to the server with the ability to add a callback for the responses, and expose an event emitter for the messages that the server pushes. Check the examples to see how it works.

LbeClient only has a single dependency, ws.

To use LbeClient.js you must import and instantiate it like so: const lbe = new (require('./LbeClient'))(options); The options object has the following properties:

• address - URL - the address of the LDAF server (REQUIRED)
• serviceDefs - array of Service definition - the definitions of services used by this device (REQUIRED)
• typeLen - {1, 2, 4} - the number of bytes to use for the type (Default: 1)
• seqLen - {1, 2, 4} - the number of bytes to use for the sequence number (Default: 1)

A new way to connect to LDAF, made for low-power devices such as the Arduino.

Instead of defining services in the headers, an initialization message will be sent just after connecting.

It is expected that this will replace WS as the main connection type

A new message encoder/decoder written in C++ instead of JS and that will support different typeLen and seqLen for each connection.

If a server receives a message it knows it's a request, and if a client receives a message it knows it's either a response or a push message. The new number of types a connection will have to support will be max(count(req), count(res) + count(psh)) instead of just the count of all types. This is a typeCount saving of up to 50%.

Along with specifying which services are required, devices will be able to connect to multiple servers.

Connect to LDAF via The Things Network.