pyngsi is a Python framework that helps you write a pipeline for your Fiware dataflow.

Writing a NGSI agent that relies on pyngsi avoids all the plumbing so you can focus on writing your own logic to build NGSI entities.

• NGSI v2 support
• Map Python-native data to NGSI entities
• Write NGSI entities to Fiware Orion
• Handle incoming data through a common interface
• Compute statistics
• Allow visualization/debugging facilities

The source code is currently hosted on GitHub at : https://github.com/pixel-ports/pyngsi

Binary installer for the latest released version is available at the Python package index.

pip install pyngsi

from pyngsi.ngsi import DataModel

m = DataModel(id="Room1", type="Room")
m.add_url("dataProvider", "https://fiware-orion.readthedocs.io/en/master/user/walkthrough_apiv2/index.html#entity-creation")
m.add("pressure", 720)
m.add("temperature", 23.0)

m.pprint()

The resulting JSON looks like this :

{
  "id": "Room1",
  "type": "Room",
  "dataProvider": {
    "value": "https://fiware-orion.readthedocs.io/en/master/user/walkthrough_apiv2/index.html#entity-creation",
    "type": "URL"
  },
  "pressure": {
    "value": 720,
    "type": "Integer"
  },
  "temperature": {
    "value": 23.0,
    "type": "Float"
  }
}

from pyngsi.sink import SinkOrion

sink = SinkOrion()
sink.write(m.json())

Let's quickly create a CSV file to store values from our room sensors

echo -e "Room1;23;720\nRoom2;21;711" > room.csv

Let's code a function that converts incoming rows to NGSI entities

def build_entity(row: Row) -> DataModel:
    id, temperature, pressure = row.record.split(';')
    m = DataModel(id=id, type="Room")
    m.add_url("dataProvider", row.provider)
    m.add("temperature", float(temperature))
    m.add("pressure", int(pressure))
    return m

Let's use it in in our new NGSI Agent

from pyngsi.sources.source import Source, Row
from pyngsi.sink import SinkOrion
from pyngsi.agent import NgsiAgent

src = Source.from_file("room.csv")
sink = SinkOrion()
agent = NgsiAgent.create_agent(src, sink, process=build_entity)
agent.run()

This basic example shows how the pyngsi framework is used to build a NGSI Agent.
Here data are stored on the local filesystem.
By changing just one line you could retrieve incoming data from a FTP server or HTTP server.

from pyngsi.sources.source import Source, Row
from pyngsi.sources.server import ServerHttpUpload
from pyngsi.sink import SinkOrion
from pyngsi.agent import NgsiAgent

src = ServerHttpUpload() # line updated !
sink = SinkOrion()
agent = NgsiAgent.create_agent(src, sink, process=build_entity)
agent.run()

The HTTP server is running. Now you can send the file to the endpoint.

curl -F file=@room.csv http://127.0.0.1:8880/upload

JSON and text formats are natively supported.
Many sources and sinks are provided, i.e. SinkStdout to just displays entities, eliminating the need of having an Orion server running.
One could create a custom Source to handle custom data. The MicrosoftExcelSource is given as exemple.
One could extend the framework according to his needs.

• loguru
• requests
• requests-toolbelt
• shortuuid
• more_itertools
• geojson
• flask
• cherrypy
• schedule
• openpyxl

Apache 2.0

The official documentation is hosted at https://pixel-ports.github.io/pyngsi-tutorial.html

SourceMicrosoftExcel may fail to open some odd Excel files due to an openpyxl bug (i.e. sometimes cannot read graphs).
In this case try to remove the offending sheet or prefer working with CSV files.

Work on pyngsi started at Orange in 2019 for the needs of the PIXEL european project.

pyngsi has been developed as part of the PIXEL project, H2020, funded by the EC under Grant Agreement number 769355.