This document describes the technical details of the Masterportal's sensor layer based on the SensorThingsAPI.
The OGC's (Open Geospatial Consortium) SensorThingsAPI "provides an open standard-based and geospatial-enabled framework to interconnect the Internet of Things devices, data, and applications over the Web." (source) The framework provides a data model describing the connection between the "Broker" (Server), a network of "Publishers" (Sensors), and "Clients" (in this case, the Masterportal application).
For more information on the open standard-based SensorThingsAPI, visit:
For a quick overview of the data model, see The Sensing Entities.
The FROST Server is an open source SensorThings Server developed by the Fraunhofer Institut. It is "a Server implementation of the OGC SensorThings API." (source) It acts as the Broker, establishing a link between Publishers (sensors) and the Client (Masterportal, browser). Calls to the FROST Server can be in pure http to use its REST API, or you may establish a bidirectional link via mqtt or CoAP.
To subscribe to required IDs of Things, use a HTTP REST call to fetch all required IDs.
⚠️
expand
,filter
, and so on as URL query parameters are usable with HTTP REST calls only. With mqtt, you may subscribe to a plain path, and URL queries ("?" and beyond) will be ignored.
Basic examples for data calls via REST API:
The FROST Server implements a REST API that allows you to expand
and filter
the query based on a query language comparable to SQL. To join tables, use the $expand
tag as URL query parameter, and separate multiple joins with a comma.
To filter things without knowing their identifier, use $filter
as URL query parameter.
To order things, use $orderby
. This can e.g. be used to retrieve the latest Observation by ordering Observations descending by date and adding $top=1
to fetch the first element only.
You may also use nested statements:
To retrieve Things within an extent, use a POLYGON
:
URL in detail:
You will only receive Things located within the given polygon. Use this to increase network request speed by only retrieving and subscribing to Things in the user's current view.
mqtt is a protocol developed for the Internet of Things to keep an open connection to servers and communicate with pull (commands from client to server) and push (messages from server to client) requests, using an established connection that does not close in the meantime.
In the browser, this might e.g. be implemented by using socket.io.
If you use npm, refer to the mqtt package instead.
The Client uses the mqtt protocol to subscribe to a Topic. A Topic is a plain path to something, e.g. v1.0/Datastreams(74)/Observations
. Note that the host is given to mqtt during the connect operation, and is omitted during further interaction.
After subscribing to a Topic (e.g. v1.0/Datastreams(74)/Observations
), the server will push every new message (in this case, a new Observation in Datastream 74), using the opened mqtt connection, to the Client.
As mqtt may only subscribe and unsubscribe Topics, you have to use HTTP requests (as shown above) to assemble the parts of your Topic. All entities of the SensorThingsAPI can be requested as Topic.
As mentioned before, you can only subscribe to plain REST URLs. Everything in the query part will be ignored:
mqtt://iot.hamburg.de/v1.0/Datastreams(74)/Observations
mqtt://iot.hamburg.de/v1.0/Datastreams(74)?$expand=Observations
Currently used mqtt versions:
The SensorThingsAPI provides automatic splitting of server responses to chunks to avoid overly large payloads. This allows displaying the progress of SensorThingsAPI calls for improved user experience. See Automatic Split for details.
The request can be minimized further by limiting it to the extent currently visible in the browser. See Automatic call in Extent for details.
The Masterportal implements a software layer called SensorThingsHttp
that provides the both split and extent handling.
⚠️ Please mind that automatic progress and "call in extent" are only available if your server side implementation of the SensorThingsAPI (e.g. the FROST Server) provides - and has set to active! - the skip and geography functions.
Your server configuration should activate the automatic splitting of responses. When activated, responses too large for a single response will contain a follow-up link ("@iot.nextLink") to the next data chunk. The total number of chunks is included as "@iot.count" value.
Using the SensorThingsHttp.get()
function, the SensorThingsHttp
layer handles "@iot.nextLink" (see The "@iot.nextLink" Value) and "@iot.count" (see The "@iot.count" Value) for you.
Here is a basic implementation of SensorThingsHttp
, using basic events of the Masterportal, to show its functionality:
import {SensorThingsHttp} from "@src/utils/sensorThingsHttp";
import LoaderOverlay from "/src/utils/loaderOverlay";
const http = new SensorThingsHttp(),
url = "https://iot.hamburg.de/v1.0/Things";
http.get(url, function (response) {
// onsuccess
// do something with the complete response
}, function () {
// onstart
LoaderOverlay.show();
}, function () {
// oncomplete (always called finally)
LoaderOverlay.hide();
}, function (error) {
// onerror
console.warn(error);
}, function (progress) {
// onprogress
// the progress (percentage = Math.round(progress * 100)) to update your progress bar with
});
Please note that the http.get
call in itself is asynchronous. All parameters of SensorThingsHttp.get()
, except for url
, are optional. At least a function for onsuccess
should be provided anyway, or the response is lost.
SensorThingsHttp can be configured with two parameters via constructor.
name | mandatory | type | default | description | example |
---|---|---|---|---|---|
removeIotLinks | No | Boolean | false | removes all "@iot.navigationLink" and "@iot.selfLink" from the response to reduce the size of the result | const http = new SensorThingsHttp({removeIotLinks: true}); |
httpClient | No | Function | null | can be used to change the default http handler (in our case axios), e.g. for testing | const http = new SensorThingsHttp({httpClient: (url, onsuccess, onerror) => {}}); |
If you don't want to use SensorThingsHttp
to automatically split the data, here are some hints regarding your implementation.
If a call's response contains too many datasets, the server splits the result into chunks, indicated by a "@iot.nextLink" for the follow-up chunk. You can follow through all "@iot.nextLink" URLs, gathering the responses until the end of data is received. If no follow-up link ("@iot.nextLink") exists, all chunks have been retrieved.
The following URL will fetch 100 datasets, and the response will include an "@iot.nextLink" to the next chunk: All Things
{
"@iot.nextLink" : "https://iot.hamburg.de/v1.0/Things?$skip=100",
"value" : [{
"...": "..."
}]
}
Calling the nextLink (https://iot.hamburg.de/v1.0/Things?$skip=100) provides you with the next chunk of data and another follow-up link ("@iot.nextLink") and so on, until the last dataset is reached.
If you don't want to use SensorThingsHttp
to automatically follow @iot.nextLinks, please mind the following hint.
Complex calls to the SensorThingsAPI may result in many chunks. The FROST Server is capable to split any delivered array - also in sub-structures - and provide them with an @iot.nextLink. These links hold the split array's key as prefix; e.g. Observations@iot.nextLink, or Datastreams@iot.nextLink.
$top=2
to enforce splitting with [prefix]@iot.nextLink on any expanded sublevel.{
"@iot.nextLink" : "https://udh-hh-iot-qs.germanynortheast.cloudapp.microsoftazure.de/v1.0/Things?$skip=100&$expand=Datastreams%28%24top%3D2%3B%24expand%3DObservations%28%24top%3D2%29%29",
"value" : [ {
"Datastreams" : [ {
"Observations" : [...],
"Observations@iot.nextLink" : "https://udh-hh-iot-qs.germanynortheast.cloudapp.microsoftazure.de/v1.0/Datastreams(13976)/Observations?$top=2&$skip=2",
}, {
"Observations" : [...],
"Observations@iot.nextLink" : "https://udh-hh-iot-qs.germanynortheast.cloudapp.microsoftazure.de/v1.0/Datastreams(13978)/Observations?$top=2&$skip=2",
} ],
"Datastreams@iot.nextLink" : "https://udh-hh-iot-qs.germanynortheast.cloudapp.microsoftazure.de/v1.0/Things(5432)/Datastreams?$top=2&$skip=2&$expand=Observations%28%24top%3D2%29",
}]
}
On following the Datastreams@iot.nextLink, a structure describing further Datastreams is returned:
{
"@iot.nextLink" : "https://udh-hh-iot-qs.germanynortheast.cloudapp.microsoftazure.de/v1.0/Things(5432)/Datastreams?$top=2&$skip=4&$expand=Observations%28%24top%3D2%29",
"value" : [ {
"Observations" : [...],
"Observations@iot.nextLink" : "https://udh-hh-iot-qs.germanynortheast.cloudapp.microsoftazure.de/v1.0/Datastreams(13980)/Observations?$top=2&$skip=2",
}]
}
Keep in mind that a single Thing has neither an @iot.nextLink, nor a "value" key. E.g. this link returns such a feature. Still, this case contains [prefix]@iot.nextLinks to follow in nested structures.
The end of @iot.nextLink follow-ups is marked by the absence of a next @iot.nextLink to follow.
However: If you limit the response using $top=X
(with X being the number of entities to load), an @iot.nextLink may exist.
Following these links will lead to a cascade of server calls - for example, $top=1
on a request that would return 1000 entities would start 1000 server calls, slowing down the system immensely.
Example call for this scenario
{
"@iot.nextLink" : "https://udh-hh-iot-qs.germanynortheast.cloudapp.microsoftazure.de/v1.0/Datastreams(13980)/Observations?$top=1&$skip=1",
"value" : [ {...} ]
}
Unfortunately, you may not simply ignore @iot.nextLinks if you find a $top=X
in the @iot.nextLink, as the X in $top=X
may exceed "the service-driven pagination limitation", and multiple requests are required to actually retrieve X entities:
"In addition, if the $top value exceeds the service-driven pagination limitation (...), the $top query option SHALL be discarded and the server-side pagination limitation SHALL be imposed." (source)
An @iot.nextLink search for $top=X
or %24top=X
in combination with $skip=Y
or %24skip=Y
will do the trick, as any $top=X
not related to the root structure is url encoded with "%3D" instead of "=".
$top=X
: /[\$|%24]top=([0-9]+)/
$skip=X
: /[\$|%24]skip=([0-9]+)/
Use this pseudo-code as guideline for your additional depth barrier:
// pseudo code, some nextLink is given
int topX = fetchTopFromNextLink(nextLink);
int skipX = fetchSkipFromNextLink(nextLink);
if (topX > 0 && topX <= skipX) {
// do not follow (depth barrier reached)
}
The number of expected chunks can be requested by adding $count=true
to the call, which will fill the value for "@iot.count" in the response.
This total number in combination with the current skip value can be used to calculate the loading progress of the application, which may then be shown to the user by a loading bar or other UI elements.
To get the total number of datasets to expect from a call, simply add $count=true
to any SensorThingsAPI URL: https://iot.hamburg.de/v1.0/Things?$count=true
{
"@iot.count" : 4723,
"@iot.nextLink" : "https://iot.hamburg.de/v1.0/Things?$skip=100&$count=true",
"value" : [ {
"...": "..."
}]
}
Combining the absolute number ("@iot.count") and the value of the current $skip
parameter gives you the progress with 1 / @iot.count * skip
.
You may want your server implementation of the SensorThingsAPI (e.g. the FROST Server) to return data only within a given extent (e.g. a polygon). The FROST Server provides you with this functionality. To use this feature, the SensorThingsHttp
layer provides a method SensorThingsHttp.getInExtent()
to retrieve data only within the given extent.
Using SensorThingsHttp.getInExtent()
, you may also use the splitting progress explained above. The SensorThingsHttp
layer creates the correct URL query parameter st_within(Locations/location,geography'POLYGON ((...))')
(see The use of POLYGON) for you.
The extent needs to be described including its source projection and target projection. The following extent options are mandatory for the use of SensorThingsHttp.getInExtent()
:
name | mandatory | type | default | description | example |
---|---|---|---|---|---|
extent | yes | Number[] | - | the extent of your current view | [556925.7670922858, 5925584.829527992, 573934.2329077142, 5942355.170472008] |
sourceProjection | yes | String | - | the extent's projection | "EPSG:25832" |
targetProjection | yes | String | - | projection expected by the SensorThingsAPI server | "EPSG:4326" |
See this basic implementation of SensorThingsHttp
to receive data within the browser's current view extent only, using Masterportal events to show its functionality, as an example:
import {SensorThingsHttp} from "../../../shared/js/api/sensorThingsHttp";
import store from "../../../app-store";
const http = new SensorThingsHttp(),
extent = store.getters["Maps/extent"],
projection = mapCollection.getMapView("2D").getProjection().getCode(),
epsg = this.get("epsg"),
url = "https://iot.hamburg.de/v1.0/Things";
http.getInExtent(url, {
extent: extent,
sourceProjection: projection,
targetProjection: epsg
}, function (response) {
// on success
// do something with the response
}, function () {
// on start (always called)
console.log("start")
}, function () {
// on complete (always called)
console.log("end")
}, function (error) {
// on error
console.warn(error);
}, function (progress) {
// on wait
// the progress to update your progress bar with
// to get the percentage use Math.round(progress * 100)
});
When using SensorThingsHttp.getInExtent()
, the url
and extent
parameters are mandatory. To retrieve the response you need to set the third parameter as an on success function. The others are optional.
An optional eighth parameter httpClient
exists that can be used to replace the default HTTP handler, which is axios
. This optional httpClient
, if used, must be a function with parameters url
, onsuccess
, and onerror
.
POLYGON
If you don't want to use SensorThingsHttp
software layer to access sensors in the current map view, consider these hints for your convenience.
To receive data only in a specified extent, the SensorThingsAPI provides certain geospatial functions using POINT
or POLYGON
structures. See the documentation for more details. You may set your extent by using such a POLYGON
, using the Location of Things to filter them by,
Basic example:
⚠️ Convert your projection to the projection used by the SensorThingsAPI. If the server uses "EPSG:4326", but your Masterportal is set to "EPSG:25832", you must use OpenLayers (or
masterportalAPI/src/crs
, exporting atransform
function) to convert the coordinates.
Example to transform a Location from your current projection into "EPSG:4326":
import crs from "@masterportal/masterportalapi/src/crs";
import store from "../../../app-store";
const extent = store.getters["Maps/extent"],
projection = mapCollection.getMapView("2D").getProjection().getCode(),
epsg = "EPSG:4326",
topLeftCorner = crs.transform(projection, epsg, {x: extent[0], y: extent[1]}),
bottomRightCorner = crs.transform(projection, epsg, {x: extent[2], y: extent[3]});
This way you will get the top left and bottom right corner of the view. To draw yourself a POLYGON
to be used with SensorThingsAPI from that, the rectangle needs to be constructed as follows:
import store from "../../../app-store";
const extent = store.getters["Maps/extent"],
polygon = [
{x: extent[0], y: extent[1]},
{x: extent[2], y: extent[1]},
{x: extent[2], y: extent[3]},
{x: extent[0], y: extent[3]},
{x: extent[0], y: extent[1]}
];
The Masterportal SensorThings software layer is capable of handling mqtt subscriptions for mqtt 3.1
, mqtt 3.1.1
, and mqtt 5.0
. The mqtt version running on the server to be used has to be known and used in SensorThingsMqtt
's constructor.
This is a basic example for mqtt 5.0
:
import {SensorThingsMqtt} from "../../../shared/js/api/sensorThingsMqtt";
const mqtt = new SensorThingsMqtt({
mqttUrl: "wss://iot.hamburg.de/mqtt",
mqttVersion: "5.0",
context: this
});
mqtt.on("message", (topic, message, packet) => {
// handler
console.log("received message:", topic, message, packet);
}, error => {
// onerror
console.warn(error);
});
mqtt.subscribe("v1.0/Datastreams(1234)/Observations", {
rh: 0
}, () => {
// onsuccess
console.log("success");
}, error => {
// onerror
console.warn(error);
});
This is a basic example for mqtt 3.1
and 3.1.1
:
import {SensorThingsMqtt} from "../../../shared/js/api/sensorThingsMqtt";
const mqtt = new SensorThingsMqtt({
mqttUrl: "wss://iot.hamburg.de/mqtt",
mqttVersion: "3.1.1", // "3.1" respective
rhPath: "https://iot.hamburg.de",
context: this
});
mqtt.on("message", (topic, message, packet) => {
// handler
console.log("received message:", topic, message, packet);
}, error => {
// onerror
console.warn(error);
});
mqtt.subscribe("v1.0/Datastreams(1234)/Observations", {
rh: 0
}, () => {
// onsuccess
console.log("success");
}, error => {
// onerror
console.warn(error);
});
Please note that Messages are not received when using "subscribe", but will come in via an on(message)
event.
The on(message)
event's messages must be redirected to your processes with help of the supplied topics.
The software layer SensorThingsMqtt
is a class to be configured at construction time. Creating a new instance, the connection to the mqtt Server is established once per instance in the background.
name | mandatory | type | default | description | example |
---|---|---|---|---|---|
mqttUrl | yes | String | "" | The url to your mqtt server. | "wss://iot.hamburg.de/mqtt" |
mqttVersion | no | String | "3.1.1" | The mqtt version your server runs on. | "3.1", "3.1.1", "5.0" |
rhPath | no | String | "" | For 3.1 and 3.1.1 only, you need to set the basic http path to your SensorThingsAPI to simulate Retained Handling. | "https://iot.hamburg.de" |
context | no | JavaScript Scope | The scope to run the events in. | If you set the context to this , you can use this in your event functions to reach your current module. |
this |
mqttUrl is the URL to connect to the mqtt service. The URL may use any of the protocols 'mqtt', 'mqtts', 'tcp', 'tls', 'ws', or 'wss'. See the mqtt package documentation for additional details.
The mqttVersion will trigger different behavior of the SensorThingsMqtt
software layer.
on("disconnect")
is provided as feature for 5.0The rhPath is used to simulate Retained Handling on mqtt versions 3.1 and 3.1.1, and has to be set to protocol plus domain. To figure out your rhPath, think of it as the missing prefix for a Topic.
E.g., if you accessed your SensorThingsAPI via "https://iot.hamburg.de/v1.0/Things(1234)/Datastreams", you'd subscribe to a Topic via mqtt with "v1.0/Things(1234)/Datastreams". The rhPath is the leftover URL part missing to actually receive data via http. In this case, "https://iot.hamburg.de" is the rhPath.
Be aware that your http path might differ from your mqtt path depending on the protocol to be used; e.g. "wss://iot.hamburg.de/mqtt" could be an rhPath to subscribe to "v1.0/Things(1234)/Datastreams".
Examples:
After construction, you can subscribe using the instance of SensorThingsMqtt
.
name | mandatory | type | default | description | example |
---|---|---|---|---|---|
qos | no | Number | 0 | Quality of service subscription level, see documentation. | 0, 1, or 2 |
rh | no | Number | 2 | "This option specifies whether retained messages are sent on subscription." (source) | 0, 1, or 2 |
Retained Handling (rh) between Client and Server is only available for mqtt 5.0, since previous version to not support this feature.
However, for 3.1 and 3.1.1, SensorThingsMqtt
may simulate Retained Messages by bypassing mqtt with http, internally using SensorThingsHttp
to receive the latest sensor message.
The Retained Handling can be configured as rh := 0, 1, or 2.
An important option for mqtt subscriptions is the so-called "Retained Handling" (rh).
A "Retained Message" is a Sensor message sent in the past, but stored by the server to send immediately after subscription.
import {SensorThingsMqtt} from "../../../shared/js/api/sensorThingsMqtt";
const mqtt = new SensorThingsMqtt({
mqttUrl: "wss://iot.hamburg.de/mqtt",
mqttVersion: "5.0",
context: this
});
mqtt.on("message", (topic, message, packet) => {
if (packet.retain === 1) {
console.log("this is a retained message");
}
else {
console.log("this is a new message");
}
});
mqtt.subscribe("v1.0/Datastreams(1234)/Observations", {rh: 0});
As this might be an unwanted behavior, Retained Handling is inactive by default, that is, rh is set to 2 by default.
import {SensorThingsMqtt} from "../../../shared/js/api/sensorThingsMqtt";
const mqtt = new SensorThingsMqtt({
mqttUrl: "wss://iot.hamburg.de/mqtt",
mqttVersion: "5.0",
context: this
});
mqtt.on("message", (topic, message, packet) => {
if (packet.retain === 1) {
console.log("this will never happen");
}
else {
console.log("this is a new message");
}
});
mqtt.subscribe("v1.0/Datastreams(1234)/Observations");
To identify whether a message is a Retained Message, check the packet.retain
flag included.
import {SensorThingsMqtt} from "../../../shared/js/api/sensorThingsMqtt";
const mqtt = new SensorThingsMqtt({
mqttUrl: "wss://iot.hamburg.de/mqtt",
mqttVersion: "5.0",
context: this
});
mqtt.on("message", (topic, message, packet) => {
if (topic === "v1.0/Datastreams(1234)/Observations" && packet.retain === 1) {
console.log("this is for the second subscription only");
}
else if (topic === "v1.0/Datastreams(1234)/Observations") {
console.log("this is for the first and second subscription");
}
else if (topic === "v1.0/Things(4321)/Datastreams") {
console.log("this is for the third subscription, retain flag is", packet.retain);
}
});
// first subscription
mqtt.subscribe("v1.0/Datastreams(1234)/Observations", {rh: 2});
// second subscription
mqtt.subscribe("v1.0/Datastreams(1234)/Observations", {rh: 0});
// third subscription
mqtt.subscribe("v1.0/Things(4321)/Datastreams", {rh: 0});
To close a mqtt connection, execute end
on the SensorThingsMqtt
instance.
import {SensorThingsMqtt} from "../../../shared/js/api/sensorThingsMqtt";
const mqtt = new SensorThingsMqtt({
mqttUrl: "wss://iot.hamburg.de/mqtt",
mqttVersion: "5.0",
context: this
});
mqtt.end(false, {}, () => {
console.log("finished");
});
For more information on the end
function parameters, visit the end function documentation of mqtt; this called is passed through to the mqtt package without further effects.