OGC Testbed-13: Asynchronous Services ER

Compared to previous testbeds, new classes of use cases are addressed in Testbed 13:

Extensions for adding asynchronous communication to WFSs are not yet available. The approaches for Asynchronous Web Feature Services implemented in this Testbed implement and extend an approach developed in the previous Testbed 12 (see Section 7, OGC 16-023r3) using additional query parameters indicating that the operation should be executed in an asynchronous communication mode. The approaches described in this engineering report extend the previous work as follows:

While most current OGC standards are based on synchronous communication patterns, there are several applications, which need asynchronous client-server interaction patterns. This way clients do not have to keep the connection to the server continuously open in order to wait for responses. This ER summarizes the results from the different activities in Testbed 13 dealing with asynchronous service responses and discusses and relates them to previous related activities. Special focus was given to the specific requirements for asynchronous communication in the two planned use case types, i.e. communication in situations with denied, degraded, intermittent or limited bandwidth and automatic notification of users if new or updated data sets become available. Solutions have been provided for the OGC Web Feature Service that reuse and extend the concepts developed in Testbed 12. The ER documents the relevant decisions, implementations, and findings in order to help to enable asynchronous communication in OGC based infrastructures.

All questions regarding this document should be directed to the editor or the contributors:

The following issues are considered as future work:

A more detailed description of these issues is given in Section 9.

Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. The Open Geospatial Consortium shall not be held responsible for identifying any or all such patent rights.

Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the standard set forth in this document, and to provide supporting documentation.

In the polling approach, the client sends a request to the service indicating that a service operation should be run in asynchronous mode. The server responds with a status message indicating that the operation has been started. Subsequently, the client can request status information about the operation’s execution. Once the operation is finished, the server will respond with the actual result of the operation. In this approach, the client needs to actively query the status of the operation running on the server.

The OGC WPS (OGC 14-065) supports this approach for asynchronous communication by defining an additional service parameter mode parameter for its Execute operation. The sequence of operations is shown in Figure 2.

In Testbed 12, the WPS and its polling approach have been evaluated to support asynchronous retrieval of feature data sets from a WFS and WCS. In comparison, an additional WFS query parameter responseFormat was specified to directly support asynchronous communication for WFSs. As an example, the following URL can be used to indicate that a GetFeature operation should be executed in asynchronous mode and the result should be mailed to a certain email address:

The concepts for both approaches and the evaluation results are described in detail in the OGC Testbed 12 Implementing Asynchronous Services Response Engineering Report (OGC 16-023). In a nutshell, asynchronous messaging was successfully implemented using both approaches. Both provide a way to add asynchronous messaging to OGC Web Services in a more lightweight fashion compared to the OGC Publish/Subscribe specification. One potential drawback of the method with WPS facade is that the usual service requests need to be wrapped in a WPS request and response which causes a little overhead. However, the bigger the data sets, the less overhead occurs. An advantage is that the WPS facade approach can be used with the different OGC data services (WFS, WCS, SOS). Besides the advantage of less communication overhead, direct support of asynchronous communication in WFS using additional query parameters was also implemented without polling, but with a notification endpoint, where the operation’s result should be sent to.

This approach is following the publish/subscribe messaging pattern that is commonly implemented by message brokers and specified for SOAP Web Services by OASIS in the Web Service Notification (WS-N) framework. Message providers, called publishers, publish their messages to a server. Clients can subscribe for these messages, but do not need to continuously query information. Instead, the clients provide an interface for receiving messages. The publish/subscribe operations may be either directly provided by an OGC Web Service or provided in an external component, the message (or notification) broker.

As mentioned above, the previously defined Sensor Event Service and Sensor Alert Service were following the publish/subscribe pattern and were evaluated in Testbeds 6 to 10. With the release of the OGC Publish/Subscribe Interface Standard in 1.0 (OGC 13-131r1) in 2016, a general standard is now available that can be used to extend existing OGC Web Services with asynchronous messaging.

Figure 3 shows the general interaction between subscribers and publishers. OGC Web Services would usually act as a Publisher, where clients (Subscriber) can subscribe for certain information, e.g. in case a feature set in a WFS is updated or new features are available. The publisher is responsible for checking all subscriptions and sending messages to subscribers using Sender and Receiver components. Subscribers can also renew subscriptions or unsubscribe.

In Testbed 12, a publish/subscribe extension for the OGC Catalogue Service, version 2.02 (OGC 07-006r1) has been specified and evaluated. The results are described in the OGC Testbed 12 PubSub/Catalog ER (OGC 16-137). Furthermore, the application of the OGC PubSub standard in the aviation domain has also been evaluated and reported in the corresponding OGC Testbed 12 Asynchronous Messaging for Aviation ER (OGC 16-017).

The OGC SensorThings API (OGC 15-078r6) takes a slightly different approach than utilizing the OGC Publish/Subscribe standard. Instead, an MQTT extension is specified that describes how to apply the Message Queue Telemetry Transport (MQTT) protocol for asynchronous communication in the SensorThings API. The interactions for creating and subscribing to observations in the SensorThings MQTT extension are shown in Figure 4.

Clients can subscribe to data streams/observations. Different sensor sources can push new observations to the SensorThings API. If clients are subscribed, they will receive the new observations via MQTT. MQTT has been originally designed for machine-to-machine communication and thus for messages with high frequency and small payload. As such, it is well-suited for publish/subscribe in sensor applications.

In OGC Testbed 13, participants assessed the ability of an Asynchronous WFS to support simulated users in a Mass Migration Scenario over Syria and Jordan. In this scenario, large numbers of people have been displaced from the Daraa region of Syria to the Zaatari refugee camp in Jordan due to ongoing conflict. As the conflict ends 'de-escalation zones' are established by major powers and plans are made to return displaced people from refugee camps to the region. Understanding the situation on the ground and the infrastructure, as well as transporting these people from refugee camps into a former conflict zone is a major challenge for relief agencies. To accomplish this task, they must understand the environment and infrastructure in the region between Zaatari refugee camp and the Daraa region.

The following examples provide a brief sample of the scenario involved in testing the Asynchronous WFS approach.

Years of war have displaced hundreds of thousands from Darra city and region. Thousands have sought shelter at the sprawling Zaatari refugee camp (Figure 5).

A ceasefire has been declared this year. Government and non-government organizations seek to assist displaced populations at Zaatari in returning to Daraa, as illustrated in Figure 6.

The demonstration focuses on one type of user, the Analyst User.

Analyst User needs to execute the following tasks, as illustrated in Figure 7:

Hence, the following User Stories are considered for the Analyst User (Figure 8):

For integration testing, The Carbon Project implemented an Asynchronous Web Feature Server (Async WFS) according to the architecture outlined in section 5 of this ER. The Async WFS system components were hosted using cloud computing services technology based on The Carbon Project’s CarbonCloud® geospatial data platform.

CarbonCloud® uses a cloud-based operating system called Windows Azure to run its 'fabric layer' - a cluster hosted at Microsoft data centers that manages computing and storage resources of the computers and provisions the resources to applications like Async WFS. Scaling, reliability, memory resources and load balancing are controlled by the Azure cloud - so developers can focus on deploying applications like Async WFS on the CarbonCloud platform.

Key Windows Azure technical features used by the Async WFS components include:

For Testbed 13, the Async WFS system components implemented OSM data in a schema based on the National System for Geospatial-Intelligence (NSG) Topographic Data Store (TDS) model, as illustrated in Figure 9. For the Async WFS-1 work item OSM was deployed as Geography Markup Language (GML) in the TDS Content Specification. Systems participating within the NSG may use the NSG TDS Content Specification to help ensure consistent geospatial data semantics, adopt common conditions for geospatial intelligence collection/exchange, support net-centric geospatial services, and achieve geospatial data interoperability. GML provides NSG TDS OSM data in XML form and JavaScript Object Notation (JSON) provides it in a lightweight exchange format.

In Testbed 13, The Carbon Project’s Async WFS-1 was tested in a humanitarian aid and mass migration scenario described in the Demonstration Scenario section. In particular, testing showed that it is feasible to request feature data to support relevant needs and have it delivered to users asynchronously to users via email and simple data download (see Figure 10 below).

The data delivered to users asynchronously implemented the National System for Geospatial-Intelligence (NSG) Topographic Data Store (TDS) OpenStreetMap (OSM) model using The Carbon Project’s free Gaia application. The Structure Points (structpts) shapefile data from the NSG TDS OSM was used for testing and delivered as GML for the Asynchronous WFS in the example below (Figure 11).

The following example shows the use of the mailto: response handler in the Asynchronous WFS request:

In this example, the Asynchronous WFS request is sent to a REST WFS endpoint with a GML output format. Other formats such as JSON may also be specified. An example of TDS OSM JSON output is included in that section that follows.

The following is an example of the acknowledgment message that the server generates in response to an Asynchronous WFS request:

Please note that the future acknowledgment message that the server generates in response to an Asynchronous WFS request may be in JSON format.

The following is an example of the notification message that the server generates in response to an Asynchronous WFS request and sends to the mailto: response handler:

The notification message from the server includes the following link to download the results of the Asynchronous WFS request:

When the user clicks the link in the notification email the data from Job 45 is downloaded. The GML data may be rendered in the geospatial application online or offline. For reference purposes it is shown in the Gaia application below (Figure 12) on top of satellite imagery tiles for the Daraa region.

For the Async WFS-1 work item OSM was also deployed as JSON in the TDS Content Specification. JSON provides a lightweight exchange format and GeoJSON is a geospatial data interchange format based JSON. It defines several types of JSON objects and the manner in which they are combined to represent data about geographic features, their properties, and their spatial extents. GeoJSON uses a geographic coordinate reference system, World Geodetic System 1984, and units of decimal degrees.

GeoJSON objects may represent a region of space (a Geometry), a spatially-bounded entity (a Feature), or a list of features (a Feature Collection). Features in GeoJSON contain a geometry object and additional properties, and a Feature Collection that contains a list of features. An example of a prototype TDS OSM JSON is shown below. It should be noted that the TDS data model is extensive and it may be simplified since many of the properties are not able to be populated with the information provided in OSM.

The scenario for the Async WFS-2 is also to download a large data set from a WFS. Instead of providing OSM data in the TDS format, data from the Ordnance Survey of Great Britain is provided by the WFS implementing the DGIWG WFS profile.

In December 2015, Northwest of England suffered a severe flood, many places were flooded, including the historic city of Yorkshire. On December 26th, the Yorkshire flood control system suddenly opened the flood gates, so that the disaster became more serious and made the local people dissatisfied. The authorities explained that they were concerned that the electronic pumps inside were flooded by water and decided to open the flood gates [1]. The situation in the Yorkshire area worsened and thousands of residents needed to be evacuated, causing local resident’s indignation. Illustrations of the floods are shown in Figure 13 and Figure 14.

In this case, if the authorities could pre-simulate the degree of flooding with real-time WFS service, they could avoid a worsening disaster. In Figure 15, Government units can also provide real-time WFS services for hydro-node and flooding map, so that citizens can understand the current flooding situation through the Application. The scientist or application providers can calculate and simulate the degree of flooding with overlap map from the asynchronous WFS services, so they could send a warning to the citizens. The raw OSM map is shown in Figure 15, an overlay with flooded areas provided by the Async WFS is shown in Figure 16.

The Asynchronous WFS-2 provides four service operations: GetCapabilities, GetFeature, GetStatus, and Cancel. These four operations demonstrate the capability to deal with Big Data query scenarios as illustrated in Figure 17. GetStatus and Cancel are optional and not required. In the scenario, a user can operate client components (procedure 1.1). The client would send a GetCapabilities request to the WFS-2 (procedure 1.2 and 1.3). The Async WFS-2 will respond with the Capabilities document immediately and send it back to the Client (procedure 1.4). For the asynchronous scenario, a customized GetFeature operation with an extra mailto parameter, containing the subscription email address, needs to be sent to the request handler (procedures 2.1-2.3). The WFS-2 will then return a message with operation ID information (procedure 2.4). The client can trace the progress via the operation ID (procedures 3.1 and 3.2) or it can cancel the operation using the operation ID (procedures 3.3-3.5). If the operation is not canceled, the WFS-2 will generate the download URL (procedures 2.5 and 2.6) after all backend query processing has finished, and deliver an e-mail with URL to the subscribed user (procedures 2.7).

Web Feature Services are still mostly used in a synchronous communication fashion for rendering feature data sets in maps following the common pattern of 'request-response-parse-render'. However, the rapidly increasing availability of large feature data sets, e.g. OpenStreetMap features, require novel approaches. The Asynchronous WFSs developed in this testbed demonstrate that the extensions for asynchronous communication are able to solve the issue of transferring large feature data sets. However, the roles that such Asynchronous WFSs may play in large geospatial enterprise architectures have not been fully explored yet. Hence, we recommend exploring these roles in future testbeds, e.g. serving initial or intermediate feature data sets in geoprocessing workflows or the role of Asynchronous WFSs in Big Data architectures.

The activities regarding asynchronous messaging were focusing on OGC Web Feature Services. The approach chosen relies on additional query parameters that indicate that the operation shall be executed in an asynchronous mode. The approach has been re-used and extended from Testbed 12 (described in the previous ER on Asynchronous Service Responses (OGC 16-023r3)). Furthermore, as outlined in Section 8 additional operations may be defined to retrieve the status of the operation execution. The operations defined, e.g. GetStatus, are the same as for the OGC WPS and may also be used for other OGC services. A change request for OWS common has already been submitted in the last testbed. The extensions and applications provided in this testbed should be considered when working on this change request. We therefore added a comment to the existing change request.

The approach for asynchronous messaging described in this engineering report relies on additional request parameters. However, the OGC PubSub standard also defines a general way to add asynchronous messaging to OGC Web Services. So far, a PubSub extension for WFS has not yet been explored. We, therefore, recommend to also apply the OGC PubSub standard to Web Feature Services for asynchronous delivery of features. A PubSub extension for the purpose of Geosynchronization has been defined as a result of the Geosynchronization Service activities (see OGC Geo-Synchronization Standard (OGC 17-031)). This may serve as a basis for a general Pub-Sub extension for WFS.

As demonstrated for Asynch WFS-1 (Section 7), JSON may be a simpler and more efficient encoding than GML responses from Web Feature Services. A study of using JSON in OGC services has already been conducted in Testbed 12 (see Testbed-12 Javascript-JSON-JSON-LD Engineering Report). Binary encodings, supported by frameworks such as Google Protocol Buffers (Protobuf) or Apache Avro may even further reduce the amount of data that needs to be transferred. The usage of Protobuf and Avro for serialization of spatial data has been evaluated in Testbed 13. The results are described in OGC Testbed-13: SWAP Engineering Report (OGC 17-037).

To indicate additional response formats served by a WFS, the responseFormat parameter can already be used in a GetFeature request. However, best practices/extensions on how to add additional encodings for responses from WFS and how to use and evaluate them for the use cases described in this ER are considered as future work.