Remote subscriber data definition

This article describes how the Discovery Data Service makes data available for Remote Subscriber Databases (RSDs) and how the DDS Remote Filer application interacts with this to update an RSD. Although it is recommended that the Remote Filer application be used for updating RSDs, any other technical solution can be substituted provided it is able to match the Remote Filer behaviour as far as the points of interaction are concerned.

Subscriber database schema

DDS currently provides SQL scripts for creating subscriber databases, one for each of the two support database engines; the database scripts are the same no matter what configuration options are selected when setting up the feed from DDS (e.g. PI versus de-identified).

• MySQL
• SQL Server (or compatible, e.g. Azure SQL DB)

Please note the following: The above are links to the public GitHub repository which was in active development until early 2021. Development has moved to private GitHub repositories since this date, so the SQL schemas linked to above should only be taken as illustrations of the schema and not the latest version (which can be provided on request). The Remote Filer application currently only supports loading data into these two database engines, and part of the data feed (for reference data) is sent as raw SQL and only these two formats are supported. DDS still supports data feeds to an older version of the subscriber database known as Compass v1 (the current version being v2). Although v1 is still supported, new instances of this will not be deployed and this article specifically addresses the v2 standard. Future developments/improvements to the DDS subscriber database will be iterative upgrades to v2.

Subscriber feeds

There are two separate feeds of data that DDS sends to each RSD:

• Published Data Feed – this includes all patient data, plus some supporting data (e.g. data about clinicians and organisations) that is sent into DDS by external publishers.

• Reference Data Feed – this includes lookups and mappings for clinical codes (e.g. Read2 to Snomed) that is not directly published into DDS but nonetheless needs to be updated in subscriber databases. Because of the semi-disconnected mechanism this data is made available to DDS, a separate feed to subscribers was created.

Data for each feed is staged in a separate directory on the DDS SFTP server for each subscriber. The DDS Remote Filer application runs supports running in two different modes, one to download and process the Published Data Feed and the other to download and process the Reference Data Feed.

To replace the Remote Filer application with an alternative solution requires supporting both these feeds.