CM4D Results Transport is the collection of services that transfer data from one or more CM4D databases into the Results database. The Results database is then used by web applications, such as CM4D Peruse or third-party BI tools, for reporting and analysis.
The CM4D Results transport system uses NServiceBus for messaging between applications, databases and services. NServiceBus is a high-performance messaging system that provides the deployment and scaling options needed for large data throughput as well as supporting smaller installations.
Deployment of Services
The Results services are typically installed on the same machine as CM4D Server, but the Results services are a separate installer from CM4D Server to allow one or more of the services to be distributed to additional servers. To support scalability for larger organizations the services can run on a single computer or multiple computers, and multiple instances of each service can be added and distributed throughout your network. The Results services can be deployed as Windows services (default install option) or as docker images.
Only ONE instance of each of the three Results Services is supported per server.
The Results Loader service can work without any connection to the Site database (e.g. in the cloud connected via a RabbitMQ queue). The connection is entered during the installation of the service.
The Results Sender service includes configuration options for specifying Managed DataSources and processing options.
Service Configuration
The Results services make use of the .NET 6 configuration concept.
Visit the Microsoft webpage for background information on .NET configurations.
The services are primarily configured during the CM4D Results Services installation. Following installation, these settings are also available via the appsettings.json file in the corresponding service directory. The configuration in the file is overwritten by information from other configuration providers that define certain key/value pairs. This is mainly relevant for docker installations.
The provider options are:
| Provider | Description | Example | |
| File | The primary configuration provider uses the appsettings.json file in the corresponding service directory. | ||
| Environment Variable Configuration | Hierarchy levels are separated by a double underscore. |
{ “section1”: { |
|
| In an appsettings.json file this would be: |
set section1__section2__key=”value” |
||
| Command Line | Key/value pairs can be passed via the command line in three ways: | ||
| 1. with a -- and =: |
--section1__section2__key=”value” |
||
| 2. with a -- and a space: |
--section1__section2__key ”value” |
||
| 3. with a / and a space: |
/section1_section2__key “value” |
||
| CM4D.4ds | A CM4D.4ds provider can set the following two parameters from the CM4D Site database. The CM4D.4ds file is located via the registry key HKEY_LOCALMACHINE\Software\Applied Tech Systems\CM4D (Windows machines only) or the location of the service assembly. The file path can be set explicitly via the configuration key “CM4D4DSFilePath” in the appsettings.json file. |
{ “ConnectionStrings”: { |
|
See also, Results Services.
Connection Strings
The connections strings for the CM4D Site Database and CM4D Results Database are the most important settings as they allow the services to connect to one another.
- Results Sender Service - Requires the database connection string for NServiceBus transport/persistence and the CM4D Site database connection string.
- Results Loader/StreamCacheSmith Services - Require the database connection string for NServiceBus transport/persistence and the CM4D Results database connection string.
If CM4D is installed on the same machine, the database connections are typically set via the CM4D.4ds file. However, this is not the case for Docker containers where the settings are typically set via environment variables. The Results Loader service can work without any connection to the CM4D Site database (e.g. in the cloud connected via a RabbitMQ queue) if the CM4DDSFilePath parameter it is explicitly set in the appsettings.json or via an environment variable.
Configure the same database for the ServiceControl configuration you are using.
Transport Options
The NServiceBus transport is the way the messages are sent from one node to another. The system can be a distributed deployment, such as a local CM4D Site database and a cloud-based Results database. There are multiple transport options available. The transport is configured by selecting the Transport option during the Results service installation. Set the same transport for the ServiceControl configuration you are using.
Site/SQL Database
The SQL transport uses database tables to implement the message queues that connect the different services. This service is simple to configure as it can use the SQL Server that is used for CM4D databases. However, it does not provide the same throughput and scalability as other transport options.
The CM4D Results Services installation uses the SQL transport as the default configuration. Replace the empty transport connection string with the CM4D Site database connection string. The database connection string is configured in the appsettings.json of each component.
"Transport": {
“Name": "SQL",
"ConnectionString": "Data Source=SERVER;Initial Catalog=DATABASE;User ID=USER;
Password=PASSWORD; TrustServerCertificate=True;MultipleActiveResultSets = True"
}
Configure the same database for the ServiceControl configuration you are using.
RabbitMQ
RabbitMQ is a high-performance open-source message broker with wide industry usage. RabbitMQ can be installed on-premises on a server, as a container or used as a service at service providers like CloudAQMP. The free plan is sufficient to test this transport. A RabbitMQ connection string requires a host, username, password and a virtual host. RabbitMQ has a message size limit of 2GB.
The transport is configured by setting the Name to RabbitMQ and by setting the connection string.
"Transport": {
“Name": "RabbitMQ",
"ConnectionString": “host=SERVER;username=USER;password=PASSWORD;VirtualHost=VHOST"
}
Configure the same transport for the ServiceControl configuration you are using.
Azure Storage Queue
Azure storage queues are a transport option for cloud-based or hybrid deployments. This option requires an Azure Storage Account. The connection string can be copied from the Azure storage account access key page. Azure has a message size limit of 64KB.
The transport is configured by setting the Name to AzureStorageQueues and by setting the connection string.
"Transport": {
“Name": "AzureStorageQueues",
"ConnectionString": “DefaultEndpointsProtocol=https;AccountName=ACCOUNTNAME;AccountKey=ACCOUNTKEY"
}
Persistency Options
Information about messages might need to be stored to ensure transactional consistency of long-running transactions. The persistency configuration defines where the data is stored. It is recommended to store it in the CM4D Site database, but another database can be configured if necessary. Additional persistency options will be available in future releases.
SQL
The CM4D installer sets SQL as the default configuration. Replace the empty persistence connection string with the CM4D Site database connection string.
"Persitence": {
“Name": "SQL",
"ConnectionString": "Data Source=SERVER;Initial Catalog=DATABASE;User ID=USER;
Password=PASSWORD; TrustServerCertificate=True;MultipleActiveResultSets = True"
}
The database connection string is configured in the appsettings.json file for each installed service.
ServiceControl
NServiceBus (used by CM4D Results and EventSmith services) provides powerful error handling and monitoring tools:
- ServiceControl - Collects data from endpoints (CM4D Services).
- ServicePulse - A web-based application to inspect failed messages and resubmit them and to monitor the availability and performance of endpoints.
- ServiceInsight - Supports auditing of message details but has a major performance impact.
At a minimum, it is strongly recommended to use ServiceControl and ServicePulse. You need to configure at least one instance of ServiceControl and Auditing and one Monitoring instance for the transport method chosen during the installation of the Results service.
The downloads and installation instructions for the NServiceBus applications can be found on the Particular Software website.
ServiceControl Queues/Subscription Tables
During installation of the CM4D Results services, three Queues are set up for ServiceControl monitoring.
- Error Queue (default: error)
- Monitoring Queue (default: Particular.Monitoring)
- Service Control Queue (default: Particular.ServiceControl)
ServiceControl queues configured during Results services installation.
These Queue labels must match what is been created by ServiceControl for the selected transport option:
- SQL Server - Database Tables
- RabbitMQ - Queues in RabbitMQ
- Azure Storage Queues - Queues on the storage account for Azure Storage Accounts
Queues for the Transport option Azure Storage Queue cannot use labels that include capital letters or the decimal/period (.) symbol. For example, replace the default 'Particular.Monitoring' with 'particular-monitoring'. Refer the Microsoft documentation for technical details on Azure Storage Queue Names.
Following installation, you can find the values in the appsettings.json file for the corresponding service.
Queues in the appsettings.json file for a Results service (default path: .\Applied Tech Systems\ATS CM4D Results Services\).
Configure the connection strings entered when creating new monitoring instances in ServiceControl Manager to match the queue labels provided to the Results services.
Transport Queues in ServiceControl instance Transport Configuration, defined as a Subscription Table.
The ServiceControl queues may be configured in the appsettings.json for each service.
- ErrorQueue, ServiceControlQueue and MonitoringQueue must match the settings in the service control instance created.
- Leave the AuditQueue empty unless asked to enable it by support.
"ServiceControl": {
"AuditQueue": "",
"ErrorQueue": "Particular.Error",
"ServiceControlQueue": "Particular.ServiceControl",
"MonitoringQueue": "Particular.Monitoring"
},
Queues for the Transport option Azure Storage Queue cannot use labels that include capital letters or the decimal/period (.) symbol. For example, replace the default 'Particular.Monitoring' with 'particular-monitoring'. Refer the Microsoft documentation for technical details on Azure Storage Queue Names.
Logging
Logging of information is important for troubleshooting of services. The following logging options are available:
- Event: Common for windows-based systems. However, it is not recommended to log more debug or trace information to the event log.
- Console: The default for Docker-based systems.
- File: Allows more detailed logging into rolling files.
- Application Insight: A Microsoft Azure technology that can be used for logging and monitoring.
"Serilog": {
"Using": [ "Serilog.Sinks.Console" ],
"MinimumLevel": "Information",
"EnableAppInsight": false,
"InsightKey": "",
"EventLogRestrictedToMinimumLevel": "Warning",
"EventLogOutputTemplate": "{Message}{NewLine}{Exception}",
"EventLogManageEventSource": false,
"WriteTo": [
{ "Name": "Console" },
{
"Name": "File",
"Args": {
"path": ".\\Logs\\ResultSenderLog_.txt",
"outputTemplate": "{Timestamp:o} [{Level:u3}] ({Application}/{MachineName}) {Message}{NewLine}{Exception}",
"rollingInterval": "Day",
"fileSizeLimitBytes": "2097152",
"rollOnFileSizeLimit": true,
"retainedFileCountLimit": 2000
}
}
],
"Enrich": [ "FromLogContext", "WithMachineName", "WithThreadId" ],
"Properties": {
"Application": "ATS.CM4D.ResultsSender",
"MachineName": "MachineATS"
}
}
- MinimumLevel: Set the level that is sent to the log. The available options are:
- Critical
- Error
- Warning
- Information
- Debug
- Trace
- None
- EnableAppInsight: Set to "True" to use Application Insights
- InsightKey: The key of an Application Insights Instance
- EventLogRestictedToMinimumLevel: Set the minimum level for event log logging
- EventLogManagedEventSource: Set to "True" for the service to register the event source. This requires local admin rights for the user running the services.