Unable to Retrieve List of OPC Servers During OPC DA Channel Configuration
If you receive the message "Failed to get a list of OPC Servers for host…" while configuring an OPC DA channel then the ATS Bus Browsing service is not running on the system that hosts the OPC DA Server.
Certificate related issues when working with OPC UA channels
When working with OPC UA Channels, ensure that the certificate thumbprints match the ones configured in the ATS Bus DataService and OT bus stop application configuration. Also make sure that the OPC UA Server has 2 or more ATS Bus OPC UA Client entries in the trusted clients tab; one for the data service and another one for the bus stop.
OPC Channel throws BadTooManySubscriptions error when connected to OPC UA server
Each OPC UA channel message that has its acquisition type set to OnChange and each OPC UA channel message handshake result in an OPC UA subscription. OPC UA servers can configure the number of subscriptions they can handle. The ATS Bus logging will show an error containing Opc.Ua.ServiceResultException: BadTooManySubscriptions at … if the number of allowed subscriptions configured by OPC UA server is too low. This issue can be resolved by:
Allowing the OPC UA server to support more subscriptions, this is a change in the OPC UA server configuration
Reconfigure ATS Bus and add one or more extra channels and divide the channel messages across them. Each channel results in a new OPC UA session.
It is known that the WinCC OPC UA server is configured to support 100 subscriptions for all sessions. In this case, it does not make sense to reconfigure ATS Bus (dividing the channel messages over extra channels).
More information: Siemens FAQ 109755860
The bus stop won’t start due to MSMQ Errors
The Windows Logs\Applications or Applications and Services logs\Applied Tech Systems event log may show 3 types of error messages:
System.Messaging.MessageQueueException: Insufficient resources to perform operation.
The event log might show an error containing System.Messaging.MessageQueueException: Insufficient resources to perform operation. when sending a message to another bus stop, ServiceControl or the monitoring instance. This message indicates that MSMQ reached its maximum capacity because one of the queues might contain a lot of unhandled messages. The following list shows the resolutions per queue:
Bus stop queue: a bus stop did not start and did not consume messages from its queue. Start the bus stop or review the event log to figure out why the bus stop stopped.
Particular.servicecontrol: the ServiceControl service is stopped. Start ServiceControl so that it consumes messages from the queue. Review the Windows Logs/Applications event log if ServiceControl does not start, it will show a clear error message.
Particular.monitoring: The monitoring instance is stopped. Start the monitoring instance. Review the Windows Logs/Applications event log if the monitoring instance does not start, it will show a clear error message.
Dead letter / transactional dead letter messages: A bus stop did not consume the messages within the given amount of time. The class type will how the exact cause of why a message ended up in this queue. Further analysis of the event log is required. This queue should be purged to start up MSMQ again.
Outgoing queues: Review the state and connection history to determine why messages are queued here. Most likely a TCPIP/hostname issue is the cause of the problem. Reconfiguring ATS Bus with the proper hostname or IP address will solve the issue.
Audit.log: This queue is used by audit logging; this feature should have been disabled via the ServiceControl management application because there is no application configured to read this queue.
Error.log: This queue is used by error logging; this feature should have been disabled via the ServiceControl management application because there is no application configured to read this queue.
System.Messaging.MessageQueueException: Message Queue service is not available.
This message states that the Message Queueing service is not running. Go to the services and start the Message queueing service.
NServiceBus.Unicast.Queuing.QueueNotFoundException: Failed to subscribe.
This error message indicates that the destination queue is not available, and this is the case when the destination bus stop did not create the queue yet. The bus stop will not start because sending the subscription message is essential. Starting the destination bus stop will resolve the issue.
The bus stop won’t start due to SQL Errors
There are situations that the bus stop does not want to start because of a fatal error. One of these errors is:
System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) ---> System.ComponentModel.Win32Exception (0x80004005): The system cannot find the file specified
The message above shows a SQL Server specific error code, 40. This error may occur when:
The user provided invalid connection details for the IT Database channel.
The user provided an invalid connection string for the NServiceBus persistence.
The SQL Server instance is not running, make sure the service is started.
A firewall blocks traffic to the SQL server.
SQL server does not allow TCP/IP connections, please go to Computer Management, Service and Application, SQL Server xxxx Configuration, Network Configuration and enable the TCP/IP protocol. Also ensure that the SQL SERVER port is set to the default 1433.
The bus stop or even the data service might show other error codes too. In these events, visit Google and search for “SQL server error <xx>” where ‘<xx>’ is the error code. The search results will provide an answer to the problem.
Error Message Due to Firewall Settings
A wide variety of error messages may occur when the firewall is not configured properly. Make sure that the following ports are configured in the firewall:
8700 (ATS Bus data service default port)
8708 (Browsing service)
8800 (OT bus stop monitoring port)
8802 (IT bus stop monitoring port)
8804 (Orderhandling bus stop monitoring port)
8806 (Archive bus stop monitoring port)
4840 (OPC UA Local discovery service)
49320 (Default OPC UA Endpoint)
Service Dependencies
The bus stops and ServiceControl depend on other services to start properly. The bus stops have these dependencies configured already but ServiceControl does not. The bus stops require:
MSMQ
Particular ServiceControl requires:
MSMQ
Incorrect OrderId, LotId, SerialNumber or EquipmentId Used In Data Type Scope
When converting the channel message to a shopfloor message, the OT bus stop copies some tag values into a section named DataScopes which is used to implement the Data Type Scopes functionality. It allows one instance of OrderId, LotId, SerialNumber or EquipmentId.
A message definition on the other hand allows multiple message fields within the same message definition to be named 'OrderId', 'LotId', 'SerialNumber' or 'EquipmentId' and this may cause problems. The code that converts the channel message to a shopfloor message will only take the first instance of a message field named OrderId, LotId, SerialNumber or EquipmentId it encounters.
Multiple entries of 'OrderId', 'LotId', 'SerialNumber' or 'EquipmentId' are caused by the fact that there are 2 different user input validation engines in the OT bus stop. The first one checks if a non-indexed message field name has a duplicate and the second engine validates if an indexed message field name has a duplicate with the same index.
For example, a message definition may contain multiple instances of SerialNumber:
And a channel message may refer to this message definition:
The order in which the message fields are processed is not guaranteed so the ‘wrong’ tag value may be copied into the DataScopes section of the shopfloor message and this causes the bus stop to use the ‘wrong’ value to be used to select a work order for instance. This issue only affects work order related channel message action i.e. UpdateBatch.
Best practice is to use the ‘reserved’ message field names carefully. Make sure that there is only one instance of the name 'OrderId', 'LotId', 'SerialNumber' or 'EquipmentId'.
Error during Installation
When installing a Bus Stop module a dialog box similar to the following may be displayed stating: 'ATS Bus XXXXX' could not be installed. Verify that you have sufficient privileges to install system services. This may happen when the services.msc, compmgmt.msc (Computer Management) or Task Manager is open.
Ensure that these applications are closed and press the Retry button.
The ServiceControl database becomes too large
ServiceControl is used to read audit and error messages from a queue and store them into a database. The size of the database depends on the number of messages and the size of each message that are stored within the retention times. The error and audit retention are configurable and can be changed using the ServiceControl management application. By default, ServiceControl is setup using a retention time of 30 days for audit messages and 15 days for error messages.
It is always possible to lower the audit and error retention but the size of the ServiceControl database is not reduced automatically, this requires several actions to be executed and the time required to execute them depends on the database size. The actions are executed in 2 stages:
Stage 1: Lower the audit and error retention times in the ServiceControl management tool:
Open the ServiceControl management application and open the configuration dialog of the ‘Particular.ServiceControl’ instance.
Navigate to the Audit retention and set the value of days to 5.
Navigate to the Error retention and set the value of days to 10.
Save the changes. This will restart the ‘Particular.ServiceControl’ instance.
Go to ServiceInsight and wait till the last audit message (the one with the green dot in front) is dated 5 days ago. ServiceControl will clean up the database and discard messages older than 5 days once it is restarted. ServiceControl is not done cleaning up if the last message is older than 5 days, you need to wait with the remaining actions if the clean-up is not completed. I don’t know how long this takes. You cannot get the maximum compression if you don’t let the clean-up finish.
Stage 2: Backup and Compress/Defragment the ServiceControl database.
Once the clean-up is finished, stop the ‘Particular.ServiceControl’ service via the services dialog or via the ServiceControl management tool like we did this afternoon.
Go to the ServiceControl data directory (C:\ProgramData\Particular\ServiceControl\Particular.ServiceControl) and create a backup of the directory named ‘DB’. I don’t know how long it takes to create a backup of the DB directory; this depends per system.
You can copy the complete directory to another drive or directory, or you can create an archive using the ‘Send to à Compressed (zipped) folder’ option via the context menu. Creating an archive is the preferred method because this compresses the data resulting in a smaller copy.
Once the backup is completed, go into the DB directory, you should now be in ‘C:\ProgramData\Particular\ServiceControl\Particular.ServiceControl\DB’.
Execute the following command: esentutl /r RVN /l logs /s system and wait for it to complete. This may take a few seconds. Make sure the Particular.ServiceControl service is still stopped.
Execute the following command when the previous command completed successfully: esentutl /d Data
Restart the ‘Particular.ServiceControl’ services via the services dialog or via the ServiceControl management tool.
Knowledge base article a summary of the preferred procedure given at: https://docs.particular.net/servicecontrol/db-compaction
Creating Bus.Contracts.DerivedContracts failed
A bus stop does not receive subscribed bus messages when they are published on the bus and the event log shows: “Creating Bus.Contracts.DerivedContracts failed…” or "No handler for message xxx".
On start, a bus stop generates an assembly named ‘Bus.Contracts.DerivedContracts.dll’ in a directory named ‘C:\ProgramData\Applied Tech Systems\ATS Bus\<BUS_STOP_NAME>’. This assembly is then loaded into memory. This assembly is used when subscribing to and publishing bus messages.
Sometimes it may happen that a bus stop does not properly stop, and this prevents the assembly from being unloaded from memory. This will show during the start of the bus stop, the following error is logged in the event log:
2020-02-10 15:03:44.643 | C:\Program Files (x86)\ATS Global B.V\ATS Bus\Bus Stops\OTBusStop\Bus.Busstop.OT.exe 2.6.20034.1 | ERROR [5] - Creating Bus.Contracts.DerivedContracts failed: (0,0: error CS0016: Could not write to output file 'c:\ProgramData\Applied Tech Systems\ATS Bus\OTBusStop\Bus.Contracts.DerivedContracts.dll' -- 'The process cannot access the file because it is being used by another process. '
Execute the following steps to resolve the issue:
Open the Windows task manager and find the Bus.BusStop.<xx>.exe file that represents the bus stop. The value <xx> can be: OT, IT, Cloud or ADOS. The error message in the event log should provide the process that caused the error.
Terminate the Bus.Busstop.<xx>.exe process.
Open the process explorer (by SysInternals) and find open file handles to the file ‘C:\ProgramData\Applied Tech Systems\ATS Bus\<BUS_STOP_NAME>\ Bus.Contracts.DerivedContracts.dll’. Stopping the bus stop should release the file handles to ‘Bus.Contracts.DerivedContracts.dll’.
Open the windows services application (by executing services.msc from the command line) and start the bus stop when there are no open file handles to ‘Bus.Contracts.DerivedContracts.dll’
The issue should be resolved and a new ‘Bus.Contracts.DerivedContracts.dll’ should be created when the bus stop starts.
SQL Server performance
ATS Bus uses SQL Server for SQL Persistence or for its database channels. The event viewer may show time-out related error messages. These errors often occur when SQL Express is used in conjunction with OT Bus stops that are under a very high load. The following items should be reviewed:
Check if the Auto Close property is set to ‘True’. This can be checked by going to the Options page in the database properties using SQL Management Studio. Set this option to false. The following SQL statement does the same:
USE [master]
GO
ALTER DATABASE [DatabaseCausingTimeouts] SET AUTO_CLOSE OFF;
GO
Check if the Recovery model property is set to ‘Simple’. This can be checked by going to the Options page in the database properties using SQL Management Studio. The recovery option can be selected at the top of the dialog. Set this option to ‘Simple’. The following SQL statement does the same:
USE [master]
GO
ALTER DATABASE [DatabaseCausingTimeouts] SET RECOVERY SIMPLE;
GO
Although ATS Bus supports SQL Express, it comes with a lot of limitations making it not suitable for high load production environments.
Webservice Server channel fails to start
On startup the channel may fail to start. This may be due to it attempting to register unnecessary settings for the Windows HTTP protocol when linking the certificate (thumbprint of certificate) with the IP address and Port.
This can fail due to insufficient rights assigned to the user running the ATS Bus Stop service.
A network administrator will be required to execute the following script:
cmd> netsh http add sslcert ipport=0.0.0.0:9999 appid={ED3CE4E1-E134-49FC-AA7F-E4E1D142B0D4} certhash=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
In the example above, 9999 refers to the configured port and x refers to the thumbprint of the installed certificate, which can be found in the Certificates snap-in in the Microsoft Management Console or by running certlm.msc.
The bus stop does not restart due to a corrupted user.config file
A bus stop may fail to restart when the user.config file is corrupted. Upon (re)start, the following dialog is shown:
The message itself does not give that much information but the Application section in the Windows Logs in the Windows Event Viewer should show a .NET Runtime (source column) error:
Application: Bus.Busstop.IT.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Xml.XmlException
at System.Xml.XmlTextReaderImpl.Throw(System.Exception)
…
at System.Configuration.BaseConfigurationRecord.InitConfigFromFile()
Exception Info: System.Configuration.ConfigurationErrorsException
at System.Configuration.ConfigurationSchemaErrors.ThrowIfErrors(Boolean)
…
at System.Configuration.ClientConfigurationSystem.OnConfigRemoved(System.Object, System.Configuration.Internal.InternalConfigEventArgs)
Exception Info: System.Configuration.ConfigurationErrorsException
at System.Configuration.ClientConfigurationSystem.OnConfigRemoved(System.Object, System.Configuration.Internal.InternalConfigEventArgs)
at System.Configuration.Internal.InternalConfigRoot.OnConfigRemoved(System.Configuration.Internal.InternalConfigEventArgs)
…
at System.Configuration.ApplicationSettingsBase.get_Item(System.String)
at Bus.Busstop.IT.Properties.Settings.get_BusStopName()
at Bus.Busstop.IT.Program.Main()
The error message shown in the event log indicates that the user.config file is corrupted. The location of this file depends on which user is executing the bus stop service. By default, the LOCAL SYSTEM account runs the bus stop but the services dialog will show the account running the service in the Log On As column:
The user.config file is stored in the %LOCALAPPDATA%\{Vendor}\{Application}\{Version} folder. The NT AUTHORITY\SYSTEM (Local System) account uses C:\Windows\system32\config\systemprofile\AppData\Local\{Vendor}\{Program}\{Version} or C:\Windows\SysWOW64\config\systemprofile\AppData\Local\{Vendor}\{Program}\{Version} when running 32-bit application on a 64-bit system which is the case with current ATS Bus versions.
The bus stop will start again after the user.config file is deleted from the C:\Windows\SysWOW64\config\systemprofile\AppData\Local\ATS_Global_B.V\ Bus.Busstop.IT.exe_xxxxxxxxxxxx\{Version}\ directory.
Can't get global settings data from WebAPI REST endpoint https://localhost:8704 (HTTP Bad Request)
On startup, the bus stop may throw the following error:
ERROR [16] - Can't get global settings data from WebAPI REST endpoint https://localhost:8704. See details:
The web service returned 'HTTP BadRequest (Bad Request)': URL = https://localhost:8704/, Parameters = /api/v1/settings/globalsettings, ResponseContent = <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"http://www.w3.org/TR/html4/strict.dtd>
<HTML><HEAD><TITLE>Bad Request</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Bad Request - Invalid Hostname</h2>
<hr><p>HTTP Error 400. The request hostname is invalid.</p>
</BODY></HTML>
The error message shows that the bus stop tries to read the configuration from a URL that contains a localhost and the URL also contains the HTTPS scheme. The connection to the data service is one encrypted using a certificate and the ‘localhost’ is not configured in the SAN (subject alternative name) of the certificate. Replacing ‘localhost’ with the FQDN that is listed in the SAN of the certificate will solve the issue. The hostname in the highlighted section of the file named ‘bus.busstop.xx.exe.config’ should be updated where ‘xx’ equals be ADOS, Cloud, IT or OT:
<applicationSettings>
<Bus.Busstop.XX.Properties.Settings>
<setting name="BusStopName" serializeAs="String">
<value>xxxxxxx</value>
</setting>
<setting name="BusStopGUID" serializeAs="String">
<value>xxxxxxx</value>
</setting>
<setting name="ConfigurationEndpoint" serializeAs="String">
<value>https://localhost:8704</value>
</setting>
</Bus.Busstop.Ados.Properties.Settings>
</applicationSettings>
Unable to get security manager endpoint neither from data service nor from backup configuration
ATS Bus 2.8 has the license manager and security manager endpoints configured in the global settings of the ATS Bus configuration database. This may raise the following error after a recent upgrade to ATS Bus 2.8:
ERROR [3] - Unable to get security manager endpoint neither from data service nor from backup configuration.
There may be another error in the event log like ‘Can't get global settings data from WebAPI REST endpoint https://localhost:8704’ stating why the bus stop cannot connect to the data service or the backup configuration in ‘C:\ProgramData\Applied Tech Systems\ATS Bus\<bus stop name>’ does contain an invalid URL to the security manager endpoint. The backup configuration should be renamed/removed and the bus stop should update the backup configuration file by connecting to the data service once.
Can't get data from WebAPI REST endpoint https://computername:port (HTTP Not Found)
On startup, the bus stop may throw the following error:
ERROR [X] - Can't get data from WebAPI REST endpoint https://computername:port. See details: The web service returned 'HTTP NotFound (Not Found)': URL = https://computername:port/, Parameters = /api/v1/configuration/it/ITBusStop, ResponseContent =
The error message shows that the configuration database does not have a bus stop named ‘ITBusStop’ configured.
Failed to retrieve the '<bus stop name>' bus stop configuration from the configuration endpoint :https://computername:port
On startup, the bus stop may throw the following error:
ERROR [X] Failed to retrieve the 'ITBusStop' bus stop configuration from the configuration endpoint :https://atsvminsp219.ats-global.local:8704
A bus stop named ‘ITBusStop’ is not configured in the ATS Bus configuration database. The following error is also shown in the logging:
ERROR [X] - Can't get data from WebAPI REST endpoint https://computername:port. See details: The web service returned 'HTTP NotFound (Not Found)': URL = https:// computername:port/, Parameters = /api/v1/configuration/it/ITBusStop, ResponseContent =
Can't get data from WebAPI REST endpoint https://computername:port (HTTP Not Found)
On startup, the bus stop may throw the following error:
FATAL [5] - The service is shutting down due to the following critical error: Failed to start the bus.
Exception details:
Raven.Abstractions.Connection.ErrorResponseException: An error occurred while sending the request. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:8084
at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Raven.Client.Connection.Implementation.HttpJsonRequest.<>c__DisplayClass36_0.<<SendRequestInternal>b__0>d.MoveNext()
--- End of inner exception stack trace ---
at Raven.Client.Connection.Implementation.HttpJsonRequest.<>c__DisplayClass36_0.<<SendRequestInternal>b__0>d.MoveNext()
The error message contains ‘Raven.Client.Connection’ which indicates that RavenDB is used as persister. RavenDB has a REST endpoint and the error ‘No connection could be made because the target machine actively refused it 127.0.0.1:8084’ indicates that RavenDB is not running. Starting the RavenDB service will resolve the issue.
Relink message fields - Correct errors introduced during the update of exported channel message configurations
On occasion, configuration errors are introduced during the process of updating channel message configurations outside of Cockpit.
This is usually as a result of the message field data type property, present in the message definition, being omitted from the acquisition tag configuration as a MessageFieldDataType property.
To resolve this issue, go to the OT message definition configuration in Cockpit, select a channel message and press the Relink button or press the Relink all button without selecting a message definition. The button executes a process that reviews all OT channel messages that refer to the selected message definition and ensures that the properties related to that message definition are correctly defined. The Relink all button does this for all message definitions.
The Relink button is able to solve most configuration issues, however, it will fail when an acquisition tag doesn't have a message field configured or it links to a non-existing message field. In this instance, the user is required to go to the channel message configuration and fix the specific acquisition tag configuration.
It is not recommended to update channel message configurations outside of cockpit due to the possibility of introducing configuration errors.
For further information on managing message definitions, please click the link. 
Error on starting up ATS Bus Dataservice 'the service did not respond to the start or control request in a timely fashion' (1053)
An error may occur upon starting the ATS Bus Dataservice with the following message:
'The service did not respond to the start or control request in a timely fashion (1053)'.
This can be caused by the request exceeding the timeout period for service responses from Windows, which is set by default to 30 seconds. If the service is not responding within the defined time, the process will be canceled and the service will not start.
The timeout period can be increased by creating a registry entry as follows:
Locate and open regedit.exe to load the Registry Editor.
Locate and click the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control
Right click Control, point to New and then click DWORD Value.
In the New Value #1 box, type ServicesPipeTimeout and then press enter.
Right click ServicesPipeTimeout, and then click Modify.
Click Decimal, type the number of milliseconds that you want to wait until the service times out and then click OK.
For example, to wait 60 seconds before the service times out, type 60000.
Quit the Registry Editor and restart the computer.
Timeseries database channel does not write measurement and raises ‘At least one field must be specified’ error message.
The Influx database currently supported by the time series database channel may throw the following error message:
“System.ArgumentException: At least one field must be specified at …”.
This message indicates that the TimeSeries channel message is trying to write an empty or ‘null’ tag value to the Influx measurement. The ‘null’ or empty tag value may be caused by a configuration error. Please use ServiceInsight to check that all tag values are populated along with the ATS Bus configuration for possible configuration errors.
Bus stop does not start, it shows the following fatal error ‘System.Net.Sockets.SocketException: An attempt was made to access a socket in a way forbidden by its access permissions’
The bus stop refuses to start and the logfile shows the following fatal error message:
yyyy-mm-dd hh:mm:ss.sss | C:\Program Files (x86)\ATS Global B.V\ATS Bus\Bus Stops\OTBusStop\Bus.Busstop.OT.exe 2.x.yyyyy.z | FATAL [22] - The service is shutting down due to the following critical error: Failed to start the bus.
Exception details:
System.ServiceModel.CommunicationException: A TCP error (10013: An attempt was made to access a socket in a way forbidden by its access permissions) occurred while listening on IP Endpoint=0.0.0.0:8800. ---> System.Net.Sockets.SocketException: An attempt was made to access a socket in a way forbidden by its access permissions
at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Bind(EndPoint localEP)
at System.ServiceModel.Channels.SocketConnectionListener.Listen()
--- End of inner exception stack trace ---
To resolve this, first execute the following command on the command prompt (where 8800 is the port number listed in the IP endpoint in the error message):
netstat -ona |findstr 8800
This may return a result similar to the below:
TCP 0.0.0.0:8800 0.0.0.0:0 LISTENING 4
TCP [::]:8800 [::]:0 LISTENING 4
The last digit is the PID of the process binding to port 8800, in this case 4 which is the Windows System process.
This indicates that there is another service already binding to port 8800.
If ATS Inspect is installed on the same system, please refer to Default Service Settings for Inspect and ensure that ATS Bus is not using one of the configured ports specified. In the above example, port 8800 is used for the ATS Inspect CAD Report Service. ATS Bus should therefore be configured to use a different port.
Inspect Checklist question does not seem to be published on the bus
An ADOS bus stop has an ADOS Inspect channel configured which processes Inspect Checklist events. These incoming messages will appear as ‘ProcessInspectEvent’ messages in ServiceInsight. The ADOS bus stop has a channel message action named ‘PublishAsWorkPerformance’ configured that publishes the Checklist question as ProcesWorkPerformance on the bus.
The user would expect that each ProcessInspectEvent message is followed by a ProcessWorkPerformance but the screen capture below shows 2 individual ProcessInspectEvent messages and one of them is followed by a ProcessWorkPerformance.
The situation above may indicate one of the following issues:
The receiving bus stop may have a message handler configured to handle the incoming bus messages but a condition on the incoming bus message handler may instruct the bus stop to ignore the message. Check if the handler has a condition configured and check if any of the data sent matches the condition.
The receiving bus stop is not started, all messages are queued on the bus stop transport queue. In this case, none of the ProcessWorkPerformance messages will show. Check if the bus stop is started.
Another bus stop with the same name but on a different computer may have subscribed to the same message but it has been stopped. This situation is only valid for bus stops that have MSMQ configured as transport. In this situation, the sending bus stop will publish the first message to the first computer and the second message to the second one. Messages will queue on the second bus stop since it hasn't started. Go to the persister (RavenDB, SQL) and check the subscriptions table for multiple entries for the same bus stop.
The receiving bus stop may have switched off audit logging in the NServiceBus tab of the bus stop. Open ATS Bus Cockpit and go to the NServiceBus tab in the bus stop configuration and check if auditing is enabled.
It is important to understand that the ProcessWorkPerformance is only shown in ServiceInsight when the subscribed bus stop handled it. The message handler will send the incoming message to the audit queue once processed.
The bus stop stops and an error message shows that the max pool size was reached
An ATS Bus bus stop may throw the following error:
2023-01-13 14:18:43.357 | C:\Program Files (x86)\ATS Global B.V\ATS Bus\Bus Stops\MYBUSSTOP\Bus.Busstop.OT.exe 2.8.22252.1 | ERROR [31] - An error occurred while publishing a ShopfloorEvent on the bus while processing a bus stop event: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and the max pool size was reached.
Exception details:
System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
at System.Data.Common.ADP.ExceptionWithStackTrace(Exception e)
--- End of stack trace from the previous location where exception was thrown ---
The cause of the problem is that the SQL connection pool of 100 connections is exhausted, there are no more connections available to the SQL database.
Possible solutions:
The ATS Bus Persistence or ATS Bus Transport database is slow because the recovery policies are not properly set. Use the following SQL scripts to set the recovery policy to Simple.
ALTER DATABASE AtsBusPersistence SET RECOVERY SIMPLE
ALTER DATABASE AtsBusTransport SET RECOVERY SIMPLE
GO
The poolsize for each connectionstring is set to 100 connections. 100 connections does not mean 100 bus messages but 50 because the other 50 are used to transport audit messages. The connection pool size can be increased to cover for a message burst. Updating the connection pool size is done by modifying the ‘Max Pool Size’ argument in the connection string. (See screenshot below for an example).
Whilst this is not a solution; adding the application name to the connection string is advised as the application name is shown with the queries in the SQL profiler. It is also advised to name the applications AtsBusTransport and AtsBusPersitence in ATS bus Cockpit. Finally, revisit the connection strings used in ServiceControl, name the connections ServiceControl, ServiceControlAudit and Monitoring to be able to identify ServiceControl and Monitoring instances.
To monitor connections open to the concrete database set “Application Name” keyword in the connection string for better monitoring. (See screenshot below for an example)
See connections to concrete database using the following script, just change the database name (<DATABASE>):
select rtrim(p.program_name) as ProgramName, count(*) as ConnectionCount
from sys.sysprocesses p with (readpast)
where p.dbid > 0 and rtrim(db_name(p.dbid)) = '<DATABASE>'
group by p.program_name
order by p.program_name
It is important to monitor the number of connections and investigate why they take so much time to execute or close.
Error: System.ArgumentException: Format of the initialization string does not conform to specification starting at index 70
The log file shows:
2023-01-20 21:01:56.176 | C:\Program Files (x86)\ATS Global B.V\ATS Bus\Bus Stops\MYBUSSTOP\Bus.Busstop.OT.exe 2.8.22252.1 | ERROR [5] - Exception when starting bus stop. Exception details: System.ArgumentException: Format of the initialization string does not conform to specification starting at index 70. at System.Data.Common.DbConnectionOptions.GetKeyValuePair(String connectionString, Int32 currentPosition, StringBuilder buffer, Boolean useOdbcRules, String& keyname, String& keyvalue) at System.Data.Common.DbConnectionOptions.ParseInternal(Hashtable parsetable, String connectionString, Boolean buildChain, Hashtable synonyms, Boolean firstKey) at System.Data.Common.DbConnectionOptions..ctor(String connectionString, Hashtable synonyms, Boolean useOdbcRules) at System.Data.Common.DbConnectionStringBuilder.set_ConnectionString(String value) at NServiceBus.SqlServerTransport.
The error message above shows 2 things:
A format error at position 70 in the connection string
‘at NServiceBus.SqlServerTransport.’ which indicates the error is in the ATS Bus transport connection string.
To solve the issue, open the NServiceBus configuration and identify the ATS Bus Transport connection string and locate position 70. The connection string may only contain valid ASCII characters and a possible problem is that non-ASCII characters are included in the connection string.
Another option is to open C:\ProgramData\Applied Tech Systems\ATS Bus\MYBUSSTOP\MYBUSSTOP_BusStopConfig.xml or C:\ProgramData\ATS Global\ATS Bus\MYBUSSTOP\MYBUSSTOP_BusStopConfig.xml. This file will show us the latest connection string used by the bus stop. The red underlined equals sign is a non-ASCII character.
Unable to login to the Configuration manager, it shows “The application cannot authenticate because it cannot connect to the ATS Security Manager site…”
After installing the configuration and security manager, the following message might be displayed: “The application cannot authenticate because it cannot connect to the ATS Security Manager site…”.
Open the Services application, locate the ATS Security Manager service and start it. It is most likely stopped.