In this Topic Hide
Unable to Retrieve List of OPC Servers During OPC DA Channel Configuration
Unable to Retrieve List of OPC Servers During OPC UA Channel Configuration
Certificate related issues when working with OPC UA channels
The bus stop won’t start due to MSMQ Errors
The bus stop won’t start due to SQL Errors
Error Message Due to Firewall Settings
Incorrect OrderId, LotId, SerialNumber or EquipmentId Used In Data Type Scope
The ServiceControl database becomes too large
Creating Bus.Contracts.DerivedContracts failed
Webservice Server channel fails to start
The bus stop does not restart due to a corrupted user.config file
If you receive the message "Failed to get a list of OPC Servers for host…" while configuring an OPC DA channel then the ATS Bus Browsing service is not running on the system that hosts the OPC DA Server.
When configuring an OPC UA Channel in the ATS Bus Cockpit the following message is displayed: "Failed to get list of OPC Servers for host '<SERVERNAME>'." The event log will show something like the following:
Opc.Ua.ServiceResultException: Thumbprint was explicitly specified in the configuration. Cannot generate a new certificate
This indicates that the OPC UA certificate thumbprint in the ATS Bus DataService application configuration does not match the one in the ATS Bus OPC UA Client certificate that can be found in the windows certificate store.
1. Open the Windows certification store. This can be found by searching for Certificates.
2. Open the UA Applications folder.
3. Double click the ATS Bus OPC UA Client certificate.
4. Select the Details tab.
5. Copy the Thumbprint.
There will be hidden characters present when copying the thumbprint so copy all but the first digit and remember the first digit.
6. Edit ATS.Bus.DataServiceHost.exe.config in the ATS Bus installation directory (C:\Program Files (x86)\Applied Tech Systems\ATS Bus\Services).
This is the ATS Bus DataService configuration file.
7. Find the ApplicationCertificateThumbprint element and replace the contents with what you just copied including the first digit you had to remember and remove the whitespaces.
8. Restart the ATS Bus DataService and configure the OPC UA channel again.
When working with OPC UA Channels, ensure that the certificate thumbprints match the ones configured in the ATS Bus DataService and OT bus stop application configuration. Also make sure that the OPC UA Server has 2 or more ATS Bus OPC UA Client entries in the trusted clients tab; one for the data service and another one for the bus stop.
● System.Messaging.MessageQueueException: Insufficient resources to perform operation.
The event log might show an error containing System.Messaging.MessageQueueException: Insufficient resources to perform operation. when sending a message to another bus stop, ServiceControl or the monitoring instance. This message indicates that MSMQ reached its maximum capacity because one of the queues might contain a lot of unhandled messages. The following list shows the resolutions per queue:o Bus stop queue: a bus stop did not start and did not consume messages from its queue. Start the bus stop or review the event log to figure out why the bus stop stopped.
o Particular.servicecontrol: the ServiceControl service is stopped. Start ServiceControl so that it consumes messages from the queue. Review the Windows Logs/Applications event log if ServiceControl does not start, it will show a clear error message.
o Particular.monitoring: The monitoring instance is stopped. Start the monitoring instance. Review the Windows Logs/Applications event log if the monitoring instance does not start, it will show a clear error message.
o Dead letter / transactional dead letter messages: A bus stop did not consume the messages within the given amount of time. The class type will how the exact cause of why a message ended up in this queue. Further analysis of the event log is required. This queue should be purged to start up MSMQ again.
o Outgoing queues: Review the state and connection history to determine why messages are queued here. Most likely a TCPIP/hostname issue is the cause of the problem. Reconfiguring ATS Bus with the proper hostname or IP address will solve the issue.
o Audit.log: This queue is used by audit logging; this feature should have been disabled via the ServiceControl management application because there is no application configured to read this queue.
o Error.log: This queue is used by error logging; this feature should have been disabled via the ServiceControl management application because there is no application configured to read this queue.
● System.Messaging.MessageQueueException: Message Queue service is not available.
This message states that the Message Queueing service is not running. Go to the services and start the Message queueing service.
● NServiceBus.Unicast.Queuing.QueueNotFoundException: Failed to subscribe.
This error message indicates that the destination queue is not available, and this is the case when the destination bus stop did not create the queue yet. The bus stop will not start because sending the subscription message is essential. Starting the destination bus stop will resolve the issue.
There are situations that the bus stop does not want to start because of a fatal error. One of these errors is:
System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) ---> System.ComponentModel.Win32Exception (0x80004005): The system cannot find the file specified
The message above shows a SQL Server specific error code, 40. This error may occur when:
● The user provided invalid connection details for the IT Database channel.
● The user provided an invalid connection string for the NServiceBus persistence.
● The SQL Server instance is not running, make sure the service is started.
● A firewall blocks traffic to the SQL server.
● SQL server does not allow TCP/IP connections, please go to Computer Management, Service and Application, SQL Server xxxx Configuration, Network Configuration and enable the TCP/IP protocol. Also ensure that the SQL SERVER port is set to the default 1433.
The bus stop or even the data service might show other error codes too. In these events, visit Google and search for “SQL server error <xx>” where ‘<xx>’ is the error code. The search results will provide an answer to the problem.
A wide variety of error messages may occur when the firewall is not configured properly. Make sure that the following ports are configured in the firewall:
● 8700 (ATS Bus data service default port)
● 8708 (Browsing service)
● 8800 (OT bus stop monitoring port)
● 8802 (IT bus stop monitoring port)
● 8804 (Orderhandling bus stop monitoring port)
● 8806 (Archive bus stop monitoring port)
● 4840 (OPC UA Local discovery service)
● 49320 (Default OPC UA Endpoint)
The bus stops and ServiceControl depend on other services to start properly. The bus stops have these dependencies configured already but ServiceControl does not. The bus stops require:
● MSMQ
● Particular ServiceControl requires:
● MSMQ
When converting the channel message to a shopfloor message, the OT bus stop copies some tag values into a section named DataScopes which is used to implement the Data Type Scopes functionality. It allows one instance of OrderId, LotId, SerialNumber or EquipmentId.
A message definition on the other hand allows multiple message fields within the same message definition to be named 'OrderId', 'LotId', 'SerialNumber' or 'EquipmentId' and this may cause problems. The code that converts the channel message to a shopfloor message will only take the first instance of a message field named OrderId, LotId, SerialNumber or EquipmentId it encounters.
Multiple entries of 'OrderId', 'LotId', 'SerialNumber' or 'EquipmentId' are caused by the fact that there are 2 different user input validation engines in the OT bus stop. The first one checks if a non-indexed message field name has a duplicate and the second engine validates if an indexed message field name has a duplicate with the same index.
For example, a message definition may contain multiple instances of SerialNumber:
And a channel message may refer to this message definition:
The order in which the message fields are processed is not guaranteed so the ‘wrong’ tag value may be copied into the DataScopes section of the shopfloor message and this causes the bus stop to use the ‘wrong’ value to be used to select a work order for instance. This issue only affects work order related channel message action i.e. UpdateBatch.
Best practice is to use the ‘reserved’ message field names carefully. Make sure that there is only one instance of the name 'OrderId', 'LotId', 'SerialNumber' or 'EquipmentId'.
When installing a Bus Stop module a dialog box similar to the following may be displayed stating: 'ATS Bus XXXXX' could not be installed. Verify that you have sufficient privileges to install system services. This may happen when the services.msc, compmgmt.msc (Computer Management) or Task Manager is open.
Ensure that these applications are closed and press the Retry button.
ServiceControl is used to read audit and error messages from a queue and store them into a database. The size of the database depends on the number of messages and the size of each message that are stored within the retention times. The error and audit retention are configurable and can be changed using the ServiceControl management application. By default, ServiceControl is setup using a retention time of 30 days for audit messages and 15 days for error messages.
It is always possible to lower the audit and error retention but the size of the ServiceControl database is not reduced automatically, this requires several actions to be executed and the time required to execute them depends on the database size. The actions are executed in 2 stages:
Stage 1: Lower the audit and error retention times in the ServiceControl management tool:
1. Open the ServiceControl management application and open the configuration dialog of the ‘Particular.ServiceControl’ instance.
2. Navigate to the Audit retention and set the value of days to 5.
3. Navigate to the Error retention and set the value of days to 10.
4. Save the changes. This will restart the ‘Particular.ServiceControl’ instance.
5. Go to ServiceInsight and wait till the last audit message (the one with the green dot in front) is dated 5 days ago. ServiceControl will clean up the database and discard messages older than 5 days once it is restarted. ServiceControl is not done cleaning up if the last message is older than 5 days, you need to wait with the remaining actions if the clean-up is not completed. I don’t know how long this takes. You cannot get the maximum compression if you don’t let the clean-up finish.
Stage 2: Backup and Compress/Defragment the ServiceControl database.
1. Once the clean-up is finished, stop the ‘Particular.ServiceControl’ service via the services dialog or via the ServiceControl management tool like we did this afternoon.
2. Go to the ServiceControl data directory (C:\ProgramData\Particular\ServiceControl\Particular.ServiceControl) and create a backup of the directory named ‘DB’. I don’t know how long it takes to create a backup of the DB directory; this depends per system.
You can copy the complete directory to another drive or directory, or you can create an archive using the ‘Send to à Compressed (zipped) folder’ option via the context menu. Creating an archive is the preferred method because this compresses the data resulting in a smaller copy.
3. Once the backup is completed, go into the DB directory, you should now be in ‘C:\ProgramData\Particular\ServiceControl\Particular.ServiceControl\DB’.
4. Execute the following command: esentutl /r RVN /l logs /s system and wait for it to complete. This may take a few seconds. Make sure the Particular.ServiceControl service is still stopped.
5. Execute the following command when
the previous command completed successfully: esentutl /d Data
6. Restart the ‘Particular.ServiceControl’ services via the services dialog or via the ServiceControl management tool.
Knowledge base article a summary of the preferred procedure given at: https://docs.particular.net/servicecontrol/db-compaction
A bus stop does not receive subscribed bus messages when they are published on the bus and the event log shows: “Creating Bus.Contracts.DerivedContracts failed…” or "No handler for message xxx".
On start, a bus stop generates an assembly named ‘Bus.Contracts.DerivedContracts.dll’ in a directory named ‘C:\ProgramData\Applied Tech Systems\ATS Bus\<BUS_STOP_NAME>’. This assembly is then loaded into memory. This assembly is used when subscribing to and publishing bus messages.
Sometimes it may happen that a bus stop does not properly stop, and this prevents the assembly from being unloaded from memory. This will show during the start of the bus stop, the following error is logged in the event log:
2020-02-10 15:03:44.643 | C:\Program Files (x86)\ATS Global B.V\ATS Bus\Bus Stops\OTBusStop\Bus.Busstop.OT.exe 2.6.20034.1 | ERROR [5] - Creating Bus.Contracts.DerivedContracts failed: (0,0: error CS0016: Could not write to output file 'c:\ProgramData\Applied Tech Systems\ATS Bus\OTBusStop\Bus.Contracts.DerivedContracts.dll' -- 'The process cannot access the file because it is being used by another process. '
Execute the following steps to resolve the issue:
1. Open the Windows task manager and find the Bus.BusStop.<xx>.exe file that represents the bus stop. The value <xx> can be: OT, IT, Cloud or ADOS. The error message in the event log should provide the process that caused the error.
2. Terminate the Bus.Busstop.<xx>.exe process.
3. Open the process explorer (by SysInternals) and find open file handles to the file ‘C:\ProgramData\Applied Tech Systems\ATS Bus\<BUS_STOP_NAME>\ Bus.Contracts.DerivedContracts.dll’. Stopping the bus stop should release the file handles to ‘Bus.Contracts.DerivedContracts.dll’.
4. Open the windows services application (by executing services.msc from the command line) and start the bus stop when there are no open file handles to ‘Bus.Contracts.DerivedContracts.dll’
5. The issue should be resolved and a new ‘Bus.Contracts.DerivedContracts.dll’ should be created when the bus stop starts.
ATS Bus uses SQL Server for SQL Persistence or for its database channels. The event viewer may show time-out related error messages. These errors often occur when SQL Express is used in conjunction with OT Bus stops that are under a very high load. The following items should be reviewed:
● Check if the Auto Close property is set to ‘True’. This can be checked by going to the Options page in the database properties using SQL Management Studio. Set this option to false. The following SQL statement does the same:
USE [master]
GO
ALTER DATABASE [DatabaseCausingTimeouts] SET AUTO_CLOSE OFF;
GO
● Check if the Recovery model property is set to ‘Simple’. This can be checked by going to the Options page in the database properties using SQL Management Studio. The recovery option can be selected at the top of the dialog. Set this option to ‘Simple’. The following SQL statement does the same:
USE [master]
GO
ALTER DATABASE [DatabaseCausingTimeouts] SET RECOVERY SIMPLE;
GO
On startup the channel may fail to start. This may be due to it attempting to register unnecessary settings for the Windows HTTP protocol when linking the certificate (thumbprint of certificate) with the IP address and Port.
This can fail due to insufficient rights assigned to the user running the ATS Bus Stop service.
A network administrator will be required to execute the following script:
cmd> netsh http add sslcert ipport=0.0.0.0:9999 appid={ED3CE4E1-E134-49FC-AA7F-E4E1D142B0D4} certhash=18de9bbe9ad571ca7fcab58f467e89682eba020e
Update the highlighted sections above using the information found in the Webservice Server Channel configuration as indicated below.
A bus stop may fail to restart when the user.config file is corrupted. Upon (re)start, the following dialog is shown:
The message itself does not give that much information but the Application section in the Windows Logs in the Windows Event Viewer should show a .NET Runtime (source column) error:
Application: Bus.Busstop.IT.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Xml.XmlException
at System.Xml.XmlTextReaderImpl.Throw(System.Exception)
…
at System.Configuration.BaseConfigurationRecord.InitConfigFromFile()
Exception Info: System.Configuration.ConfigurationErrorsException
at System.Configuration.ConfigurationSchemaErrors.ThrowIfErrors(Boolean)
…
at System.Configuration.ClientConfigurationSystem.OnConfigRemoved(System.Object, System.Configuration.Internal.InternalConfigEventArgs)
Exception Info: System.Configuration.ConfigurationErrorsException
at System.Configuration.ClientConfigurationSystem.OnConfigRemoved(System.Object, System.Configuration.Internal.InternalConfigEventArgs)
at System.Configuration.Internal.InternalConfigRoot.OnConfigRemoved(System.Configuration.Internal.InternalConfigEventArgs)
…
at System.Configuration.ApplicationSettingsBase.get_Item(System.String)
at Bus.Busstop.IT.Properties.Settings.get_BusStopName()
at Bus.Busstop.IT.Program.Main()
The error message shown in the event log indicates that the user.config file is corrupted. The location of this file depends on which user is executing the bus stop service. By default, the LOCAL SYSTEM account runs the bus stop but the services dialog will show the account running the service in the Log On As column:
The user.config file is stored in the %LOCALAPPDATA%\{Vendor}\{Application}\{Version} folder. The NT AUTHORITY\SYSTEM (Local System) account uses C:\Windows\system32\config\systemprofile\AppData\Local\{Vendor}\{Program}\{Version} or C:\Windows\SysWOW64\config\systemprofile\AppData\Local\{Vendor}\{Program}\{Version} when running 32-bit application on a 64-bit system which is the case with current ATS Bus versions.
The bus stop will start again after the user.config file is deleted from the C:\Windows\SysWOW64\config\systemprofile\AppData\Local\ATS_Global_B.V\ Bus.Busstop.IT.exe_xxxxxxxxxxxx\{Version}\ directory.