Troubleshooting

 

In this Topic Hide

ServiceControl Does Not Start

Unable to Retrieve List of OPC Servers During OPC DA Channel Configuration

Unable to Retrieve List of OPC Servers During OPC UA Channel Configuration

Certificate related issues when working with OPC UA channels

The bus stop won’t start

Misconfigured ServiceControl, private MSMQ audit.log is full

Particular.ServiceControl is not started

Error Message Due to Firewall Settings

Service Dependencies

Incorrect OrderId, LotId, SerialNumber or EquipmentId Used In Data Type Scope

ServiceControl Does Not Start

Perform the actions provided at this web page if ServiceControl does not start and the following message is shown in the event log:

“System.Messaging.MessageQueueException (0x80004005): Insufficient resources to perform operation.

at System.Messaging.MessageQueue.SendInternal(Object obj, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)”

If the MSMQ storage exceeds the maximum capacity and messages cannot be processed anymore carry out the following steps.

1.     Start the Computer Management desktop application (%windir%\system32\compmgmt.msc /s).

2.     On the left hand side expand/select Services and Applications > Message Queueing > Private Queues. The right hand side of the screen will show the amount of messages in each private queue. You now have 2 options:

a.     Purge the queue: expand the queue in the left hand side pane and right click the Queue Messages. Select All Tasks > Purge to clear the queue.

b.     Increase the storage assigned to the Message Queueing Service: Right click Message Queueing in the left hand side pane and select Properties. Now you can assign more diskspace to MSMQ. This requires a reboot of the system or restart of the MSMQ service.

Unable to Retrieve List of OPC Servers During OPC DA Channel Configuration

If you receive the message "Failed to get a list of OPC Servers for host…” while configuring an OPC DA channel then the ATS Bus Browsing service is not running on the system that hosts the OPC DA Server.

Unable to Retrieve List of OPC Servers During OPC UA Channel Configuration

Problem

When configuring an OPC UA Channel in the ATS Bus Cockpit the following message is displayed: “Failed to get list of OPC Servers for host '<SERVERNAME>'.” The event log will show something like the following:

Opc.Ua.ServiceResultException: Thumbprint was explicitly specified in the configuration. Cannot generate a new certificate

This indicates that the OPC UA certificate thumbprint in the ATS Bus DataService application configuration does not match the one in the ATS Bus OPC UA Client certificate that can be found in the windows certificate store.

Solution

1.     Open the Windows certification store. This can be found by searching for Certificates.

2.     Open the UA Applications folder.

3.     Double click the ATS Bus OPC UA Client certificate.

4.     Select the Details tab.

5.     Copy the Thumbprint.

6.     Edit ATS.Bus.DataServiceHost.exe.config in the ATS Bus installation directory (C:\Program Files (x86)\Applied Tech Systems\ATS Bus\Services).

This is the ATS Bus DataService configuration file.

7.     Find the ApplicationCertificateThumbprint element and replace the contents with what you just copied including the first digit you had to remember and remove the whitespaces.

8.     Restart the ATS Bus DataService and configure the OPC UA channel again.

Certificate related issues when working with OPC UA channels

When working with OPC UA Channels, ensure that the certificate thumbprints match the ones configured in the ATS Bus DataService and OT bus stop application configuration. Also make sure that the OPC UA Server has 2 or more ATS Bus OPC UA Client entries in the trusted clients tab; one for the data service and another one for the bus stop.

The bus stop won’t start

There are multiple reasons why the bus stop would fail to start:

Misconfigured ServiceControl, private MSMQ audit.log is full

It may happen that ServiceControl is not configured properly and that audit logging has been enabled. This will fill up the private MSMQ named audit.log.

Purge the queue, disable audit log the ServiceControl configuration and restart ServiceControl.

Particular.ServiceControl is not started

Sometimes, the bus stops fail to start because Particular.ServiceControl is not started, one of the following bus stop warnings or errors is displayed in the Applications and Services logs\Applied Tech Systems eventlog:

2016-11-17 09:00:41.585 | Bus.BusStop.Archive.exe 1.0.16322.1 | WARN  [9] - Unable to send heartbeat to ServiceControl: System.Exception: Failed to send message to address: particular.servicecontrol@<HOSTNAME> ---> System.Messaging.MessageQueueException: Insufficient resources to perform operation.

Or:

2016-11-17 09:00:41.597 | Bus.BusStop.Archive.exe 1.0.16322.1 | WARN  [9] - Unable to register endpoint startup with ServiceControl. Going to reattempt registration after 00:01:00. System.Exception: Failed to send message to address: particular.servicecontrol@<HOSTNAME> ---> System.Messaging.MessageQueueException: Insufficient resources to perform operation.

And the “Windows Logs\Applications” may contain the following error message:

Application: Bus.BusStop.Archive.exe

Framework Version: v4.0.30319

Description: The process was terminated due to an unhandled exception.

Exception Info: System.Messaging.MessageQueueException

   at System.Messaging.MessageQueue.SendInternal(System.Object, System.Messaging.MessageQueueTransaction, System.Messaging.MessageQueueTransactionType)

   at System.Messaging.MessageQueue.Send(System.Object, System.Messaging.MessageQueueTransactionType)

   at NServiceBus.Transports.Msmq.MsmqMessageSender.Send(NServiceBus.TransportMessage, NServiceBus.Unicast.SendOptions)

Exception Info: System.Exception

   at NServiceBus.Transports.Msmq.MsmqMessageSender.ThrowFailedToSendException(NServiceBus.Address, System.Exception)

   at NServiceBus.Transports.Msmq.MsmqMessageSender.Send(NServiceBus.TransportMessage, NServiceBus.Unicast.SendOptions)

   at NServiceBus.Unicast.Subscriptions.MessageDrivenSubscriptions.SubscriptionManager.SendSubscribeMessageWithRetries(NServiceBus.Address, NServiceBus.TransportMessage, System.String, Int32)

   at NServiceBus.Unicast.Subscriptions.MessageDrivenSubscriptions.SubscriptionManager+<>c__DisplayClass1.<Subscribe>b__0(System.Object)

   at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(System.Object)

   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()

   at System.Threading.ThreadPoolWorkQueue.Dispatch()

   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

Particular ServiceControl should have raised the following error in “Windows Logs\Applications”:

Service cannot be started. System.InvalidOperationException: There is a problem with the input queue: FormatName:DIRECT=OS:<HOSTNAME>\private$\Particular.ServiceControl. See the enclosed exception for details. ---> System.Messaging.MessageQueueException: Message Queue service is not available.

   at System.Messaging.MessageQueue.MQCacheableInfo.get_Transactional()

   at System.Messaging.MessageQueue.get_Transactional()

   at NServiceBus.Transports.Msmq.MsmqDequeueStrategy.QueueIsTransactional() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Transports\Msmq\MsmqDequeueStrategy.cs:line 157

--- End of inner exception stack trace ---

   at NServiceBus.Transports.Msmq.MsmqDequeueStrategy.QueueIsTransactional() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Transports\Msmq\MsmqDequeueStrategy.cs:line 161

   at NServiceBus.Transports.Msmq.MsmqDequeueStrategy.Init(Address address, TransactionSettings settings, Func`2 tryProcessMessage, Action`2 endProcessMessage) in C:\BuildAgent\work\3206e2123f54fce4\sr...

The reason that the bus stop did not start is because the bus stop needs ServiceControl but could not reach it because ServiceControl was not started because the MSMQ service was not started.

The bus stops and service control depend on MSMQ. The bus stops have these dependencies set but ServiceControl does not have this dependency set. It may happen that MSMQ is started later during a reboot of the system, so ServiceControl will start and fail and will require a manual restart.

Error Message Due to Firewall Settings

A wide variety of error messages may occur when the firewall is not configured properly. Make sure that the following ports are configured in the firewall:

       8700 (ATS Bus data service default port)

       8708 (Browsing service)

       8800 (OT bus stop monitoring port)

       8802 (IT bus stop monitoring port)

       8804 (Orderhandling bus stop monitoring port)

       8806 (Archive bus stop monitoring port)

       4840 (OPC UA Local discovery service)

       49320 (Default OPC UA Endpoint)

Service Dependencies

The bus stops and ServiceControl depend on other services to start properly. The bus stops have these dependencies configured already but ServiceControl does not. The bus stops require:

       MSMQ

       MSDTC

       Performance counters

       Particular ServiceControl requires:

       MSMQ

Incorrect OrderId, LotId, SerialNumber or EquipmentId Used In Data Type Scope

When converting the channel message to a shopfloor message, the OT bus stop copies some tag values into a section named DataScopes which is used to implement the Data Type Scopes functionality. It allows one instance of OrderId, LotId, SerialNumber or EquipmentId.

A message definition on the other hand allows multiple message fields within the same message definition to be named 'OrderId', 'LotId', 'SerialNumber' or 'EquipmentId' and this may cause problems. The code that converts the channel message to a shopfloor message will only take the first instance of a message field named OrderId, LotId, SerialNumber or EquipmentId it encounters.

Multiple entries of 'OrderId', 'LotId', 'SerialNumber' or 'EquipmentId' are caused by the fact that there are 2 different user input validation engines in the OT bus stop. The first one checks if a non-indexed message field name has a duplicate and the second engine validates if an indexed message field name has a duplicate with the same index.

For example, a message definition may contain multiple instances of SerialNumber:

And a channel message may refer to this message definition:

The order in which the message fields are processed is not guaranteed so the ‘wrong’ tag value may be copied into the DataScopes section of the shopfloor message and this causes the bus stop to use the ‘wrong’ value to be used to select a work order for instance. This issue only affects work order related channel message action i.e. UpdateBatch.

Best practice is to use the ‘reserved’ message field names carefully. Make sure that there is only one instance of the name 'OrderId', 'LotId', 'SerialNumber' or 'EquipmentId'.