Windows app hanging, ppmacserver not running, have to restart PPMAC to clear conditio

I developed a system using the PPMAC. My customer created the Windows based GUI using the Delta Tau Dot-Net development library. System is up and working as expected. However, once in a while the customer's Windows GUI app locks up due to exceptions being thrown due to an index somehow exceeding array bounds. Once it locks up, shutting it down and restarting will not allow it to reconnect to the PPMAC. After rebooting the Windows computer we are still unable to clear the condition and connect to the PPMAC. I have the IDE installed on the same computer. It has no problem connecting at any time even when the customer's application is locked up and unable to connect.

On an impulse, the last time the customer's app locked up, I closed the app, and then cycled power to the PPMAC and then restarted the customer's app. The customer's app was immediately able to connect. Coincidence? Customer's app hasn't locked up since so I have not had an opportunity to duplicate this.

But this has me wondering if something is happening at the PPMAC to prevent reconnection. In a case where I have both the customer's app and the IDE running on the same machine, is Linux or the PPMAC "aware" that these are separate communication threads? Is it possible that when the customer's app throws an exception and gets shut down the customer's application is leaving this thread open and the PPMAC or Linux is still "connected" to this orphaned thread and, therefore, blocks attempts by the customer's app to reestablish this thread? The fact that restarting the PPMAC instantly allowed the customer's app to connect leads me to believe something like this might be going on.

Suggestions?

Thanks:

mshaver:

Could you provide the details of the exception(s) that are getting thrown by the .NET program? This info would probably be very helpful in figuring out what the problem is.

Also, if the connection to the Power PMAC cannot be reestablished until it has been rebooted, this suggests that something on the Power PMAC might be crashing. Has anyone logged in to the Power PMAC with a terminal and verified that the Delta Tau processes are still running (gppmac, ppmacserver) after the comms stop working?

Even if the .NET program is leaving the TCP connection open, the Power PMAC should automatically shut it down after a timeout. But even so, I don't think this wouldn't prevent re-connections as the default settings allow many simultaneous connections.

I developed a system using the PPMAC. My customer created the Windows based GUI using the Delta Tau Dot-Net development library. System is up and working as expected. However, once in a while the customer's Windows GUI app locks up due to exceptions being thrown due to an index somehow exceeding array bounds. Once it locks up, shutting it down and restarting will not allow it to reconnect to the PPMAC. After rebooting the Windows computer we are still unable to clear the condition and connect to the PPMAC. I have the IDE installed on the same computer. It has no problem connecting at any time even when the customer's application is locked up and unable to connect.

 

On an impulse, the last time the customer's app locked up, I closed the app, and then cycled power to the PPMAC and then restarted the customer's app. The customer's app was immediately able to connect. Coincidence? Customer's app hasn't locked up since so I have not had an opportunity to duplicate this.

 

But this has me wondering if something is happening at the PPMAC to prevent reconnection. In a case where I have both the customer's app and the IDE running on the same machine, is Linux or the PPMAC "aware" that these are separate communication threads? Is it possible that when the customer's app throws an exception and gets shut down the customer's application is leaving this thread open and the PPMAC or Linux is still "connected" to this orphaned thread and, therefore, blocks attempts by the customer's app to reestablish this thread? The fact that restarting the PPMAC instantly allowed the customer's app to connect leads me to believe something like this might be going on.

 

Suggestions?

Thanks:

can you give me more details from the exception like Stack trace, inner exception etc to see which function is throwing it? What is the frequency of this problem ? Is the exception thrown on user interaction or the exception is thrown while the opraetor monitoring the screen? Are you runing and C background programs and sending the strings to your applications ? I will need more information.

When next time this happens can you check;

1. Can you communicate with IDE?

2. can you disable all the PLC inculding C app and then try to launch the application.

Please let me know.

Thanks,

Atul

1. I'll try to get details of the exception being thrown by the Windows app.

2. How do I verify that the Delta Tau processes are still running (gppmac, ppmacserver)?

3. The windows application updates quite a bit of information from the Delta Tau very frequently. IE; multiple requests for information per second and sometimes it locks up within a few seconds of starting up so there have been no operator actions yet. Are there any upper limits on these background information request rates that would cause something to lock up?

4. In all cases, the IDE continues to work fine. The windows app has locked up with and without the IDE running so I don't think the IDE has anything to do with it. However, if the IDE was running, it continues to communicate fine. If it was not running, I am always able to start it up and communicate without incident.

5. I have no C programs running in the PPMAC.

1. I'll try to get details of the exception being thrown by the Windows app.

2. How do I verify that the Delta Tau processes are still running (gppmac, ppmacserver)?

3. The windows application updates quite a bit of information from the Delta Tau very frequently. IE; multiple requests for information per second and sometimes it locks up within a few seconds of starting up so there have been no operator actions yet. Are there any upper limits on these background information request rates that would cause something to lock up?

4. In all cases, the IDE continues to work fine. The windows app has locked up with and without the IDE running so I don't think the IDE has anything to do with it. However, if the IDE was running, it continues to communicate fine. If it was not running, I am always able to start it up and communicate without incident.

5. I have no C programs running in the PPMAC.

2. How do I verify that the Delta Tau processes are still running (gppmac, ppmacserver)?

You can check the process by opning the taskManager from IDE. Select Tools/Taskmanager from the IDE and then select OS Resource Tab.

Point 3,4,and 5:

Looks like it is specific to your Windows application. In this case my suggestion will be run the application using debugger to identify the problem. I will try my best to help you.

Thanks,

Atul

Here is some very fresh information that I hope the Delta Tau experts can make some sense of:

1. The customer's windows application locked up several times today while using it to jog and move axis in the PPMAC for testing purposes. In each case, the following occurred.

2. We were able to shut down the windows app and restart it but it would not communicate with the PPMAC.

3. The IDE was running at the time and it continued to communicate with the PPMAC without any problems.

4. We shut down and restarted the Windows computer.

5. The customer's Windows app still would not communicate with the PPMAC but the IDE would.

6. We checked the PPMAC OS Processes from the IDE.

> gppmac was running.

> ppmacserver WAS NOT RUNNING.

7. We then cycled power to the PPMAC.

8. The IDE connected without incident.

9. The customer's Windows app connected and communicated normally.

10. We checked the PPMAC OS Processes from the IDE.

> BOTH gppmac AND ppmacserver ARE NOW RUNNING.

> The customer's Windows app is running and communicating normally.

So tell me about ppmacserver. What is it. What does it take to make it crash or shut down?

Is the fact that it is disappearing a symptom or a cause of the problem with the customer's app?

Below is some information being logged by the customer's app. I don't know if the innermost exception is being thrown by a function in the Delta Tau Dot-Net component classes or in the customer's code.

================================

First Excerpt From Application Log

================================

//This is an excerpt from one type of log that the customer's app creates. First part is normal logging of polling activities, then the exception gets thrown.

6/2/2014 12:46:14.186, Debug, DeltaTauPowerPmac, 0, < M4004=425999

6/2/2014 12:46:14.186, Info, DeltaTauPowerPmac, 0, CheckAxisStatus complete

6/2/2014 12:46:14.186, Info, DeltaTauPowerPmac, 0, CheckControllerStatus complete

6/2/2014 12:46:14.451, Debug, DeltaTauPowerPmac, 0, > M4000 // application polling PMAC for data in M4000

6/2/2014 12:46:14.451, Debug, DeltaTauPowerPmac, 0, < M4000=425995

// PMAC responding

6/2/2014 12:46:14.498, Debug, DeltaTauPowerPmac, 0, > M4001

6/2/2014 12:46:14.498, Debug, DeltaTauPowerPmac, 0, < M4001=425999

6/2/2014 12:46:14.498, Info, DeltaTauPowerPmac, 0, CheckAxisStatus complete

6/2/2014 12:46:18.881, Error, DeltaTauPowerPmac, 0, Status = Failed Comm Error with Command: M4002 // Attepting to poll M4002

// ONCE THE ERROR ABOVE OCCURS, THE WINDOWS APPLICATION WILL BE UNABLE TO COMMUNICATE WITH THE PPMAC UNTIL POWER IS CYCLED TO THE PPMAC

// THE IDE WILL CONTINUE TO COMMUNICATE WITHOUT ANY PROBLEMS.

6/2/2014 12:46:18.881, Info, DeltaTauPowerPmac, 0, Status = Failed Comm Error with Command: M4002

6/2/2014 12:46:18.881, Debug, DeltaTauPowerPmac, 0, > M4002

6/2/2014 12:46:18.881, Exception, DeltaTauPowerPmac, 0, Exception in SendGetResponse: Index was out of range. Must be non-negative and less than the size of the collection.

Parameter name: index

6/2/2014 12:46:18.881, Error, DeltaTauPowerPmac, 0, CheckAxisStatus error

6/2/2014 12:46:22.953, Error, DeltaTauPowerPmac, 0, Status = Failed Comm Error with Command: M4003

6/2/2014 12:46:22.953, Info, DeltaTauPowerPmac, 0, Status = Failed Comm Error with Command: M4003

6/2/2014 12:46:22.953, Debug, DeltaTauPowerPmac, 0, > M4003

6/2/2014 12:46:22.953, Exception, DeltaTauPowerPmac, 0, Exception in SendGetResponse: Index was out of range. Must be non-negative and less than the size of the collection.

Parameter name: index

6/2/2014 12:46:22.953, Error, DeltaTauPowerPmac, 0, CheckAxisStatus error

6/2/2014 12:46:26.993, Error, DeltaTauPowerPmac, 0, Status = Failed Comm Error with Command: M4004

6/2/2014 12:46:26.993, Info, DeltaTauPowerPmac, 0, Status = Failed Comm Error with Command: M4004

6/2/2014 12:46:26.993, Debug, DeltaTauPowerPmac, 0, > M4004

6/2/2014 12:46:26.993, Exception, DeltaTauPowerPmac, 0, Exception in SendGetResponse: Index was out of range. Must be non-negative and less than the size of the collection.

//and on and on

//once the error above occurs, the Windows app is no longer able to coumminicate with the PPMAC until power to the PPMAC is cycled.

==============================

Second Excerpt:

==============================

// This is an excerpt from a different type of log that the customer's app creates.

// Note two types of exceptions being logged.

// Once we see these types of exceptions in the customer's application log, the application will no longer communicate with the PPMAC

// Until power is cycled to the PPMAC.

6/2/2014 12:28:17.851, Exception, DeltaTauPowerPmac, 0, Exception in GetPosition: Conversion from string "-inf" to type 'Double' is not valid.: System.InvalidCastException: Conversion from string "-inf" to type 'Double' is not valid. ---> System.FormatException: Input string was not in a correct format.

at Microsoft.VisualBasic.CompilerServices.Conversions.ParseDouble(String Value, NumberFormatInfo NumberFormat)

at Microsoft.VisualBasic.CompilerServices.Conversions.ToDouble(String Value, NumberFormatInfo NumberFormat)

--- End of inner exception stack trace ---

at Microsoft.VisualBasic.CompilerServices.Conversions.ToDouble(String Value, NumberFormatInfo NumberFormat)

at Microsoft.VisualBasic.CompilerServices.Conversions.ToDouble(String Value)

at DeltaTauPowerPmacController.classDeltaTauPowerPmacController.GetAxisPosition(Int32 AxisNumber, Double& Position)

at DeltaTauPowerPmacController.classDeltaTauPowerPmacController.GetPosition(String AxisName, Double& CurrentPosition)

6/2/2014 12:46:18.881, Exception, DeltaTauPowerPmac, 0, Exception in SendGetResponse: Index was out of range. Must be non-negative and less than the size of the collection.

Parameter name: index: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.

Parameter name: index

at System.ThrowHelper.ThrowArgumentOutOfRangeException()

at System.Collections.Generic.List`1.get_Item(Int32 index)

at DeltaTauPowerPmacController.classDeltaTauPowerPmacController.SendGetResponse(String CommandString, String& ResponseString)

Here is some very fresh information that I hope the Delta Tau experts can make some sense of:

 

1. The customer's windows application locked up several times today while using it to jog and move axis in the PPMAC for testing purposes. In each case, the following occurred.

2. We were able to shut down the windows app and restart it but it would not communicate with the PPMAC.

3. The IDE was running at the time and it continued to communicate with the PPMAC without any problems.

4. We shut down and restarted the Windows computer.

5. The customer's Windows app still would not communicate with the PPMAC but the IDE would.

6. We checked the PPMAC OS Processes from the IDE.

> gppmac was running.

> ppmacserver WAS NOT RUNNING.

7. We then cycled power to the PPMAC.

8. The IDE connected without incident.

9. The customer's Windows app connected and communicated normally.

10. We checked the PPMAC OS Processes from the IDE.

> BOTH gppmac AND ppmacserver ARE NOW RUNNING.

> The customer's Windows app is running and communicating normally.

 

So tell me about ppmacserver. What is it. What does it take to make it crash or shut down?

 

Is the fact that it is disappearing a symptom or a cause of the problem with the customer's app?

 

Below is some information being logged by the customer's app. I don't know if the innermost exception is being thrown by a function in the Delta Tau Dot-Net component classes or in the customer's code.

 

================================

First Excerpt From Application Log

================================

//This is an excerpt from one type of log that the customer's app creates. First part is normal logging of polling activities, then the exception gets thrown.

6/2/2014 12:46:14.186, Debug, DeltaTauPowerPmac, 0, < M4004=425999

 

6/2/2014 12:46:14.186, Info, DeltaTauPowerPmac, 0, CheckAxisStatus complete

6/2/2014 12:46:14.186, Info, DeltaTauPowerPmac, 0, CheckControllerStatus complete

6/2/2014 12:46:14.451, Debug, DeltaTauPowerPmac, 0, > M4000 // application polling PMAC for data in M4000

6/2/2014 12:46:14.451, Debug, DeltaTauPowerPmac, 0, < M4000=425995

// PMAC responding

6/2/2014 12:46:14.498, Debug, DeltaTauPowerPmac, 0, > M4001

6/2/2014 12:46:14.498, Debug, DeltaTauPowerPmac, 0, < M4001=425999

 

6/2/2014 12:46:14.498, Info, DeltaTauPowerPmac, 0, CheckAxisStatus complete

6/2/2014 12:46:18.881, Error, DeltaTauPowerPmac, 0, Status = Failed Comm Error with Command: M4002 // Attepting to poll M4002

 

// ONCE THE ERROR ABOVE OCCURS, THE WINDOWS APPLICATION WILL BE UNABLE TO COMMUNICATE WITH THE PPMAC UNTIL POWER IS CYCLED TO THE PPMAC

// THE IDE WILL CONTINUE TO COMMUNICATE WITHOUT ANY PROBLEMS.

 

 

6/2/2014 12:46:18.881, Info, DeltaTauPowerPmac, 0, Status = Failed Comm Error with Command: M4002

6/2/2014 12:46:18.881, Debug, DeltaTauPowerPmac, 0, > M4002

6/2/2014 12:46:18.881, Exception, DeltaTauPowerPmac, 0, Exception in SendGetResponse: Index was out of range. Must be non-negative and less than the size of the collection.

Parameter name: index

6/2/2014 12:46:18.881, Error, DeltaTauPowerPmac, 0, CheckAxisStatus error

6/2/2014 12:46:22.953, Error, DeltaTauPowerPmac, 0, Status = Failed Comm Error with Command: M4003

6/2/2014 12:46:22.953, Info, DeltaTauPowerPmac, 0, Status = Failed Comm Error with Command: M4003

6/2/2014 12:46:22.953, Debug, DeltaTauPowerPmac, 0, > M4003

6/2/2014 12:46:22.953, Exception, DeltaTauPowerPmac, 0, Exception in SendGetResponse: Index was out of range. Must be non-negative and less than the size of the collection.

Parameter name: index

6/2/2014 12:46:22.953, Error, DeltaTauPowerPmac, 0, CheckAxisStatus error

6/2/2014 12:46:26.993, Error, DeltaTauPowerPmac, 0, Status = Failed Comm Error with Command: M4004

6/2/2014 12:46:26.993, Info, DeltaTauPowerPmac, 0, Status = Failed Comm Error with Command: M4004

6/2/2014 12:46:26.993, Debug, DeltaTauPowerPmac, 0, > M4004

6/2/2014 12:46:26.993, Exception, DeltaTauPowerPmac, 0, Exception in SendGetResponse: Index was out of range. Must be non-negative and less than the size of the collection.

//and on and on

//once the error above occurs, the Windows app is no longer able to coumminicate with the PPMAC until power to the PPMAC is cycled.

 

==============================

Second Excerpt:

==============================

// This is an excerpt from a different type of log that the customer's app creates.

// Note two types of exceptions being logged.

// Once we see these types of exceptions in the customer's application log, the application will no longer communicate with the PPMAC

// Until power is cycled to the PPMAC.

6/2/2014 12:28:17.851, Exception, DeltaTauPowerPmac, 0, Exception in GetPosition: Conversion from string "-inf" to type 'Double' is not valid.: System.InvalidCastException: Conversion from string "-inf" to type 'Double' is not valid. ---> System.FormatException: Input string was not in a correct format.

at Microsoft.VisualBasic.CompilerServices.Conversions.ParseDouble(String Value, NumberFormatInfo NumberFormat)

at Microsoft.VisualBasic.CompilerServices.Conversions.ToDouble(String Value, NumberFormatInfo NumberFormat)

--- End of inner exception stack trace ---

at Microsoft.VisualBasic.CompilerServices.Conversions.ToDouble(String Value, NumberFormatInfo NumberFormat)

at Microsoft.VisualBasic.CompilerServices.Conversions.ToDouble(String Value)

at DeltaTauPowerPmacController.classDeltaTauPowerPmacController.GetAxisPosition(Int32 AxisNumber, Double& Position)

at DeltaTauPowerPmacController.classDeltaTauPowerPmacController.GetPosition(String AxisName, Double& CurrentPosition)

 

6/2/2014 12:46:18.881, Exception, DeltaTauPowerPmac, 0, Exception in SendGetResponse: Index was out of range. Must be non-negative and less than the size of the collection.

Parameter name: index: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.

Parameter name: index

at System.ThrowHelper.ThrowArgumentOutOfRangeException()

at System.Collections.Generic.List`1.get_Item(Int32 index)

at DeltaTauPowerPmacController.classDeltaTauPowerPmacController.SendGetResponse(String CommandString, String& ResponseString)

The function GetAxisPosition, GetPosition and SendgetResponse are not the PowerPMACComLib function but someone build the wrapper class using PowerPMACCommLib.

Did you develop this class ? You may want to debug the wrapper class to find out how the commands are send

Thanks

Atul