Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Sendall Coordinate System Race Condition?
#1
I'm receiving coordinate system runtime errors (Coord[n].RuntimeError) sporadically after many minutes to hours of running. We're running four motors in four separate coordinate systems. The four coordinate systems run an identical program which simultaneously moves the motors at the same time using external time base.

'list apc' is showing that the program counter was pointing at 'sendall' immediately following a 'send' command at the time of the abort in all four coordinate systems.

I've tried increasing Sys.PreCalc as recommended by other threads here to no avail. I also wrote a plc to monitor the high water mark of the Coord[n].BufferWarn variable - this never exceeded 0. There are also no additional errors such as FeLimit. 'getsends' is running on all communications ports being used.

I'm at a loss to explain the RuntimeError and am thinking now that perhaps this is due to a race amongst the four simultaneous calls to 'sendall'. Any help or insight would be greatly appreciated.
Reply
#2
Using "Sendall" in a realtime routine might cause this if your Sys.RtIntPeriod=0, you have a high servo rate, and a lot of text being sent out.

Can you please try implementing process locks around the "sendall" command in your motion program?

The syntax would look like this:

Code:
while(Sys.Lock[0] == 1){}  // Wait for this program instance's turn to use "sendall"
sendall;
Sys.Lock[0] = 0; // Release the process control

If you have a lot of these in your program, you can make a subprogram call something like:

Code:
open subprog SafeSendall(locknumber)
while(Sys.Lock[locknumber] == 1){}  // Wait for this program instance's turn to use "sendall"
sendall;
Sys.Lock[locknumber] = 0; // Release the process control
close

and then call it with

Code:
call SafeSendall(0); // 0 is the desired lock number, can be 0 to 31

See page 594 for more details on process locks.

An alternate solution would be to place small timer delays between SEND commands. A timer subprogram we use often in the office is below:

Code:
OPEN SUBPROG Timer(delay_time) // Open subroutine buffer
    LOCAL EndTime; // Local variable
    EndTime = Sys.Time + delay_time; // Store current time + user specified delay
    WHILE (EndTime > Sys.Time){}; // Wait until real time reaches stored value
    return
close

You can call it like

Code:
call Timer(0.001); // Wait 0.001 seconds

This is better than using a Dwell since it will not affect motion sequencing if you keep the amount of wait time very small. You may need to increase Coord[x].GoBack (takes values of 0 to 255) to use this in a motion program without breaking blending. Same applies to waiting for the locks in a while loop.
Reply
#3
Thanks for the recommendations Charles. I went ahead and tried your first suggestion but to no avail. Oddly, all 4 coordinate systems stopped at the same time with "CoordExecStatus[n] - Stopped at breakpoint...", this is despite having no breakpoints defined.

I have verified that the sendall is the culprit - removing it removes the symptoms at least.

If I get a chance, I'll try the timer suggestion as well. Thanks.
Reply
#4
OK; sounds good. Could you also post a short excerpt of the kind of code that is causing you a problem? I can test it here on my system.
Reply
#5
(06-23-2015, 04:38 PM)CharlesP Wrote: Could you also post a short excerpt of the kind of code that is causing you a problem? I can test it here on my system.

It's pretty simple, basically just an infinite loop with a move, delay, send and sendall. The same code runs in four different coordinate systems with a single motor each. All coordinate systems are synchronized with a single time-base trigger and remain synchronized via an external time-base such that all four motors move at exactly the same time (down to the same servo period AFAICT).

Code:
// ... Setup time-base trigger & arming...
while ( done != 1 )
{
    // ... calculate move pos...
    
    x(movePos) ta(moveTime/2) td(moveTime/2) ts(0) tm(moveTime)
    delay(delayTime)
    send 0 "This is a status update"
    sendall
}

This will run for random amounts of time before triggering the runtime-error (I've seen anywhere from 5 to 45 minutes). getsends is always running, etc...[/code]
Reply
#6
Hi again,

OK, I have been playing with your code throughout the day and experienced program failures as well. My initial suggestions do not seem to help.

I think due to the nature of the send and sendall commands, it might not be a good idea to use them in a motion program directly. Instead, I propose that you set a csglobal flag in the motion program that indicates to a waiting background Script PLC to send your status updates via send and sendall. Is that a possibility for your architecture? That way, the realtime routine isn't used to perform a task (sending the string) that is time-consuming and does not fit well in a realtime routine, and instead the asynchronous background PLC can take care of it.

For example, the motion program:

Code:
open prog SendFlagProg
linear inc;
movePos=1; // user units
moveTime = 200; // msec
enable plc SendPLC;
while (done == 0)
{
    x(movePos) ta(moveTime/2) td(moveTime/2) ts(0) tm(moveTime)
    ReadyToSend == 1; // Indicate to waiting PLC that the motion program wants to send an update
    delay(delayTime)
    // PLC then sets this flag to 0 after sending the update
}
close

And the PLC:

Code:
open plc SendPLC
local CSNumber = 1; // Used to index coordinate system numbers
while(CSNumber < 5)
{
    Ldata.Coord = CSNumber; // Change this PLC's coordinate system number
    if(ReadyToSend > 0)
    {
        send 0"Coordinate System %u Sends a Status Update",CSNumber;
        sendall;
        ReadyToSend = 0;
    }    
    CSNumber++;
    if(CSNumber == 5)
        CSNumber = 1;
}



close

And the global definitions:

Code:
csglobal moveTime = 1;
csglobal movePos = 1;
csglobal delayTime = 10;
csglobal done = 0;
csglobal ReadyToSend = 0;
Reply
#7
(06-24-2015, 04:20 PM)CharlesP Wrote: OK, I have been playing with your code throughout the day and experienced program failures as well. My initial suggestions do not seem to help.

Great! I'm glad to hear you can reproduce this.

(06-24-2015, 04:20 PM)CharlesP Wrote: I think due to the nature of the send and sendall commands, it might not be a good idea to use them in a motion program directly. Instead, I propose that you set a csglobal flag in the motion program that indicates to a waiting background Script PLC to send your status updates via send and sendall. Is that a possibility for your architecture?

Yes this is possible, sounds reasonable and will be easy to implement. My only question is whether or not I need to worry about the plc sporadically dying for the same reason?

Thanks for the code samples too.
Reply
#8
I'm virtually certain the PLC will not fault out. I will let a test run all day on my box here in the office and let you know at the end of the day.
Reply
#9
(06-25-2015, 10:24 AM)CharlesP Wrote: I'm virtually certain the PLC will not fault out. I will let a test run all day on my box here in the office and let you know at the end of the day.
Cool. This is a small data point, but I've ran all day off your recommended plc with no failures.
Reply
#10
As have I. 22290 seconds and counting. Looks like this is an acceptable solution!

Let me know if you have any further issues.
Reply


Forum Jump:


Users browsing this thread: 2 Guest(s)