Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Open Servo CPU load
#1
Hi All
I just tuned an Open Servo controller on a 240 MHz UMAC that is controlling one axis. The CPU load (monitored via the PMAC CPU Resources from PEWIN) tells me that the % of FG time is near 22%.

How can I scale this value to a system that is controlling 6 axis via this Open Servo? The 6 axis are all equals, so there's no need to add to the algorithm further computation. Should I expect a 6x my 22%? Or this percentage is fixed, regarding the number of axis connected?

thanks
ciao
gg
Reply
#2
I usually do this emperically. Just activate another axis and have it call the algorithm. As long as you disable the FFE (ixx11) the amp fault and position limits (ixx24) the motor will go to closed loop state and then it is calling the routine and you can see how much this adds.

I expect it will be at least half again what you are seeing as the compiled open servo code is not so efficient. It is good to test your logic but in the end you might need to write directly in assembly code as this is much more efficient.
Reply
#3
(09-20-2013, 09:25 AM)bradp Wrote: I usually do this emperically. Just activate another axis and have it call the algorithm. As long as you disable the FFE (ixx11) the amp fault and position limits (ixx24) the motor will go to closed loop state and then it is calling the routine and you can see how much this adds.

I expect it will be at least half again what you are seeing as the compiled open servo code is not so efficient. It is good to test your logic but in the end you might need to write directly in assembly code as this is much more efficient.

After several weeks of test, we finally have a real good candidate controller for our application. Unfortunately, the CPU time is quite high, and the thumb-rule that you explained is not working. In a single axis operation I have a Servo Interrupt CPU time of 14.9%. When I create the fake secondary axis, the servo Interrupt CPU time goes up to 27.3 %. Unfortunately, the system has 6 axis, so I expect that the servo time for all the system will go up to 90%. I think this is indicating that the open servo needs to be written directly in assembly.

ciao
gigi
Reply
#4
It does sound that way. If you want you can mail me the code and I can take a quick look to see if I see any optimizations that could be done. It does not sound like this could bring enough extra time but I am willing to look and be sure.

Other than that I can pass the developement environmet and some samples for the assembly coding.
Reply
#5
(01-09-2014, 03:38 AM)bradp Wrote: It does sound that way. If you want you can mail me the code and I can take a quick look to see if I see any optimizations that could be done. It does not sound like this could bring enough extra time but I am willing to look and be sure.

Other than that I can pass the developement environmet and some samples for the assembly coding.

ok, thanks for your help.
Reply
#6
(01-09-2014, 03:38 AM)bradp Wrote: Other than that I can pass the developement environmet and some samples for the assembly coding.

Hi Brad
I am in front of the Hexapod system where I plan to put the Open Servo algorithm. It looks like that the CPU load is almost linear with the number of axis activated: the load on one axis is around 12% and on the full 6 axis is 72 %.
This is only the Servo Task load. Please keep in mind that the UMAC is also running forward and inverse kinematics as well as several watchdog PLCCs.
On the Turbo PMAC manual it is stated "It is generally recommended that the portion of processor time devoted to phase and servo tasks not exceed 50%, in order to allow sufficient time for lower-priority tasks, such as motion program and PLC program calculations. To find out how much processor time an Open Servo algorithm occupies, refer to the section Evaluating Turbo PMAC’s Computational Load."

Is that valid also for this UMAC / Turbo PMAC 2? Should I be concerned about that load?

thanks
gigi
Reply
#7
UMAC is a Turbo PMAC so the statement holds.

Although some people run with CPU higher than 50% when you get to that point you must carefully evaluate how the entire system behaves because you are in the region where tasks might get starved of time. All this causes is run time errors and slow communication and slow execution of PLCs/PLCCs and perhaps a watchdog. So you must be sure that when everything else is going the system is robust for the application you have written. Our statement means that you are in the zone where such faults are more likely to occur.
Reply
#8
(03-21-2014, 01:26 AM)bradp Wrote: UMAC is a Turbo PMAC so the statement holds.

Although some people run with CPU higher than 50% when you get to that point you must carefully evaluate how the entire system behaves because you are in the region where tasks might get starved of time. All this causes is run time errors and slow communication and slow execution of PLCs/PLCCs and perhaps a watchdog. So you must be sure that when everything else is going the system is robust for the application you have written. Our statement means that you are in the zone where such faults are more likely to occur.

So, I ran some test last week on the Hexapod machine, and the results are not promising.
Here's a brief description of what we tested on the system, we'd like to have your opinion on that, because what we saw was at some point really scary.

Apart from the OpenServo, the system is loaded with:
- kinematics computation
- 1 PLCC that performs the forward kinematics, to compute actual hexapod position from actual position of the 6 encoder
- 2 PLCCs that compute the variance of the following error, used as watchdog to stop oscillation (1 PLCC for fast oscillations, 1 PLCC for slow oscillations with a different timing)
- 2 PLCs that monitor external sensor variables (ADC input) and in case stop the motion
- 2 PLCs for communication (one checks if a command has been sent, one to move the variables to return to the host in the proper register position)


First test: we run the hexapod with the Open Servo controller that I tuned on a single axis and run on my lab for several weeks. The servo CPU load is 72% for all the 6 axis. I sent small motion commands to the hexapod, at it behaves correctly: the system stayed in close loop for several minutes without any sort of error.

Second test: because of the "50% suggested limit" we removed some filters on the OpenServo algorithm and we used it as a carbon copy of the standard UMAC PID (just to test that if it behaves like the regular PID).
The CPU power consumption for the 6 axis is 47%. Again, the system stayed in close loop and has been moved for several minutes without any error.

We then tryied to perform a PVT test that help us in measuring the transfer function of the actuator; it is a constant velocity command with an overimposed noise. The PVT file is generated by a Matlab script.
To perform this test, the kinematics is turned off and all the actuators are assigned the X axis, so all the actuators are moved simultaneously. This test has been performed hundreds times in the past, so we are confident on the correct execution of the PVT command. The PVT program is quite long (5000 lines); the code of the PVT test is the following:

Code:
CLOSE
END GATHER
DELETE GATHER
DELETE TRACE
OPEN PROG 21 CLEAR
ABS
PVT2
X 0.000000 : 100.000000
X 0.200000 : 100.000000
X 0.400000 : 100.000000
X 0.600000 : 100.000000
X 0.800000 : 100.000000
X 1.000000 : 100.000000
X 1.200000 : 100.000000
X 1.400000 : 100.000000
X 1.600000 : 100.000000
X 1.746460 : 74.844406
X 1.896149 : 196.944243
X 2.290038 : 72.480201
X 2.434998 : 58.546872
X 2.552092 : 95.639992
X 2.743372 : 163.561903
X 3.070495 : 42.032994
X 3.154561 : 107.092339
...

Here begins the strange behavior of the system: at first, we tried to run the PVT test on the 6 actuators with teh OpenServo enabled on all of them. The maneuvre did not succeed because some actuators went in following error right after the PVT commands had been issued.
We then decided to load the OpenServo only on 1 actuator, in order to have the smallest CPU load possible. The total ServoTime measured was 8 %.
We run again the PVT test, and here the system started to behave crazy. In attach you can see a couple of plot showing positions and currents gathered t servo-loop rate directly on the UMAC.
   

The actual position (red line), measured by the encoder, has several points at 0.0 (not even a bit set to 1) and 1 point at -2.8e8 (basically -inf, that number cannot exist). These measured skipped all the low-level safety systems such as the Encoder Conversion Table filtering, Following Error, etc.

   
The corresponding currents measured are quite scary too: the plot shows the commanded and actual currents as read by the ADC. It seems that the ADC read two points at full-scale (30A) and the commands has several points at 0 A or out of the limit imposed on the Open Servo (7A, clearly visible in the flat part of the plot). These kind of error appeared only in the first part of the PVT maneuvre; the remaining part of the test (10seconds) is ok.
Because the hexapod is mounted on the final system with all the other instruments, we obviously decided to go back to the standard PID configuration and do not explore the Open Servo on the hexapod system until we got a description on what happened on the machine.

The OpenServo that generated that error is the following.
FYI, we decided to go to the openservo solution because we needed:
- an extra D action with a notch filtering
- a velocity feedforward with a limited threshold
- an extra filtering while the hexapod is inposition

We'd like to know if and how we get this malfunction of the UMAC: is it an error on our side due to CPU load? The PVT with 2ms pacing is too fast and it takes too much CPU time? Is there any other way to perform the filtering that we need on the PID?

In the code that follows, we use only the standard PID branch as implemented in the UMAC (all the rest is commented).
Code:
CLOSE
END GATHER
DELETE GATHER
DELETE TRACE
#define AddrOffset        P3000    ; offset of the address, used to address all the motors in the proper variables (P3100 to P3199 for motor 1, etc)
#define MaxOut            P3004    ; output saturation limit
#define ActualOut        P3005    ; actual output (temporary variable)
#define FFGain            P3009
#define FFLim            P2999

#define FEOffset        2        ;
#define ServoCycle L0
L0->X:$0,0,24,S ; Servo cycle counter
#define LastServoCycle L1[MTRNUM-1]
L1->X:$010400[32] ; Register array in UBUFFER
#define ServoExtension L2
L2->Y:(R1-$21)
#define PIDout F1[MTRNUM-1]
F1 ->L:$010000[32]            ; size 32 is MANDATORY!!
#define Dout F2[MTRNUM-1]
F2 ->L:$010020[32]
#define Mout F3[MTRNUM-1]
F3->L:$010040[32]
#define PREVouta0 F4[MTRNUM-1]
F4->L:$010060[32]
#define PIDa2 F5[MTRNUM-1]
F5->L:$10080[32]
#define PIDa3 F6[MTRNUM-1]
F6->L:$100A0[32]
#define PIDb1 F7[MTRNUM-1]
F7->L:$100C0[32]
#define PIDb2 F8[MTRNUM-1]
F8->L:$100E0[32]
#define PIDb3 F9[MTRNUM-1]
F9->L:$10100[32]
#define Da2 F10[MTRNUM-1]
F10->L:$10120[32]
#define Da3 F11[MTRNUM-1]
F11->L:$10140[32]
#define Da4 F12[MTRNUM-1]
F12->L:$10160[32]
#define Db1 F13[MTRNUM-1]
F13->L:$10180[32]
#define Db2 F14[MTRNUM-1]
F14->L:$101A0[32]
#define Db3 F15[MTRNUM-1]
F15->L:$101C0[32]
#define Db4 F16[MTRNUM-1]
F16->L:$101E0[32]
#define PREVouta1 F17[MTRNUM-1]
F17->L:$010060[32]

OPEN SERVO
CLEAR
COPYREG P3010    ; P3010 to P3015 are {ActualVel; DesiredVel; FollowingError; ActualPos; DesiredPos} units are the same as ESA, so FE is (1/(1xx08 * 32)) cts
; reset out e input variables: OPTIMIZE THIS!!!
IF (ServoCycle-LastServoCycle!=ServoExtension+1)
    PIDa2 = 0
    PIDa3 = 0
    PIDb1 = 0
    PIDb2 = 0
    PIDb3 = 0
    Da2 = 0
    Da3 = 0
    Da4 = 0
    Db1 = 0
    Db2 = 0
    Db3 = 0
    Db4 = 0
ENDIF
LastServoCycle=ServoCycle ; Store for next cycle




; P path
;    numerator * FolError
PIDout = (P3012 * 2.885683)  + (PIDa2 * -5.388260) + (PIDa3 * 2.503395) + (PIDb2)
;    shift prev errors
PIDa3 = PIDa2
PIDa2 = P3012
;    shift prev out
;    set prev P output  
PIDb2 = PIDout;


; Derivative path
;    numerator * FolError
; Dout = (P3012 * 2.503395) + (Da2 * -7.346922) + (Da3 * 7.346922) + (Da4 * -2.503395) - (Db2 * -1.818224) - (Db3 * 0.879512)
; Dout = ((P3012-Da4) * 2.503395) + ((Da3-Da2) * 7.346922) - (Db2 * -1.818224) - (Db3 * 0.879512)
;    shift prev errors
; Da4 = Da3
; Da3 = Da2
; Da2 = P3012
;    shift prev out
; Db3 = Db2
;    set prev D output  
; Db2 = Dout; take only previous D out         ; set actual out as prev out for the next cycle


;End of servo loop

; ========== modding on assist
;if (M(ITOF(MTRNUM * 100)+40) != 0)                                              ; != 0 invece che == 1 fa risparmiare 0.01%
;    Mout = PREVouta0 * 1.9 - PREVouta1*0.9025 + PIDout * 0.0025                    ; check this filter in matlab as tf(0.1, [1 -0.9], 1/2250, 'variable', 'z^-1')
;else
;    Mout = PIDout + FLIMIT(P3011 * 50, 179200); + Dout                            ; extra vel FF with 1 A limit
;endif
;PREVouta1 = PREVouta0
;PREVouta0 = Mout
; ========= end modding on assist

Mout = PIDout

RETURN(FTOI(FLIMIT(Mout, 1488640)))        ;
CLOSE
Reply
#9
did you look at any status bits such as servo error and real time interrupt re-entry? I would not be surprised to see servo error coming on meaning the servo code is not always finishing in one servo cycle.
Reply
#10
(03-25-2014, 05:03 AM)bradp Wrote: did you look at any status bits such as servo error and real time interrupt re-entry? I would not be surprised to see servo error coming on meaning the servo code is not always finishing in one servo cycle.

No, I'm sorry, I had not look at the servo error flags. But is it possible that an error on the servo generates such a random behavior on the current and encoder readings? I don't understand why, with a 8% load, the CPU has crashed under the PLCCs and PVT maneuvres.

ciao
gg
Reply
#11
I think it is possible. I have not gone through all your code and at the moment I can not. But a quick look tells me that if your code skips a servo cycle then the first IF condition will set data to zero when you were not expecting it. Can this cause such a problem? It is never good when the servo skips.

Since this does not happen when you use the built in servo then it is related to the open servo somehow. And since it does not happen until you run this program it seems related to CPU load, otherwise if it was just the open servo code it would always happen. So CPU and open servo bring the question if the servo is skipping.

It does not look like you are using the PMAC plotting. What are you gathering and how are you recording it? This might give some clues.
Reply
#12
(03-25-2014, 07:20 AM)bradp Wrote: It does not look like you are using the PMAC plotting. What are you gathering and how are you recording it? This might give some clues.

These are the addresses that I gather and are plotted in the previous post:
i5001=$80008B // Actual Position i5002=$8000A5 // Commanded position
...
i5005=$4000BF // I command value
i5006=$4000B9 // Actual current
Reply
#13
Can you look into the actual data gathered and uploaded from Pmac before any processing was done? Since data gathering happens in the servo task if a servo skipped the gathering could also skip for that cycle. In that situation I imagine you would just not have any data points and you would see in the data which usually has the servo count included a jump in the servo count that is too large and would indicate the gathering did not happen at that cycle.
Reply
#14
(03-26-2014, 01:33 AM)bradp Wrote: In that situation I imagine you would just not have any data points and you would see in the data which usually has the servo count included a jump in the servo count that is too large and would indicate the gathering did not happen at that cycle.

I am looking at the servo count included in the data gathering: the Time coordinate on the plots is exactly the servo count scaled into Seconds for more readability.
It seems that the gathering has been performed without problems: the time frame between each measure is constant during all the test (1 servo count each datum gathered).
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)