Thread Rating:
  • 2 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Manually Scheduling Threads in Power PMAC
#1
Attached is a document that explains how to manually schedule POSIX threads in Power PMAC by means of a C Application. This is helpful if the user wants periodic, multithreaded applications that run at a different interval than the default intervals offered by Power PMAC's RTICPLCs, Background CPLCs, Servo Routines, and Phase Routines.

Much credit to KEJR, an active contributor to our Power PMAC forum, for helping so much with this.

If something in the document is not clear, please feel free to post your questions.

UPDATE (11/8/11): Added the source and header files (inside customthreading.zip below) for download so you do not have to copy/paste from the document.

UPDATE (12/16/11): Updated the source code and instructions in the .pdf to include an AdditionalStackBytes parameter in CreateThread(). With this parameter, the user can add bytes to the thread's default stack allocation size if needed (i.e. if the thread will use more than 1065410 bytes of stack), in order to prevent "program termination and/or corrupted data" when the thread's allocated stack size is exceeded as this POSIX tutorial says.

UPDATE (12/22/11): Fixed a typo in the passing data to threads "Through Structures" example.


Attached Files
.pdf   Manually Scheduling Threads in Power PMAC.pdf (Size: 512.39 KB / Downloads: 343)
.zip   customthreading.zip (Size: 2.62 KB / Downloads: 128)
Reply
#2
You are welcome Charles, it was good working with you on this.

I just wanted to add that in addition to being able to schedule different thread periods this code has the following advantages:

- It cleans up the complex pthread functions.
- Foundation of your own company code library.
- All your threads can run in the same code base/memory space.
- You can organize your thread files how you want in the IDE.
-----> i.e. one mega code file or individual broken up files.
- You can easily use global C variables across your app.
- Your multithreaded RTOS app can be one executable.
----->Easy to copy, download, backup, etc.

Note: Please read up on mutex locks when using global vars across different threads (especially writes)! With great power comes great responsibility!

Thanks to Delta Tau for allowing us the keys to this race car. I think this has allowed us to do some cool things that you usually have to resort to using an embedded RTOS for and the fact that you get a world class motion controller included is a real perk. ;o)

KEJR
Reply
#3
Bump for 12/16/11 update (see first post in this thread)
Reply
#4
I just wanted to give other users a heads-up that the stack size change was failing in my code. For some reason even though we were using the POSIX macro for the min stack size it was failing to set the stack properly and going back to the 8mb stack size default. I tried modifying the code to use a 1mb stack size and it worked see below:

Code:
size_t stacksize = PTHREAD_STACK_MIN + 1048576;//1meg for now...

//Set Stack size to something reasonable
SysAdv("set stack size returned: %d", pthread_attr_setstacksize(&attr, stacksize));

of course, in the end it should be set to something like this:

Code:
if(pthread_attr_setstacksize(&attr, stacksize))
    SysError("Problem setting stack size for thread: %s", ThreadName);

I was OK with the 1mb stack but it is overkill for most apps. If anyone wants to try setting the stack size down until it stops working then it woudl be good to know where the real min value is. I'm not sure why PTHREAD_STACK_MIN is not giving us an acceptable value.

KEJR
Reply
#5
This is a very useful document. Could you comment on the necessity of using SCHED_FIFO and priority=49?

I have inherited some code that creates a number of threads in a Background C Program in order to log to the disk and also to communicate over a socket to a remote computer. This code does not call pthread_setschedparam() to change the scheduler or priority. This seems like it should be OK, because these tasks are not time critical.

Could this cause us trouble?
Reply
#6
(06-19-2014, 12:18 AM)smr99 Wrote: This is a very useful document. Could you comment on the necessity of using SCHED_FIFO and priority=49?

I have inherited some code that creates a number of threads in a Background C Program in order to log to the disk and also to communicate over a socket to a remote computer. This code does not call pthread_setschedparam() to change the scheduler or priority. This seems like it should be OK, because these tasks are not time critical.

Could this cause us trouble?

The FIFO scheduler is just a function of the Xenomai RTOS scheduling method. The first thread to be available at the current priority level will get run in the order that they became ready to run (as they come out of sleep or mutex, etc).

It is hard to say for sure without knowing your application, but you probably don't have to worry about the RTOS functionality from what I'm hearing. If all you are doing is IO calls (file, socket) then you will be getting kicked out of the RTOS (Xenomai) scheduler while those calls are made (That is if you set up the thread to run as hard realtime). By default the background C programs run with the Linux scheduler which is ideal for logging and socket IO.

As with any multiprocess/multithreaded system, however, remember that you will have all of the issues with thread syncronization (or lack of). Particularly be wary of the classical "Read-Modify-Write" Scenario if both a realtime/PMAC thread and your background linux thread are modifying the same shared memory. In PPMAC terms you would not want a C PLC incrementing a global "P" variable and a background C program incrementing the same variable because they will trample on each other if no care is taken to safeguard this. You probably don't have this situation, but I wanted to mention it.

The fortunate thing is that the PPMAC will read an entire 64 bit memory location in one clock cycle, so there is not a problem with mixing up bytes in your words because the CPU has to fetch it in two instructions (Potentially getting interrupted between instruction cycles and having that data word get updated by the higher priority thread). Because of this, make sure that any array indexing you do on shared memory is done with 'double' types instead of 8/16/32 bit char/int/float types. I had a problem where I was having a non realtime background thread periodically backing up blocks or shared memory to disk and I started out with storing it as bytes and I was getting corrupted data. I went to fetching as 64 bit words and the problem went away. If you want to learn more about that I think this is in another thread on this site.

The thread library I developed was because I prefer to have precise control over my scheduling and it allows me to interact with all of my threads in one process space with access to mutex, sockets, file IO, etc. (For my applications accessing IO can be dropped down to lower priority as it is mostly error handling when things have stopped). I can do things like have one piece of code run multiple identical pieces of hardware, and do almost all of my code without writing a single script or C PLC. For me this is cleaner but YMMV.

KEJR
Reply
#7
(06-19-2014, 04:52 AM)KEJR Wrote: It is hard to say for sure without knowing your application, but you probably don't have to worry about the RTOS functionality from what I'm hearing. If all you are doing is IO calls (file, socket) then you will be getting kicked out of the RTOS (Xenomai) scheduler while those calls are made (That is if you set up the thread to run as hard realtime). By default the background C programs run with the Linux scheduler which is ideal for logging and socket IO.

The application in question is basically a bridge to our HMI running on a different machine -- thus the socket -- and yes, it is mainly doing logging and socket I/O. We use the pushm to send commands down to the RTI and servo routines (all written in C) and read back status. Additionally, we have a pair of FIFOs sending data streams from RTI to the bridge process. Apart from pushm and a couple other calls to start/stop the RTI, the code looks like a standard POSIX program.

Given that, I was quite surprised that its threads show up in /proc/xenomai/sched at all. As a guess: I suspect the main thread is registered due to something in InitLibrary(), while the other three are due to wrapping of pthread_create(). They all appear as PRI=1, which intrigues me because "gpascii" instances have PRI=0.

The reason I was asking about threads is that our bridge suffers from lockups, sometimes followed by a hard watchdog. During one lockup, with the help of /proc/PID/status and /proc/PID/wchan, I was able to determine that the process was stuck in the system call "xnshadow_harden" (a Xenomai call). As I say, I was quite surprised that these threads had anything to do with Xenomai. But /proc/xenomai/stat shows that 3 of the 4 threads are performing a huge number of mode switches -- almost the same number as the context switches -- which probably explains the call to xnshadow_harden(). Since it was stuck in the syscall, I wasn't able to get gdb to tell me anything useful (it didn't succeed to break into the program).

In another thread (http://forums.deltatau.com/showthread.php?tid=1654) I posted about the fact that calls in the bridge itself are wrapped but calls in a shared library are not. In particular, the bridge main thread uses pselect() on the FIFO and socket file descriptors. The FIFO read() calls are done in the bridge (hence wrapped) but the socket send/recv are done in the library and are not wrapped. For that matter, the FIFO write() calls in the RTI CPLC are not wrapped either. Have you any thoughts on whether this is a problem?

(06-19-2014, 04:52 AM)KEJR Wrote: I had a problem where I was having a non realtime background thread periodically backing up blocks or shared memory to disk and I started out with storing it as bytes and I was getting corrupted data. I went to fetching as 64 bit words and the problem went away. If you want to learn more about that I think this is in another thread on this site.

Are you referring to http://forums.deltatau.com/showthread.php?tid=652 ? That was an informative thread, thanks!
Reply
#8
(06-20-2014, 12:10 AM)smr99 Wrote: The application in question is basically a bridge to our HMI running on a different machine -- thus the socket -- and yes, it is mainly doing logging and socket I/O. We use the pushm to send commands down to the RTI and servo routines (all written in C) and read back status.

Is your RTI code doing alot of servo stuff, or are you using it to just get fast updates (monitoring, etc)? The reason I mention is that you can do some pretty fast processing using the background C threads and you can have it all in one protected memory space.

I wonder if Xenomai needs to schedule the threads simply because they are using the shared memory and/or FIFO facilities. This might explain your entry in the xenomai scheduler.

I can't comment about the makefile --wrap. I can only assume that they are needed on some files but not others due to the include files and libraries linked in. I trust Shansen's replies in the other thread, he is definitely one of the more knowledgable members here regarding the make process on the PPMAC as he has done his own make process in Eclipse. (I'm tempted to go there, but cling to the support of the PPMAC IDE.... :o)

Yeah, you found the atomicity thread. I never knew CS410 would be so useful at the time I took the course! Knowing your thread synchronization and potential dead lock conditions are crucial in these applications. :o)

I'd encourage you to hook up a scope and start pulsing some outputs at different places in your threads (preferably different outputs per thread, etc). A multichannel scope or logic analyzer can be helpful to see if your code is really jittery or if is executing at times when you don't expect it to. Do you have a sleep call in your threads, or do they block on a FIFO read or something like that? Try to look in your code to see if somehow you might be starving something in a busy loop. I know these are basic things, but I can't think of anything else to try.

Have you thought about using GPASCII for much of your HMI updates? I have found that I can poll GPASII for a few dozen things and the update rate is really good. I tend to only update numbers in my HMI that are on the active page so it limits the traffic vs. sending all the data back and forth for the application (whether it is shown or not). It sounds like you have a ton of code existing so I know it might be difficult to look at different alternatives but sometimes you have to weigh the redesign vs. fix-existing dilemma.

How is your code reading and writing? If it is going in and out of realtime that much I would suspect that it is reading one value from FIFO/SHM and then writing it out to Socket/File IO in a loop or something like that. I'd try reading in all data to a memory buffer and then spitting it out to the IO resource after it is all captured. In this way if it is in fact going in and out of realtime at least its once per update cycle. I'm not sure if this would apply to shared memory, but maybe I'm sure the FIFO has some synchronization that needs the xenomai scheduler. I'm kind of grasping at straws, but this is the kind of thing I would start to look given the info you gave me.

Good luck.

Ken
Reply
#9
Hello CharlesP,

Would you share the project which can be made by "Manually Scheduling Threads in Power PMAC.pdf" step by step? If you have! Not only the most important two files.

I try that as "the Manually Scheduling Threads in Power PMAC.pdf",but IDE give me some error. So a complete project will a good beginning of me.

Thanks
Reply
#10
When build the project,IDE tell me some error happenned:
Syntax error,unexpected '.'
Maybe caused by this line:
Void sendf(int sendportnum , const char *fmt, ...)

What should I do?
Reply
#11
Hi,

Can you send me the full code that you actually tried to download? Please email it to support at deltatau dot com.

Also please make sure you are using the latest IDE and Firmware.

Also note that you capitalized "Void", which you must not do.
Reply
#12
Hi dzrong,
If you are talking about the error in the IDE code window (shown by the red underline) - the syntax checker in the IDE is incorrectly flagging the "..." as a syntax error. This will still build and download normally.
Reply
#13
Yes,you are right.If this file is closed in IDE,no error happenned.
Reply
#14
One of the firmware engineers asked me to post this in response:

You need to make the C APP a non RT type as shown in the IDE Sample Projects. Specifically the following code:

struct sched_param param;
struct timespec sleeptime = {0};
sleeptime.tv_nsec = NANO_1MSEC; // #defines NANO_5MSEC & NANO_10MSEC are available

//----------------------------------------------------------------------------
// Required Startup Code: Insures that this APP is run as an RT or NON-RT APP
// otherwise depending upon how it is started it will inherit the scheduling
// priority of the task that started it.
//----------------------------------------------------------------------------
#ifndef RUN_AS_RT_APP
//-----------------------------
// Runs as a NON RT Linux APP
//-----------------------------
param.__sched_priority = 0;
pthread_setschedparam(pthread_self(), SCHED_OTHER, &param);
#else
//----------------------------------------------------------------------------
// Runs as a RT Linux APP at the same priority as the Background Script PLCs
// To run at a lower priority use BACKGROUND_SCRIPT_PLC_PRIORITY - 10
// To run at a higher priority use BACKGROUND_SCRIPT_PLC_PRIORITY + 1
//----------------------------------------------------------------------------
param.__sched_priority = BACKGROUND_RT_PRIORITY - 10;
pthread_setschedparam(pthread_self(), SCHED_FIFO, &param);
#endif
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)