Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Writing data to file using the gather buffer
#1
Hi,

I'm trying to make a gather function that keeps on running and adding the data to a txt file. The code I have in the background c app is this:

Code:
fpGather = fopen("/var/ftp/gather/gather.txt", "w");

while (!done)
    {
        pGatherBuffer = (char *)(pshm->Gather.Buffer);
        samples = pshm->Gather.Index;

        if(pshm->P[xsaveValues] == 1)
        {
            for (i = 0; i < samples; i++)
            {
                memcpy(&time, (unsigned char*)pGatherBuffer, sizeof(int));
                pGatherBuffer += sizeof(int);
                fprintf(fpGather, "%d \r\n", time);
            }
            pshm->P[xsaveValues] = 0;
        }
    }

The value pshm->P[xsaveValues] is changed in my servo loop so it is only updated when the loop is used.

the gather values are the following:
Code:
gather.addr[0] = sys.servocount.a
gather.items = 1
gather.period = 1
gather.maxlines = 1048576
gather.maxsamples = 1048575
gather.enable = 2

The problem I have is that the servo loop runs at 5kHz, so it will update 5000 times a second. I timed the saving of values and in 10 second I got a .txt file of 53 MB with about 5.6 million times the servocount printed, where I excpected 50.000 lines.

The first value I get is 2.453.738 which is normal, and it goes up by one until it reaches 3.373.594 on line 919857. After that, on the next line it starts again at 2.453.738 until it reaches 3381841. This pattern continues until the last line where it ends at the highest value of 3.415.154.

I don't know where it goes wrong, to me it looks correct. Maybe someone has seen this before and knows how to solve it.
Reply
#2
It is unclear to me why you are doing this. In the other post about this subject (http://forums.deltatau.com/showthread.php?tid=2067) the gather feature was explained about how to stream the gathered data to the standard IO or to a file. Since this is a feature already provided why do you need to recreate the feature in your Capp?
Reply
#3
I tried the thing Charles said, but I didn't see a file anywhere on the file system. When I looked further I saw this on the forum:

(08-27-2014, 02:23 PM)steve.milici Wrote: Currently the only customer supported usage is:
gather_csv [-u] [filename]

The “-u” will stream the contents of the most recent data gather to the standard output as in the current SSH, Telnet or serial port connection. The “-u” with a file name will stream it to that file. I would suggest using the “/var/ftp/usrflash/” directory as this is already writable and will allow ftp uploads without any permission changes. For example:
gather_csv -u /var/ftp/usrflash/steve

This puts the last gathered data in a text file named “steve” in the “/var/ftp/usrflash/” directory where you now have ftp access.

Keep in mind the gather files can be very large.

This is the only thing I could find about how to use the gathered data. This was a while ago so I dont know if there are different way to do this now.

Than I found this tutorial you made to gather data using a capp.

(04-15-2011, 09:27 AM)bradp Wrote: The Gather Buffer is just a series of bytes so to access the Gather Buffer Data from a C App you must know what type of variables you gathered. Then you can access the buffer directly and offset your pointer the correct number of bytes for each element that was gathered.


pGatherBuffer = (char *)(pSharedMem->Gather.Buffer);
samples = pSharedMem->Gather.Index;
*timeConstant = 0.0;

fpGather = fopen ("/var/ftp/gather/autotune_gather.txt", "w");


//--------------------------------------------------------------------------
// Read in gathered data and compute the following error and min max
// following error
//--------------------------------------------------------------------------
for (i = 0; i < samples; i++)
{
memcpy(&t, (unsigned char*)pGatherBuffer, sizeof(unsigned));
pGatherBuffer += sizeof(unsigned);
memcpy(&cmdpos,(unsigned char*) pGatherBuffer, sizeof(double));
pGatherBuffer += sizeof(double);
memcpy(&actpos, (unsigned char*)pGatherBuffer, sizeof(double));
pGatherBuffer += sizeof(double);
memcpy(&dac, (unsigned char*)pGatherBuffer, sizeof(float));
pGatherBuffer += sizeof(float);
fprintf(fpGather, "%u %lf %lf %lf\r\n", time, cmdpos, actpos, dac);
}
fclose(fpGather);
That was why I tried it in a capp.
If there is a way without it I would be thrilled, but the way Charles explained it, didn't work for me.
Reply
#4
I wondered about what was written. The command is not gather_csv it is just gather. So you should do gather -u /var/ftp/gather/ggg.txt to send the already gathered data from the structures inside the ppmac to the file ggg.txt.

I just tested this and it is working. My test was simple. I open the plot program and gathered some data and plotted it to be sure the gather worked. Then I ran the command and saw my file was created. Then I listed the file contents to verify it had the data inside.
Reply
#5
The way you're describing it works for me too, but the problem is that I want to gather more data than the software normaly does. That is why I tried to read directly from the buffer, so that could be overwriten. Is there a way to do that?
Reply
#6
I am not sure I understand what you are saying. What I understand is that you plan to store the data on the PPmac, therefore you can not be planning to gather more data than you have disk space on the PPmac. PPmac gives you 1Mb of internal buffer space for gathering. If you need more than this you must use streaming either to a file on the PPmac disk or to another host on the stdio. See Gather.Enable, Gather.MaxLines, Gather.MaxSamples for a start at looking at how this works. This of course means the streaming must be active while the gather is running.

If you are planning on gathering more than can be stored in the internal PPmac buffer then you must stream the data to a host or a file on disk and this is also done with the gather program but you use Gather.Enable=3 then you start the gather {filename} and you must make sure not to overflow the disk. Note in this streaming mode the -u option is not used.
Reply
#7
That worked, thank you for your help.

As a side note, I don't know if I could't find it or if it's not available, but I could find not much documentation on the Gather function. I could find that it was available, but not how to use it. This guide(http://forums.deltatau.com/showthread.php?tid=99) did help, but that was about the most information I could find.
Reply
#8
These type of "appware" that we have in the controller are not well documented. Most only have a limited embedded help with the --h runtime switch. Some do not have that. I have already mentioned this to the documentation dept. based on your recent post so a future manual will hopefully have this documented.
Reply
#9
(11-12-2015, 03:09 AM)bradp Wrote: If you are planning on gathering more than can be stored in the internal PPmac buffer then you must stream the data to a host or a file on disk and this is also done with the gather program but you use Gather.Enable=3 then you start the gather {filename} and you must make sure not to overflow the disk. Note in this streaming mode the -u option is not used.

Hi all
this gathering mode is really interesting and useful.
However, I'm trying to automate some programs and plcs, and I cannot make the PPMAC start the streaming gathering from a prog.
Inside of a program I have:
Code:
Gather.Enable = 3;        // start rotary gathering
dwell(2000);
system "gather /var/ftp/usrflash/stepXXX.txt" //
but the gather save only an empty file. I think that's because the gather software must run without interruptions in order to work in background.
I tried all the following commands without success:
Code:
system "nohup gather /var/ftp/usrflash/stepXXX.txt"
system "gather /var/ftp/usrflash/stepXXX.txt & "
system "nohup gather /var/ftp/usrflash/stepXXX.txt & "
system "nohup gather /var/ftp/usrflash/stepXXX.txt \046 "
am I missing some information regarding the system command?

thanks
gigi
Reply
#10
gigi,

I just tested and it works as described.
1. I first went to the plot program and configured my items to gather.
2. I then test by pressing Enable Gather, wait a bit, Stop, upload and plot. I see I get the data correctly.
3. In the terminal I enter Gather.Enable=3
4. Then from a puTTy SSH session and enter gather /var/ftp/Temp/g1.txt
5. I wait a bit and enter gather.enable=0 from the IDE terminal
6. I look at the file on the ftp and see it has data.

This is not a perfect test but shows the function works. You should make sure this works for you before we go further. It might give some hint.
Reply
#11
Hi Brad
(06-30-2017, 07:22 AM)bradp Wrote: gigi,

I just tested and it works as described.
1. I first went to the plot program and configured my items to gather.
2. I then test by pressing Enable Gather, wait a bit, Stop, upload and plot. I see I get the data correctly.
3. In the terminal I enter Gather.Enable=3
4. Then from a puTTy SSH session and enter gather /var/ftp/Temp/g1.txt
5. I wait a bit and enter gather.enable=0 from the IDE terminal
6. I look at the file on the ftp and see it has data.

sorry, I think I explained myself wrong. The procedure you are describing is the one that I am using right now, and it is working.
However, I'd like to start the gathering from a PROG or a PLC, not using SSH. And here comes my problem with the SYSTEM function: it seems that stop the execution of 'gather', even if I use the linux parameters to run in background.

Thanks
gigi
Reply
#12
looks to me like the system command in firmware 2.3.1.0 and perhaps earlier is broken in a program. Or I just do not know how to use it and neither does the SRM. Anyways I am testing on a motion machine. Other ppmac types might behave differently. Here is the problem I see. I do as the manual explains

open plc 1
Gather.enable=3
p11=Sys.Time+2
while(p11>Sys.Time){}
system "/opt/ppmac/gather_csv/gather_csv /var/ftp/Temp/g5.txt"
p11=Sys.Time+5
while(p11>Sys.Time){}
Gather.enable=0
disable plc 1
close

But the file g5.txt never gets created. So I do a workaround since from the IDE terminal the file would get created. I do this.

open plc 2
Gather.enable=3
p11=Sys.Time+2
while(p11>Sys.Time){}
cmd"system /opt/ppmac/gather_csv/gather_csv /var/ftp/Temp/g5.txt"
p11=Sys.Time+5
while(p11>Sys.Time){}
Gather.enable=0
disable plc 1
close

Now the file g5.txt is created.

But either from the terminal or from PLC 2 the file g5.txt is always 0 bytes.

I can not go further with this. Someone at ODT factory will need to test and look at this and tell use how to make this work from a script PLC.
Reply
#13
You will need to "build" the string and then "execute" as in the following:
cx p1=sprintf(256, "/opt/ppmac/gather_csv/gather_csv /var/ftp/usrflash/Temp/g5.txt")
cx system "%s", 256
Reply
#14
Corrections / additions:

don't forget the "-u" switch:
cx p1=sprintf(256, "/opt/ppmac/gather_csv/gather_csv -u /var/ftp/usrflash/Temp/g5.txt")
cx system "%s", 256

And when using the "Plot" application to set it up make sure to set "Legacy Mode" to create the old style gather parameters setup file.

Also don't rely on Windows "IE" or "Explorer" to view or copy the files as it LIES. Use a good ftp or sftp client like FileZilla to see the data in the generated file. I just tried this on my UMAC 465, 2.3.1.0 and it worked well.
Reply
#15
Actually it should be OK without "Legacy Mode".
Reply
#16
the -u is for upload. The original question was for streaming. And in streaming the -u should not be used as was told.
Reply
#17
just tested. This building the string solution does not work on motion machine. What did you test with? What firmware?
Reply
#18
(07-01-2017, 11:58 AM)bradp Wrote: just tested. This building the string solution does not work on motion machine. What did you test with? What firmware?

I also cannot use the CX command since it is only a online command, I need to put that on a Prog.

Thanks
ciao
gigi
Reply
#19
Here's my latest findings: if I use the following code in a prog, the behavior is the one that I want, but the gather function is using 150% of CPU, making the IDE pretty unstable

Code:
system "nohup gather /var/ftp/usrflash/stepscript.txt > /dev/null  <&- &"

This is equivalent in commanding on the shell:
Code:
nohup gather /var/ftp/usrflash/stepscript.txt > /dev/null  <&- &
and that's using only 70% of CPU time. Any idea on why the prog is doing that?

thanks
ciao
gigi
Reply
#20
I will test this on IPC today.
Reply
#21
(07-05-2017, 09:22 AM)steve.milici Wrote: I will test this on IPC today.

Hi Steve
do you have any news on that topic?

thanks a lot

Bests
gigi
Reply
#22
This issue should be fixed on the next firmware release Vs 2.3.2.5.

This should be within the next few days.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)