FileWriter

In the previous sections, application monitoring was based on the use of the Logger, which is a powerful tool to log any information from the application, but it has some obvious limitations. Namely, it does not store the files and the output can quickly become overwhelming if the application is running for a long time.

In this section, data will be stored in a file using the FileWriter DataSource. This DataSource can be configured to store the data in a binary or text CSV format.

Many MARTe2 data sources, including the FileWriter data source, have decoupling mechanisms so that the writing of data does not interfere with the real-time execution of the application.

These mechanisms typically involve buffering the data in memory and writing it to the file asynchronously on a separate thread (which may be allocated to a different CPU core).

In this example, data is stored in a CSV file, which can be easily opened with spreadsheet software or plotted with any plotting tool (e.g. gnuplot). The configuration of the FileWriter DataSource is shown in the following listing.

FileWriter configuration.
 1        +FileWriter = {
 2            Class = FileDataSource::FileWriter
 3            NumberOfBuffers = 100
 4            CPUMask = 0x1 
 5            StackSize = 10000000 
 6            Filename = "../Test/Integrated/MassSpring-18.csv" 
 7            Overwrite = "yes" 
 8            FileFormat = "csv"
 9            CSVSeparator = "," 
10            StoreOnTrigger = 0 
11            RefreshContent = 0 
12            Signals = {
13                Time = {
14                    Type = uint32
15                }
16                ReferencePosition = {

Note

Many of these data sources also have the built-in capability of writing the data based on an external trigger, also allowing specification of the number of pre- and post-samples to be stored when the trigger occurs.

Running the application

Start the application with:

./MARTeApp.sh -f ../Configurations/MassSpring/RTApp-MassSpring-18.cfg -l RealTimeLoader -s State1

Once the application is running, inspect the screen output and verify that the application is running without any issues. The log should show entries similar to the following:

$ [Warning - Threads.cpp:181]: Failed to change the thread priority (likely due to insufficient permissions)
$ [Information - RealTimeLoader.cpp:111]: Started application in state State1
$ [Information - MARTeApp.cpp:135]: Application starting
...

Open another terminal and check the content of the log file (e.g. ../Test/Integrated/MassSpring-18.csv) and verify that the data is being stored in the file as expected. The log should show entries similar to the following:

less ../Test/Integrated/MassSpring-18.csv

#Time (uint32)[1],ReferencePosition (float64)[1],Position (float64)[1],PositionDisturbed (float64)[1],PositionFiltered (float64)[1],PositionM (float64)[1],Velocity (float64)[1],VelocityM (float64)[1],PositionErr (float64)[1],Force (float64)[1],Thread1CycleTime (uint32)[1],GAMTimer_ReadTime (uint32)[1],GAMTimer_ExecTime (uint32)[1],GAMTimer_WriteTime (uint32)[1],GAMReference_ExecTime (uint32)[1],GAMReference_WriteTime (uint32)[1],GAMController_ReadTime (uint32)[1],GAMController_ExecTime (uint32)[1],GAMController_WriteTime (uint32)[1],GAMSpringMass_ReadTime (uint32)[1],GAMSpringMass_ExecTime (uint32)[1],GAMSpringMass_WriteTime (uint32)[1],GAMPerfMonitor_ReadTime (uint32)[1],GAMPerfMonitor_ExecTime (uint32)[1],GAMPerfMonitor_WriteTime (uint32)[1],GAMFileWriter_ReadTime (uint32)[1],GAMFileWriter_ExecTime (uint32)[1],GAMFileWriter_WriteTime (uint32)[1],Thread1CycleTimeAverage (uint32)[1],Thread1CycleTimeMovingAverage (float32)[1],Thread1CycleTimeStdDev (uint32)[1],Thread1CycleTimeMax (uint32)[1],Thread1CycleTimeMin (uint32)[1],Thread1CycleTimeHistogram (uint32)[11],Thread1FreeTimeHistogram (uint32)[11],GAMsExecutionTime (uint32)[1],ForceAverage (float64)[1],ForceStdDev (float64)[1],ForceMax (float64)[1],ForceMin (float64)[1]
 0,2.000000,0,0,0,0,0,0.300000,-2.000000,30.000000,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,{ 0 0 0 0 0 0 0 0 0 0 0 } ,{ 0 0 0 0 0 0 0 0 0 0 0 } ,0,0.150000,2.116010,30.000000,30.000000
 20000,2.000000,0.001496,0.009511,0.009097,0.003000,0.299177,0.598500,-1.990903,30.000000,0,9849,9850,9851,9852,9853,9874,9875,9876,9876,9883,9883,5,7,10,9932,9933,9941,0,0,0,0,0,{ 0 0 0 0 0 0 0 0 0 0 0 } ,{ 0 0 0 0 0 0 0 0 0 0 0 } ,92,0.300000,2.984962,30.000000,30.000000
 ...

The data can be plotted using any plotting tool (e.g. gnuplot) and should show the expected behaviour of the system.

A Python-based plotting script is also available in ../Test/Integrated/plot_mass_spring_csv.py. To run the script, execute the following command:

python ../Test/Integrated/plot_mass_spring_csv.py -f ../Test/Integrated/MassSpring-18.csv -s ReferencePosition Position

Exercises

Ex. 1: Log execution every second

As can be seen when executing the application, the log is now totally silent. Add a GAMDisplay to log the time once per second.

Also make sure that data is only written to the file for the first 10 seconds of the application execution, using the DataSource built-in triggering mechanism.

  1. Edit the file ../Configurations/MassSpring/RTApp-MassSpring-19.cfg and add a GAM to compute the display trigger. Note that the MathExpressionGAM does not implement the mod operator. A possible solution is to divide the Time by the desired period (in uint32) and then multiply the result by the period again. If the result is equal to the original time, then the trigger condition is met.

  2. Add a GAM to trigger the Time into the LoggerDataSource.

  3. Modify the FileWriter DataSource to use the trigger signal to only write data for the first 10 seconds of the application execution.

  4. Modify the FileWriter DataSource to write into a file named ../Test/Integrated/MassSpring-19.csv.

  5. Check that the application is only writing to the display once per second and that data is only being written to the file for the first 10 seconds of the application execution.

./MARTeApp.sh -f ../Configurations/MassSpring/RTApp-MassSpring-19.cfg -l RealTimeLoader -s State1
Solution

The solution is to add a MathExpressionGAM implementing the equation defined above.

MathExpressionGAM to compute the display trigger.
 1        +GAMMathTriggerSecond = {
 2            Class = MathExpressionGAM
 3            Expression = "
 4                TriggerEveryMicroSecs = (uint32)1000000;
 5                TruncatedTime = (Time / TriggerEveryMicroSecs) * TriggerEveryMicroSecs;
 6                TriggerDisplay = (uint8)(Time == TruncatedTime);"
 7            InputSignals = {
 8                Time = {
 9                    DataSource = DDB1
10                    Type = uint32
11                }
12            }
13            OutputSignals = {
14                TruncatedTime = {
15                    DataSource = DDB1
16                    Type = uint32
17                }
18                TriggerDisplay = {
19                    DataSource = DDB1
20                    Type = uint8
21                }
22            }
23        }

To modify the FileWriter DataSource to use the trigger signal, the Trigger parameter needs to be added to the DataSource configuration.

Updated FileWriter with Trigger signal.
 1        +FileWriter = {
 2            Class = FileDataSource::FileWriter
 3            NumberOfBuffers = 100
 4            CPUMask = 0x1 
 5            StackSize = 10000000 
 6            Filename = "../Test/Integrated/MassSpring-19.csv" 
 7            Overwrite = "yes" 
 8            FileFormat = "csv"
 9            CSVSeparator = "," 
10            StoreOnTrigger = 1 
11            RefreshContent = 0 
12            NumberOfPreTriggers = 0
13            NumberOfPostTriggers = 0
14            Signals = {
15                TriggerWriter = {
16                    Type = uint8
17                }
18                Time = {
19                    Type = uint32
20                }

Ex. 2: Use the Refresh for live monitoring

The FileWriter DataSource has a built-in capability to refresh the file content instead of appending to it. This can be used to monitor the application behaviour in real time using an external tool (e.g. watch -n 1 cat ../Test/Integrated/MassSpring-20-stats.csv).

  1. Edit the file ../Configurations/MassSpring/RTApp-MassSpring-20.cfg and add a FileWriter DataSource to store the signals Thread1CycleTime, Thread1CycleTimeAverage, Thread1CycleTimeMovingAverage, Thread1CycleTimeStdDev, Thread1CycleTimeMax, Thread1CycleTimeMin, Thread1CycleTimeHistogram, Thread1FreeTimeHistogram, GAMsExecutionTime.

  2. Make sure that the property RefreshContent is set to 1 in the DataSource configuration.

  3. Write the data into a file named ../Test/Integrated/MassSpring-20-stats.csv.

  4. Add a GAM to write the signals to the DataSource.

  5. Add the GAM to the execution list.

  6. Check that the file is being refreshed and that the content is updated with the latest values of the signals.

./MARTeApp.sh -f ../Configurations/MassSpring/RTApp-MassSpring-20.cfg -l RealTimeLoader -s State1
Solution

The solution is to add a FileWriterStats with the signals.

FileWriterStats to store the signals.
 1        +FileWriterStats = {
 2            Class = FileDataSource::FileWriter
 3            NumberOfBuffers = 100
 4            CPUMask = 0x1 
 5            StackSize = 10000000 
 6            Filename = "../Test/Integrated/MassSpring-20-stats.csv" 
 7            Overwrite = "yes" 
 8            FileFormat = "csv"
 9            CSVSeparator = "," 
10            StoreOnTrigger = 0 
11            RefreshContent = 1 
12            NumberOfPreTriggers = 0
13            NumberOfPostTriggers = 0
14            Signals = {
15                Thread1CycleTime = {
16                    Type = uint32
17                }
18                Thread1CycleTimeAverage = {
19                    Type = uint32
20                }
21                Thread1CycleTimeMovingAverage = {
22                    Type = float32
23                }
24                Thread1CycleTimeStdDev = {
25                    Type = uint32
26                }
27                Thread1CycleTimeMax = {
28                    Type = uint32
29                }
30                Thread1CycleTimeMin = {
31                    Type = uint32
32                }
33                Thread1CycleTimeHistogram = {
34                    Type = uint32
35                    NumberOfElements = 11
36                }
37                Thread1FreeTimeHistogram = {
38                    Type = uint32
39                    NumberOfElements = 11
40                }
41                GAMsExecutionTime = {
42                    Type = uint32
43                }
44            }
45        }

Add the GAM to write the signals to the DataSource.

GAM to write the signals to the DataSource.
 1        +GAMWriterStats = {
 2            Class = IOGAM
 3            InputSignals = {
 4                Thread1CycleTime = {
 5                    DataSource = DDB1
 6                    Type = uint32
 7                }
 8                Thread1CycleTimeAverage = {
 9            OutputSignals = {
10                Thread1CycleTime = {
11                    DataSource = FileWriterStats
12                    Type = uint32
13                }
14                Thread1CycleTimeAverage = {