1 /*! \page tracing Tracing Simulations
4 Tracing is widely used to observe and understand the behavior of
5 parallel applications and distributed algorithms. Usually, this is
6 done in a two-step fashion: the user instruments the application and
7 the traces are analyzed after the end of the execution. The analysis
8 can highlights unexpected behaviors, bottlenecks and sometimes can be
9 used to correct distributed algorithms. The SimGrid team has
10 instrumented the library in order to let users trace their simulations
11 and analyze them. This part of the user manual explains how the
12 tracing-related features can be enabled and used during the
13 development of simulators using the SimGrid library.
15 \section tracing_tracing_enabling Enabling using CMake
17 With the sources of SimGrid, it is possible to enable the tracing
18 using the parameter <b>-Denable_tracing=ON</b> when the cmake is
19 executed. The sections \ref instr_category_functions, \ref
20 instr_mark_functions, and \ref instr_uservariables_functions describe
21 all the functions available when this Cmake options is
22 activated. These functions will have no effect if SimGrid is
23 configured without this option (they are wiped-out by the
27 $ cmake -Denable_tracing=ON .
31 \section instr_category_functions Tracing categories functions
33 The SimGrid library is instrumented so users can trace the platform
34 utilization using MSG, SimDAG and SMPI interfaces. It registers how
35 much power is used for each host and how much bandwidth is used for
36 each link of the platform. The idea with this type of tracing is to
37 observe the overall view of resources utilization in the first place,
38 especially the identification of bottlenecks, load-balancing among
41 Another possibility is to trace resource utilization by
42 categories. Categorized resource utilization tracing gives SimGrid
43 users to possibility to classify MSG and SimDAG tasks by category,
44 tracing resource utilization for each of the categories. The functions
45 below let the user declare a category and apply it to tasks. <em>The
46 tasks that are not classified according to a category are not
47 traced</em>. Even if the user does not specify any category, the
48 simulations can still be traced in terms of resource utilization by
49 using a special parameter that is detailed below (see section \ref
50 tracing_tracing_options).
52 \li \c TRACE_category(const char *category)
53 \li \c TRACE_category_with_color(const char *category, const char *color)
54 \li \c MSG_task_set_category(msg_task_t task, const char *category)
55 \li \c MSG_task_get_category(msg_task_t task)
56 \li \c SD_task_set_category(SD_task_t task, const char *category)
57 \li \c SD_task_get_category(SD_task_t task)
59 \section instr_mark_functions Tracing marks functions
60 \li \c TRACE_declare_mark(const char *mark_type)
61 \li \c TRACE_mark(const char *mark_type, const char *mark_value)
63 \section instr_uservariables_functions Tracing user variables functions
67 \li \c TRACE_host_variable_declare(const char *variable)
68 \li \c TRACE_host_variable_declare_with_color(const char *variable, const char *color)
69 \li \c TRACE_host_variable_set(const char *host, const char *variable, double value)
70 \li \c TRACE_host_variable_add(const char *host, const char *variable, double value)
71 \li \c TRACE_host_variable_sub(const char *host, const char *variable, double value)
72 \li \c TRACE_host_variable_set_with_time(double time, const char *host, const char *variable, double value)
73 \li \c TRACE_host_variable_add_with_time(double time, const char *host, const char *variable, double value)
74 \li \c TRACE_host_variable_sub_with_time(double time, const char *host, const char *variable, double value)
78 \li \c TRACE_link_variable_declare(const char *variable)
79 \li \c TRACE_link_variable_declare_with_color(const char *variable, const char *color)
80 \li \c TRACE_link_variable_set(const char *link, const char *variable, double value)
81 \li \c TRACE_link_variable_add(const char *link, const char *variable, double value)
82 \li \c TRACE_link_variable_sub(const char *link, const char *variable, double value)
83 \li \c TRACE_link_variable_set_with_time(double time, const char *link, const char *variable, double value)
84 \li \c TRACE_link_variable_add_with_time(double time, const char *link, const char *variable, double value)
85 \li \c TRACE_link_variable_sub_with_time(double time, const char *link, const char *variable, double value)
87 For links, but use source and destination to get route:
89 \li \c TRACE_link_srcdst_variable_set(const char *src, const char *dst, const char *variable, double value)
90 \li \c TRACE_link_srcdst_variable_add(const char *src, const char *dst, const char *variable, double value)
91 \li \c TRACE_link_srcdst_variable_sub(const char *src, const char *dst, const char *variable, double value)
92 \li \c TRACE_link_srcdst_variable_set_with_time(double time, const char *src, const char *dst, const char *variable, double value)
93 \li \c TRACE_link_srcdst_variable_add_with_time(double time, const char *src, const char *dst, const char *variable, double value)
94 \li \c TRACE_link_srcdst_variable_sub_with_time(double time, const char *src, const char *dst, const char *variable, double value)
96 \section tracing_tracing_options Tracing configuration Options
98 To check which tracing options are available for your simulator, you
99 can just run it with the option \verbatim --help-tracing \endverbatim
100 to get a very detailed and updated explanation of each tracing
101 parameter. These are some of the options accepted by the tracing
102 system of SimGrid, you can use them by running your simulator with the
103 <b>--cfg=</b> switch:
108 Safe switch. It activates (or deactivates) the tracing system.
109 No other tracing options take effect if this one is not activated.
117 It activates the categorized resource utilization tracing. It should
118 be enabled if tracing categories are used by this simulator.
120 --cfg=tracing/categorized:1
124 tracing/uncategorized
126 It activates the uncategorized resource utilization tracing. Use it if
127 this simulator do not use tracing categories and resource use have to be
130 --cfg=tracing/uncategorized:1
136 A file with this name will be created to register the simulation. The file
137 is in the Paje format and can be analyzed using Viva or Paje visualization
138 tools. More information can be found in these webpages:
139 <a href="http://github.com/schnorr/viva/">http://github.com/schnorr/viva/</a>
140 <a href="http://github.com/schnorr/pajeng/">http://github.com/schnorr/pajeng/</a>
142 --cfg=tracing/filename:mytracefile.trace
144 If you do not provide this parameter, the trace file will be named simgrid.trace.
149 By default, the tracing system uses all routes in the platform file
150 to re-create a "graph" of the platform and register it in the trace file.
151 This option let the user tell the tracing system to use only the routes
152 that are composed with just one link.
154 --cfg=tracing/onelink_only:1
160 This option only has effect if this simulator is SMPI-based. Traces the MPI
161 interface and generates a trace that can be analyzed using Gantt-like
162 visualizations. Every MPI function (implemented by SMPI) is transformed in a
163 state, and point-to-point communications can be analyzed with arrows.
171 This option only has effect if this simulator is SMPI-based. The processes
172 are grouped by the hosts where they were executed.
174 --cfg=tracing/smpi/group:1
180 This option only has effect if this simulator is MSG-based. It traces the
181 behavior of all categorized MSG processes, grouping them by hosts. This option
182 can be used to track process location if this simulator has process migration.
184 --cfg=tracing/msg/process:1
190 This option put some events in a time-ordered buffer using the
191 insertion sort algorithm. The process of acquiring and releasing
192 locks to access this buffer and the cost of the sorting algorithm
193 make this process slow. The simulator performance can be severely
194 impacted if this option is activated, but you are sure to get a trace
195 file with events sorted.
197 --cfg=tracing/buffer:1
203 This option changes the way SimGrid register its platform on the trace
204 file. Normally, the tracing considers all routes (no matter their
205 size) on the platform file to re-create the resource topology. If this
206 option is activated, only the routes with one link are used to
207 register the topology within an AS. Routes among AS continue to be
210 --cfg=tracing/onelink_only:1
214 tracing/disable_destroy
216 Disable the destruction of containers at the end of simulation. This
217 can be used with simulators that have a different notion of time
218 (different from the simulated time).
220 --cfg=tracing/disable_destroy:1
226 Some visualization tools are not able to parse correctly the Paje file format.
227 Use this option if you are using one of these tools to visualize the simulation
228 trace. Keep in mind that the trace might be incomplete, without all the
229 information that would be registered otherwise.
231 --cfg=tracing/basic:1
237 Use this to add a comment line to the top of the trace file.
239 --cfg=tracing/comment:my_string
245 Use this to add the contents of a file to the top of the trace file as comment.
247 --cfg=tracing/comment_file:textual_file.txt
253 This option generates a graph configuration file for Viva considering
254 categorized resource utilization.
256 --cfg=viva/categorized:graph_categorized.plist
262 This option generates a graph configuration file for Viva considering
263 uncategorized resource utilization.
265 --cfg=viva/uncategorized:graph_uncategorized.plist
268 Please pass \verbatim --help-tracing \endverbatim to your simulator
269 for the updated list of tracing options.
271 \section tracing_tracing_example_parameters Case studies
273 Some scenarios that might help you decide which tracing options
274 you should use to analyze your simulator.
276 \li I want to trace the resource utilization of all hosts
277 and links of the platform, and my simulator <b>does not</b> use
278 the tracing API. For that, you can run a uncategorized trace
279 with the following parameters (it will work with <b>any</b> Simgrid
284 --cfg=tracing/uncategorized:1 \
285 --cfg=tracing/filename:mytracefile.trace \
286 --cfg=viva/uncategorized:uncat.plist
289 \li I want to trace only a subset of my MSG (or SimDAG) tasks.
290 For that, you will need to create tracing categories using the
291 <b>TRACE_category (...)</b> function (as explained above),
292 and then classify your tasks to a previously declared category
293 using the <b>MSG_task_set_category (...)</b>
294 (or <b>SD_task_set_category (...)</b> for SimDAG tasks). After
295 recompiling, run your simulator with the following parameters:
299 --cfg=tracing/categorized:1 \
300 --cfg=tracing/filename:mytracefile.trace \
301 --cfg=viva/categorized:cat.plist
305 \section tracing_tracing_example Example of Instrumentation
307 A simplified example using the tracing mandatory functions.
310 int main (int argc, char **argv)
312 MSG_init (&argc, &argv);
314 //(... after deployment ...)
316 //note that category declaration must be called after MSG_create_environment
317 TRACE_category_with_color ("request", "1 0 0");
318 TRACE_category_with_color ("computation", "0.3 1 0.4");
319 TRACE_category ("finalize");
321 msg_task_t req1 = MSG_task_create("1st_request_task", 10, 10, NULL);
322 msg_task_t req2 = MSG_task_create("2nd_request_task", 10, 10, NULL);
323 msg_task_t req3 = MSG_task_create("3rd_request_task", 10, 10, NULL);
324 msg_task_t req4 = MSG_task_create("4th_request_task", 10, 10, NULL);
325 MSG_task_set_category (req1, "request");
326 MSG_task_set_category (req2, "request");
327 MSG_task_set_category (req3, "request");
328 MSG_task_set_category (req4, "request");
330 msg_task_t comp = MSG_task_create ("comp_task", 100, 100, NULL);
331 MSG_task_set_category (comp, "computation");
333 msg_task_t finalize = MSG_task_create ("finalize", 0, 0, NULL);
334 MSG_task_set_category (finalize, "finalize");
343 \section tracing_tracing_analyzing Analyzing SimGrid Simulation Traces
345 A SimGrid-based simulator, when executed with the correct parameters
346 (see above) creates a trace file in the Paje file format holding the
347 simulated behavior of the application or the platform. You have
348 several options to analyze this trace file:
350 - Dump its contents to a CSV-like format using `pj_dump` (see <a
351 href="https://github.com/schnorr/pajeng/wiki/pj_dump">PajeNG's wiki
352 on pj_dump</a> and more generally the <a
353 href="https://github.com/schnorr/pajeng/">PajeNG suite</a>) and use
354 gnuplot to plot resource usage, time spent on blocking/executing
355 functions, and so on. Filtering capabilities are at your hand by
356 doing `grep`, with the best regular expression you can provide, to
357 get only parts of the trace (for instance, only a subset of
358 resources or processes).
360 - Derive statistics from trace metrics (the ones built-in with any
361 SimGrid simulation, but also those metrics you injected in the trace
362 using the TRACE module) using the <a
363 href="http://www.r-project.org/">R project</a> and all its
364 modules. You can also combine R with <a
365 href="http://ggplot2.org/">ggplot2</a> to get a number of high
366 quality plots from your simulation metrics. You need to `pj_dump`
367 the contents of the SimGrid trace file to use R.
369 - Visualize the behavior of your simulation using classic space/time
370 views (gantt-charts) provided by the <a
371 href="https://github.com/schnorr/pajeng/">PajeNG suite</a> and any
372 other tool that supports the <a
373 href="http://paje.sourceforge.net/download/publication/lang-paje.pdf">Paje
374 file format</a>. Consider this option if you need to understand the
375 causality of your distributed simulation.
377 - Visualize the behavior of your simulation with treemaps (specially
378 if your simulation has a platform with several thousand resources),
379 provided by the <a href="http://github.com/schnorr/viva/">Viva</a>
380 visualization tool. See <a
381 href="https://github.com/schnorr/viva/wiki">Viva's wiki</a> for
382 further details on what is a treemap and how to use it.
384 - Correlate the behavior of your simulator with the platform topology
385 with an interactive, force-directed, and hierarchical graph
386 visualization, provided by <a
387 href="http://github.com/schnorr/viva/">Viva</a>. Check <a
388 href="https://github.com/schnorr/viva/wiki">Viva's wiki</a> for
389 further details. This <a
390 href="http://hal.inria.fr/hal-00738321/">research report</a>,
391 published at ISPASS 2013, has a detailed description of this
392 visualization technique.
394 - You can also check our online <a
395 href="http://simgrid.gforge.inria.fr/tutorials.html"> tutorial
396 section</a> that contains a dedicated tutorial with several
397 suggestions on how to use the tracing infrastructure. Look for the
398 SimGrid User::Visualization 101 tutorial.
400 - Ask for help on the <a
401 href="mailto:simgrid-user@lists.gforge.inria.fr">simgrid-user@lists.gforge.inria.fr</a>
402 mailing list, giving us a detailed explanation on what your
403 simulator does and what kind of information you want to trace. You
404 can also check the <a
405 href="http://lists.gforge.inria.fr/pipermail/simgrid-user/">mailing
406 list archive</a> for old messages regarding tracing and analysis.
408 \subsection tracing_viva_analysis Viva Visualization Tool
410 This subsection describe some of the concepts regarding the <a
411 href="http://github.com/schnorr/viva/">Viva Visualization Tool</a> and
412 its relation with SimGrid traces. You should refer to Viva's website
413 for further details on all its visualization techniques.
415 \subsubsection tracing_viva_time_slice Time Slice
417 The analysis of a trace file using the tool always takes into account
418 the concept of the <em>time-slice</em>. This concept means that what
419 is being visualized in the screen is always calculated considering a
420 specific time frame, with its beggining and end timestamp. The
421 time-slice is configured by the user and can be changed dynamically
422 through the window called <em>Time Interval</em> that is opened
423 whenever a trace file is being analyzed. Users are capable to select
424 the beggining and size of the time slice.
426 \subsubsection tracing_viva_graph Hierarchical Graph View
428 As stated above (see section \ref tracing_tracing_analyzing), one
429 possibility to analyze SimGrid traces is to use Viva's graph view with
430 a graph configuration to customize the graph according to the
431 traces. A valid graph configuration (we are using the non-XML <a
432 href="http://en.wikipedia.org/wiki/Property_list">Property List
433 Format</a> to describe the configuration) can be created for any
434 SimGrid-based simulator using the
435 <em>--cfg=viva/uncategorized:graph_uncategorized.plist</em> or
436 <em>--cfg=viva/categorized:graph_categorized.plist</em> (if the
437 simulator defines resource utilization categories) when executing the
440 \subsubsection basic_conf Basic Graph Configuration
442 The basic description of the configuration is as follows:
445 node = (LINK, HOST, );
446 edge = (HOST-LINK, LINK-HOST, LINK-LINK, );
449 The nodes of the graph will be created based on the <i>node</i>
450 parameter, which in this case is the different <em>"HOST"</em>s and
451 <em>"LINK"</em>s of the platform used to simulate. The <i>edge</i>
452 parameter indicates that the edges of the graph will be created based
453 on the <em>"HOST-LINK"</em>s, <em>"LINK-HOST"</em>s, and
454 <em>"LINK-LINK"</em>s of the platform. After the definition of these
455 two parameters, the configuration must detail how the nodes
456 (<em>HOST</em>s and <em>LINK</em>s) should be drawn.
458 For that, the configuration must have an entry for each of
459 the types used. For <em>HOST</em>, as basic configuration, we have:
465 values = (power_used);
469 The parameter <em>size</em> indicates which variable from the trace
470 file will be used to define the size of the node HOST in the
471 visualization. If the simulation was executed with availability
472 traces, the size of the nodes will be changed according to these
473 traces. The parameter <em>type</em> indicates which geometrical shape
474 will be used to represent HOST, and the <em>values</em> parameter
475 indicates which values from the trace will be used to fill the shape.
477 For <em>LINK</em> we have:
483 values = (bandwidth_used);
488 The same configuration parameters are used here: <em>type</em> (with a
489 rhombus), the <em>size</em> (whose value is from trace's bandwidth
490 variable) and the <em>values</em>.
492 \subsubsection custom_graph Customizing the Graph Representation
494 Viva is capable to handle a customized graph representation based on
495 the variables present in the trace file. In the case of SimGrid, every
496 time a category is created for tasks, two variables in the trace file
497 are defined: one to indicate node utilization (how much power was used
498 by that task category), and another to indicate link utilization (how
499 much bandwidth was used by that category). For instance, if the user
500 declares a category named <i>request</i>, there will be variables
501 named <b>p</b><i>request</i> and a <b>b</b><i>request</i> (<b>p</b>
502 for power and <b>b</b> for bandwidth). It is important to notice that
503 the variable <i>prequest</i> in this case is only available for HOST,
504 and <i>brequest</i> is only available for LINK. <b>Example</b>:
505 suppose there are two categories for tasks: request and compute. To
506 create a customized graph representation with a proportional
507 separation of host and link utilization, use as configuration for HOST
514 values = (prequest, pcomputation);
519 values = (brequest, bcomputation);
523 This configuration enables the analysis of resource utilization by MSG
524 tasks through the identification of load-balancing issues and network
525 bottlenecks, for instance.