1 MPI Network calibration
2 ***********************
4 This tutorial demonstrates how to properly calibrate SimGrid to reflect
5 the performance of MPI operations in a Grid’5000 cluster. However, the same
6 approach can be performed to calibrate any other environment.
8 This tutorial is the result of the effort from many people along the years.
9 Specially, it is based on Tom Cornebize’s Phd thesis
10 (https://tel.archives-ouvertes.fr/tel-03328956).
12 You can execute the notebook `network_calibration_tutorial.ipynb <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/network_calibration_tutorial.ipynb>`_) by yourself using the docker image
13 available at: `Dockerfile <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/Dockerfile>`_. For that, run the
14 following commands in the tutorial folder inside simgrid's code source (``docs/source/tuto_network_calibration``):
18 docker build -t tuto_network .
19 docker run -p 8888:8888 tuto_network
24 Performing a realistic simulation is hard and therefore the correct SimGrid calibration requires some work.
26 We briefly present these steps here and detail some of them later. Please, refer to the different links and the original notebook
29 1. **Execution of tests in a real platform**
31 Executing the calibration code in a real platform to obtain the raw data
32 to be analyzed and inject in SimGrid.
34 2. **MPI Async/Sync modes: Identifying threshold**
36 Identify the threshold of the asynchronous and synchronous mode of MPI.
40 Identify the semantic breakpoints of each MPI operation.
44 Aggregating the points inside each segment to create noise models.
46 In this tutorial, we propose 2 alternatives to automatically do the clustering:
47 *ckmeans.1d.dp* and *dhist*. You must choose one, test and maybe adapt it
48 manually depending on your platform.
50 5. **Description of the platform in SimGrid**
52 Writing your platform file using the models created by this notebook.
54 6. **SimGrid execution and comparison**
56 Re-executing the calibration code in SimGrid and comparing the simulation and real world.
58 *This tutorial focuses on steps 3 to 6. For other steps, please see the
61 1. Execution of tests in a real platform
62 ========================================
64 The first step is running tests in a real platform to obtain the data to be used in the calibration.
66 The platform-calibration project provides a tool to run MPI experiments. In a few words, the tool will run a
67 bunch of MPI operations in the nodes to gather their performance. In this tutorial, we are interested in 4 measures
68 related to network operations:
70 - **MPI_Send**: measures the time spent in blocking MPI_Send command.
71 - **MPI_Isend**: same for non-blocking MPI_Isend command.
72 - **MPI_Recv**: time spent in MPI_Recv.
73 - **Ping-pong**: measures the elapsed time to perform a MPI_Send followed by a MPI_Recv.
75 The first 3 tests (MPI_Send, MPI_Isend and MPI_Recv) are used to calibrate the SMPI options
76 (:ref:`smpi/os<cfg=smpi/os>`, :ref:`smpi/or<cfg=smpi/or>`, :ref:`smpi/ois<cfg=smpi/ois>`) while
77 the Ping-pong is used for network calibration (:ref:`network/latency-factor<cfg=network/latency-factor>`
78 and :ref:`network/bandwidth-factor<cfg=network/bandwidth-factor>`).
80 For more detail about this step, please refer to:
81 https://framagit.org/simgrid/platform-calibration
83 The result of this phase can be seen in the figure below. These are the results for the
84 calibration on Grid'5000 dahu cluster at Grenoble/France.
86 .. image:: /tuto_network_calibration/plot_op_raw.png
89 We can see a huge variability in the measured elapsed time for each MPI operation, specially:
91 - **Performance leaps**: at some points, MPI changes its operation mode and the duration can increase drastically.
92 This is mainly due to the different implementation of the MPI.
93 - **Noise/variability**: for a same message size, we have different elapsed times, forming the horizontal lines you can see in the figure.
95 In order to do a correct simulation, we must be able to identify and model these different phenomena.
98 2. MPI Async/Sync modes: Identifying threshold
99 ==============================================
101 MPI communications can operate in different modes
102 (asynchronous/synchronous), depending on the message size of your
103 communication. In asynchronous mode, the MPI_Send will return
104 immediately while in synchronous it’ll wait for respective MPI_Recv
105 starts before returning. See Section `2.2
106 SimGrid/SMPI <https://tel.archives-ouvertes.fr/tel-03328956/document>`__
109 The first step is identifying the message size from which MPI starts
110 operating in synchronous mode. This is important to determine which
111 dataset to use in further tests (individual MPI_Send/MPI_Recv or
112 PingPong operations).
114 In this example, we set the threshold to **63305**, because it’s the data
115 available in our tests and matches the output of the segmentation tool.
117 However the real threshold for this platform is 64000. To be
118 able to identify it, another study would be necessary and the adjustment
119 of the breakpoints needs to be made. We refer to the Section `5.3.2
121 breakpoints <https://tel.archives-ouvertes.fr/tel-03328956/document>`__
128 The objective of the segmentation phase is identify the **performance leaps** in MPI operations.
129 The first step for segmentation is removing the noise by averaging the duration for each message size.
131 .. image:: /tuto_network_calibration/plot_op_average.png
133 Visually, you can already identify some of the segments (e.g. around 1e5 for MPI_Isend).
136 However, we use a tool `pycewise <https://github.com/Ezibenroc/pycewise>`_ that makes this job and finds the correct vertical lines which divide each segment.
138 We present here a summarized version of the results for MPI_Send and Ping-Pong operations. For detailed version, please see "Segmentation" section in `network_calibration_tutorial.ipynb <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/network_calibration_tutorial.ipynb>`_.
146 .dataframe tbody tr th:only-of-type {
147 vertical-align: middle;
150 .dataframe tbody tr th {
154 .dataframe thead th {
158 <table border="1" class="dataframe">
160 <tr style="text-align: right;">
173 <td>2.064276e-07</td>
174 <td>6.785879e-09</td>
180 <td>3.126291e-07</td>
181 <td>7.794590e-11</td>
187 <td>7.346840e-40</td>
188 <td>1.458088e-10</td>
194 <td>4.052195e-06</td>
195 <td>1.042737e-10</td>
201 <td>8.556209e-06</td>
202 <td>1.262608e-10</td>
210 This is the example of the pycewise's output for **MPI_Send** operation. Each line represents one segment which is characterized by:
212 - **interval** (min_x, max_x): the message size interval for this segment
213 - **intercept**: output of the *linear model* of this segment
214 - **coefficient**: output of the *linear model* of this segment
216 The average duration of each segment is characterized by the formula: :math:`coefficient*msg\_size + intercept`.
220 In the ping-pong case, we are interested only in the synchronous mode, so we keep the segments
221 with message size greater than 65503.
227 .dataframe tbody tr th:only-of-type {
228 vertical-align: middle;
231 .dataframe tbody tr th {
235 .dataframe thead th {
239 <table border="1" class="dataframe">
241 <tr style="text-align: right;">
255 <td>1.621952e-10</td>
263 **Setting the base bandwidth and latency for our platform**
265 We use the ping-pong results to estimate the bandwidth and latency for
266 our dahu cluster. These values are passed to SimGrid in the JSON files
267 and are used later to calculate network factors.
269 To obtain similar timing in SimGrid simulations, your platform must use
270 these values when describing the links.
272 In this case, the hosts in dahu are interconnected through
273 a single link with this bandwidth and latency.
277 bandwidth_base = (1.0/reg_pingpong_df.iloc[0]["coefficient"])*2.0
278 latency_base = reg_pingpong_df.iloc[0]['intercept']/2.0
279 print("Bandwidth: %e" % bandwidth_base)
280 print("Latency: %e" % latency_base)
285 Bandwidth: 1.233082e+10
286 Latency: 1.292490e-05
289 3.1. Segmentation results
290 -------------------------
292 The figure below presents the results of the segmentation phase for the dahu calibration.
294 At this phase, you may need to adjust the segments and select those to keep. You can for example
295 do the union of the different segments for each MPI operation to keep them uniform.
297 For simplicity, we do nothing in this tutorial.
299 .. image:: /tuto_network_calibration/plot_op_segmented.png
301 The linear models are sufficient to emulate the average duration of each operation.
303 However, you may be interested in a more realistic model capable of generating the noise and variability for each message size.
305 For that, it's necessary the clustering phase to create specific models for the noise inside each segment.
310 We present 2 tool options for creating the noise models for MPI
311 communications: **ckmeans** and **dhist**.
313 You probably want to try both and see which one is better in your
314 environment. Note that a manual tuning of the results may be needed.
316 The output of the clustering phase is injected in SimGrid. To make this
317 easier, we export the different models using JSON files.
319 Again, we present here just a few results to illustrate the process. For complete information, please see "Clustering" section in `network_calibration_tutorial.ipynb <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/network_calibration_tutorial.ipynb>`_. Also, you can check the 2 individual notebooks that are used for the clustering: `clustering_ckmeans.ipynb <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/clustering_ckmeans.ipynb>`_ and `clustering_dhist.ipynb <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/clustering_dhist.ipynb>`_.
321 4.1. Ckmeans.1d.dp (alternative 1)
322 ----------------------------------
324 The noise is modeled here by a mixture of normal distributions. For each
325 segmented found by pycewise, we have a set of normal distributions (with
326 their respective probabilities) that describes the noise.
328 Ckmeans is used to aggregate the points together. One mixture of normal
329 distributions is created for each cluster.
331 .. image:: /tuto_network_calibration/plot_ckmeans_PingPong.png
334 The figure above presents the output for ping-pong. The process involves 4 phases:
336 1. **Quantile regression**: a quantile regression is made to have our baseline linear model. A quantile regression is used to avoid having negative intercepts and consequently negative estimate duration times.
337 2. **Intercept residuals**: from the quantile regression, we calculate the intercept for each message size (:math:`intercept = duration - coefficient*msg\_size`)
338 3. **Ckmeans**: creates a set of groups based on our intercept residuals. In the figure, each color represents a group.
339 4. **Normal distributions**: for each group found by ckmeans, we calculate the mean and standard deviation of that group. The probabilities are drawn from the density of each group (points in group/total number of points).
345 Ping-pong measures give us the round-trip estimated time, but we need
346 the elapsed time in 1 direction to inject in SimGrid.
348 For simplicity, we just scale down the normal distributions.
349 However, a proper calculation may be necessary at this step.
353 pingpong_models["coefficient"] = pingpong_models["coefficient"]/2
354 pingpong_models["mean"] = pingpong_models["mean"]/2
355 pingpong_models["sd"] = pingpong_models["sd"]/numpy.sqrt(2)
363 .dataframe tbody tr th:only-of-type {
364 vertical-align: middle;
367 .dataframe tbody tr th {
371 .dataframe thead th {
375 <table border="1" class="dataframe">
377 <tr style="text-align: right;">
391 <td>4.356809e-07</td>
393 <td>8.049632e-11</td>
395 <td>3.402823e+38</td>
400 <td>5.219426e-07</td>
402 <td>8.049632e-11</td>
404 <td>3.402823e+38</td>
409 <td>1.673437e-06</td>
411 <td>8.049632e-11</td>
413 <td>3.402823e+38</td>
418 <td>2.023256e-06</td>
420 <td>8.049632e-11</td>
422 <td>3.402823e+38</td>
427 <td>2.530620e-06</td>
429 <td>8.049632e-11</td>
431 <td>3.402823e+38</td>
436 <td>3.533823e-06</td>
438 <td>8.049632e-11</td>
440 <td>3.402823e+38</td>
448 This table presents the clustering results for Ping-pong. Each line represents a normal distribution that characterizes the noise along with its probability.
450 At our simulator, we'll draw our noise following these probabilities/distributions.
453 Finally, we dump the results in a JSON format. Below, we present the `pingpong_ckmeans.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/pingpong_ckmeans.json>`_ file.
455 This file will be read by your simulator later to generate the proper factor for network operations.
459 {'bandwidth_base': 12330818795.43382,
460 'latency_base': 1.2924904864614219e-05,
461 'seg': [{'mean': 1.1503128856516448e-05,
462 'sd': 4.3568091437319533e-07,
463 'prob': 0.49970588235294106,
464 'coefficient': 8.04963230919345e-11,
466 'max_x': 3.4028234663852886e+38},
467 {'mean': 1.2504551284320949e-05,
468 'sd': 5.219425841751762e-07,
469 'prob': 0.385196078431373,
470 'coefficient': 8.04963230919345e-11,
472 'max_x': 3.4028234663852886e+38},
473 {'mean': 1.879472592512515e-05,
474 'sd': 1.6734369316865939e-06,
475 'prob': 0.0733137254901961,
476 'coefficient': 8.04963230919345e-11,
478 'max_x': 3.4028234663852886e+38},
479 {'mean': 2.451754075327485e-05,
480 'sd': 2.0232563328989863e-06,
481 'prob': 0.0241078431372549,
482 'coefficient': 8.04963230919345e-11,
484 'max_x': 3.4028234663852886e+38},
485 {'mean': 3.004149952883e-05,
486 'sd': 2.5306204869242285e-06,
487 'prob': 0.0116960784313725,
488 'coefficient': 8.04963230919345e-11,
490 'max_x': 3.4028234663852886e+38},
491 {'mean': 3.688584189653765e-05,
492 'sd': 3.5338234385210185e-06,
493 'prob': 0.00598039215686275,
494 'coefficient': 8.04963230919345e-11,
496 'max_x': 3.4028234663852886e+38}]}
499 The same is done for each one of the MPI operations, creating the different input files: `pingpong_ckmeans.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/pingpong_ckmeans.json>`_, `isend_ckmeans.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/isend_ckmeans.json>`_, `recv_ckmeans.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/recv_ckmeans.json>`_, `send_ckmeans.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/send_ckmeans.json>`_.
502 4.2. Dhist (alternative 2)
503 --------------------------
505 Alternatively, we can model the noise using non-uniform histograms.
507 Diagonally cut histograms are used in this case, one histogram for each
510 The noise is later sampled according to these histograms.
512 Note: For better results, we had to apply a log function on the elapsed
513 time before running the dhist algorithm. However, it’s not clear why
514 this manipulation gives better results.
516 .. image:: /tuto_network_calibration/plot_dhist_PingPong.png
519 The figure presents the histogram for the ping-pong operation.
521 In the x-axis, we have the intercept residuals calculated using the linear models found by pycewise.
523 The vertical lines are the bins found by dhist. Note that the size of each bin varies depending on their density.
527 Ping-pong measures give us the round-trip estimated time, but we need
528 the elapsed time in 1 direction to inject in SimGrid. As we applied the log function on our data, we need a minor trick to calculate the elapsed time.
530 :math:`\frac{e^x}{2}` = :math:`e^{x + log(\frac{1}{2})}`
534 for i in pingpong_dhist:
535 i["xbr"] = [v + numpy.log(1/2) for v in i["xbr"]]
538 pingpong_dhist = {"bandwidth_base": bandwidth_base, "latency_base" : latency_base, "seg": pingpong_dhist}
544 {'bandwidth_base': 12330818795.43382,
545 'latency_base': 1.2924904864614219e-05,
546 'seg': [{'log': True,
548 'max_x': 3.4028234663852886e+38,
549 'xbr': [-11.541562041539144,
568 'height': [28047.5350565562,
586 'coeff': 8.10976153806028e-11}]}
588 This JSON file is read by the simulator to create the platform and generate the appropriate noise.
589 The same is done for each one of the MPI operations, creating the different input files: `pingpong_dhist.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/pingpong_dhist.json>`_, `isend_dhist.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/isend_dhist.json>`_, `recv_dhist.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/recv_dhist.json>`_, `send_dhist.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/send_dhist.json>`_.
591 5. Description of the platform in SimGrid
592 =========================================
594 At this point we have done the analysis and extracted the models in the several JSON files. It's possible now to create our platform file that will be used by SimGrid later.
596 The platform is created using the C++ interface from SimGrid. The result is a library file (.so) which is loaded by SimGrid when running the application.
598 The best to understand is reading the C++ code in `docs/source/tuto_network_calibration <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/>`_, the main files are:
600 - `dahu_platform_ckmeans.cpp <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/dahu_platform_ckmeans.cpp>`_: create the dahu platform using the JSON files from ckmeans.
601 - `dahu_platform_dhist.cpp <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/dahu_platform_dhist.cpp>`_: same for dhist output.
602 - `Utils.cpp <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/Utils.cpp>`_/`Utils.hpp <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/Utils.hpp>`_: some auxiliary classes used by both platforms to handle the segmentation and sampling.
603 - `CMakeLists.txt <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/CMakeLists.txt>`_: create the shared library to be loaded by SimGrid
605 Feel free to re-use and adapt these files according to your needs.
607 6. SimGrid execution and comparison
608 ===================================
613 **Ckmeans.1d.dp** and **Dhist**
615 The execution is similar for both modes. The only change is the platform library to be used: **libdahu_ckmeans.so** or **libdhist.so**.
622 cd /source/simgrid.git/docs/source/tuto_network_calibration/
624 smpirun --cfg=smpi/simulate-computation:0 \
625 --cfg=smpi/display-timing:yes \
626 -platform ./libdahu_ckmeans.so \
627 -hostfile /tmp/host.txt -np 2 \
628 /source/platform-calibration/src/calibration/calibrate -d /tmp/exp -m 1 -M 1000000 -p exp -s /tmp/exp.csv
633 Read bandwidth_base: 1.233082e+10 latency_base: 1.292490e-05
634 Starting parsing file: pingpong_ckmeans.json
635 Starting parsing file: send_ckmeans.json
636 Starting parsing file: isend_ckmeans.json
637 Starting parsing file: recv_ckmeans.json
639 [0] nb_exp=115200, largest_size=980284
640 [0] Alloc size: 1960568
642 [1] nb_exp=115200, largest_size=980284
643 [1] Alloc size: 1960568
644 [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/privatization' to '1'
645 [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/np' to '2'
646 [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/hostfile' to '/tmp/host.txt'
647 [0.000000] [xbt_cfg/INFO] Configuration change: Set 'surf/precision' to '1e-9'
648 [0.000000] [xbt_cfg/INFO] Configuration change: Set 'network/model' to 'SMPI'
649 [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/simulate-computation' to '0'
650 [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/display-timing' to 'yes'
651 [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/tmpdir' to '/tmp'
652 [0.000000] [smpi_config/INFO] You did not set the power of the host running the simulation. The timings will certainly not be accurate. Use the option "--cfg=smpi/host-speed:<flops>" to set its value. Check https://simgrid.org/doc/latest/Configuring_SimGrid.html#automatic-benchmarking-of-smpi-code for more information.
653 [6.845963] [smpi_utils/INFO] Simulated time: 6.84596 seconds.
655 The simulation took 71.6111 seconds (after parsing and platform setup)
656 1.77771 seconds were actual computation of the application
663 Finally, let’s compare the SimGrid results the real ones. The red points are the real data while the blue ones are the output from our simulator.
667 .. image:: /tuto_network_calibration/plot_op_simgrid_ckmeans.png
671 .. image:: /tuto_network_calibration/plot_op_simgrid_dhist.png
676 Note that for ping-ping tests, we have an important gap between the real
677 performance (in red) and SimGrid (in blue) for messages below our
678 sync/async threshold (63305).
680 This behavior is explained by how we measure the extra cost for each
681 MPI_Send/MPI_Recv operations.
683 In `calibrate.c <https://framagit.org/simgrid/platform-calibration/-/blob/master/src/calibration/calibrate.c>`_ in platform-calibration, the ping-pong test is as follows
684 (considering the processes are synchronized):
687 .. image:: /tuto_network_calibration/fig/pingpong_real.png
689 We can see that we measure the delay at **Process 1**, just before the
690 first *MPI_Send-1* until the end of respective *MPI_Recv-2*. Moreover,
691 the extra cost of MPI operations is paid concurrently with the network
694 In this case, it doesn't matter when the *MPI_Send-2* will finish.
695 Despite we expect that it finished before the *MPI_Recv-2*, we couldn't
698 Also, both processes are running in parallel, so we can expect that the
699 measure time will be:
700 :math:`max(\text{MPI_Send-1}, \text{MPI_Recv-1}) + \text{MPI_Recv-2}` -
701 :math:`max(\text{MPI_Send-1}, \text{MPI_Recv-1})`: since we cannot start
702 *MPI_Recv-2* or *MPI_Send_2* before finishing both commands -
703 :math:`\text{MPI_Recv-2}`: because we measure just after the finishing
706 However, the simulation world is a little more stable. The same
707 communication occurs in the following way:
710 .. image:: /tuto_network_calibration/fig/pingpong_simgrid.png
712 In SimGrid, the extra costs are paid sequentially. That means, initially
713 we pay the extra cost for *MPI_Send-1*, after the network communication
714 cost, followed by the extra cost for *MPI-Recv-1*.
716 This effect leads to a total time of: *MPI_Send-1* + *MPI_Recv-1* +
717 *MPI_Send-2* + *MPI_Recv-2* which is slightly higher than the real cost.
719 The same doesn't happen for largest messages because we don’t pay the
720 extra overhead cost for each MPI operation (the communication is limited
721 by the network capacity).