1 MPI Network calibration
2 ***********************
4 This tutorial demonstrates how to properly calibrate SimGrid to reflect
5 the performance of MPI operations in a Grid’5000 cluster. However, the same
6 approach can be performed to calibrate any other environment.
8 This tutorial is the result of the effort from many people along the years.
9 Specially, it is based on Tom Cornebize’s Phd thesis (https://tel.archives-ouvertes.fr/tel-03328956).
11 You can execute the notebook `network_calibration_tutorial.ipynb <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/network_calibration_tutorial.ipynb>`_) by yourself using the docker image
12 available at: `Dockerfile <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/Dockerfile>`_. For that, run the
13 following commands in the tutorial folder inside simgrid's code source (``docs/source/tuto_network_calibration``):
17 docker build -t tuto_network .
18 docker run -p 8888:8888 tuto_network
20 Please also refer to https://framagit.org/simgrid/platform-calibration/ for more complete information.
26 Performing a realistic simulation is hard and therefore the correct SimGrid calibration requires some work.
28 We briefly present these steps here and detail some of them later. Please, refer to the different links and the original notebook
31 1. **Execution of tests in a real platform**
33 Executing the calibration code in a real platform to obtain the raw data
34 to be analyzed and inject in SimGrid.
36 2. **MPI Async/Sync modes: Identifying threshold**
38 Identify the threshold of the asynchronous and synchronous mode of MPI.
42 Identify the semantic breakpoints of each MPI operation.
46 Aggregating the points inside each segment to create noise models.
48 In this tutorial, we propose 2 alternatives to automatically do the clustering:
49 *ckmeans.1d.dp* and *dhist*. You must choose one, test and maybe adapt it
50 manually depending on your platform.
52 5. **Description of the platform in SimGrid**
54 Writing your platform file using the models created by this notebook.
56 6. **SimGrid execution and comparison**
58 Re-executing the calibration code in SimGrid and comparing the simulation and real world.
60 *This tutorial focuses on steps 3 to 6. For other steps, please see the
63 1. Execution of tests in a real platform
64 ========================================
66 The first step is running tests in a real platform to obtain the data to be used in the calibration.
68 The platform-calibration project provides a tool to run MPI experiments. In a few words, the tool will run a
69 bunch of MPI operations in the nodes to gather their performance. In this tutorial, we are interested in 4 measures
70 related to network operations:
72 - **MPI_Send**: measures the time spent in blocking MPI_Send command.
73 - **MPI_Isend**: same for non-blocking MPI_Isend command.
74 - **MPI_Recv**: time spent in MPI_Recv.
75 - **Ping-pong**: measures the elapsed time to perform a MPI_Send followed by a MPI_Recv.
77 The first 3 tests (MPI_Send, MPI_Isend and MPI_Recv) are used to calibrate the SMPI options
78 (:ref:`smpi/os<cfg=smpi/os>`, :ref:`smpi/or<cfg=smpi/or>`, :ref:`smpi/ois<cfg=smpi/ois>`) while
79 the Ping-pong is used for network calibration (:ref:`network/latency-factor<cfg=network/latency-factor>`
80 and :ref:`network/bandwidth-factor<cfg=network/bandwidth-factor>`).
82 For more detail about this step, please refer to:
83 https://framagit.org/simgrid/platform-calibration
85 The result of this phase can be seen in the figure below. These are the results for the
86 calibration on Grid'5000 dahu cluster at Grenoble/France.
88 .. image:: /tuto_network_calibration/plot_op_raw.png
91 We can see a huge variability in the measured elapsed time for each MPI operation, specially:
93 - **Performance leaps**: at some points, MPI changes its operation mode and the duration can increase drastically.
94 This is mainly due to the different implementation of the MPI.
95 - **Noise/variability**: for a same message size, we have different elapsed times, forming the horizontal lines you can see in the figure.
97 In order to do a correct simulation, we must be able to identify and model these different phenomena.
100 2. MPI Async/Sync modes: Identifying threshold
101 ==============================================
103 MPI communications can operate in different modes
104 (asynchronous/synchronous), depending on the message size of your
105 communication. In asynchronous mode, the MPI_Send will return
106 immediately while in synchronous it’ll wait for respective MPI_Recv
107 starts before returning. See Section `2.2
108 SimGrid/SMPI <https://tel.archives-ouvertes.fr/tel-03328956/document>`__
111 The first step is identifying the message size from which MPI starts
112 operating in synchronous mode. This is important to determine which
113 dataset to use in further tests (individual MPI_Send/MPI_Recv or
114 PingPong operations).
116 In this example, we set the threshold to **63305**, because it’s the data
117 available in our tests and matches the output of the segmentation tool.
119 However the real threshold for this platform is 64000. To be
120 able to identify it, another study would be necessary and the adjustment
121 of the breakpoints needs to be made. We refer to the Section `5.3.2
123 breakpoints <https://tel.archives-ouvertes.fr/tel-03328956/document>`__
130 The objective of the segmentation phase is identify the **performance leaps** in MPI operations.
131 The first step for segmentation is removing the noise by averaging the duration for each message size.
133 .. image:: /tuto_network_calibration/plot_op_average.png
135 Visually, you can already identify some of the segments (e.g. around 1e5 for MPI_Isend).
138 However, we use a tool `pycewise <https://github.com/Ezibenroc/pycewise>`_ that makes this job and finds the correct vertical lines which divide each segment.
140 We present here a summarized version of the results for MPI_Send and Ping-Pong operations. For detailed version, please see "Segmentation" section in `network_calibration_tutorial.ipynb <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/network_calibration_tutorial.ipynb>`_.
148 .dataframe tbody tr th:only-of-type {
149 vertical-align: middle;
152 .dataframe tbody tr th {
156 .dataframe thead th {
160 <table border="1" class="dataframe">
162 <tr style="text-align: right;">
175 <td>2.064276e-07</td>
176 <td>6.785879e-09</td>
182 <td>3.126291e-07</td>
183 <td>7.794590e-11</td>
189 <td>7.346840e-40</td>
190 <td>1.458088e-10</td>
196 <td>4.052195e-06</td>
197 <td>1.042737e-10</td>
203 <td>8.556209e-06</td>
204 <td>1.262608e-10</td>
212 This is the example of the pycewise's output for **MPI_Send** operation. Each line represents one segment which is characterized by:
214 - **interval** (min_x, max_x): the message size interval for this segment
215 - **intercept**: output of the *linear model* of this segment
216 - **coefficient**: output of the *linear model* of this segment
218 The average duration of each segment is characterized by the formula: :math:`coefficient*msg\_size + intercept`.
222 In the ping-pong case, we are interested only in the synchronous mode, so we keep the segments
223 with message size greater than 65503.
229 .dataframe tbody tr th:only-of-type {
230 vertical-align: middle;
233 .dataframe tbody tr th {
237 .dataframe thead th {
241 <table border="1" class="dataframe">
243 <tr style="text-align: right;">
257 <td>1.621952e-10</td>
265 **Setting the base bandwidth and latency for our platform**
267 We use the ping-pong results to estimate the bandwidth and latency for
268 our dahu cluster. These values are passed to SimGrid in the JSON files
269 and are used later to calculate network factors.
271 To obtain similar timing in SimGrid simulations, your platform must use
272 these values when describing the links.
274 In this case, the hosts in dahu are interconnected through
275 a single link with this bandwidth and latency.
279 bandwidth_base = (1.0/reg_pingpong_df.iloc[0]["coefficient"])*2.0
280 latency_base = reg_pingpong_df.iloc[0]['intercept']/2.0
281 print("Bandwidth: %e" % bandwidth_base)
282 print("Latency: %e" % latency_base)
287 Bandwidth: 1.233082e+10
288 Latency: 1.292490e-05
291 3.1. Segmentation results
292 -------------------------
294 The figure below presents the results of the segmentation phase for the dahu calibration.
296 At this phase, you may need to adjust the segments and select those to keep. You can for example
297 do the union of the different segments for each MPI operation to keep them uniform.
299 For simplicity, we do nothing in this tutorial.
301 .. image:: /tuto_network_calibration/plot_op_segmented.png
303 The linear models are sufficient to emulate the average duration of each operation.
305 However, you may be interested in a more realistic model capable of generating the noise and variability for each message size.
307 For that, it's necessary the clustering phase to create specific models for the noise inside each segment.
312 We present 2 tool options for creating the noise models for MPI
313 communications: **ckmeans** and **dhist**.
315 You probably want to try both and see which one is better in your
316 environment. Note that a manual tuning of the results may be needed.
318 The output of the clustering phase is injected in SimGrid. To make this
319 easier, we export the different models using JSON files.
321 Again, we present here just a few results to illustrate the process. For complete information, please see "Clustering" section in `network_calibration_tutorial.ipynb <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/network_calibration_tutorial.ipynb>`_. Also, you can check the 2 individual notebooks that are used for the clustering: `clustering_ckmeans.ipynb <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/clustering_ckmeans.ipynb>`_ and `clustering_dhist.ipynb <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/clustering_dhist.ipynb>`_.
323 4.1. Ckmeans.1d.dp (alternative 1)
324 ----------------------------------
326 The noise is modeled here by a mixture of normal distributions. For each
327 segmented found by pycewise, we have a set of normal distributions (with
328 their respective probabilities) that describes the noise.
330 Ckmeans is used to aggregate the points together. One mixture of normal
331 distributions is created for each cluster.
333 .. image:: /tuto_network_calibration/plot_ckmeans_PingPong.png
336 The figure above presents the output for ping-pong. The process involves 4 phases:
338 1. **Quantile regression**: a quantile regression is made to have our baseline linear model. A quantile regression is used to avoid having negative intercepts and consequently negative estimate duration times.
339 2. **Intercept residuals**: from the quantile regression, we calculate the intercept for each message size (:math:`intercept = duration - coefficient*msg\_size`)
340 3. **Ckmeans**: creates a set of groups based on our intercept residuals. In the figure, each color represents a group.
341 4. **Normal distributions**: for each group found by ckmeans, we calculate the mean and standard deviation of that group. The probabilities are drawn from the density of each group (points in group/total number of points).
347 Ping-pong measures give us the round-trip estimated time, but we need
348 the elapsed time in 1 direction to inject in SimGrid.
350 For simplicity, we just scale down the normal distributions.
351 However, a proper calculation may be necessary at this step.
355 pingpong_models["coefficient"] = pingpong_models["coefficient"]/2
356 pingpong_models["mean"] = pingpong_models["mean"]/2
357 pingpong_models["sd"] = pingpong_models["sd"]/numpy.sqrt(2)
365 .dataframe tbody tr th:only-of-type {
366 vertical-align: middle;
369 .dataframe tbody tr th {
373 .dataframe thead th {
377 <table border="1" class="dataframe">
379 <tr style="text-align: right;">
393 <td>4.356809e-07</td>
395 <td>8.049632e-11</td>
397 <td>3.402823e+38</td>
402 <td>5.219426e-07</td>
404 <td>8.049632e-11</td>
406 <td>3.402823e+38</td>
411 <td>1.673437e-06</td>
413 <td>8.049632e-11</td>
415 <td>3.402823e+38</td>
420 <td>2.023256e-06</td>
422 <td>8.049632e-11</td>
424 <td>3.402823e+38</td>
429 <td>2.530620e-06</td>
431 <td>8.049632e-11</td>
433 <td>3.402823e+38</td>
438 <td>3.533823e-06</td>
440 <td>8.049632e-11</td>
442 <td>3.402823e+38</td>
450 This table presents the clustering results for Ping-pong. Each line represents a normal distribution that characterizes the noise along with its probability.
452 At our simulator, we'll draw our noise following these probabilities/distributions.
455 Finally, we dump the results in a JSON format. Below, we present the `pingpong_ckmeans.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/pingpong_ckmeans.json>`_ file.
457 This file will be read by your simulator later to generate the proper factor for network operations.
461 {'bandwidth_base': 12330818795.43382,
462 'latency_base': 1.2924904864614219e-05,
463 'seg': [{'mean': 1.1503128856516448e-05,
464 'sd': 4.3568091437319533e-07,
465 'prob': 0.49970588235294106,
466 'coefficient': 8.04963230919345e-11,
468 'max_x': 3.4028234663852886e+38},
469 {'mean': 1.2504551284320949e-05,
470 'sd': 5.219425841751762e-07,
471 'prob': 0.385196078431373,
472 'coefficient': 8.04963230919345e-11,
474 'max_x': 3.4028234663852886e+38},
475 {'mean': 1.879472592512515e-05,
476 'sd': 1.6734369316865939e-06,
477 'prob': 0.0733137254901961,
478 'coefficient': 8.04963230919345e-11,
480 'max_x': 3.4028234663852886e+38},
481 {'mean': 2.451754075327485e-05,
482 'sd': 2.0232563328989863e-06,
483 'prob': 0.0241078431372549,
484 'coefficient': 8.04963230919345e-11,
486 'max_x': 3.4028234663852886e+38},
487 {'mean': 3.004149952883e-05,
488 'sd': 2.5306204869242285e-06,
489 'prob': 0.0116960784313725,
490 'coefficient': 8.04963230919345e-11,
492 'max_x': 3.4028234663852886e+38},
493 {'mean': 3.688584189653765e-05,
494 'sd': 3.5338234385210185e-06,
495 'prob': 0.00598039215686275,
496 'coefficient': 8.04963230919345e-11,
498 'max_x': 3.4028234663852886e+38}]}
501 The same is done for each one of the MPI operations, creating the different input files: `pingpong_ckmeans.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/pingpong_ckmeans.json>`_, `isend_ckmeans.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/isend_ckmeans.json>`_, `recv_ckmeans.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/recv_ckmeans.json>`_, `send_ckmeans.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/send_ckmeans.json>`_.
504 4.2. Dhist (alternative 2)
505 --------------------------
507 Alternatively, we can model the noise using non-uniform histograms.
509 Diagonally cut histograms are used in this case, one histogram for each
512 The noise is later sampled according to these histograms.
514 Note: For better results, we had to apply a log function on the elapsed
515 time before running the dhist algorithm. However, it’s not clear why
516 this manipulation gives better results.
518 .. image:: /tuto_network_calibration/plot_dhist_PingPong.png
521 The figure presents the histogram for the ping-pong operation.
523 In the x-axis, we have the intercept residuals calculated using the linear models found by pycewise.
525 The vertical lines are the bins found by dhist. Note that the size of each bin varies depending on their density.
529 Ping-pong measures give us the round-trip estimated time, but we need
530 the elapsed time in 1 direction to inject in SimGrid. As we applied the log function on our data, we need a minor trick to calculate the elapsed time.
532 :math:`\frac{e^x}{2}` = :math:`e^{x + log(\frac{1}{2})}`
536 for i in pingpong_dhist:
537 i["xbr"] = [v + numpy.log(1/2) for v in i["xbr"]]
540 pingpong_dhist = {"bandwidth_base": bandwidth_base, "latency_base" : latency_base, "seg": pingpong_dhist}
546 {'bandwidth_base': 12330818795.43382,
547 'latency_base': 1.2924904864614219e-05,
548 'seg': [{'log': True,
550 'max_x': 3.4028234663852886e+38,
551 'xbr': [-11.541562041539144,
570 'height': [28047.5350565562,
588 'coeff': 8.10976153806028e-11}]}
590 This JSON file is read by the simulator to create the platform and generate the appropriate noise.
591 The same is done for each one of the MPI operations, creating the different input files: `pingpong_dhist.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/pingpong_dhist.json>`_, `isend_dhist.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/isend_dhist.json>`_, `recv_dhist.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/recv_dhist.json>`_, `send_dhist.json <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/send_dhist.json>`_.
593 5. Description of the platform in SimGrid
594 =========================================
596 At this point we have done the analysis and extracted the models in the several JSON files. It's possible now to create our platform file that will be used by SimGrid later.
598 The platform is created using the C++ interface from SimGrid. The result is a library file (.so) which is loaded by SimGrid when running the application.
600 The best to understand is reading the C++ code in `docs/source/tuto_network_calibration <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/>`_, the main files are:
602 - `dahu_platform_ckmeans.cpp <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/dahu_platform_ckmeans.cpp>`_: create the dahu platform using the JSON files from ckmeans.
603 - `dahu_platform_dhist.cpp <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/dahu_platform_dhist.cpp>`_: same for dhist output.
604 - `Utils.cpp <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/Utils.cpp>`_/`Utils.hpp <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/Utils.hpp>`_: some auxiliary classes used by both platforms to handle the segmentation and sampling.
605 - `CMakeLists.txt <https://framagit.org/simgrid/simgrid/tree/master/docs/source/tuto_network_calibration/CMakeLists.txt>`_: create the shared library to be loaded by SimGrid
607 Feel free to re-use and adapt these files according to your needs.
609 6. SimGrid execution and comparison
610 ===================================
615 **Ckmeans.1d.dp** and **Dhist**
617 The execution is similar for both modes. The only change is the platform library to be used: **libdahu_ckmeans.so** or **libdhist.so**.
624 cd /source/simgrid.git/docs/source/tuto_network_calibration/
626 smpirun --cfg=smpi/simulate-computation:0 \
627 --cfg=smpi/display-timing:yes \
628 -platform ./libdahu_ckmeans.so \
629 -hostfile /tmp/host.txt -np 2 \
630 /source/platform-calibration/src/calibration/calibrate -d /tmp/exp -m 1 -M 1000000 -p exp -s /tmp/exp.csv
635 Read bandwidth_base: 1.233082e+10 latency_base: 1.292490e-05
636 Starting parsing file: pingpong_ckmeans.json
637 Starting parsing file: send_ckmeans.json
638 Starting parsing file: isend_ckmeans.json
639 Starting parsing file: recv_ckmeans.json
641 [0] nb_exp=115200, largest_size=980284
642 [0] Alloc size: 1960568
644 [1] nb_exp=115200, largest_size=980284
645 [1] Alloc size: 1960568
646 [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/privatization' to '1'
647 [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/np' to '2'
648 [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/hostfile' to '/tmp/host.txt'
649 [0.000000] [xbt_cfg/INFO] Configuration change: Set 'surf/precision' to '1e-9'
650 [0.000000] [xbt_cfg/INFO] Configuration change: Set 'network/model' to 'SMPI'
651 [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/simulate-computation' to '0'
652 [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/display-timing' to 'yes'
653 [0.000000] [xbt_cfg/INFO] Configuration change: Set 'smpi/tmpdir' to '/tmp'
654 [0.000000] [smpi_config/INFO] You did not set the power of the host running the simulation. The timings will certainly not be accurate. Use the option "--cfg=smpi/host-speed:<flops>" to set its value. Check https://simgrid.org/doc/latest/Configuring_SimGrid.html#automatic-benchmarking-of-smpi-code for more information.
655 [6.845963] [smpi_utils/INFO] Simulated time: 6.84596 seconds.
657 The simulation took 71.6111 seconds (after parsing and platform setup)
658 1.77771 seconds were actual computation of the application
665 Finally, let’s compare the SimGrid results the real ones. The red points are the real data while the blue ones are the output from our simulator.
669 .. image:: /tuto_network_calibration/plot_op_simgrid_ckmeans.png
673 .. image:: /tuto_network_calibration/plot_op_simgrid_dhist.png
678 Note that for ping-ping tests, we have an important gap between the real
679 performance (in red) and SimGrid (in blue) for messages below our
680 sync/async threshold (63305).
682 This behavior is explained by how we measure the extra cost for each
683 MPI_Send/MPI_Recv operations.
685 In `calibrate.c <https://framagit.org/simgrid/platform-calibration/-/blob/master/src/calibration/calibrate.c>`_ in platform-calibration, the ping-pong test is as follows
686 (considering the processes are synchronized):
689 .. image:: /tuto_network_calibration/fig/pingpong_real.png
691 We can see that we measure the delay at **Process 1**, just before the
692 first *MPI_Send-1* until the end of respective *MPI_Recv-2*. Moreover,
693 the extra cost of MPI operations is paid concurrently with the network
696 In this case, it doesn't matter when the *MPI_Send-2* will finish.
697 Despite we expect that it finished before the *MPI_Recv-2*, we couldn't
700 Also, both processes are running in parallel, so we can expect that the
701 measure time will be:
702 :math:`max(\text{MPI_Send-1}, \text{MPI_Recv-1}) + \text{MPI_Recv-2}` -
703 :math:`max(\text{MPI_Send-1}, \text{MPI_Recv-1})`: since we cannot start
704 *MPI_Recv-2* or *MPI_Send_2* before finishing both commands -
705 :math:`\text{MPI_Recv-2}`: because we measure just after the finishing
708 However, the simulation world is a little more stable. The same
709 communication occurs in the following way:
712 .. image:: /tuto_network_calibration/fig/pingpong_simgrid.png
714 In SimGrid, the extra costs are paid sequentially. That means, initially
715 we pay the extra cost for *MPI_Send-1*, after the network communication
716 cost, followed by the extra cost for *MPI-Recv-1*.
718 This effect leads to a total time of: *MPI_Send-1* + *MPI_Recv-1* +
719 *MPI_Send-2* + *MPI_Recv-2* which is slightly higher than the real cost.
721 The same doesn't happen for largest messages because we don’t pay the
722 extra overhead cost for each MPI operation (the communication is limited
723 by the network capacity).