\usepackage{graphicx}
\usepackage[american]{babel}
% Extension pour les liens intra-documents (tagged PDF)
-% et l'affichage correct des URL (commande \url{http://example.com})
+% et l'affichage correct des UR (commande \url{http://example.com})
%\usepackage{hyperref}
\usepackage{url}
iterative ones. Performing large scale experiments with different kind of
network parameters is not easy because with supercomputers such parameters are
fixed. So, one solution consists in using simulations first in order to analyze
-what parameters could influence or not the behaviors of an algorithm. In this
-paper, we show that it is interesting to use SimGrid to simulate the behaviors
+what parameters could influence or not the behavior of an algorithm. In this
+paper, we show that it is interesting to use SimGrid to simulate the behavior
of asynchronous iterative algorithms. For that, we compare the behavior of a
synchronous GMRES algorithm with an asynchronous multisplitting one with
simulations which let us easily choose some parameters. Both codes are real MPI
\section{Motivations and scientific context}
-As exposed in the introduction, parallel iterative methods are now widely used
-in many scientific domains. They can be classified in three main classes
-depending on how iterations and communications are managed (for more details
-readers can refer to~\cite{bcvc06:ij}). In the synchronous iterations model,
-data are exchanged at the end of each iteration. All the processors must begin
-the same iteration at the same time and important idle times on processors are
+As described in the introduction, parallel iterative methods are now widely used
+in many scientific domains. They can be classified in three main classes
+depending on how iterations and communications are managed (for more details
+readers can refer to~\cite{bcvc06:ij}). In the synchronous iterations model,
+data are exchanged at the end of each iteration. All the processors must begin
+the same iteration at the same time and important and useless idle times used for synchronization on processors are
generated. It is possible to use asynchronous communications, in this case, the
-model can be compared to the previous one except that data required on another
-processor are sent asynchronously i.e. without stopping current computations.
-This technique allows communications to be partially overlapped by computations but
-unfortunately, the overlapping is only partial and important idle times remain.
-It is clear that, in a grid computing context, where the number of computational
-nodes is large, heterogeneous and widely distributed, the idle times generated
-by synchronizations are very penalizing. One way to overcome this problem is to
-use the asynchronous iterations model. Here, local computations do not need to
-wait for required data. Processors can then perform their iterations with the
-data present at that time. Figure~\ref{fig:aiac} illustrates this model where
-the gray blocks represent the computation phases. With this algorithmic model,
-the number of iterations required before the convergence is generally greater
-than for the two former classes. But, and as detailed in~\cite{bcvc06:ij},
-asynchronous iterative algorithms can significantly reduce overall execution
-times by suppressing idle times due to synchronizations especially in a grid
-computing context.
+model can be compared to the previous one except that data required on another
+processor are sent asynchronously i.e. without stopping current computations.
+This technique allows communications to be partially overlapped by computations
+but unfortunately, the overlapping is only partial and useless idle times used for synchronization remain.
+It is clear that, in a grid computing context, where the number of
+computational nodes is large, heterogeneous and widely distributed, the idle
+times generated by synchronizations are very penalizing. One way to overcome
+this problem is to use the asynchronous iterations model. Here, local
+computations do not need to wait for required data. Processors can then perform
+their iterations with the data present at that time. Figure~\ref{fig:aiac}
+illustrates this model where the gray blocks represent the computation phases.
+With this algorithmic model, the number of iterations required before the
+convergence is generally greater than for the two former classes. But, and as
+detailed in~\cite{bcvc06:ij}, asynchronous iterative algorithms can
+significantly reduce overall execution times by suppressing idle times due to
+synchronizations especially in a grid computing context.
\begin{figure}[!t]
\centering
multisplitting code will obtain the solution more or less quickly. Of course,
the GMRES method also depends on the same parameters. As it is difficult to have
access to many clusters, grids or supercomputers with many different network
-parameters, it is interesting to be able to simulate the behaviors of
+parameters, it is interesting to be able to simulate the behavior of
asynchronous iterative algorithms before being able to run real experiments.
\section{SimGrid}
-SimGrid~\cite{SimGrid,casanova+legrand+quinson.2008.simgrid} is a simulation
-framework to study the behavior of large-scale distributed systems. As its name
-suggests, it emanates from the grid computing community, but is nowadays used to
-study grids, clouds, HPC or peer-to-peer systems. The early versions of SimGrid
-date back from 1999, but it is still actively developed and distributed as an open
-source software. Today, it is one of the major generic tools in the field of
-simulation for large-scale distributed systems.
+SimGrid~\cite{SimGrid,casanova+legrand+quinson.2008.simgrid,casanova+giersch+legrand+al.2014.versatile}
+is a simulation framework to study the behavior of large-scale distributed
+systems. As its name suggests, it emanates from the grid computing community,
+but is nowadays used to study grids, clouds, HPC or peer-to-peer systems. The
+early versions of SimGrid date back from 1999, but it is still actively
+developed and distributed as an open source software. Today, it is one of the
+major generic tools in the field of simulation for large-scale distributed
+systems.
SimGrid provides several programming interfaces: MSG to simulate Concurrent
Sequential Processes, SimDAG to simulate DAGs of (parallel) tasks, and SMPI to
run real applications written in MPI~\cite{MPI}. Apart from the native C
interface, SimGrid provides bindings for the C++, Java, Lua and Ruby programming
-languages. SMPI is the interface that has been used for the work exposed in
+languages. SMPI is the interface that has been used for the work described in
this paper. The SMPI interface implements about \np[\%]{80} of the MPI 2.0
standard~\cite{bedaride+degomme+genaud+al.2013.toward}, and supports
-applications written in C or Fortran, with little or no modifications.
+applications written in C or Fortran, with little or no modifications (cf Section IV - paragraph B).
Within SimGrid, the execution of a distributed application is simulated by a
single process. The application code is really executed, but some operations,
like communications, are intercepted, and their running time is computed
according to the characteristics of the simulated execution platform. The
description of this target platform is given as an input for the execution, by
-the mean of an XML file. It describes the properties of the platform, such as
+means of an XML file. It describes the properties of the platform, such as
the computing nodes with their computing power, the interconnection links with
their bandwidth and latency, and the routing strategy. The scheduling of the
simulated processes, as well as the simulated running time of the application
the local convergence on each cluster $\ell$ is detected when the following
condition is satisfied
\begin{equation*}
- (k\leq \MI) \text{ or } (\|X_\ell^k - X_\ell^{k+1}\|_{\infty}\leq\epsilon)
+ (k=\MI) \text{ or } (\|X_\ell^k - X_\ell^{k+1}\|_{\infty}\leq\epsilon)
\end{equation*}
where $\MI$ is the maximum number of outer iterations and $\epsilon$ is the
tolerance threshold of the error computed between two successive local solution
-$X_\ell^k$ and $X_\ell^{k+1}$.
+$X_\ell^k$ and $X_\ell^{k+1}$. It should be noted that with asynchronous iterative algorithms, we cannot use a classical norm (which would require to synchronize all processors), such as $\|X_\ell^k - X_\ell^{k+1}\|_{2}$ for example. Nevertheless, in our experiments, we check that the final result is correct, for this we compute the precision with $max_i | A*x-b |_i$.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-We did not encounter major blocking problems when adapting the multisplitting algorithm previously described to a simulation environment like SimGrid unless some code
-debugging. Indeed, apart from the review of the program sequence for asynchronous exchanges between processors within a cluster or between clusters, the algorithm was executed successfully with SMPI and provided identical outputs as those obtained with direct execution under MPI. For the synchronous GMRES method, the execution of the program raised no particular issue but in the asynchronous multisplitting method, the review of the sequence of \texttt{MPI\_Isend, MPI\_Irecv} and \texttt{MPI\_Waitall} instructions
+We did not encounter major blocking problems when adapting the multisplitting algorithm previously described to a simulation environment like SimGrid. Only, some code
+debugging has been required. Indeed, apart from the review of the program sequence for asynchronous exchanges between processors within a cluster or between clusters, the algorithm was executed successfully with SMPI and provided identical outputs as those obtained with direct execution under MPI. For the synchronous GMRES method, the execution of the program raised no particular issue but in the asynchronous multisplitting method, the review of the sequence of \texttt{MPI\_Isend, MPI\_Irecv} and \texttt{MPI\_Waitall} instructions
and with the addition of the primitive \texttt{MPI\_Test} was needed to avoid a memory fault due to an infinite loop resulting from the non-convergence of the algorithm.
%\CER{On voulait en fait montrer la simplicité de l'adaptation de l'algo a SimGrid. Les problèmes rencontrés décrits dans ce paragraphe concerne surtout le mode async}\LZK{OK. J'aurais préféré avoir un peu plus de détails sur l'adaptation de la version async}
%\CER{Le problème majeur sur l'adaptation MPI vers SMPI pour la partie asynchrone de l'algorithme a été le plantage en SMPI de Waitall après un Isend et Irecv. J'avais proposé un workaround en utilisant un MPI\_wait séparé pour chaque échange a la place d'un waitall unique pour TOUTES les échanges, une instruction qui semble bien fonctionner en MPI. Ce workaround aussi fonctionne bien. Mais après, tu as modifié le programme avec l'ajout d'un MPI\_Test, au niveau de la routine de détection de la convergence et du coup, l'échange global avec waitall a aussi fonctionné.}
\begin{itemize}
\item At the network level, we found that the most critical values are the
bandwidth and the network latency.
-\item Hosts processors power (GFlops) can also influence the results.
+\item Host processor power (GFlops) can also influence the results.
\item Finally, when submitting job batches for execution, the arguments values
passed to the program like the maximum number of iterations or the precision are critical. They allow us to ensure not only the convergence of the
algorithm but also to get the main objective in getting an execution time with the asynchronous multisplitting less than with synchronous GMRES.
increasing the matrix size up to $100^3$ elements, it was necessary to increase the
CPU power by \np[\%]{50} to \np[GFlops]{1.5} to get the algorithm convergence and the same order of asynchronous mode efficiency. Maintaining a relative gain of $2.5$ and such processor power but increasing network throughput inter cluster up to \np[Mbit/s]{50}, is obtained with
high external precision of \np{E-11} for a matrix size from $110^3$ to $150^3$ side
-elements.
+elements.
%For the 3 clusters architecture including a total of 100 hosts,
%Table~\ref{tab.cluster.3x33} shows that it was difficult to have a combination
%\CER{Définitivement, les paramètres réseaux variables ici se rapportent au réseau INTER cluster.}
\section{Conclusion}
The simulation of the execution of parallel asynchronous iterative algorithms on large scale clusters has been presented.
-In this work, we show that SimGrid is an efficient simulation tool that has enabled us to
+In this work, we show that SimGrid is one of efficient simulation tool that has enabled us to
reach the following two objectives:
\begin{enumerate}
