X-Git-Url: https://bilbo.iut-bm.univ-fcomte.fr/and/gitweb/hpcc2014.git/blobdiff_plain/0d3de76a4d7b1dae39911fa91801f25f009d3cf9..3fa7ab9f6999f6f13a72e4342cfa175e6e74803f:/hpcc.tex?ds=inline

diff --git a/hpcc.tex b/hpcc.tex
index 5e4bea6..edba67f 100644
--- a/hpcc.tex
+++ b/hpcc.tex
@@ -1,10 +1,11 @@
+
 \documentclass[conference]{IEEEtran}
 
 \usepackage[T1]{fontenc}
 \usepackage[utf8]{inputenc}
 \usepackage{amsfonts,amssymb}
 \usepackage{amsmath}
-\usepackage{algorithm}
+%\usepackage{algorithm}
 \usepackage{algpseudocode}
 %\usepackage{amsthm}
 \usepackage{graphicx}
@@ -25,6 +26,14 @@
 \usepackage[textsize=footnotesize]{todonotes}
 \newcommand{\AG}[2][inline]{%
   \todo[color=green!50,#1]{\sffamily\textbf{AG:} #2}\xspace}
+\newcommand{\DL}[2][inline]{%
+  \todo[color=yellow!50,#1]{\sffamily\textbf{DL:} #2}\xspace}
+\newcommand{\LZK}[2][inline]{%
+  \todo[color=blue!10,#1]{\sffamily\textbf{LZK:} #2}\xspace}
+\newcommand{\RC}[2][inline]{%
+  \todo[color=red!10,#1]{\sffamily\textbf{RC:} #2}\xspace}
+\newcommand{\CER}[2][inline]{%
+  \todo[color=pink!10,#1]{\sffamily\textbf{CER:} #2}\xspace}
 
 \algnewcommand\algorithmicinput{\textbf{Input:}}
 \algnewcommand\Input{\item[\algorithmicinput]}
@@ -32,318 +41,696 @@
 \algnewcommand\algorithmicoutput{\textbf{Output:}}
 \algnewcommand\Output{\item[\algorithmicoutput]}
 
+\newcommand{\MI}{\mathit{MaxIter}}
+\newcommand{\Time}[1]{\mathit{Time}_\mathit{#1}}
 
 \begin{document}
 
-\title{Simulation of Asynchronous Iterative Numerical Algorithms Using SimGrid}
+\title{Simulation of Asynchronous Iterative Algorithms Using SimGrid}
 
 \author{%
   \IEEEauthorblockN{%
-    Charles Emile Ramamonjisoa and
-    David Laiymani and
-    Arnaud Giersch and
-    Lilia Ziane Khodja and
-    RaphaÃ«l Couturier
+    Charles Emile Ramamonjisoa\IEEEauthorrefmark{1},
+    Lilia Ziane Khodja\IEEEauthorrefmark{2},
+    David Laiymani\IEEEauthorrefmark{1},
+    Arnaud Giersch\IEEEauthorrefmark{1} and
+    RaphaÃ«l Couturier\IEEEauthorrefmark{1}
+  }
+  \IEEEauthorblockA{\IEEEauthorrefmark{1}%
+    Femto-ST Institute -- DISC Department\\
+    UniversitÃ© de Franche-ComtÃ©,
+    IUT de Belfort-MontbÃ©liard\\
+    19 avenue du MarÃ©chal Juin, BP 527, 90016 Belfort cedex, France\\
+    Email: \email{{charles.ramamonjisoa,david.laiymani,arnaud.giersch,raphael.couturier}@univ-fcomte.fr}
   }
-  \IEEEauthorblockA{%
-    Femto-ST Institute - DISC Department\\
-    UniversitÃ© de Franche-ComtÃ©\\
-    Belfort\\
-    Email: \email{{raphael.couturier,arnaud.giersch,david.laiymani,charles.ramamonjisoa}@univ-fcomte.fr}
+  \IEEEauthorblockA{\IEEEauthorrefmark{2}%
+    Inria Bordeaux Sud-Ouest\\
+    200 avenue de la Vieille Tour, 33405 Talence cedex, France \\
+    Email: \email{lilia.ziane@inria.fr}
   }
 }
 
 \maketitle
 
-\AG{Ordre des autheurs pas dÃ©finitif}
 \begin{abstract}
-The abstract goes here.
+
+Synchronous  iterative  algorithms  are  often less  scalable  than  asynchronous
+iterative  ones.  Performing  large  scale experiments  with  different kind  of
+network parameters is not easy  because with supercomputers such parameters are
+fixed. So one  solution consists in using simulations first  in order to analyze
+what parameters  could influence or not  the behaviors of an  algorithm. In this
+paper, we show  that it is interesting to use SimGrid  to simulate the behaviors
+of asynchronous  iterative algorithms. For that,  we compare the  behaviour of a
+synchronous  GMRES  algorithm  with  an  asynchronous  multisplitting  one  with
+simulations  in  which we  choose  some parameters.   Both  codes  are real  MPI
+codes. Simulations allow us to see when the multisplitting algorithm can be more
+efficient than the GMRES one to solve a 3D Poisson problem.
+
+
+% no keywords for IEEE conferences
+% Keywords: Algorithm distributed iterative asynchronous simulation SimGrid
 \end{abstract}
 
 \section{Introduction}
 
-Parallel computing and high performance computing (HPC) are becoming 
-more and more imperative for solving various problems raised by 
-researchers on various scientific disciplines but also by industrial in 
-the field. Indeed, the increasing complexity of these requested 
-applications combined with a continuous increase of their sizes lead to 
-write distributed and parallel algorithms requiring significant hardware 
-resources (grid computing, clusters, broadband network, etc\dots{}) but
-also a non-negligible CPU execution time. We consider in this paper a
-class of highly efficient parallel algorithms called iterative executed 
-in a distributed environment. As their name suggests, these algorithm 
-solves a given problem that might be NP- complete complex by successive 
-iterations ($X_{n +1} = f(X_{n})$) from an initial value $X_{0}$ to find
-an approximate value $X^*$ of the solution with a very low
-residual error. Several well-known methods demonstrate the convergence 
-of these algorithms. Generally, to reduce the complexity and the 
-execution time, the problem is divided into several \emph{pieces} that will
-be solved in parallel on multiple processing units. The latter will 
-communicate each intermediate results before a new iteration starts 
-until the approximate solution is reached. These distributed parallel 
-computations can be performed either in \emph{synchronous} communication mode
-where a new iteration begin only when all nodes communications are 
-completed, either \emph{asynchronous} mode where processors can continue
-independently without or few synchronization points. Despite the 
-effectiveness of iterative approach, a major drawback of the method is 
-the requirement of huge resources in terms of computing capacity, 
-storage and high speed communication network. Indeed, limited physical 
-resources are blocking factors for large-scale deployment of parallel 
-algorithms. 
-
-In recent years, the use of a simulation environment to execute parallel 
-iterative algorithms found some interests in reducing the highly cost of 
-access to computing resources: (1) for the applications development life 
-cycle and in code debugging (2) and in production to get results in a 
-reasonable execution time with a simulated infrastructure not accessible 
-with physical resources. Indeed, the launch of distributed iterative 
-asynchronous algorithms to solve a given problem on a large-scale 
-simulated environment challenges to find optimal configurations giving 
-the best results with a lowest residual error and in the best of 
-execution time. According our knowledge, no testing of large-scale 
-simulation of the class of algorithm solving to achieve real results has 
-been undertaken to date. We had in the scope of this work implemented a 
-program for solving large non-symmetric linear system of equations by 
-numerical method GMRES (Generalized Minimal Residual) in the simulation
-environment SimGrid. The simulated platform had allowed us to launch
-the application from a modest computing infrastructure by simulating 
-different distributed architectures composed by clusters nodes 
-interconnected by variable speed networks. In addition, it has been 
-permitted to show the effectiveness of asynchronous mode algorithm by 
-comparing its performance with the synchronous mode time. With selected 
-parameters on the network platforms (bandwidth, latency of inter cluster 
-network) and on the clusters architecture (number, capacity calculation 
-power) in the simulated environment, the experimental results have
-demonstrated not only the algorithm convergence within a reasonable time 
-compared with the physical environment performance, but also a time 
-saving of up to \np[\%]{40} in asynchronous mode.
-
-This article is structured as follows: after this introduction, the next 
-section will give a brief description of iterative asynchronous model. 
-Then, the simulation framework SimGrid will be presented with the
-settings to create various distributed architectures. The algorithm of 
-the multi -splitting method used by GMRES written with MPI primitives 
-and its adaptation to SimGrid with SMPI (Simulated MPI) will be in the
-next section. At last, the experiments results carried out will be
-presented before the conclusion which we will announce the opening of 
-our future work after the results.
- 
-\section{The asynchronous iteration model}
+Parallel computing and high performance computing (HPC) are becoming  more and more imperative for solving various
+problems raised by  researchers on various scientific disciplines but also by industrial in  the field. Indeed, the
+increasing complexity of these requested  applications combined with a continuous increase of their sizes lead to  write
+distributed and parallel algorithms requiring significant hardware  resources (grid computing, clusters, broadband
+network, etc.) but also a non-negligible CPU execution time. We consider in this paper a class of highly efficient
+parallel algorithms called \emph{iterative algorithms} executed in a distributed environment. As their name
+suggests, these algorithms solve a given problem by successive iterations ($X_{n +1} = f(X_{n})$) from an initial value
+$X_{0}$ to find an approximate value $X^*$ of the solution with a very low residual error. Several well-known methods
+demonstrate the convergence of these algorithms~\cite{BT89,Bahi07}.
+
+Parallelization of such algorithms generally involve the division of the problem
+into  several  \emph{blocks}  that  will  be  solved  in  parallel  on  multiple
+processing units. The latter will communicate each intermediate results before a
+new  iteration starts  and until  the  approximate solution  is reached.   These
+parallel computations can be performed either in \emph{synchronous} mode where a
+new iteration  begins only  when all nodes  communications are completed,  or in
+\emph{asynchronous}  mode where  processors can  continue independently  with no
+synchronization points~\cite{bcvc06:ij}. In this case, local computations do not
+need to  wait for  required data. Processors  can then perform  their iterations
+with the  data present at that time.  Even if the number  of iterations required
+before  the convergence  is generally  greater  than for  the synchronous  case,
+asynchronous  iterative algorithms  can significantly  reduce  overall execution
+times by  suppressing idle  times due to  synchronizations especially in  a grid
+computing context (see~\cite{Bahi07} for more details).
+
+Parallel applications  based on a (synchronous or  asynchronous) iteration model
+may have different configuration and deployment requirements.  Quantifying their
+resource  allocation  policies and  application  scheduling  algorithms in  grid
+computing environments under varying load,  CPU power and network speeds is very
+costly,       very        labor       intensive       and        very       time
+consuming~\cite{Calheiros:2011:CTM:1951445.1951450}.   The case  of asynchronous
+iterative algorithms  is even more problematic  since they are  very sensible to
+the  execution environment  context.  For instance,  variations  in the  network
+bandwidth (intra and  inter-clusters), in the number and the  power of nodes, in
+the number  of clusters\dots{} can lead  to very different  number of iterations
+and so  to very  different execution times.   Then, it  appears that the  use of
+simulation tools to explore various  platform scenarios and to run large numbers
+of  experiments quickly  can  be very  promising.  In  this  way, the  use of  a
+simulation  environment  to execute  parallel  iterative  algorithms found  some
+interests in reducing the highly cost  of access to computing resources: (1) for
+the  applications  development life  cycle  and in  code  debugging  (2) and  in
+production  to get  results  in a  reasonable  execution time  with a  simulated
+infrastructure not  accessible with physical  resources.  Indeed, the  launch of
+distributed  iterative asynchronous  algorithms to  solve a  given problem  on a
+large-scale  simulated  environment challenges  to  find optimal  configurations
+giving  the best  results  with  a lowest  residual  error and  in  the best  of
+execution time.
+
+
+To our knowledge,  there is no existing work on the  large-scale simulation of a
+real asynchronous  iterative application.  {\bf The contribution  of the present
+  paper can be  summarised in two main points}.  First we  give a first approach
+of the simulation  of asynchronous iterative algorithms using  a simulation tool
+(i.e.    the   SimGrid   toolkit~\cite{SimGrid}).    Second,  we   confirm   the
+effectiveness  of the  asynchronous  multisplitting algorithm  by comparing  its
+performance   with  the   synchronous  GMRES   (Generalized   Minimal  Residual)
+\cite{ref1}.  Both  these codes can  be used to  solve large linear  systems. In
+this  paper, we  focus  on  a 3D  Poisson  problem.  We  show,  that with  minor
+modifications of the initial MPI code,  the SimGrid toolkit allows us to perform
+a  test campaign  of  a  real asynchronous  iterative  application on  different
+computing architectures.
+% The  simulated results  we
+%obtained are  in line with real  results exposed in  ??\AG[]{ref?}. 
+SimGrid  had  allowed us  to  launch the  application  from  a modest  computing
+infrastructure  by simulating  different distributed  architectures  composed by
+clusters  nodes interconnected by  variable speed  networks.  Parameters  of the
+network  platforms  are   the  bandwidth  and  the  latency   of  inter  cluster
+network. Parameters on the cluster's architecture are the number of machines and
+the  computation power  of a  machine.  Simulations show  that the  asynchronous
+multisplitting algorithm  can solve the  3D Poisson problem  approximately twice
+faster than GMRES with two distant clusters.
+
+
+
+This article is structured as follows: after this introduction, the next section
+will  give a  brief  description  of iterative  asynchronous  model.  Then,  the
+simulation framework  SimGrid is presented  with the settings to  create various
+distributed architectures.  Then, the  multisplitting method is presented, it is
+based  on GMRES to  solve each  block obtained  of the  splitting. This  code is
+written with MPI  primitives and its adaptation to  SimGrid with SMPI (Simulated
+MPI) is  detailed in the next  section. At last, the  simulation results carried
+out will be presented before some concluding remarks and future works.
 
-DÃ©crire le modÃ¨le asynchrone. Je m'en charge (DL)
-
-\section{SimGrid}
+ 
+\section{Motivations and scientific context}
+
+As exposed in  the introduction, parallel iterative methods  are now widely used
+in  many scientific  domains.   They can  be  classified in  three main  classes
+depending on  how iterations  and communications are  managed (for  more details
+readers  can refer  to~\cite{bcvc06:ij}). In  the synchronous  iterations model,
+data are exchanged  at the end of each iteration. All  the processors must begin
+the same iteration  at the same time and important idle  times on processors are
+generated.  It is possible to use asynchronous communications, in this case, the
+model can be  compared to the previous one except that  data required on another
+processor are  sent asynchronously i.e.  without  stopping current computations.
+This technique  allows to partially  overlap communications by  computations but
+unfortunately, the overlapping is only  partial and important idle times remain.
+It is clear that, in a grid computing context, where the number of computational
+nodes is large,  heterogeneous and widely distributed, the  idle times generated
+by synchronizations are very penalizing. One  way to overcome this problem is to
+use the asynchronous iterations model.   Here, local computations do not need to
+wait for  required data. Processors can  then perform their  iterations with the
+data present  at that time.  Figure~\ref{fig:aiac} illustrates  this model where
+the gray blocks represent the  computation phases.  With this algorithmic model,
+the number  of iterations required  before the convergence is  generally greater
+than  for the  two former  classes.  But,  and as  detailed in~\cite{bcvc06:ij},
+asynchronous  iterative algorithms  can significantly  reduce  overall execution
+times by  suppressing idle  times due to  synchronizations especially in  a grid
+computing context.
+
+\begin{figure}[!t]
+  \centering
+    \includegraphics[width=8cm]{AIAC.pdf}
+  \caption{The asynchronous iterations model}
+  \label{fig:aiac}
+\end{figure}
+
+
+%% It is very challenging to develop efficient applications for large scale,
+%% heterogeneous and distributed platforms such as computing grids. Researchers and
+%% engineers have to develop techniques for maximizing application performance of
+%% these multi-cluster platforms, by redesigning the applications and/or by using
+%% novel algorithms that can account for the composite and heterogeneous nature of
+%% the platform. Unfortunately, the deployment of such applications on these very
+%% large scale systems is very costly, labor intensive and time consuming. In this
+%% context, it appears that the use of simulation tools to explore various platform
+%% scenarios at will and to run enormous numbers of experiments quickly can be very
+%% promising. Several works\dots{}
+
+%% \AG{Several works\dots{} what?\\
+%  Le paragraphe suivant se trouve dÃ©jÃ  dans l'intro ?}
+In the context of asynchronous algorithms, the number of iterations to reach the
+convergence depends on  the delay of messages. With  synchronous iterations, the
+number of  iterations is exactly  the same than  in the sequential mode  (if the
+parallelization process does  not change the algorithm). So  the difficulty with
+asynchronous iteratie algorithms comes from the fact it is necessary to run the algorithm
+with real data. In fact, from an execution to another the order of messages will
+change and the  number of iterations to reach the  convergence will also change.
+According  to all  the parameters  of the  platform (number  of nodes,  power of
+nodes,  inter  and  intra clusrters  bandwith  and  latency,  ....) and  of  the
+algorithm  (number   of  splitting  with  the   multisplitting  algorithm),  the
+multisplitting code  will obtain the solution  more or less  quickly. Or course,
+the GMRES method also depends of the same parameters. As it is difficult to have
+access to  many clusters,  grids or supercomputers  with many  different network
+parameters,  it  is  interesting  to  be  able  to  simulate  the  behaviors  of
+asynchronous iterative algoritms before being able to runs real experiments.
 
-DÃ©crire SimGrid (Arnaud)
 
 
 
 
 
+\section{SimGrid}
 
+SimGrid~\cite{SimGrid,casanova+legrand+quinson.2008.simgrid} is a simulation
+framework to study the behavior of large-scale distributed systems.  As its name
+says, it emanates from the grid computing community, but is nowadays used to
+study grids, clouds, HPC or peer-to-peer systems.  The early versions of SimGrid
+date from 1999, but it's still actively developed and distributed as an open
+source software.  Today, it's one of the major generic tools in the field of
+simulation for large-scale distributed systems.
+
+SimGrid provides several programming interfaces: MSG to simulate Concurrent
+Sequential Processes, SimDAG to simulate DAGs of (parallel) tasks, and SMPI to
+run real applications written in MPI~\cite{MPI}.  Apart from the native C
+interface, SimGrid provides bindings for the C++, Java, Lua and Ruby programming
+languages.  SMPI is the interface that has been used for the work exposed in
+this paper.  The SMPI interface implements about \np[\%]{80} of the MPI 2.0
+standard~\cite{bedaride:hal-00919507}, and supports applications written in C or
+Fortran, with little or no modifications.
+
+Within SimGrid, the execution of a distributed application is simulated on a
+single machine.  The application code is really executed, but some operations
+like the communications are intercepted, and their running time is computed
+according to the characteristics of the simulated execution platform.  The
+description of this target platform is given as an input for the execution, by
+the mean of an XML file.  It describes the properties of the platform, such as
+the computing nodes with their computing power, the interconnection links with
+their bandwidth and latency, and the routing strategy.  The simulated running
+time of the application is computed according to these properties.
+
+To compute the durations of the operations in the simulated world, and to take
+into account resource sharing (e.g. bandwidth sharing between competing
+communications), SimGrid uses a fluid model.  This allows to run relatively fast
+simulations, while still keeping accurate
+results~\cite{bedaride:hal-00919507,tomacs13}.  Moreover, depending on the
+simulated application, SimGrid/SMPI allows to skip long lasting computations and
+to only take their duration into account.  When the real computations cannot be
+skipped, but the results have no importance for the simulation results, there is
+also the possibility to share dynamically allocated data structures between
+several simulated processes, and thus to reduce the whole memory consumption.
+These two techniques can help to run simulations at a very large scale.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \section{Simulation of the multisplitting method}
 %DÃ©crire le problÃ¨me (algo) traitÃ© ainsi que le processus d'adaptation Ã  SimGrid.
-Let $Ax=b$ be a large sparse system of $n$ linear equations in $\mathbb{R}$, where $A$ is a sparse square and nonsingular matrix, $x$ is the solution vector and $y$ is the right-hand side vector. We use a multisplitting method based on the block Jacobi partitioning to solve this linear system on a large scale platform composed of $L$ clusters of processors. In this case, we apply a row-by-row splitting without overlapping  
-\[
-\left(\begin{array}{ccc}
-A_{11} & \cdots & A_{1L} \\
-\vdots & \ddots & \vdots\\
-A_{L1} & \cdots & A_{LL}
-\end{array} \right)
-\times 
-\left(\begin{array}{c}
-X_1 \\
-\vdots\\
-X_L
-\end{array} \right)
-=
-\left(\begin{array}{c}
-Y_1 \\
-\vdots\\
-Y_L
-\end{array} \right)\] 
-in such a way that successive rows of matrix $A$ and both vectors $x$ and $b$ are assigned to one cluster, where for all $l,i\in\{1,\ldots,L\}$ $A_{li}$ is a rectangular block of $A$ of size $n_l\times n_i$, $X_l$ and $Y_l$ are sub-vectors of $x$ and $y$, respectively, each of size $n_l$ and $\sum_{l} n_l=\sum_{i} n_i=n$.
-
-The multisplitting method proceeds by iteration to solve in parallel the linear system by $L$ clusters of processors, in such a way each sub-system
+Let $Ax=b$ be a large sparse system of $n$ linear equations in $\mathbb{R}$, where $A$ is a sparse square and nonsingular matrix, $x$ is the solution vector and $b$ is the right-hand side vector. We use a multisplitting method based on the block Jacobi splitting to solve this linear system on a large scale platform composed of $L$ clusters of processors~\cite{o1985multi}. In this case, we apply a row-by-row splitting without overlapping  
+\begin{equation*}
+  \left(\begin{array}{ccc}
+      A_{11} & \cdots & A_{1L} \\
+      \vdots & \ddots & \vdots\\
+      A_{L1} & \cdots & A_{LL}
+    \end{array} \right)
+  \times
+  \left(\begin{array}{c}
+      X_1 \\
+      \vdots\\
+      X_L
+    \end{array} \right)
+  =
+  \left(\begin{array}{c}
+      B_1 \\
+      \vdots\\
+      B_L
+    \end{array} \right)
+\end{equation*}
+in such a way that successive rows of matrix $A$ and both vectors $x$ and $b$
+are assigned to one cluster, where for all $\ell,m\in\{1,\ldots,L\}$, $A_{\ell
+  m}$ is a rectangular block of $A$ of size $n_\ell\times n_m$, $X_\ell$ and
+$B_\ell$ are sub-vectors of $x$ and $b$, respectively, of size $n_\ell$ each,
+and $\sum_{\ell} n_\ell=\sum_{m} n_m=n$.
+
+The multisplitting method proceeds by iteration to solve in parallel the linear system on $L$ clusters of processors, in such a way each sub-system
 \begin{equation}
-\left\{
-\begin{array}{l}
-A_{ll}X_l = Y_l \mbox{,~such that}\\
-Y_l = B_l - \displaystyle\sum_{i=1,i\neq l}^{L}A_{li}X_i,
-\end{array}
-\right.
-\label{eq:4.1}
+  \label{eq:4.1}
+  \left\{
+    \begin{array}{l}
+      A_{\ell\ell}X_\ell = Y_\ell \text{, such that}\\
+      Y_\ell = B_\ell - \displaystyle\sum_{\substack{m=1\\ m\neq \ell}}^{L}A_{\ell m}X_m
+    \end{array}
+  \right.
 \end{equation}
-is solved independently by a cluster and communication are required to update the right-hand side sub-vectors $Y_l$, such that the sub-vectors $X_i$ represent the data dependencies between the clusters. As each sub-system (\ref{eq:4.1}) is solved in parallel by a cluster of processors, our multisplitting method uses an iterative method as an inner solver which is easier to parallelize and more scalable than a direct method. In this work, we use the parallel GMRES method~\cite{ref1} which is one of the most used iterative method by many researchers. 
-
-\begin{algorithm}
-\caption{A multisplitting solver with inner iteration GMRES method}
+is solved independently by a cluster and communications are required to update
+the right-hand side sub-vector $Y_\ell$, such that the sub-vectors $X_m$
+represent the data dependencies between the clusters. As each sub-system
+(\ref{eq:4.1}) is solved in parallel by a cluster of processors, our
+multisplitting method uses an iterative method as an inner solver which is
+easier to parallelize and more scalable than a direct method. In this work, we
+use the parallel algorithm of GMRES method~\cite{ref1} which is one of the most
+used iterative method by many researchers.
+
+\begin{figure}[!t]
+  %%% IEEE instructions forbid to use an algorithm environment here, use figure
+  %%% instead
 \begin{algorithmic}[1]
-\Input $A_l$ (local sparse matrix), $B_l$ (local right-hand side), $x^0$ (initial guess)
-\Output $X_l$ (local solution vector)\vspace{0.2cm}
-\State Load $A_l$, $B_l$, $x^0$
-\State Initialize the shared vector $\hat{x}=x^0$
-\For {$k=1,2,3,\ldots$ until the global convergence}
-\State $x^0=\hat{x}$
-\State Inner iteration solver: \Call{InnerSolver}{$x^0$, $k$}
-\State Exchange the local solution ${X}_l^k$ with the neighboring clusters and copy the shared vector elements in $\hat{x}$
+\Input $A_\ell$ (sparse sub-matrix), $B_\ell$ (right-hand side sub-vector)
+\Output $X_\ell$ (solution sub-vector)\medskip
+
+\State Load $A_\ell$, $B_\ell$
+\State Set the initial guess $x^0$
+\For {$k=0,1,2,\ldots$ until the global convergence}
+\State Restart outer iteration with $x^0=x^k$
+\State Inner iteration: \Call{InnerSolver}{$x^0$, $k+1$}
+\State\label{algo:01:send} Send shared elements of $X_\ell^{k+1}$ to neighboring clusters
+\State\label{algo:01:recv} Receive shared elements in $\{X_m^{k+1}\}_{m\neq \ell}$
 \EndFor
 
 \Statex
 
 \Function {InnerSolver}{$x^0$, $k$}
-\State Compute the local right-hand side: $Y_l = B_l - \sum^L_{i=1,i\neq l}A_{li}X_i^0$
-\State Solving the local splitting $A_{ll}X_l^k=Y_l$ using the parallel GMRES method, such that $X_l^0$ is the local initial guess
-\State \Return $X_l^k$
+\State Compute local right-hand side $Y_\ell$:
+       \begin{equation*}
+         Y_\ell = B_\ell - \sum\nolimits^L_{\substack{m=1\\ m\neq \ell}}A_{\ell m}X_m^0
+       \end{equation*}
+\State Solving sub-system $A_{\ell\ell}X_\ell^k=Y_\ell$ with the parallel GMRES method
+\State \Return $X_\ell^k$
 \EndFunction
 \end{algorithmic}
+\caption{A multisplitting solver with GMRES method}
 \label{algo:01}
-\end{algorithm}
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-
-
-
-
+\end{figure}
+
+Algorithm on Figure~\ref{algo:01} shows the main key points of the
+multisplitting method to solve a large sparse linear system. This algorithm is
+based on an outer-inner iteration method where the parallel synchronous GMRES
+method is used to solve the inner iteration. It is executed in parallel by each
+cluster of processors. For all $\ell,m\in\{1,\ldots,L\}$, the matrices and
+vectors with the subscript $\ell$ represent the local data for cluster $\ell$,
+while $\{A_{\ell m}\}_{m\neq \ell}$ are off-diagonal matrices of sparse matrix
+$A$ and $\{X_m\}_{m\neq \ell}$ contain vector elements of solution $x$ shared
+with neighboring clusters. At every outer iteration $k$, asynchronous
+communications are performed between processors of the local cluster and those
+of distant clusters (lines~\ref{algo:01:send} and~\ref{algo:01:recv} in
+Figure~\ref{algo:01}). The shared vector elements of the solution $x$ are
+exchanged by message passing using MPI non-blocking communication routines.
+
+\begin{figure}[!t]
+\centering
+  \includegraphics[width=60mm,keepaspectratio]{clustering}
+\caption{Example of three clusters of processors interconnected by a virtual unidirectional ring network.}
+\label{fig:4.1}
+\end{figure}
+
+The global convergence of the asynchronous multisplitting solver is detected
+when the clusters of processors have all converged locally. We implemented the
+global convergence detection process as follows. On each cluster a master
+processor is designated (for example the processor with rank 1) and masters of
+all clusters are interconnected by a virtual unidirectional ring network (see
+Figure~\ref{fig:4.1}). During the resolution, a Boolean token circulates around
+the virtual ring from a master processor to another until the global convergence
+is achieved. So starting from the cluster with rank 1, each master processor $i$
+sets the token to \textit{True} if the local convergence is achieved or to
+\textit{False} otherwise, and sends it to master processor $i+1$. Finally, the
+global convergence is detected when the master of cluster 1 receives from the
+master of cluster $L$ a token set to \textit{True}. In this case, the master of
+cluster 1 broadcasts a stop message to masters of other clusters. In this work,
+the local convergence on each cluster $\ell$ is detected when the following
+condition is satisfied
+\begin{equation*}
+  (k\leq \MI) \text{ or } (\|X_\ell^k - X_\ell^{k+1}\|_{\infty}\leq\epsilon)
+\end{equation*}
+where $\MI$ is the maximum number of outer iterations and $\epsilon$ is the
+tolerance threshold of the error computed between two successive local solution
+$X_\ell^k$ and $X_\ell^{k+1}$.
+
+
+
+In this paper, we solve the 3D Poisson problem whose the mathematical model is 
+\begin{equation}
+\left\{
+\begin{array}{l}
+\nabla^2 u = f \text{~in~} \Omega \\
+u =0 \text{~on~} \Gamma =\partial\Omega
+\end{array}
+\right.
+\label{eq:02}
+\end{equation}
+where $\nabla^2$ is the Laplace operator, $f$ and $u$ are real-valued functions, and $\Omega=[0,1]^3$. The spatial discretization with a finite difference scheme reduces problem~(\ref{eq:02}) to a system of sparse linear equations. Our multisplitting method solves the 3D Poisson problem using a seven point stencil whose the general expression could be written as
+\begin{equation}
+\begin{array}{l}
+u(x-1,y,z) + u(x,y-1,z) + u(x,y,z-1)\\+u(x+1,y,z)+u(x,y+1,z)+u(x,y,z+1) \\ -6u(x,y,z)=h^2f(x,y,z),
+%u(x,y,z)= & \frac{1}{6}\times [u(x-1,y,z) + u(x+1,y,z) + \\
+ %         & u(x,y-1,z) + u(x,y+1,z) + \\
+  %        & u(x,y,z-1) + u(x,y,z+1) - \\ & h^2f(x,y,z)],
+\end{array}
+\label{eq:03}
+\end{equation} 
+where $h$ is the distance between two adjacent elements in the spatial discretization scheme and the iteration matrix $A$ of size $N_x\times N_y\times N_z$ of the discretized linear system is sparse, symmetric and positive definite. 
 
+The parallel solving of the 3D Poisson problem with our multisplitting method requires a data partitioning of the problem between clusters and between processors within a cluster. We have chosen the 3D partitioning instead of the row-by-row partitioning in order to reduce the data exchanges at sub-domain boundaries. Figure~\ref{fig:4.2} shows an example of the data partitioning of the 3D Poisson problem between two clusters of processors, where each sub-problem is assigned to a processor. In this context, a processor has at most six neighbors within a cluster or in distant clusters with which it shares data at sub-domain boundaries. 
 
+\begin{figure}[!t]
+\centering
+  \includegraphics[width=80mm,keepaspectratio]{partition}
+\caption{Example of the 3D data partitioning between two clusters of processors.}
+\label{fig:4.2}
+\end{figure}
 
-\section{Experimental results}
 
-When the ``real'' application runs in the simulation environment and produces
-the expected results, varying the input parameters and the program arguments
-allows us to compare outputs from the code execution. We have noticed from this
-study that the results depend on the following parameters: (1) at the network
-level, we found that the most critical values are the bandwidth (bw) and the
-network latency (lat). (2) Hosts power (GFlops) can also influence on the
-results. And finally, (3) when submitting job batches for execution, the
-arguments values passed to the program like the maximum number of iterations or
-the ``external'' precision are critical to ensure not only the convergence of the
-algorithm but also to get the main objective of the experimentation of the
-simulation in having an execution time in asynchronous less than in synchronous
-mode, in others words, in having a ``speedup'' less than 1 (Speedup = Execution
-time in synchronous mode / Execution time in asynchronous mode).
 
-A priori, obtaining a speedup less than 1 would be difficult in a local area
-network configuration where the synchronous mode will take advantage on the rapid
-exchange of information on such high-speed links. Thus, the methodology adopted
-was to launch the application on clustered network. In this last configuration,
-degrading the inter-cluster network performance will \emph{penalize} the synchronous
-mode allowing to get a speedup lower than 1. This action simulates the case of
-clusters linked with long distance network like Internet.
 
-As a first step, the algorithm was run on a network consisting of two clusters
-containing fifty hosts each, totaling one hundred hosts. Various combinations of
-the above factors have providing the results shown in Table~\ref{tab.cluster.2x50} with a matrix size
-ranging from Nx = Ny = Nz = 62 to 171 elements or from $62^{3} = \np{238328}$ to
-$171^{3} = \np{5211000}$ entries.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+We did not encounter major blocking problems when adapting the multisplitting algorithm previously described to a simulation environment like SimGrid unless some code 
+debugging. Indeed, apart from the review of the program sequence for asynchronous exchanges between processors within a cluster or between clusters, the algorithm was executed successfully with SMPI and provided identical outputs as those obtained with direct execution under MPI. In synchronous 
+mode, the execution of the program raised no particular issue but in asynchronous mode, the review of the sequence of MPI\_Isend, MPI\_Irecv and MPI\_Waitall instructions
+and with the addition of the primitive MPI\_Test was needed to avoid a memory fault due to an infinite loop resulting from the non-convergence of the algorithm.
+%\CER{On voulait en fait montrer la simplicitÃ© de l'adaptation de l'algo a SimGrid. Les problÃ¨mes rencontrÃ©s dÃ©crits dans ce paragraphe concerne surtout le mode async}\LZK{OK. J'aurais prÃ©fÃ©rÃ© avoir un peu plus de dÃ©tails sur l'adaptation de la version async} 
+%\CER{Le problÃ¨me majeur sur l'adaptation MPI vers SMPI pour la partie asynchrone de l'algorithme a Ã©tÃ© le plantage en SMPI de Waitall aprÃ¨s un Isend et Irecv. J'avais proposÃ© un workaround en utilisant un MPI\_wait sÃ©parÃ© pour chaque Ã©change a la place d'un waitall unique pour TOUTES les Ã©changes, une instruction qui semble bien fonctionner en MPI. Ce workaround aussi fonctionne bien. Mais aprÃ¨s, tu as modifiÃ© le programme avec l'ajout d'un MPI\_Test, au niveau de la routine de dÃ©tection de la convergence et du coup, l'Ã©change global avec waitall a aussi fonctionnÃ©.}
+Note here that the use of SMPI functions optimizer for memory footprint and CPU usage is not recommended knowing that one wants to get real results by simulation.
+As mentioned, upon this adaptation, the algorithm is executed as in the real life in the simulated environment after the following minor changes. The scope of all declared 
+global variables have been moved to local to subroutine. Indeed, global variables generate side effects arising from the concurrent access of 
+shared memory used by threads simulating each computing unit in the SimGrid architecture. 
+%Second, some compilation errors on MPI\_Waitall and MPI\_Finalize primitives have been fixed with the latest version of SimGrid.
+%\AG{compilation or run-time error?}
+In total, the initial MPI program running on the simulation environment SMPI gave after a very simple adaptation the same results as those obtained in a real 
+environment. We have successfully executed the code in synchronous mode using parallel GMRES algorithm compared with our multisplitting algorithm in asynchronous mode after few modifications. 
+
+
+
+\section{Simulation results}
+
+When the \textit{real} application runs in the simulation environment and produces the expected results, varying the input
+parameters and the program arguments allows us to compare outputs from the code execution. We have noticed from this
+study that the results depend on the following parameters:  
+\begin{itemize}
+\item At the network level, we found that the most critical values are the
+  bandwidth and the network latency.
+\item Hosts processors power (GFlops) can also influence on the results.
+\item Finally, when submitting job batches for execution, the arguments values
+  passed to the program like the maximum number of iterations or the precision are critical. They allow us to ensure not only the convergence of the
+  algorithm but also to get the main objective in getting an execution time in asynchronous communication less than in
+  synchronous mode. The ratio between the execution time of synchronous
+  compared to the asynchronous mode ($t_\text{sync} / t_\text{async}$) is defined as the \emph{relative gain}. So,
+  our objective running the algorithm in SimGrid is to obtain a relative gain
+  greater than 1.
+\end{itemize}
 
-Then we have changed the network configuration using three clusters containing
-respectively 33, 33 and 34 hosts, or again by on hundred hosts for all the
-clusters. In the same way as above, a judicious choice of key parameters has
-permitted to get the results in Table~\ref{tab.cluster.3x33} which shows the speedups less than 1 with
-a matrix size from 62 to 100 elements.
+A priori, obtaining a relative gain greater than 1 would be difficult in a local
+area network configuration where the synchronous mode will take advantage on the
+rapid exchange of information on such high-speed links. Thus, the methodology
+adopted was to launch the application on a clustered network. In this
+configuration, degrading the inter-cluster network performance will penalize the
+synchronous mode allowing to get a relative gain greater than 1.  This action
+simulates the case of distant clusters linked with long distance network as in grid computing context.
+
+
+% As a first step, 
+The algorithm was run on a two clusters based network with 50 hosts each, totaling 100 hosts. Various combinations of the above
+factors have provided the results shown in Table~\ref{tab.cluster.2x50}. The algorithm convergence with a 3D
+matrix size ranging from $N_x = N_y = N_z = \text{62}$ to 150 elements (that is from
+$\text{62}^\text{3} = \text{\np{238328}}$ to $\text{150}^\text{3} =
+\text{\np{3375000}}$ entries), is obtained in asynchronous in average 2.5 times faster than in the synchronous mode.
+\AG{Expliquer comment lire les tableaux.}
+\CER{J'ai reformulÃ© la phrase par la lecture du tableau. Plus de dÃ©tails seront lus dans la partie InterprÃ©tations et commentaires}
+% use the same column width for the following three tables
+\newlength{\mytablew}\settowidth{\mytablew}{\footnotesize\np{E-11}}
+\newenvironment{mytable}[1]{% #1: number of columns for data
+  \renewcommand{\arraystretch}{1.3}%
+  \begin{tabular}{|>{\bfseries}r%
+                  |*{#1}{>{\centering\arraybackslash}p{\mytablew}|}}}{%
+    \end{tabular}}
+
+\begin{table}[!t]
+  \centering
+  \caption{2 clusters, each with 50 nodes}
+  \label{tab.cluster.2x50}
 
-In a final step, results of an execution attempt to scale up the three clustered
-configuration but increasing by two hundreds hosts has been recorded in Table~\ref{tab.cluster.3x67}.
+  \begin{mytable}{5}
+    \hline
+    bandwidth (Mbit/s)
+    & 5         & 5         & 5         & 5         & 5         \\
+    \hline
+    latency (ms)
+    & 0.02      & 0.02      & 0.02      & 0.02      & 0.02      \\
+    \hline
+    power (GFlops)
+    & 1         & 1         & 1         & 1.5       & 1.5       \\
+    \hline
+    size $(n^3)$
+    & 62        & 62        & 62        & 100       & 100       \\
+    \hline
+    Precision
+    & \np{E-5}  & \np{E-8}  & \np{E-9}  & \np{E-11} & \np{E-11} \\
+    \hline
+    \hline
+    Relative gain
+    & 2.52      & 2.55      & 2.52      & 2.57      & 2.54      \\
+    \hline
+  \end{mytable}
+
+  \bigskip
+
+  \begin{mytable}{5}
+    \hline
+    bandwidth (Mbit/s)
+    & 50        & 50        & 50        & 50        & 50 \\ %       & 10        & 10 \\
+    \hline
+    latency (ms)
+    & 0.02      & 0.02      & 0.02      & 0.02      & 0.02 \\ %      & 0.03      & 0.01 \\
+    \hline
+    Power (GFlops)
+    & 1.5       & 1.5       & 1.5       & 1.5       & 1.5 \\ %      & 1         & 1.5 \\
+    \hline
+    size $(n^3)$
+    & 110       & 120       & 130       & 140       & 150  \\ %     & 171       & 171 \\
+    \hline
+    Precision
+    & \np{E-11} & \np{E-11} & \np{E-11} & \np{E-11} & \np{E-11} \\ % & \np{E-5}  & \np{E-5} \\
+    \hline
+    \hline
+    Relative gain
+    & 2.53      & 2.51     & 2.58     & 2.55     & 2.54   \\ %  & 1.59      & 1.29 \\
+    \hline
+  \end{mytable}
+\end{table}
+  
+%Then we have changed the network configuration using three clusters containing
+%respectively 33, 33 and 34 hosts, or again by on hundred hosts for all the
+%clusters. In the same way as above, a judicious choice of key parameters has
+%permitted to get the results in Table~\ref{tab.cluster.3x33} which shows the
+%relative gains greater than 1 with a matrix size from 62 to 100 elements.
+
+\CER{En accord avec RC, on a pour le moment enlevÃ© les tableaux 2 et 3 sachant que les rÃ©sultats obtenus sont limites. De mÃªme, on a enlevÃ© aussi les deux derniÃ¨res colonnes du tableau I en attendant une meilleure performance et une meilleure precision}
+%\begin{table}[!t]
+%  \centering
+%  \caption{3 clusters, each with 33 nodes}
+%  \label{tab.cluster.3x33}
+%
+%  \begin{mytable}{6}
+%    \hline
+%    bandwidth 
+%    & 10       & 5        & 4        & 3        & 2        & 6 \\
+%    \hline
+%    latency
+%    & 0.01     & 0.02     & 0.02     & 0.02     & 0.02     & 0.02 \\
+%    \hline
+%    power
+%    & 1        & 1        & 1        & 1        & 1        & 1 \\
+%    \hline
+%    size
+%    & 62       & 100      & 100      & 100      & 100      & 171 \\
+%    \hline
+%    Prec/Eprec
+%    & \np{E-5} & \np{E-5} & \np{E-5} & \np{E-5} & \np{E-5} & \np{E-5} \\
+%    \hline
+%    \hline
+%    Relative gain
+%    & 1.003    & 1.01     & 1.08     & 1.19     & 1.28     & 1.01 \\
+%    \hline
+%  \end{mytable}
+%\end{table}
+
+%In a final step, results of an execution attempt to scale up the three clustered
+%configuration but increasing by two hundreds hosts has been recorded in
+%Table~\ref{tab.cluster.3x67}.
+
+%\begin{table}[!t]
+%  \centering
+%  \caption{3 clusters, each with 66 nodes}
+%  \label{tab.cluster.3x67}
+%
+%  \begin{mytable}{1}
+%    \hline
+%    bandwidth  & 1 \\
+%    \hline
+%    latency    & 0.02 \\
+%    \hline
+%    power      & 1 \\
+%    \hline
+%    size       & 62 \\
+%    \hline
+%    Prec/Eprec & \np{E-5} \\
+%    \hline
+%    \hline
+%    Relative gain    & 1.11 \\
+%    \hline
+%  \end{mytable}
+%\end{table}
 
 Note that the program was run with the following parameters:
 
 \paragraph*{SMPI parameters}
 
 \begin{itemize}
-	\item HOSTFILE: Hosts file description.
-	\item PLATFORM: file description of the platform architecture : clusters (CPU power,
-\dots{}), intra cluster network description, inter cluster network (bandwidth bw,
-lat latency, \dots{}).
+\item HOSTFILE: Text file containing the list of the processors units name. Here 100 hosts;
+\item PLATFORM: XML file description of the platform architecture : two clusters (cluster1 and cluster2) with the following characteristics :
+  \begin{itemize}
+  \item Processor unit power: \np[GFlops]{1.5};
+  \item Intracluster network bandwidth: \np[Gbit/s]{1.25} and latency:
+    \np[$\mu$s]{0.05};
+  \item Intercluster network bandwidth: \np[Mbit/s]{5} and latency:
+    \np[$\mu$s]{5};
+  \end{itemize}
 \end{itemize}
 
 
 \paragraph*{Arguments of the program}
 
 \begin{itemize}
-	\item Description of the cluster architecture;
-	\item Maximum number of internal and external iterations;
-	\item Internal and external precisions;
-	\item Matrix size NX, NY and NZ;
-	\item Matrix diagonal value = 6.0;
-	\item Execution Mode: synchronous or asynchronous.
+\item Description of the cluster architecture matching the format <Number of
+  cluster> <Number of hosts in cluster1> <Number of hosts in cluster2>;
+\item Maximum number of iterations;
+\item Precisions on the residual error;
+\item Matrix size $N_x$, $N_y$ and $N_z$;
+\item Matrix diagonal value: $6$ (See~(\ref{eq:03}));
+\item Matrix off-diagonal value: $-1$;
+\item Communication mode: asynchronous.
 \end{itemize}
 
-\begin{table}
-  \centering
-  \caption{2 clusters X 50 nodes}
-  \label{tab.cluster.2x50}
-  \AG{Les images manquent dans le dÃ©pÃ´t Git. Si ce sont vraiment des tableaux, utiliser un format vectoriel (eps ou pdf), et surtout pas de jpeg!}
-  \includegraphics[width=209pt]{img1.jpg}
-\end{table}
-
-\begin{table}
-  \centering
-  \caption{3 clusters X 33 nodes}
-  \label{tab.cluster.3x33}
-  \AG{Le fichier manque.}
-  \includegraphics[width=209pt]{img2.jpg}
-\end{table}
-
-\begin{table}
-  \centering
-  \caption{3 clusters X 67 nodes}
-  \label{tab.cluster.3x67}
-  \AG{Le fichier manque.}
-%  \includegraphics[width=160pt]{img3.jpg}
-  \includegraphics[scale=0.5]{img3.jpg}
-\end{table}
-
 \paragraph*{Interpretations and comments}
 
-After analyzing the outputs, generally, for the configuration with two or three
-clusters including one hundred hosts (Tables~\ref{tab.cluster.2x50} and~\ref{tab.cluster.3x33}), some combinations of the
-used parameters affecting the results have given a speedup less than 1, showing
-the effectiveness of the asynchronous performance compared to the synchronous
-mode.
-
-In the case of a two clusters configuration, Table~\ref{tab.cluster.2x50} shows that with a
-deterioration of inter cluster network set with \np[Mbits/s]{5} of bandwidth, a latency
-in order of a hundredth of a millisecond and a system power of one GFlops, an
-efficiency of about \np[\%]{40} in asynchronous mode is obtained for a matrix size of 62
-elements. It is noticed that the result remains stable even if we vary the
-external precision from \np{E-5} to \np{E-9}. By increasing the problem size up to 100
-elements, it was necessary to increase the CPU power of \np[\%]{50} to \np[GFlops]{1.5} for a
-convergence of the algorithm with the same order of asynchronous mode efficiency.
-Maintaining such a system power but this time, increasing network throughput
-inter cluster up to \np[Mbits/s]{50}, the result of efficiency of about \np[\%]{40} is
-obtained with high external  precision of \np{E-11} for a matrix size from 110 to 150
-side elements.
-
-For the 3 clusters architecture including a total of 100 hosts, Table~\ref{tab.cluster.3x33} shows
-that it was difficult to have a combination which gives an efficiency of
-asynchronous below \np[\%]{80}. Indeed, for a matrix size of 62 elements, equality
-between the performance of the two modes (synchronous and asynchronous) is
-achieved with an inter cluster of \np[Mbits/s]{10} and a latency of \np{E-1} ms. To
-challenge an efficiency by \np[\%]{78} with a matrix size of 100 points, it was
-necessary to degrade the inter cluster network bandwidth from 5 to 2 Mbit/s.
-
-A last attempt was made for a configuration of three clusters but more power
-with 200 nodes in total. The convergence with a speedup of \np[\%]{90} was obtained
-with a bandwidth of \np[Mbits/s]{1} as shown in Table~\ref{tab.cluster.3x67}.
-
+After analyzing the outputs, generally, for the two clusters including one hundred hosts configuration (Tables~\ref{tab.cluster.2x50}), some combinations of parameters affecting
+the results have given a relative gain more than 2.5, showing the effectiveness of the
+asynchronous performance compared to the synchronous mode.
+
+With these settings, Table~\ref{tab.cluster.2x50} shows
+that after a deterioration of inter cluster network with a bandwidth of \np[Mbit/s]{5} and a latency in order of one hundredth of millisecond and a processor power
+of one GFlops, an efficiency of about \np[\%]{40} is
+obtained in asynchronous mode for a matrix size of 62 elements. It is noticed that the result remains
+stable even we vary the residual error precision from \np{E-5} to \np{E-9}. By
+increasing the matrix size up to 100 elements, it was necessary to increase the
+CPU power of \np[\%]{50} to \np[GFlops]{1.5} to get the algorithm convergence and the same order of asynchronous mode efficiency.  Maintaining such processor power but increasing network throughput inter cluster up to
+\np[Mbit/s]{50}, the result of efficiency with a relative gain of 2.5 is obtained with
+high external precision of \np{E-11} for a matrix size from 110 to 150 side
+elements.
+
+%For the 3 clusters architecture including a total of 100 hosts,
+%Table~\ref{tab.cluster.3x33} shows that it was difficult to have a combination
+%which gives a relative gain of asynchronous mode more than 1.2. Indeed, for a
+%matrix size of 62 elements, equality between the performance of the two modes
+%(synchronous and asynchronous) is achieved with an inter cluster of
+%\np[Mbit/s]{10} and a latency of \np[ms]{E-1}. To challenge an efficiency greater than 1.2 with a matrix %size of 100 points, it was necessary to degrade the
+%inter cluster network bandwidth from 5 to \np[Mbit/s]{2}.
+\AG{Conclusion, on prend une plateforme pourrie pour avoir un bon ratio sync/async ???
+  Quelle est la perte de perfs en faisant Ã§a ?}
+
+%A last attempt was made for a configuration of three clusters but more powerful
+%with 200 nodes in total. The convergence with a relative gain around 1.1 was
+%obtained with a bandwidth of \np[Mbit/s]{1} as shown in
+%Table~\ref{tab.cluster.3x67}.
+
+%\RC{Est ce qu'on sait expliquer pourquoi il y a une telle diffÃ©rence entre les rÃ©sultats avec 2 et 3 clusters... Avec 3 clusters, ils sont pas trÃ¨s bons... Je me demande s'il ne faut pas les enlever...}
+%\RC{En fait je pense avoir la rÃ©ponse Ã  ma remarque... On voit avec les 2 clusters que le gain est d'autant plus grand qu'on choisit une bonne prÃ©cision. Donc, plusieurs solutions, lancer rapidement un long test pour confirmer ca, ou enlever des tests... ou on ne change rien :-)}
+%\LZK{Ma question est: le bandwidth et latency sont ceux inter-clusters ou pour les deux inter et intra cluster??}
+%\CER{DÃ©finitivement, les paramÃ¨tres rÃ©seaux variables ici se rapportent au rÃ©seau INTER cluster.}
 \section{Conclusion}
+The experimental results on executing a parallel iterative algorithm in 
+asynchronous mode on an environment simulating a large scale of virtual 
+computers organized with interconnected clusters have been presented. 
+Our work has demonstrated that using such a simulation tool allow us to 
+reach the following three objectives: 
+
+\begin{enumerate}
+\item To have a flexible configurable execution platform resolving the 
+hard exercise to access to very limited but so solicited physical 
+resources;
+\item to ensure the algorithm convergence with a reasonable time and
+iteration number ;
+\item and finally and more importantly, to find the correct combination 
+of the cluster and network specifications permitting to save time in 
+executing the algorithm in asynchronous mode.
+\end{enumerate}
+Our results have shown that in certain conditions, asynchronous mode is 
+speeder up to \np[\%]{40} than executing the algorithm in synchronous mode
+which is not negligible for solving complex practical problems with more 
+and more increasing size.
+
+ Several studies have already addressed the performance execution time of 
+this class of algorithm. The work presented in this paper has 
+demonstrated an original solution to optimize the use of a simulation 
+tool to run efficiently an iterative parallel algorithm in asynchronous 
+mode in a grid architecture. 
+
+\LZK{Perspectives???}
 
 \section*{Acknowledgment}
 
-
-The authors would like to thank\dots{}
-
+This work is partially funded by the Labex ACTION program (contract ANR-11-LABX-01-01).
+\todo[inline]{The authors would like to thank\dots{}}
 
 % trigger a \newpage just before the given reference
 % number - used to balance the columns on the last page
 % adjust value as needed - may need to be readjusted if
 % the document is modified later
 \bibliographystyle{IEEEtran}
-\bibliography{hpccBib}
+\bibliography{IEEEabrv,hpccBib}
+
+
 
 \end{document}
 
@@ -353,3 +740,12 @@ The authors would like to thank\dots{}
 %%% fill-column: 80
 %%% ispell-local-dictionary: "american"
 %%% End:
+
+% LocalWords:  Ramamonjisoa Laiymani Arnaud Giersch Ziane Khodja RaphaÃ«l Femto
+% LocalWords:  UniversitÃ© Franche ComtÃ© IUT MontbÃ©liard MarÃ©chal Juin Inria Sud
+% LocalWords:  Ouest Vieille Talence cedex scalability experimentations HPC MPI
+% LocalWords:  Parallelization AIAC GMRES multi SMPI SISC SIAC SimDAG DAGs Lua
+% LocalWords:  Fortran GFlops priori Mbit de du fcomte multisplitting scalable
+% LocalWords:  SimGrid Belfort parallelize Labex ANR LABX IEEEabrv hpccBib
+% LocalWords:  intra durations nonsingular Waitall discretization discretized
+% LocalWords:  InnerSolver Isend Irecv
