+In this paper, we propose two algorithms of two-stage multisplitting methods. The first algorithm is based on the asynchronous model which allows communications to be overlapped by computations and reduces the idle times resulting from the synchronizations. So in the asynchronous mode, our two-stage algorithm uses asynchronous outer iterations and asynchronous communications between clusters. The communications (i.e. lines~\ref{send} and~\ref{recv} in Figure~\ref{alg:01}) are performed by message passing using MPI non-blocking communication routines. The convergence of the asynchronous iterations is detected when all clusters have locally converged:
+\begin{equation}
+k\geq\MIM\mbox{~or~}\|x_\ell^{k+1}-x_\ell^k\|_{\infty }\leq\TOLM,
+\label{eq:04}
+\end{equation}
+where $\MIM$ is the maximum number of outer iterations and $\TOLM$ is the tolerance threshold for the two-stage algorithm.
+
+The second two-stage algorithm is based on synchronous outer iterations. We propose to use the Krylov iteration based on residual minimization to improve the slow convergence of the multisplitting methods. In this case, a $n\times s$ matrix $S$ is set using solutions issued from the inner iteration:
+\begin{equation}
+S=[x^1,x^2,\ldots,x^s],~s\ll n.
+\label{eq:05}
+\end{equation}
+At each $s$ outer iterations, the algorithm computes a new approximation $\tilde{x}=S\alpha$ which minimizes the residual:
+\begin{equation}
+\min_{\alpha\in\mathbb{R}^s}{\|b-AS\alpha\|_2}.
+\label{eq:06}
+\end{equation}
+The algorithm in Figure~\ref{alg:02} includes the procedure of the residual minimization and the outer iteration is restarted with a new approximation $\tilde{x}$ at every $s$ iterations. The least-squares problem~(\ref{eq:06}) is solved in parallel by all clusters using CGLS method~\cite{Hestenes52} such that $\MIC$ is the maximum number of iterations and $\TOLC$ is the tolerance threshold for this method (line~\ref{cgls} in Figure~\ref{alg:02}).
+
+\begin{figure}[t]
+%\begin{algorithm}[t]
+%\caption{Krylov two-stage method using block Jacobi multisplitting}
+\begin{algorithmic}[1]
+ \Input $A_\ell$ (sparse matrix), $b_\ell$ (right-hand side)
+ \Output $x_\ell$ (solution vector)\vspace{0.2cm}
+ \State Set the initial guess $x^0$
+ \For {$k=1,2,3,\ldots$ until convergence}
+ \State $c_\ell=b_\ell-\sum_{m\neq\ell}A_{\ell m}x_m^{k-1}$
+ \State $x^k_\ell=Solve_{gmres}(A_{\ell\ell},c_\ell,x^{k-1}_\ell,\MIG,\TOLG)$
+ \State $S_{\ell,k\mod s}=x_\ell^k$
+ \If{$k\mod s = 0$}
+ \State $\alpha = Solve_{cgls}(AS,b,\MIC,\TOLC)$\label{cgls}
+ \State $\tilde{x_\ell}=S_\ell\alpha$
+ \State Send $\tilde{x_\ell}$ to neighboring clusters
+ \Else
+ \State Send $x_\ell^k$ to neighboring clusters
+ \EndIf
+ \State Receive $\{x_m^k\}_{m\neq\ell}$ from neighboring clusters
+ \EndFor
+\end{algorithmic}
+\caption{Krylov two-stage method using block Jacobi multisplitting}
+\label{alg:02}
+%\end{algorithm}
+\end{figure}
+
+\subsection{Simulation of the two-stage methods using SimGrid toolkit}
+\label{sec:04.02}
+
+One of our objectives when simulating the application in Simgrid is, as in real
+life, to get accurate results (solutions of the problem) but also to ensure the
+test reproducibility under the same conditions. According to our experience,
+very few modifications are required to adapt a MPI program for the Simgrid
+simulator using SMPI (Simulator MPI). The first modification is to include SMPI
+libraries and related header files (smpi.h). The second modification is to
+suppress all global variables by replacing them with local variables or using a
+Simgrid selector called "runtime automatic switching"
+(smpi/privatize\_global\_variables). Indeed, global variables can generate side
+effects on runtime between the threads running in the same process and generated by
+Simgrid to simulate the grid environment.
+
+%\RC{On vire cette phrase ?} \RCE {Si c'est la phrase d'avant sur les threads, je pense qu'on peut la retenir car c'est l'explication du pourquoi Simgrid n'aime pas les variables globales. Si c'est pas bien dit, on peut la reformuler. Si c'est la phrase ci-apres, effectivement, on peut la virer si elle preterais a discussion}The
+%last modification on the MPI program pointed out for some cases, the review of
+%the sequence of the MPI\_Isend, MPI\_Irecv and MPI\_Waitall instructions which
+%might cause an infinite loop.
+
+
+\paragraph{Simgrid Simulator parameters}
+\ \\ \noindent Before running a Simgrid benchmark, many parameters for the
+computation platform must be defined. For our experiments, we consider platforms
+in which several clusters are geographically distant, so there are intra and
+inter-cluster communications. In the following, these parameters are described:
+
+\begin{itemize}
+ \item hostfile: hosts description file.
+ \item platform: file describing the platform architecture: clusters (CPU power,
+\dots{}), intra cluster network description, inter cluster network (bandwidth bw,
+latency lat, \dots{}).
+ \item archi : grid computational description (number of clusters, number of
+nodes/processors for each cluster).
+\end{itemize}
+\noindent
+In addition, the following arguments are given to the programs at runtime:
+
+\begin{itemize}
+ \item maximum number of inner and outer iterations;
+ \item inner and outer precisions;
+ \item maximum number of the GMRES restarts in the Arnorldi process;
+ \item maximum number of iterations and the tolerance threshold in classical GMRES;
+ \item tolerance threshold for outer and inner-iterations;
+ \item matrix size (N$_{x}$, N$_{y}$ and N$_{z}$) respectively on $x, y, z$ axis;
+ \item matrix diagonal value is fixed to $6.0$ for synchronous Krylov multisplitting experiments and $6.2$ for asynchronous block Jacobi experiments; \RC{CE tu vérifies, je dis ca de tête}
+ \item matrix off-diagonal value;
+ \item execution mode: synchronous or asynchronous;
+ \RCE {C'est ok la liste des arguments du programme mais si Lilia ou toi pouvez preciser pour les arguments pour CGLS ci dessous} \RC{Vu que tu n'as pas fait varier ce paramètre, on peut ne pas en parler}
+ \item Size of matrix S;
+ \item Maximum number of iterations and tolerance threshold for CGLS.
+\end{itemize}
+
+It should also be noticed that both solvers have been executed with the Simgrid selector \texttt{-cfg=smpi/running\_power} which determines the computational power (here 19GFlops) of the simulator host machine.
