12-10-2014 04

[GMRES2stage.git] / paper.tex

diff --git a/paper.tex b/paper.tex
index 16beac7e509d9b87c1bd444249ad71a7807fe5cd..51781ba01ad4fd0363f031d16e054de8d9a270f3 100644 (file)
--- a/paper.tex
+++ b/paper.tex
@@ -439,7 +439,7 @@ GMRES.
 \end{abstract}
 
 \begin{IEEEkeywords}
 \end{abstract}
 
 \begin{IEEEkeywords}

-Iterative Krylov methods; sparse linear systems; residual minimization; PETSc; %à voir... 
+Iterative Krylov methods; sparse linear systems; two stage iteration; least-squares residual minimization; PETSc

 \end{IEEEkeywords}
 
 
 \end{IEEEkeywords}
 
 


@@ -547,38 +547,42 @@ Iterative Krylov methods; sparse linear systems; residual minimization; PETSc; %
 % You must have at least 2 lines in the paragraph with the drop letter
 % (should never be an issue)
 
 % You must have at least 2 lines in the paragraph with the drop letter
 % (should never be an issue)
 

-Iterative methods have recently become more attractive than  direct ones to  solve very large
-sparse  linear systems.  They are more  efficient  in a  parallel
-context,  supporting  thousands  of  cores,  and they require  less  memory  and  arithmetic
-operations than direct  methods. This is why new iterative  methods are frequently 
-proposed or adapted by researchers, and the increasing need to solve very large sparse
-linear  systems  has triggered the  development  of such efficient iterative  techniques
-suitable for parallel processing.
-
-Most of the successful iterative methods currently available are based on so-called ``Krylov
-subspaces''. They  consist in forming a  basis of successive matrix
-powers multiplied by an initial vector, which can be for instance the residual. These methods use vectors orthogonality of the Krylov  subspace  basis in order to solve  linear
-systems.  The  most known iterative  Krylov subspace methods  are conjugate
-gradient and GMRES ones (Generalized Minimal RESidual).
-
-
-However,  iterative  methods suffer  from scalability  problems  on parallel
-computing  platforms  with many  processors, due  to  their need  of  reduction
-operations, and to  collective    communications   to  achieve   matrix-vector
+Iterative methods have recently become more attractive than direct ones to solve
+very large sparse  linear systems\cite{Saad2003}.  They are more  efficient in a
+parallel context,  supporting thousands of  cores, and they require  less memory
+and  arithmetic operations than  direct methods~\cite{bahicontascoutu}.  This is
+why new iterative methods are frequently proposed or adapted by researchers, and
+the increasing need to solve very  large sparse linear systems has triggered the
+development  of  such  efficient  iterative  techniques  suitable  for  parallel
+processing.
+
+Most  of the  successful  iterative  methods currently  available  are based  on
+so-called ``Krylov  subspaces''. They consist  in forming a basis  of successive
+matrix powers  multiplied by an  initial vector, which  can be for  instance the
+residual. These methods  use vectors orthogonality of the  Krylov subspace basis
+in  order to solve  linear systems.   The most  known iterative  Krylov subspace
+methods are conjugate gradient and GMRES ones (Generalized Minimal RESidual).
+
+
+However,  iterative  methods  suffer   from  scalability  problems  on  parallel
+computing  platforms  with many  processors,  due  to  their need  of  reduction
+operations,   and  to   collective  communications   to   achieve  matrix-vector

 multiplications. The  communications on large  clusters with thousands  of cores
 multiplications. The  communications on large  clusters with thousands  of cores

-and  large  sizes of  messages  can  significantly  affect the  performances  of these
-iterative methods. As a consequence, Krylov subspace iteration methods are often used
-with preconditioners in practice, to increase their convergence and accelerate their
-performances.  However, most  of the  good preconditioners  are not  scalable on
-large clusters.
-
-In this research work, a two-stage algorithm based on  two nested iterations
-called inner-outer  iterations is proposed.  This algorithm  consists in solving  the sparse
-linear system iteratively  with a small number of  inner iterations, and restarting
-the outer  step with a  new solution minimizing  some error functions  over some
-previous residuals. This algorithm is iterative and easy to parallelize on large
-clusters. Furthermore,  the   minimization  technique   improves  its   convergence  and
-performances.
+and large sizes  of messages can significantly affect  the performances of these
+iterative methods. As a consequence, Krylov subspace iteration methods are often
+used  with  preconditioners  in  practice,  to increase  their  convergence  and
+accelerate their  performances.  However, most  of the good  preconditioners are
+not scalable on large clusters.
+
+In  this research work,  a two-stage  algorithm based  on two  nested iterations
+called inner-outer  iterations is proposed.  This algorithm  consists in solving
+the sparse  linear system iteratively with  a small number  of inner iterations,
+and  restarting  the  outer step  with  a  new  solution minimizing  some  error
+functions  over some previous  residuals. For  further information  on two-stage
+iteration      methods,     interested      readers      are     invited      to
+consult~\cite{Nichols:1973:CTS}. Two-stage algorithms are easy to parallelize on
+large clusters.  Furthermore,  the least-squares minimization technique improves
+its convergence and performances.

 
 The present  article is  organized as follows.   Related works are  presented in
 Section~\ref{sec:02}. Section~\ref{sec:03} details the two-stage algorithm using
 
 The present  article is  organized as follows.   Related works are  presented in
 Section~\ref{sec:02}. Section~\ref{sec:03} details the two-stage algorithm using


@@ -597,7 +601,12 @@ is summarized while intended perspectives are provided.
 %%%*********************************************************
 \section{Related works}
 \label{sec:02} 
 %%%*********************************************************
 \section{Related works}
 \label{sec:02} 

-%Wherever Times is specified, Times Roman or Times New Roman may be used. If neither is available on your system, please use the font closest in appearance to Times. Avoid using bit-mapped fonts if possible. True-Type 1 or Open Type fonts are preferred. Please embed symbol fonts, as well, for math, etc.
+Krylov subspace iteration methods have increasingly become useful and successful techniques for solving linear and nonlinear systems and eigenvalue problems, especially since the increase development of the preconditioners~\cite{Saad2003,Meijerink77}. One reason of the popularity of these methods is their generality, simplicity and efficiency to solve systems of equations arising from very large and complex problems. %A Krylov method is based on a projection process onto a Krylov subspace spanned by vectors and it forms a sequence of approximations by minimizing the residual over the subspace formed~\cite{}.
+
+GMRES is one of the most widely used Krylov iterative method for solving sparse and large linear systems. It is developed by Saad and al.~\cite{Saad86} as a generalized method to deal with unsymmetric and non-Hermitian problems, and indefinite symmetric problems too. In its original version called full GMRES, it minimizes the residual over the current Krylov subspace until convergence in at most $n$ iterations, where $n$ is the size of the sparse matrix. It should be noted that full GMRES is too expensive in the case of large matrices since the required orthogonalization process per iteration grows quadratically with the number of iterations. For that reason, in practice GMRES is restarted after each $m\ll n$ iterations to avoid the storage of a large orthonormal basis. However, the convergence behavior of the restarted GMRES in many cases depends quite critically on the value of $m$~\cite{Huang89}. Therefore in most cases, a preconditioning technique is applied to the restarted GMRES method in order to improve its convergence.
+
+%FGMRES , GMRESR, two-stage, communication avoiding 
+

 %%%*********************************************************
 %%%*********************************************************
 
 %%%*********************************************************
 %%%*********************************************************
 


@@ -650,10 +659,10 @@ appropriate than a single direct method in a parallel context.
   \Input $A$ (sparse matrix), $b$ (right-hand side)
   \Output $x$ (solution vector)\vspace{0.2cm}
   \State Set the initial guess $x_0$
   \Input $A$ (sparse matrix), $b$ (right-hand side)
   \Output $x$ (solution vector)\vspace{0.2cm}
   \State Set the initial guess $x_0$

-  \For {$k=1,2,3,\ldots$ until convergence (error$<\epsilon_{tsirm}$)} \label{algo:conv}
+  \For {$k=1,2,3,\ldots$ until convergence ($error<\epsilon_{tsirm}$)} \label{algo:conv}

     \State  $[x_k,error]=Solve(A,b,x_{k-1},max\_iter_{kryl})$   \label{algo:solve}
     \State  $[x_k,error]=Solve(A,b,x_{k-1},max\_iter_{kryl})$   \label{algo:solve}

-    \State $S_{k \mod s}=x_k$ \label{algo:store}
-    \If {$k \mod s=0$ {\bf and} error$>\epsilon_{kryl}$}
+    \State $S_{k \mod s}=x_k$ \label{algo:store} \Comment{update column ($k \mod s$) of $S$}
+    \If {$k \mod s=0$ {\bf and} $error>\epsilon_{kryl}$}

       \State $R=AS$ \Comment{compute dense matrix} \label{algo:matrix_mul}
             \State $\alpha=Least\_Squares(R,b,max\_iter_{ls})$ \label{algo:}
       \State $x_k=S\alpha$  \Comment{compute new solution}
       \State $R=AS$ \Comment{compute dense matrix} \label{algo:matrix_mul}
             \State $\alpha=Least\_Squares(R,b,max\_iter_{ls})$ \label{algo:}
       \State $x_k=S\alpha$  \Comment{compute new solution}


@@ -671,10 +680,10 @@ method. Moreover,  a tolerance  threshold must be  specified for the  solver. In
 practice, this threshold must be  much smaller than the convergence threshold of
 the  TSIRM algorithm  (\emph{i.e.}, $\epsilon_{tsirm}$).  We also  consider that
 after the call of the $Solve$ function, we obtain the vector $x_k$ and the error
 practice, this threshold must be  much smaller than the convergence threshold of
 the  TSIRM algorithm  (\emph{i.e.}, $\epsilon_{tsirm}$).  We also  consider that
 after the call of the $Solve$ function, we obtain the vector $x_k$ and the error

-which is defined by $||Ax^k-b||_2$.
+which is defined by $||Ax_k-b||_2$.

 
   Line~\ref{algo:store},
 
   Line~\ref{algo:store},

-$S_{k \mod  s}=x^k$ consists in  copying the solution  $x_k$ into the  column $k
+$S_{k \mod  s}=x_k$ consists in  copying the solution  $x_k$ into the  column $k

 \mod s$ of $S$.   After the minimization, the matrix $S$ is  reused with the new
 values of the residuals.  To solve the minimization problem, an iterative method
 is used. Two parameters are required  for that: the maximum number of iterations
 \mod s$ of $S$.   After the minimization, the matrix $S$ is  reused with the new
 values of the residuals.  To solve the minimization problem, an iterative method
 is used. Two parameters are required  for that: the maximum number of iterations


@@ -739,41 +748,59 @@ these operations are easy to implement in PETSc or similar environment.
 
 \section{Convergence results}
 \label{sec:04}
 
 \section{Convergence results}
 \label{sec:04}

-Let us recall the following result, see~\cite{Saad86} for further readings.
-\begin{proposition}
-\label{prop:saad}
-Suppose that $A$ is a positive real matrix with symmetric part $M$. Then the residual norm provided at the $m$-th step of GMRES satisfies:
-\begin{equation}
-||r_m|| \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{m}{2}} ||r_0|| ,
-\end{equation}
-where $\alpha = \lambda_{min}(M)^2$ and $\beta = \lambda_{max}(A^T A)$, which proves 
-the convergence of GMRES($m$) for all $m$ under that assumption regarding $A$.
-\end{proposition}

 
 
 We can now claim that,
 \begin{proposition}
 
 
 We can now claim that,
 \begin{proposition}

-If $A$ is a positive real matrix and GMRES($m$) is used as solver, then the TSIRM algorithm is convergent. Furthermore, 
-let $r_k$ be the
+\label{prop:saad}
+If $A$ is either a definite positive or a positive matrix and GMRES($m$) is used as solver, then the TSIRM algorithm is convergent. 
+
+Furthermore, let $r_k$ be the

 $k$-th residue of TSIRM, then
 $k$-th residue of TSIRM, then

-we still have:
+we have the following boundaries:
+\begin{itemize}
+\item when $A$ is positive:

 \begin{equation}
 ||r_k|| \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{km}{2}} ||r_0|| ,
 \end{equation}
 \begin{equation}
 ||r_k|| \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{km}{2}} ||r_0|| ,
 \end{equation}

-where $\alpha$ and $\beta$ are defined as in Proposition~\ref{prop:saad}.
+where $M$ is the symmetric part of $A$, $\alpha = \lambda_{min}(M)^2$ and $\beta = \lambda_{max}(A^T A)$;
+\item when $A$ is positive definite:
+\begin{equation}
+\|r_k\| \leq \left( 1-\frac{\lambda_{\mathrm{min}}^2(1/2(A^T + A))}{ \lambda_{\mathrm{max}}(A^T A)} \right)^{km/2} \|r_0\|.
+\end{equation}
+\end{itemize}
+%In the general case, where A is not positive definite, we have
+%$\|r_n\| \le \inf_{p \in P_n} \|p(A)\| \le \kappa_2(V) \inf_{p \in P_n} \max_{\lambda \in \sigma(A)} |p(\lambda)| \|r_0\|, .$

 \end{proposition}
 
 \begin{proof}
 \end{proposition}
 
 \begin{proof}

-We will prove by a mathematical induction that, for each $k \in \mathbb{N}^\ast$, 
-$||r_k|| \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{mk}{2}} ||r_0||.$
+Let us first recall that the residue is under control when considering the GMRES algorithm on a positive definite matrix, and it is bounded as follows:
+\begin{equation*}
+\|r_k\| \leq \left( 1-\frac{\lambda_{\mathrm{min}}^2(1/2(A^T + A))}{ \lambda_{\mathrm{max}}(A^T A)} \right)^{k/2} \|r_0\| .
+\end{equation*}
+Additionally, when $A$ is a positive real matrix with symmetric part $M$, then the residual norm provided at the $m$-th step of GMRES satisfies:
+\begin{equation*}
+||r_m|| \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{m}{2}} ||r_0|| ,
+\end{equation*}
+where $\alpha$ and $\beta$ are defined as in Proposition~\ref{prop:saad}, which proves 
+the convergence of GMRES($m$) for all $m$ under such assumptions regarding $A$.
+These well-known results can be found, \emph{e.g.}, in~\cite{Saad86}.
+
+We will now prove by a mathematical induction that, for each $k \in \mathbb{N}^\ast$, 
+$||r_k|| \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{mk}{2}} ||r_0||$ when $A$ is positive, and $\|r_k\| \leq \left( 1-\frac{\lambda_{\mathrm{min}}^2(1/2(A^T + A))}{ \lambda_{\mathrm{max}}(A^T A)} \right)^{km/2} \|r_0\|$ when $A$ is positive definite.

 
 

-The base case is obvious, as for $k=1$, the TSIRM algorithm simply consists in applying GMRES($m$) once, leading to a new residual $r_1$ which follows the inductive hypothesis due to Proposition~\ref{prop:saad}.
+The base case is obvious, as for $k=1$, the TSIRM algorithm simply consists in applying GMRES($m$) once, leading to a new residual $r_1$ that follows the inductive hypothesis due, to the results recalled above.

 
 

-Suppose now that the claim holds for all $m=1, 2, \hdots, k-1$, that is, $\forall m \in \{1,2,\hdots, k-1\}$, $||r_m|| \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{km}{2}} ||r_0||$.
+Suppose now that the claim holds for all $m=1, 2, \hdots, k-1$, that is, $\forall m \in \{1,2,\hdots, k-1\}$, $||r_m|| \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{km}{2}} ||r_0||$ in the positive case, and $\|r_k\| \leq \left( 1-\frac{\lambda_{\mathrm{min}}^2(1/2(A^T + A))}{ \lambda_{\mathrm{max}}(A^T A)} \right)^{km/2} \|r_0\|$ in the definite positive one.

 We will show that the statement holds too for $r_k$. Two situations can occur:
 \begin{itemize}
 We will show that the statement holds too for $r_k$. Two situations can occur:
 \begin{itemize}

-\item If $k \mod m \neq 0$, then the TSIRM algorithm consists in executing GMRES once. In that case, we obtain $||r_k|| \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{m}{2}} ||r_{k-1}||\leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{km}{2}} ||r_0||$ by the inductive hypothesis.
-\item Else, the TSIRM algorithm consists in two stages: a first GMRES($m$) execution leads to a temporary $x_k$ whose residue satisfies $||r_k|| \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{m}{2}} ||r_{k-1}||\leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{km}{2}} ||r_0||$, and a least squares resolution.
+\item If $k \not\equiv 0 ~(\textrm{mod}\ m)$, then the TSIRM algorithm consists in executing GMRES once. In that case and by using the inductive hypothesis, we obtain either $||r_k|| \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{m}{2}} ||r_{k-1}||\leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{km}{2}} ||r_0||$ if $A$ is positive, or $\|r_k\| \leqslant \left( 1-\frac{\lambda_{\mathrm{min}}^2(1/2(A^T + A))}{ \lambda_{\mathrm{max}}(A^T A)} \right)^{m/2} \|r_{k-1}\|$ $\leqslant$ $\left( 1-\frac{\lambda_{\mathrm{min}}^2(1/2(A^T + A))}{ \lambda_{\mathrm{max}}(A^T A)} \right)^{km/2} \|r_{0}\|$ in the positive definite case.
+\item Else, the TSIRM algorithm consists in two stages: a first GMRES($m$) execution leads to a temporary $x_k$ whose residue satisfies:
+\begin{itemize}
+\item $||r_k|| \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{m}{2}} ||r_{k-1}||\leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{km}{2}} ||r_0||$ in the positive case, 
+\item $\|r_k\| \leqslant \left( 1-\frac{\lambda_{\mathrm{min}}^2(1/2(A^T + A))}{ \lambda_{\mathrm{max}}(A^T A)} \right)^{m/2} \|r_{k-1}\|$ $\leqslant$ $\left( 1-\frac{\lambda_{\mathrm{min}}^2(1/2(A^T + A))}{ \lambda_{\mathrm{max}}(A^T A)} \right)^{km/2} \|r_{0}\|$ in the positive definite one,
+\end{itemize}
+and a least squares resolution.

 Let $\operatorname{span}(S) = \left \{ {\sum_{i=1}^k \lambda_i v_i \Big| k \in \mathbb{N}, v_i \in S, \lambda _i \in \mathbb{R}} \right \}$ be the linear span of a set of real vectors $S$. So,\\
 $\min_{\alpha \in \mathbb{R}^s} ||b-R\alpha ||_2 = \min_{\alpha \in \mathbb{R}^s} ||b-AS\alpha ||_2$
 
 Let $\operatorname{span}(S) = \left \{ {\sum_{i=1}^k \lambda_i v_i \Big| k \in \mathbb{N}, v_i \in S, \lambda _i \in \mathbb{R}} \right \}$ be the linear span of a set of real vectors $S$. So,\\
 $\min_{\alpha \in \mathbb{R}^s} ||b-R\alpha ||_2 = \min_{\alpha \in \mathbb{R}^s} ||b-AS\alpha ||_2$
 


@@ -784,14 +811,16 @@ $\begin{array}{ll}
 & \leqslant \min_{\lambda \in \mathbb{R}} ||b-\lambda Ax_{k} ||_2\\
 & \leqslant ||b-Ax_{k}||_2\\
 & = ||r_k||_2\\
 & \leqslant \min_{\lambda \in \mathbb{R}} ||b-\lambda Ax_{k} ||_2\\
 & \leqslant ||b-Ax_{k}||_2\\
 & = ||r_k||_2\\

-& \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{km}{2}} ||r_0||,
+& \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{km}{2}} ||r_0||, \textrm{ if $A$ is positive,}\\
+& \leqslant \left( 1-\frac{\lambda_{\mathrm{min}}^2(1/2(A^T + A))}{ \lambda_{\mathrm{max}}(A^T A)} \right)^{km/2} \|r_{0}\|, \textrm{ if $A$ is}\\
+& \textrm{positive definite,} 

 \end{array}$
 \end{itemize}
 which concludes the induction and the proof.
 \end{proof}
 
 \end{array}$
 \end{itemize}
 which concludes the induction and the proof.
 \end{proof}
 

-We can remark that, at each iterate, the residue of the TSIRM algorithm is lower 
-than the one of the GMRES method.
+%We can remark that, at each iterate, the residue of the TSIRM algorithm is lower 
+%than the one of the GMRES method.

 
 %%%*********************************************************
 %%%*********************************************************
 
 %%%*********************************************************
 %%%*********************************************************


@@ -799,11 +828,13 @@ than the one of the GMRES method.
 \label{sec:05}
 
 
 \label{sec:05}
 
 

-In order to see the influence of our algorithm with only one processor, we first
-show  a comparison  with the  standard version  of GMRES  and our  algorithm. In
-Table~\ref{tab:01},  we  show  the  matrices  we  have used  and  some  of  them
-characteristics. For all  the matrices, the name, the field,  the number of rows
-and the number of nonzero elements are given.
+In order to see the behavior of the proposal when considering only one processor, a first
+comparison with GMRES or FGMRES and the new algorithm detailed previously has been experimented. 
+Matrices that have been used with their characteristics (names, fields, rows, and nonzero coefficients) are detailed in 
+Table~\ref{tab:01}.  These latter, which are real-world applications matrices, 
+have been extracted 
+ from   the  Davis  collection,   University  of
+Florida~\cite{Dav97}.

 
 \begin{table}[htbp]
 \begin{center}
 
 \begin{table}[htbp]
 \begin{center}


@@ -823,8 +854,9 @@ torso3             & 2D/3D problem & 259,156 & 4,429,042 \\
 \label{tab:01}
 \end{center}
 \end{table}
 \label{tab:01}
 \end{center}
 \end{table}

-
-The following  parameters have been chosen  for our experiments.   As by default
+Chosen parameters are detailed below.
+%The following  parameters have been chosen  for our experiments.   
+As by default

 the restart  of GMRES is performed every  30 iterations, we have  chosen to stop
 the GMRES every 30 iterations (\emph{i.e.} $max\_iter_{kryl}=30$).  $s$ is set to 8. CGLS is
 chosen  to minimize  the least-squares  problem with  the  following parameters:
 the restart  of GMRES is performed every  30 iterations, we have  chosen to stop
 the GMRES every 30 iterations (\emph{i.e.} $max\_iter_{kryl}=30$).  $s$ is set to 8. CGLS is
 chosen  to minimize  the least-squares  problem with  the  following parameters:


@@ -836,13 +868,14 @@ Core(TM) i7-3630QM CPU @ 2.40GHz with the version 3.5.1 of PETSc.
 In  Table~\ref{tab:02}, some  experiments comparing  the solving  of  the linear
 systems obtained with the previous matrices  with a GMRES variant and with out 2
 stage algorithm are  given. In the second column, it can  be noticed that either
 In  Table~\ref{tab:02}, some  experiments comparing  the solving  of  the linear
 systems obtained with the previous matrices  with a GMRES variant and with out 2
 stage algorithm are  given. In the second column, it can  be noticed that either

-gmres or fgmres is used to  solve the linear system.  According to the matrices,
-different  preconditioner is used.   With TSIRM,  the same  solver and  the same
-preconditionner are used.  This Table shows that TSIRM can drastically reduce the
-number of iterations to reach the  convergence when the number of iterations for
-the normal GMRES is more or less  greater than 500. In fact this also depends on
-tow  parameters: the  number  of iterations  to  stop GMRES  and  the number  of
-iterations to perform the minimization.
+GRMES or  FGMRES (Flexible  GMRES)~\cite{Saad:1993} is used  to solve  the linear
+system.   According to  the matrices,  different preconditioner  is  used.  With
+TSIRM, the same solver and the  same preconditionner are used.  This Table shows
+that  TSIRM  can  drastically reduce  the  number  of  iterations to  reach  the
+convergence when the  number of iterations for the normal GMRES  is more or less
+greater than  500. In fact  this also depends  on tow parameters: the  number of
+iterations  to  stop  GMRES  and   the  number  of  iterations  to  perform  the
+minimization.

 
 
 \begin{table}[htbp]
 
 
 \begin{table}[htbp]


@@ -903,8 +936,16 @@ by core. The Juqueen architecture is composed  of IBM PowerPC A2 at 1.6 GHz with
 speed network.
 
 
 speed network.
 
 

+In  many situations, using  preconditioners is  essential in  order to  find the
+solution of a linear system.  There are many preconditioners available in PETSc.
+For parallel applications all  the preconditioners based on matrix factorization
+are  not  available. In  our  experiments, we  have  tested  different kinds  of
+preconditioners, however  as it is  not the subject  of this paper, we  will not
+present results with many preconditioners. In  practise, we have chosen to use a
+multigrid (mg)  and successive  over-relaxation (sor). For  more details  on the
+preconditioner in PETSc please consult~\cite{petsc-web-page}.
+

 
 

-{\bf Description of preconditioners}\\

 
 \begin{table*}[htbp]
 \begin{center}
 
 \begin{table*}[htbp]
 \begin{center}


@@ -932,8 +973,7 @@ speed network.
 
 Table~\ref{tab:03} shows  the execution  times and the  number of  iterations of
 example ex15  of PETSc on the  Juqueen architecture. Different  numbers of cores
 
 Table~\ref{tab:03} shows  the execution  times and the  number of  iterations of
 example ex15  of PETSc on the  Juqueen architecture. Different  numbers of cores

-are  studied ranging  from  2,048  up-to 16,383.   Two  preconditioners have  been
-tested: {\it mg} and {\it sor}.   For those experiments,  the number  of components  (or unknowns  of the
+are  studied ranging  from  2,048  up-to 16,383 with the two preconditioners {\it mg} and {\it sor}.   For those experiments,  the number  of components  (or unknowns  of the

 problems)  per core  is fixed  to 25,000,  also called  weak  scaling. This
 number can seem relatively small. In fact, for some applications that need a lot
 of  memory, the  number of  components per  processor requires  sometimes  to be
 problems)  per core  is fixed  to 25,000,  also called  weak  scaling. This
 number can seem relatively small. In fact, for some applications that need a lot
 of  memory, the  number of  components per  processor requires  sometimes  to be


@@ -1000,8 +1040,40 @@ the number of iterations. So, the overall benefit of using TSIRM is interesting.
 \end{table*}
 
 
 \end{table*}
 
 

-In Table~\ref{tab:04}, some experiments with example ex54 on the Curie architecture are reported.
-
+In  Table~\ref{tab:04},  some  experiments   with  example  ex54  on  the  Curie
+architecture are reported.  For this  application, we fixed $\alpha=0.6$.  As it
+can be seen in that Table, the size of the problem has a strong influence on the
+number of iterations to reach the  convergence. That is why we have preferred to
+change the threshold.  If we set  it to $1e-3$ as with the previous application,
+only one iteration is necessray  to reach the convergence. So Table~\ref{tab:04}
+shows the results  of differents executions with differents  number of cores and
+differents thresholds. As  with the previous example, we  can observe that TSIRM
+is faster than FGMRES. The ratio greatly depends on the number of iterations for
+FMGRES to reach the threshold. The greater the number of iterations to reach the
+convergence is, the  better the ratio between our algorithm  and FMGRES is. This
+experiment is  also a  weak scaling with  approximately $25,000$  components per
+core. It can also  be observed that the difference between CGLS  and LSQR is not
+significant. Both can be good but it seems not possible to know in advance which
+one will be the best.
+
+Table~\ref{tab:05} show a strong scaling experiment with the exemple ex54 on the
+Curie  architecture. So  in  this case,  the  number of  unknownws  is fixed  to
+$204,919,225$ and the number of cores ranges from $512$ to $8192$ with the power
+of two.  The  threshold is fixed to $5e-5$ and only  the $mg$ preconditioner has
+been tested. Here  again we can see that TSIRM is  faster that FGMRES. Efficiecy
+of each algorithms is reported. It  can be noticed that FGMRES is more efficient
+than TSIRM except with $8,192$ cores and that its efficiency is greater that one
+whereas the  efficiency of TSIRM is  lower than one. Nevertheless,  the ratio of
+TSIRM  with any  version  of the  least-squares  method is  always faster.  With
+$8,192$ cores when the number of iterations is far more important for FGMRES, we
+can see that it is only slightly more important for TSIRM.
+
+In  Figure~\ref{fig:02}  we report  the  number  of  iterations per  second  for
+experiments  reported in  Table~\ref{tab:05}.  This Figure  highlights that  the
+number of iterations per  seconds is more of less the same  for FGMRES and TSIRM
+with a little advantage for FGMRES. It  can be explained by the fact that, as we
+have previously explained, that the iterations of the least-sqaure steps are not
+taken into account with TSIRM.

 
 \begin{table*}[htbp]
 \begin{center}
 
 \begin{table*}[htbp]
 \begin{center}


@@ -1032,6 +1104,26 @@ In Table~\ref{tab:04}, some experiments with example ex54 on the Curie architect
 \label{fig:02}
 \end{figure}
 
 \label{fig:02}
 \end{figure}
 

+
+Concerning the  experiments some  other remarks are  interesting.
+\begin{itemize}
+\item We  can tested other examples of  PETSc (ex29, ex45, ex49).  For all these
+  examples,  we also obtained  similar gain  between GMRES  and TSIRM  but those
+  examples are  not scalable with many  cores. In general, we  had some problems
+  with more than $4,096$ cores.
+\item We have tested many iterative  solvers available in PETSc.  In fast, it is
+  possible to use most of them with TSIRM. From our point of view, the condition
+  to  use  a  solver inside  TSIRM  is  that  the  solver  must have  a  restart
+  feature. More  precisely, the solver must  support to be  stoped and restarted
+  without decrease its  converge. That is why  with GMRES we stop it  when it is
+  naturraly  restarted (i.e.  with  $m$ the  restart parameter).   The Conjugate
+  Gradient (CG) and all its variants do not have ``restarted'' version in PETSc,
+  so they  are not  efficient.  They  will converge with  TSIRM but  not quickly
+  because if  we compare  a normal CG  with a CG  for which  we stop it  each 16
+  iterations  for example,  the  normal CG  will  be for  more efficient.   Some
+  restarted CG  or CG variant versions exist  and may be interested  to study in
+  future works.
+\end{itemize}

 %%%*********************************************************
 %%%*********************************************************
 
 %%%*********************************************************
 %%%*********************************************************
 


@@ -1054,13 +1146,14 @@ experiments up to 16,394 cores have been led to verify that TSIRM runs
 5 or  7 times  faster than GMRES.
 
 
 5 or  7 times  faster than GMRES.
 
 

-For future work, the authors' intention is to investigate 
-other kinds of matrices, problems, and inner solvers. The 
-influence of all parameters must be tested too, while 
-other methods to minimize the residuals must be regarded.
-The number of outer iterations to minimize should become 
-adaptative to improve the overall performances of the proposal.
-Finally, this solver will be implemented inside PETSc.
+For  future  work, the  authors'  intention is  to  investigate  other kinds  of
+matrices, problems, and  inner solvers. The influence of  all parameters must be
+tested too, while other methods to minimize the residuals must be regarded.  The
+number of outer  iterations to minimize should become  adaptative to improve the
+overall performances of the proposal.   Finally, this solver will be implemented
+inside PETSc. This  would be very interesting because it would  allow us to test
+all the non-linear  examples and compare our algorithm  with the other algorithm
+implemented in PETSc.

 
 
 % conference papers do not normally have an appendix
 
 
 % conference papers do not normally have an appendix