X-Git-Url: https://bilbo.iut-bm.univ-fcomte.fr/and/gitweb/GMRES2stage.git/blobdiff_plain/eab5f4a9438f75792a4317e54757ff950cd04faf..b751185010fba3e771ea8f9a6ef579aba3848e1f:/paper.tex

diff --git a/paper.tex b/paper.tex
index d0fac64..b96fa3a 100644
--- a/paper.tex
+++ b/paper.tex
@@ -601,7 +601,14 @@ is summarized while intended perspectives are provided.
 %%%*********************************************************
 \section{Related works}
 \label{sec:02} 
-%Wherever Times is specified, Times Roman or Times New Roman may be used. If neither is available on your system, please use the font closest in appearance to Times. Avoid using bit-mapped fonts if possible. True-Type 1 or Open Type fonts are preferred. Please embed symbol fonts, as well, for math, etc.
+Krylov subspace iteration methods have increasingly become useful and successful techniques for solving linear and nonlinear systems and eigenvalue problems, especially since the increase development of the preconditioners~\cite{Saad2003,Meijerink77}. One reason of the popularity of these methods is their generality, simplicity and efficiency to solve systems of equations arising from very large and complex problems. %A Krylov method is based on a projection process onto a Krylov subspace spanned by vectors and it forms a sequence of approximations by minimizing the residual over the subspace formed~\cite{}.
+
+GMRES is one of the most widely used Krylov iterative method for solving sparse and large linear systems. It is developed by Saad and al.~\cite{Saad86} as a generalized method to deal with unsymmetric and non-Hermitian problems, and indefinite symmetric problems too. In its original version called full GMRES, it minimizes the residual over the current Krylov subspace until convergence in at most $n$ iterations, where $n$ is the size of the sparse matrix. It should be noted that full GMRES is too expensive in the case of large matrices since the required orthogonalization process per iteration grows quadratically with the number of iterations. For that reason, in practice GMRES is restarted after each $m\ll n$ iterations to avoid the storage of a large orthonormal basis. However, the convergence behavior of the restarted GMRES, called GMRES($m$), in many cases depends quite critically on the value of $m$~\cite{Huang89}. Therefore in most cases, a preconditioning technique is applied to the restarted GMRES method in order to improve its convergence.
+
+In order to enhance the robustness of Krylov iterative solvers, some techniques have been proposed allowing the use of different preconditioners, if necessary, within the iteration instead of restarting. Those techniques may lead to considerable savings in CPU time and memory requirements. Van der Vorst in~\cite{Vorst94} has proposed variants of the GMRES algorithm in which a different preconditioner is applied in each iteration, so-called GMRESR family of nested methods. In fact, the GMRES method is effectively preconditioned with other iterative schemes (or GMRES itself), where the iterations of the GMRES method are called outer iterations while the iterations of the preconditioning process referred to as inner iterations. Saad in~\cite{Saad:1993} has proposed FGMRES which is another variant of the GMRES algorithm using a variable preconditioner. In FGMRES the search directions are preconditioned whereas in GMRESR the residuals are preconditioned. However in practice the good preconditioners are those based on direct methods, as ILU preconditioners, which are not easy to parallelize and suffer from the scalability problems on large clusters of thousands of cores.      
+
+% two-stage, communication avoiding 
+
 %%%*********************************************************
 %%%*********************************************************
 
@@ -654,10 +661,10 @@ appropriate than a single direct method in a parallel context.
   \Input $A$ (sparse matrix), $b$ (right-hand side)
   \Output $x$ (solution vector)\vspace{0.2cm}
   \State Set the initial guess $x_0$
-  \For {$k=1,2,3,\ldots$ until convergence (error$<\epsilon_{tsirm}$)} \label{algo:conv}
+  \For {$k=1,2,3,\ldots$ until convergence ($error<\epsilon_{tsirm}$)} \label{algo:conv}
     \State  $[x_k,error]=Solve(A,b,x_{k-1},max\_iter_{kryl})$   \label{algo:solve}
-    \State $S_{k \mod s}=x_k$ \label{algo:store} \Comment{update column (k mod s) of S}
-    \If {$k \mod s=0$ {\bf and} error$>\epsilon_{kryl}$}
+    \State $S_{k \mod s}=x_k$ \label{algo:store} \Comment{update column ($k \mod s$) of $S$}
+    \If {$k \mod s=0$ {\bf and} $error>\epsilon_{kryl}$}
       \State $R=AS$ \Comment{compute dense matrix} \label{algo:matrix_mul}
             \State $\alpha=Least\_Squares(R,b,max\_iter_{ls})$ \label{algo:}
       \State $x_k=S\alpha$  \Comment{compute new solution}
@@ -675,10 +682,10 @@ method. Moreover,  a tolerance  threshold must be  specified for the  solver. In
 practice, this threshold must be  much smaller than the convergence threshold of
 the  TSIRM algorithm  (\emph{i.e.}, $\epsilon_{tsirm}$).  We also  consider that
 after the call of the $Solve$ function, we obtain the vector $x_k$ and the error
-which is defined by $||Ax^k-b||_2$.
+which is defined by $||Ax_k-b||_2$.
 
   Line~\ref{algo:store},
-$S_{k \mod  s}=x^k$ consists in  copying the solution  $x_k$ into the  column $k
+$S_{k \mod  s}=x_k$ consists in  copying the solution  $x_k$ into the  column $k
 \mod s$ of $S$.   After the minimization, the matrix $S$ is  reused with the new
 values of the residuals.  To solve the minimization problem, an iterative method
 is used. Two parameters are required  for that: the maximum number of iterations
@@ -790,7 +797,12 @@ Suppose now that the claim holds for all $m=1, 2, \hdots, k-1$, that is, $\foral
 We will show that the statement holds too for $r_k$. Two situations can occur:
 \begin{itemize}
 \item If $k \not\equiv 0 ~(\textrm{mod}\ m)$, then the TSIRM algorithm consists in executing GMRES once. In that case and by using the inductive hypothesis, we obtain either $||r_k|| \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{m}{2}} ||r_{k-1}||\leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{km}{2}} ||r_0||$ if $A$ is positive, or $\|r_k\| \leqslant \left( 1-\frac{\lambda_{\mathrm{min}}^2(1/2(A^T + A))}{ \lambda_{\mathrm{max}}(A^T A)} \right)^{m/2} \|r_{k-1}\|$ $\leqslant$ $\left( 1-\frac{\lambda_{\mathrm{min}}^2(1/2(A^T + A))}{ \lambda_{\mathrm{max}}(A^T A)} \right)^{km/2} \|r_{0}\|$ in the positive definite case.
-\item Else, the TSIRM algorithm consists in two stages: a first GMRES($m$) execution leads to a temporary $x_k$ whose residue satisfies $||r_k|| \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{m}{2}} ||r_{k-1}||\leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{km}{2}} ||r_0||$, and a least squares resolution.
+\item Else, the TSIRM algorithm consists in two stages: a first GMRES($m$) execution leads to a temporary $x_k$ whose residue satisfies:
+\begin{itemize}
+\item $||r_k|| \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{m}{2}} ||r_{k-1}||\leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{km}{2}} ||r_0||$ in the positive case, 
+\item $\|r_k\| \leqslant \left( 1-\frac{\lambda_{\mathrm{min}}^2(1/2(A^T + A))}{ \lambda_{\mathrm{max}}(A^T A)} \right)^{m/2} \|r_{k-1}\|$ $\leqslant$ $\left( 1-\frac{\lambda_{\mathrm{min}}^2(1/2(A^T + A))}{ \lambda_{\mathrm{max}}(A^T A)} \right)^{km/2} \|r_{0}\|$ in the positive definite one,
+\end{itemize}
+and a least squares resolution.
 Let $\operatorname{span}(S) = \left \{ {\sum_{i=1}^k \lambda_i v_i \Big| k \in \mathbb{N}, v_i \in S, \lambda _i \in \mathbb{R}} \right \}$ be the linear span of a set of real vectors $S$. So,\\
 $\min_{\alpha \in \mathbb{R}^s} ||b-R\alpha ||_2 = \min_{\alpha \in \mathbb{R}^s} ||b-AS\alpha ||_2$
 
@@ -801,14 +813,16 @@ $\begin{array}{ll}
 & \leqslant \min_{\lambda \in \mathbb{R}} ||b-\lambda Ax_{k} ||_2\\
 & \leqslant ||b-Ax_{k}||_2\\
 & = ||r_k||_2\\
-& \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{km}{2}} ||r_0||,
+& \leqslant \left(1-\dfrac{\alpha}{\beta}\right)^{\frac{km}{2}} ||r_0||, \textrm{ if $A$ is positive,}\\
+& \leqslant \left( 1-\frac{\lambda_{\mathrm{min}}^2(1/2(A^T + A))}{ \lambda_{\mathrm{max}}(A^T A)} \right)^{km/2} \|r_{0}\|, \textrm{ if $A$ is}\\
+& \textrm{positive definite,} 
 \end{array}$
 \end{itemize}
 which concludes the induction and the proof.
 \end{proof}
 
-We can remark that, at each iterate, the residue of the TSIRM algorithm is lower 
-than the one of the GMRES method.
+%We can remark that, at each iterate, the residue of the TSIRM algorithm is lower 
+%than the one of the GMRES method.
 
 %%%*********************************************************
 %%%*********************************************************
@@ -816,13 +830,13 @@ than the one of the GMRES method.
 \label{sec:05}
 
 
-In order to see the influence of our algorithm with only one processor, we first
-show a comparison with GMRES or FGMRES and our algorithm. In Table~\ref{tab:01},
-we  show the  matrices we  have  used and  some of  them characteristics.  Those
-matrices  are   chosen  from   the  Davis  collection   of  the   University  of
-Florida~\cite{Dav97}. They are matrices arising in real-world applications.  For
-all the  matrices, the name,  the field,  the number of  rows and the  number of
-nonzero elements are given.
+In order to see the behavior of the proposal when considering only one processor, a first
+comparison with GMRES or FGMRES and the new algorithm detailed previously has been experimented. 
+Matrices that have been used with their characteristics (names, fields, rows, and nonzero coefficients) are detailed in 
+Table~\ref{tab:01}.  These latter, which are real-world applications matrices, 
+have been extracted 
+ from   the  Davis  collection,   University  of
+Florida~\cite{Dav97}.
 
 \begin{table}[htbp]
 \begin{center}
@@ -842,8 +856,9 @@ torso3             & 2D/3D problem & 259,156 & 4,429,042 \\
 \label{tab:01}
 \end{center}
 \end{table}
-
-The following  parameters have been chosen  for our experiments.   As by default
+Chosen parameters are detailed below.
+%The following  parameters have been chosen  for our experiments.   
+As by default
 the restart  of GMRES is performed every  30 iterations, we have  chosen to stop
 the GMRES every 30 iterations (\emph{i.e.} $max\_iter_{kryl}=30$).  $s$ is set to 8. CGLS is
 chosen  to minimize  the least-squares  problem with  the  following parameters:
@@ -923,8 +938,16 @@ by core. The Juqueen architecture is composed  of IBM PowerPC A2 at 1.6 GHz with
 speed network.
 
 
+In  many situations, using  preconditioners is  essential in  order to  find the
+solution of a linear system.  There are many preconditioners available in PETSc.
+For parallel applications all  the preconditioners based on matrix factorization
+are  not  available. In  our  experiments, we  have  tested  different kinds  of
+preconditioners, however  as it is  not the subject  of this paper, we  will not
+present results with many preconditioners. In  practise, we have chosen to use a
+multigrid (mg)  and successive  over-relaxation (sor). For  more details  on the
+preconditioner in PETSc please consult~\cite{petsc-web-page}.
+
 
-{\bf Description of preconditioners}\\
 
 \begin{table*}[htbp]
 \begin{center}
@@ -952,8 +975,7 @@ speed network.
 
 Table~\ref{tab:03} shows  the execution  times and the  number of  iterations of
 example ex15  of PETSc on the  Juqueen architecture. Different  numbers of cores
-are  studied ranging  from  2,048  up-to 16,383.   Two  preconditioners have  been
-tested: {\it mg} and {\it sor}.   For those experiments,  the number  of components  (or unknowns  of the
+are  studied ranging  from  2,048  up-to 16,383 with the two preconditioners {\it mg} and {\it sor}.   For those experiments,  the number  of components  (or unknowns  of the
 problems)  per core  is fixed  to 25,000,  also called  weak  scaling. This
 number can seem relatively small. In fact, for some applications that need a lot
 of  memory, the  number of  components per  processor requires  sometimes  to be
@@ -1020,8 +1042,40 @@ the number of iterations. So, the overall benefit of using TSIRM is interesting.
 \end{table*}
 
 
-In Table~\ref{tab:04}, some experiments with example ex54 on the Curie architecture are reported.
-
+In  Table~\ref{tab:04},  some  experiments   with  example  ex54  on  the  Curie
+architecture are reported.  For this  application, we fixed $\alpha=0.6$.  As it
+can be seen in that Table, the size of the problem has a strong influence on the
+number of iterations to reach the  convergence. That is why we have preferred to
+change the threshold.  If we set  it to $1e-3$ as with the previous application,
+only one iteration is necessray  to reach the convergence. So Table~\ref{tab:04}
+shows the results  of differents executions with differents  number of cores and
+differents thresholds. As  with the previous example, we  can observe that TSIRM
+is faster than FGMRES. The ratio greatly depends on the number of iterations for
+FMGRES to reach the threshold. The greater the number of iterations to reach the
+convergence is, the  better the ratio between our algorithm  and FMGRES is. This
+experiment is  also a  weak scaling with  approximately $25,000$  components per
+core. It can also  be observed that the difference between CGLS  and LSQR is not
+significant. Both can be good but it seems not possible to know in advance which
+one will be the best.
+
+Table~\ref{tab:05} show a strong scaling experiment with the exemple ex54 on the
+Curie  architecture. So  in  this case,  the  number of  unknownws  is fixed  to
+$204,919,225$ and the number of cores ranges from $512$ to $8192$ with the power
+of two.  The  threshold is fixed to $5e-5$ and only  the $mg$ preconditioner has
+been tested. Here  again we can see that TSIRM is  faster that FGMRES. Efficiecy
+of each algorithms is reported. It  can be noticed that FGMRES is more efficient
+than TSIRM except with $8,192$ cores and that its efficiency is greater that one
+whereas the  efficiency of TSIRM is  lower than one. Nevertheless,  the ratio of
+TSIRM  with any  version  of the  least-squares  method is  always faster.  With
+$8,192$ cores when the number of iterations is far more important for FGMRES, we
+can see that it is only slightly more important for TSIRM.
+
+In  Figure~\ref{fig:02}  we report  the  number  of  iterations per  second  for
+experiments  reported in  Table~\ref{tab:05}.  This Figure  highlights that  the
+number of iterations per  seconds is more of less the same  for FGMRES and TSIRM
+with a little advantage for FGMRES. It  can be explained by the fact that, as we
+have previously explained, that the iterations of the least-sqaure steps are not
+taken into account with TSIRM.
 
 \begin{table*}[htbp]
 \begin{center}
@@ -1052,6 +1106,26 @@ In Table~\ref{tab:04}, some experiments with example ex54 on the Curie architect
 \label{fig:02}
 \end{figure}
 
+
+Concerning the  experiments some  other remarks are  interesting.
+\begin{itemize}
+\item We  can tested other examples of  PETSc (ex29, ex45, ex49).  For all these
+  examples,  we also obtained  similar gain  between GMRES  and TSIRM  but those
+  examples are  not scalable with many  cores. In general, we  had some problems
+  with more than $4,096$ cores.
+\item We have tested many iterative  solvers available in PETSc.  In fast, it is
+  possible to use most of them with TSIRM. From our point of view, the condition
+  to  use  a  solver inside  TSIRM  is  that  the  solver  must have  a  restart
+  feature. More  precisely, the solver must  support to be  stoped and restarted
+  without decrease its  converge. That is why  with GMRES we stop it  when it is
+  naturraly  restarted (i.e.  with  $m$ the  restart parameter).   The Conjugate
+  Gradient (CG) and all its variants do not have ``restarted'' version in PETSc,
+  so they  are not  efficient.  They  will converge with  TSIRM but  not quickly
+  because if  we compare  a normal CG  with a CG  for which  we stop it  each 16
+  iterations  for example,  the  normal CG  will  be for  more efficient.   Some
+  restarted CG  or CG variant versions exist  and may be interested  to study in
+  future works.
+\end{itemize}
 %%%*********************************************************
 %%%*********************************************************
 
@@ -1074,13 +1148,14 @@ experiments up to 16,394 cores have been led to verify that TSIRM runs
 5 or  7 times  faster than GMRES.
 
 
-For future work, the authors' intention is to investigate 
-other kinds of matrices, problems, and inner solvers. The 
-influence of all parameters must be tested too, while 
-other methods to minimize the residuals must be regarded.
-The number of outer iterations to minimize should become 
-adaptative to improve the overall performances of the proposal.
-Finally, this solver will be implemented inside PETSc.
+For  future  work, the  authors'  intention is  to  investigate  other kinds  of
+matrices, problems, and  inner solvers. The influence of  all parameters must be
+tested too, while other methods to minimize the residuals must be regarded.  The
+number of outer  iterations to minimize should become  adaptative to improve the
+overall performances of the proposal.   Finally, this solver will be implemented
+inside PETSc. This  would be very interesting because it would  allow us to test
+all the non-linear  examples and compare our algorithm  with the other algorithm
+implemented in PETSc.
 
 
 % conference papers do not normally have an appendix
