X-Git-Url: https://bilbo.iut-bm.univ-fcomte.fr/and/gitweb/GMRES2stage.git/blobdiff_plain/42497688801cf1834fb46147eb2837a6fa13e5a0..e8b2791c920612ee2d944379890d2a6b76e0acea:/IJHPCN/paper.tex

diff --git a/IJHPCN/paper.tex b/IJHPCN/paper.tex
index 3aa76de..f67fa56 100644
--- a/IJHPCN/paper.tex
+++ b/IJHPCN/paper.tex
@@ -49,9 +49,7 @@
 \makeatletter
 \def\theequation{\arabic{equation}}
 
-%\JOURNALNAME{\TEN{\it Int. J. System Control and Information
-%Processing,
-%Vol. \theVOL, No. \theISSUE, \thePUBYEAR\hfill\thepage}}%
+\JOURNALNAME{\TEN{\it International Journal of High Performance Computing and Networking}}
 %
 %\def\BottomCatch{%
 %\vskip -10pt
@@ -73,10 +71,9 @@
 
 \setcounter{page}{1}
 
-\LRH{F. Wang et~al.}
+\LRH{R. Couturier, L. Ziane Khodja and C. Guyeux}
 
-\RRH{Metadata Based Management and Sharing of Distributed Biomedical
-Data}
+\RRH{TSIRM: A Two-Stage Iteration with least-squares Residual Minimization algorithm}
 
 \VOL{x}
 
@@ -86,7 +83,7 @@ Data}
 
 \BottomCatch
 
-\PUBYEAR{2012}
+\PUBYEAR{2015}
 
 \subtitle{}
 
@@ -109,19 +106,25 @@ Data}
 
 
 \begin{abstract}
-In  this article, a  two-stage iterative  algorithm is  proposed to  improve the
+In  this paper,  a  two-stage iterative  algorithm is  proposed  to improve  the
 convergence  of  Krylov  based  iterative  methods,  typically  those  of  GMRES
-variants.  The  principle of  the  proposed approach  is  to  build an  external
-iteration over the  Krylov method, and to frequently  store its current residual
+variants.   The principle  of  the proposed  approach is  to  build an  external
+iteration over the  Krylov method, and to frequently store  its current residual
 (at each GMRES restart for instance).  After a given number of outer iterations,
 a least-squares minimization step is applied on the matrix composed by the saved
-residuals, in order  to compute a better solution and to  make new iterations if
-required.  It  is proven that the  proposal has the  same convergence properties
-than the  inner embedded  method itself.  Experiments  using up to  16,394 cores
-also  show that the  proposed algorithm  runs around  5 or  7 times  faster than
-GMRES.
+residuals, in order to  compute a better solution and to  make new iterations if
+required.  It  is proven that the  proposal has the same  convergence properties
+than the inner  embedded method itself.
+%%NEW
+Several experiments  have been performed
+with  the PETSc  solver  with  linear and  nonlinear  problems.  They show  good
+speedups   compared  to   GMRES  with   up  to   16,394  cores   with  different
+preconditioners.
+%%ENDNEW
 \end{abstract}
 
+
+
 \KEYWORD{Iterative Krylov methods; sparse linear and non linear systems; two stage iteration; least-squares residual minimization; PETSc.}
 
 %\REF{to this paper should be made as follows: Rodr\'{\i}guez
@@ -131,28 +134,11 @@ GMRES.
 %Semantics and Ontologies}, Vol. x, No. x, pp.xxx\textendash xxx.}
 
 \begin{bio}
-Manuel Pedro Rodr\'iguez Bol\'ivar received his PhD in Accounting at
-the University of Granada. He is a Lecturer at the Department of
-Accounting and Finance, University of Granada. His research
-interests include issues related to conceptual frameworks of
-accounting, diffusion of financial information on Internet, Balanced
-Scorecard applications and environmental accounting. He is author of
-a great deal of research studies published at national and
-international journals, conference proceedings as well as book
-chapters, one of which has been edited by Kluwer Academic
-Publishers.\vs{9}
-
-\noindent Bel\'en Sen\'es Garc\'ia received her PhD in Accounting at
-the University of Granada. She is a Lecturer at the Department of
-Accounting and Finance, University of Granada. Her research
-interests are related to cultural, institutional and historic
-accounting and in environmental accounting. She has published
-research papers at national and international journals, conference
-proceedings as well as chapters of books.\vs{8}
-
-\noindent Both authors have published a book about environmental
-accounting edited by the Institute of Accounting and Auditing,
-Ministry of Economic Affairs, in Spain in October 2003.
+RaphaÃ«l Couturier ....
+
+\noindent Lilia Ziane Khodja ...
+
+\noindent Christophe Guyeux ...
 \end{bio}
 
 
@@ -502,6 +488,13 @@ that the proposed TSIRM converges while the GMRES($m$) does not.
 \section{Experiments using PETSc}
 \label{sec:05}
 
+%%NEW
+In this section four kinds of experiments have been performed. First, some experiments on real matrices issued from the sparse matrix florida have been achieved out. Second, some experiments in parallel with some linear problems are reported and analyzed. Third, some experiments in parallÃ¨le with som nonlinear problems are illustrated. Finally some parameters of TSIRM are studied in order to understand their influences.
+
+
+\subsection{Real matrices}
+%%ENDNEW
+
 
 In order to see the behavior of our approach when considering only one processor,
 a  first  comparison  with  GMRES  or  FGMRES and  the  new  algorithm  detailed
@@ -574,8 +567,9 @@ torso3             & fgmres / sor  & 37.70 & 565 & 34.97 & 510 \\
 \end{table*}
 
 
-
-
+%%NEW
+\subsection{Parallel linear problems}
+%%ENDNEW
 
 In order to perform larger experiments, we have tested some example applications
 of  PETSc. These  applications are  available in  the \emph{ksp}  part,  which is
@@ -614,7 +608,7 @@ However, for parallel applications, all  the preconditioners based on matrix fac
 are  not  available. In  our  experiments, we  have  tested  different kinds  of
 preconditioners, but  as it is  not the subject  of this paper, we  will not
 present results with many preconditioners. In  practice, we have chosen to use a
-multigrid (mg)  and successive  over-relaxation (sor). For  further details  on the
+multigrid (MG)  and successive  over-relaxation (SOR). For  further details  on the
 preconditioners in PETSc, readers are referred to~\cite{petsc-web-page}.
 
 
@@ -627,18 +621,18 @@ preconditioners in PETSc, readers are referred to~\cite{petsc-web-page}.
   nb. cores & precond   & \multicolumn{2}{c|}{FGMRES} & \multicolumn{2}{c|}{TSIRM CGLS} &  \multicolumn{2}{c|}{TSIRM LSQR} & best gain \\ 
 \cline{3-8}
              &                       & Time  & \# Iter.  & Time  & \# Iter. & Time  & \# Iter. & \\\hline \hline
-  2,048      & mg                    & 403.49   & 18,210    & 73.89  & 3,060   & 77.84  & 3,270  & 5.46 \\
-  2,048      & sor                   & 745.37   & 57,060    & 87.31  & 6,150   & 104.21 & 7,230  & 8.53 \\
-  4,096      & mg                    & 562.25   & 25,170    & 97.23  & 3,990   & 89.71  & 3,630  & 6.27 \\
-  4,096      & sor                   & 912.12   & 70,194    & 145.57 & 9,750   & 168.97 & 10,980 & 6.26 \\
-  8,192      & mg                    & 917.02   & 40,290    & 148.81 & 5,730   & 143.03 & 5,280  & 6.41 \\
-  8,192      & sor                   & 1,404.53 & 106,530   & 212.55 & 12,990  & 180.97 & 10,470 & 7.76 \\
-  16,384     & mg                    & 1,430.56 & 63,930    & 237.17 & 8,310   & 244.26 & 7,950  & 6.03 \\
-  16,384     & sor                   & 2,852.14 & 216,240   & 418.46 & 21,690  & 505.26 & 23,970 & 6.82 \\
+  2,048      & MG                    & 403.49   & 18,210    & 73.89  & 3,060   & 77.84  & 3,270  & 5.46 \\
+  2,048      & SOR                   & 745.37   & 57,060    & 87.31  & 6,150   & 104.21 & 7,230  & 8.53 \\
+  4,096      & MG                    & 562.25   & 25,170    & 97.23  & 3,990   & 89.71  & 3,630  & 6.27 \\
+  4,096      & SOR                   & 912.12   & 70,194    & 145.57 & 9,750   & 168.97 & 10,980 & 6.26 \\
+  8,192      & MG                    & 917.02   & 40,290    & 148.81 & 5,730   & 143.03 & 5,280  & 6.41 \\
+  8,192      & SOR                   & 1,404.53 & 106,530   & 212.55 & 12,990  & 180.97 & 10,470 & 7.76 \\
+  16,384     & MG                    & 1,430.56 & 63,930    & 237.17 & 8,310   & 244.26 & 7,950  & 6.03 \\
+  16,384     & SOR                   & 2,852.14 & 216,240   & 418.46 & 21,690  & 505.26 & 23,970 & 6.82 \\
 \hline
 
 \end{tabular}
-\caption{Comparison of FGMRES and TSIRM with FGMRES for example ex15 of PETSc with two preconditioners (mg and sor) having 25,000 components per core on Juqueen ($\epsilon_{tsirm}=1e-3$, $max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$, $\epsilon_{ls}=1e-40$),  time is expressed in seconds.}
+\caption{Comparison of FGMRES and TSIRM with FGMRES for example ex15 of PETSc/KSP with two preconditioners (MG and SOR) having 25,000 components per core on Juqueen ($\epsilon_{tsirm}=1e-3$, $max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$, $\epsilon_{ls}=1e-40$),  time is expressed in seconds.}
 \label{tab:03}
 \end{center}
 \end{table*}
@@ -646,7 +640,7 @@ preconditioners in PETSc, readers are referred to~\cite{petsc-web-page}.
 Table~\ref{tab:03} shows  the execution  times and the  number of  iterations of
 example ex15  of PETSc on the  Juqueen architecture. Different  numbers of cores
 are studied  ranging from 2,048 up-to  16,383 with the  two preconditioners {\it
-  mg}  and {\it  sor}.   For those  experiments,  the number  of components  (or
+  MG}  and {\it  SOR}.   For those  experiments,  the number  of components  (or
 unknowns  of  the problems)  per  core  is fixed  at  25,000,  also called  weak
 scaling. This number  can seem relatively small. In  fact, for some applications
 that  need a  lot of  memory, the  number of  components per  processor requires
@@ -710,7 +704,7 @@ interesting.
 \hline
 
 \end{tabular}
-\caption{Comparison of FGMRES  and TSIRM with FGMRES algorithms for ex54 of Petsc (both with the MG preconditioner) with 25,000 components per core on Curie ($max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$, $\epsilon_{ls}=1e-40$),  time is expressed in seconds.}
+\caption{Comparison of FGMRES  and TSIRM with FGMRES algorithms for ex54 of PETSc/KSP (both with the MG preconditioner) with 25,000 components per core on Curie ($max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$, $\epsilon_{ls}=1e-40$),  time is expressed in seconds.}
 \label{tab:04}
 \end{center}
 \end{table*}
@@ -769,7 +763,7 @@ taken into account with TSIRM.
 \hline
 
 \end{tabular}
-\caption{Comparison of FGMRES  and TSIRM for ex54 of PETSc (both with the MG preconditioner) with 204,919,225 components on Curie with different number of cores ($\epsilon_{tsirm}=5e-5$, $max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$, $\epsilon_{ls}=1e-40$),  time is expressed in seconds.}
+\caption{Comparison of FGMRES  and TSIRM for ex54 of PETSc/KSP (both with the MG preconditioner) with 204,919,225 components on Curie with different number of cores ($\epsilon_{tsirm}=5e-5$, $max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$, $\epsilon_{ls}=1e-40$),  time is expressed in seconds.}
 \label{tab:05}
 \end{center}
 \end{table*}
@@ -782,9 +776,215 @@ taken into account with TSIRM.
 \end{figure}
 
 
+\begin{figure}[htbp]
+\centering
+  \includegraphics[width=0.5\textwidth]{nb_iter_sec_ex45_curie}
+\caption{Number of iterations per second with ex45 and the same parameters as in Table~\ref{tab:06} (weak scaling)}
+\label{fig:03}
+\end{figure}
+
+
+%%NEW
+
+{\bf example ex45/ksp Ã  dÃ©crire et commenter en montrant que hypre est pourri avec cet exemple}
+
+\begin{table*}[htbp]
+\begin{center}
+\begin{tabular}{|r|r|r|r|r|r|r|r|} 
+\hline
+
+  nb. cores   & \multicolumn{2}{c|}{FGMRES/ASM} & \multicolumn{2}{c|}{TSIRM CGLS/ASM} & gain& \multicolumn{2}{c|}{FGMRES/HYPRE}   \\ 
+\cline{2-5} \cline{7-8}
+                    & Time  & \# Iter.  & Time  & \# Iter. &        & Time  & \# Iter.   \\\hline \hline
+   512              & 5.54      & 685    & 2.5 &       570 & 2.21   & 128.9 & 9     \\
+   2048             & 14.95     & 1,560  &  4.32 &     746 & 3.48   & 335.7 & 9 \\
+   4096             & 25.13    & 2,369   & 5.61 &   859    & 4.48   & >1000  & -- \\
+   8192             & 44.35   & 3,197   &  7.6  &  1083    &  5.84  & >1000 &  --   \\
+
+\hline
+
+\end{tabular}
+\caption{Comparison of FGMRES  and TSIRM for ex45 of PETSc/KSP with two preconditioner (ASM and HYPRE)  having 5,000 components per core on Curie ($\epsilon_{tsirm}=1e-10$, $max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$,$\epsilon_{ls}=1e-40$),  time is expressed in seconds.}
+\label{tab:06}
+\end{center}
+\end{table*}
+
+
+\subsection{Parallel nonlinear problems}
+
+With  PETSc,  linear  solvers  are  used inside  nonlinear  solvers.   The  SNES
+(Scalable Nonlinear  Equations Solvers) module  in PETSc implements easy  to use
+methods,  like  Newton-type, quasi-Newton  or  full  approximation scheme  (FAS)
+multigrid to  solve systems of  nonlinears equations.  As  SNES is based  on the
+Krylov methods of PETSc, it is interesting to investigate if the TSIRM method is
+also efficient  and scalable with non  linear problems. In PETSc,  some examples
+are provided.  An important criteria is the scalability of the initial code with
+classical solvers. Consequently, we have chosen  two of these examples: ex14 and
+ex20.  In ex14, the code solves the  Bratu (SFI - solid fuel ignition) nonlinear
+partial  difference equations  in 3  dimension.  In  ex20, the  code solves  a 3
+dimension radiative transport test problem.  For more details on these examples,
+interested readers are invited  to see the code in the  PETSc examples. For both
+these  examples,   a  weak  scaling   case  is  chosen  where   processors  have
+approximately a number of components equals to 100,000.
+
+In Table~\ref{tab:07}  we report the result  of our experiments for  the example
+ex14 with the block Jacobi preconditioner.  For TSIRM the CGLS algorithm is used
+to solve  the minimization step. In  this table, we  can see that the  number of
+iterations used by the linear solver is smaller with TSIRM compared with FGMRES.
+Consequently the execution times are smaller  with TSIRM. The gain between TSIRM
+and FGMRES  is around  6 and  7. The parameters  of TSIRM  are expressed  in the
+caption of the table.
+
+\begin{table*}[htbp]
+\begin{center}
+\begin{tabular}{|r|r|r|r|r|r|} 
+\hline
+
+  nb. cores   & \multicolumn{2}{c|}{FGMRES/BJAC} & \multicolumn{2}{c|}{TSIRM CGLS/BJAC} & gain  \\ 
+\cline{2-5}
+                    & Time         & \# Iter.  & Time   & \# Iter. &  \\\hline \hline
+   1,024              & 159.52      & 11,584    &  26.34  &     1,563  &  6.06  \\
+   2,048             & 226.24       & 16,459    &  37.23 &     2,248   &  6.08\\
+   4,096             & 391.21     & 27,794   &  50.93 &   2,911  &  7.69\\
+   8,192             & 543.23     & 37,770   &  79.21  &  4,324  & 6.86 \\
+
+\hline
+
+\end{tabular}
+\caption{Comparison of FGMRES  and TSIRM for ex14 of PETSc/SNES with a Block Jacobi  preconditioner  having 100,000 components per core on Curie ($\epsilon_{tsirm}=1e-10$, $max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$, $\epsilon_{ls}=1e-40$),  time is expressed in seconds.}
+\label{tab:07}
+\end{center}
+\end{table*}
+
+In Table~\cite{tab:08}, the results of the experiments with the example ex20 are
+reported. The block  Jacobi preconditioner has also been used  and CGLS to solve
+the minimization step for TSIRM. For this example, we can observ that the number
+of  iterations  for  FMGRES  increase  drastically  when  the  number  of  cores
+increases. With  TSIRM, we can  see that the  number of iterations  is initially
+very small compared  to the FGMRES ones  and when the number  of cores increase,
+the number  of iterations increases slighther  with TSIRM than with  FGMRES. For
+this example,  the gain  between TSIRM  and FGMRES ranges  between 8  with 1,024
+cores to more than 16 with 8,192 cores.
+
+\begin{table*}[htbp]
+\begin{center}
+\begin{tabular}{|r|r|r|r|r|r|} 
+\hline
+
+  nb. cores   & \multicolumn{2}{c|}{FGMRES/BJAC} & \multicolumn{2}{c|}{TSIRM CGLS/BJAC} & gain   \\ 
+\cline{2-5}
+                    & Time         & \# Iter.  & Time   & \# Iter.  &  \\\hline \hline
+   1,024              & 667.92      & 48,732    & 81.65  &     5,087 &  8.18 \\
+   2,048             & 966.87       & 77,177    &  90.34 &     5,716 &  10.70\\
+   4,096             & 1,742.31     & 124,411   &  119.21 &   6,905  & 14.61\\
+   8,192             & 2,739.21     & 187,626   &  168.9  &  9,000   & 16.22\\
+
+\hline
+
+\end{tabular}
+\caption{Comparison of FGMRES  and TSIRM for ex20 of PETSc/SNES with a Block Jacobi  preconditioner  having 100,000 components per core on Curie ($\epsilon_{tsirm}=1e-10$, $max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$, $\epsilon_{ls}=1e-40$),  time is expressed in seconds.}
+\label{tab:08}
+\end{center}
+\end{table*}
+
+
+
+
+
+
+
+%%NEW
+\subsection{Influence of parameters for TSIRM}
+In this section we present some experimental results in order to study the influence of some parameters on the TSIRM algorithm. We conducted experiments on $16$ cores to solve 3D problems of size $200,000$ components per core. We solved nonlinear problems token from examples of PETSc. We fixed some parameters of the TSIRM algorithm as follows: the nonlinear systems are solved with a precision of $10^{-8}$, block Jacobi preconditioner is used, the tolerance threshold $\epsilon_{tsirm}$ is $10^{-8}$ , the maximum number of iterations $max\_iter_{tsirm}$ is set to $10,000$ iterations, the FGMRES method is used as the inner solver with a tolerance threshold $\epsilon_{kryl}=10^{-10}$ and the least-squares problem is solved with a precision $\epsilon_{ls}=10^{-40}$ in the minimization process.
+
+%time mpirun ../ex48 -da_grid_x 147 -da_grid_y 147 -da_grid_z 147 -snes_rtol 1.e-8 -snes_monitor -ksp_type tsirm -ksp_pc_type bjacobi -pc_type ksp -ksp_tsirm_tol 1e-8 -ksp_tsirm_maxiter 10000 -ksp_ksp_type fgmres -ksp_tsirm_max_inner_iter 30 -ksp_tsirm_inner_restarts 30 -ksp_tsirm_inner_tol 1e-10 -ksp_tsirm_cgls 0 -ksp_tsirm_tol_ls 1.e-40 -ksp_tsirm_maxiter_ls 15 -ksp_tsirm_size_ls 10 
+\begin{figure}[htbp]
+\centering
+  \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_cgls_iter_total}
+\caption{Number of total iterations using two different methods for the minimization: LSQR and CGLS.}
+\label{fig:cgls-iter} 
+\end{figure}
+
+\begin{figure}[htbp]
+\centering
+  \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_cgls_time}
+\caption{Execution time in seconds using two different methods for the minimization: LSQR and CGLS.}
+\label{fig:cgls-time} 
+\end{figure}
+
+%time mpirun ../ex35 -da_grid_x 147 -da_grid_y 147 -da_grid_z 147 -snes_rtol 1.e-8 -snes_monitor -ksp_type tsirm -ksp_pc_type bjacobi -pc_type ksp -ksp_tsirm_tol 1e-8 -ksp_tsirm_maxiter 10000 -ksp_ksp_type fgmres -ksp_tsirm_max_inner_iter 30 -ksp_tsirm_inner_restarts 38 -ksp_tsirm_inner_tol 1e-10 -ksp_tsirm_cgls 0 -ksp_tsirm_tol_ls 1.e-40 -ksp_tsirm_maxiter_ls 15 -ksp_tsirm_size_ls 10
+\begin{figure}[htbp]
+\centering
+  \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_inner_restarts_iter_total}
+\caption{Number of total iterations with variation of restarts in the inner solver FGMRES.}
+\label{fig:inner_restarts_iter_total} 
+\end{figure}
+
+\begin{figure}[htbp]
+\centering
+  \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_inner_restarts_time}
+\caption{Execution time in seconds with variation of restarts in the inner solver FGMRES.}
+\label{fig:inner_restarts_time} 
+\end{figure}
+
+%time mpirun ../ex14 -da_grid_x 147 -da_grid_y 147 -da_grid_z 147 -snes_rtol 1.e-8 -snes_monitor -ksp_type tsirm -ksp_pc_type bjacobi -pc_type ksp -ksp_tsirm_tol 1e-8 -ksp_tsirm_maxiter 10000 -ksp_ksp_type fgmres -ksp_tsirm_max_inner_iter 1000 -ksp_tsirm_inner_restarts 30 -ksp_tsirm_inner_tol 1e-10 -ksp_tsirm_cgls 0 -ksp_tsirm_tol_ls 1.e-40 -ksp_tsirm_maxiter_ls 15 -ksp_tsirm_size_ls 10
+\begin{figure}[htbp]
+\centering
+  \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_max_inner_iter}
+\caption{Number of total iterations with variation of number of inner iterations.}
+\label{fig:max_inner_iter} 
+\end{figure}
+
+\begin{figure}[htbp]
+\centering
+  \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_max_inner_time}
+\caption{Execution time in seconds with variation of number of inner iterations.}
+\label{fig:max_inner_time} 
+\end{figure}
+
+%time mpirun ../ex14 -da_grid_x 147 -da_grid_y 147 -da_grid_z 147 -snes_rtol 1.e-8 -snes_monitor -ksp_type tsirm -ksp_pc_type bjacobi -pc_type ksp -ksp_tsirm_tol 1e-8 -ksp_tsirm_maxiter 10000 -ksp_ksp_type fgmres -ksp_tsirm_max_inner_iter 30 -ksp_tsirm_inner_restarts 30 -ksp_tsirm_inner_tol 1e-10 -ksp_tsirm_cgls 0 -ksp_tsirm_tol_ls 1.e-40 -ksp_tsirm_maxiter_ls 5 -ksp_tsirm_size_ls 10
+\begin{figure}[htbp]
+\centering
+  \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_maxiter_ls_iter}
+\caption{Number of total iterations with variation of number of iterations in the minimization process.}
+\label{fig:maxiter_ls_iter} 
+\end{figure}
+
+\begin{figure}[htbp]
+\centering
+  \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_maxiter_ls_time}
+\caption{Execution time in seconds with variation of number of iterations in the minimization process.}
+\label{fig:maxiter_ls_time} 
+\end{figure}
+
+%time mpirun ../ex14 -da_grid_x 147 -da_grid_y 147 -da_grid_z 147 -snes_rtol 1.e-8 -snes_monitor -ksp_type tsirm -ksp_pc_type bjacobi -pc_type ksp -ksp_tsirm_tol 1e-8 -ksp_tsirm_maxiter 10000 -ksp_ksp_type fgmres -ksp_tsirm_max_inner_iter 30 -ksp_tsirm_inner_restarts 30 -ksp_tsirm_inner_tol 1e-10 -ksp_tsirm_cgls 0 -ksp_tsirm_tol_ls 1.e-40 -ksp_tsirm_maxiter_ls 15 -ksp_tsirm_size_ls 2
+\begin{figure}[htbp]
+\centering
+  \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_size_ls_iter}
+\caption{Number of total iterations with variation of the size of the least-squares problem in the minimization process.}
+\label{fig:size_ls_iter} 
+\end{figure}
+
+\begin{figure}[htbp]
+\centering
+  \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_size_ls_time}
+\caption{Execution time in seconds with variation of the size of the least-squares problem in the minimization process.}
+\label{fig:size_ls_time} 
+\end{figure}
+
+%%ENDNEW
+
+
+
+
+
+\subsection{Experiments conclusions }
+
+{\bf A refaire}
+
 Concerning the  experiments some  other remarks are  interesting.
 \begin{itemize}
-\item We have tested other examples  of PETSc (ex29, ex45, ex49).  For all these
+\item We have tested other examples  of PETSc/KSP (ex29, ex45, ex49).  For all these
   examples,  we have also  obtained similar  gains between  GMRES and  TSIRM but
   those  examples are  not scalable  with many  cores. In  general, we  had some
   problems with more than $4,096$ cores.
@@ -801,34 +1001,10 @@ Concerning the  experiments some  other remarks are  interesting.
   efficient.   Some  restarted  CG or  CG  variant  versions  exist and  may  be
   interesting to study in future works.
 \end{itemize}
-%%%*********************************************************
-%%%*********************************************************
-
-
-%%NEW
-\begin{table*}[htbp]
-\begin{center}
-\begin{tabular}{|r|r|r|r|r|r|r|} 
-\hline
-
-  nb. cores   & \multicolumn{2}{c|}{FGMRES/ASM} & \multicolumn{2}{c|}{TSIRM CGLS/ASM} & \multicolumn{2}{c|}{FGMRES/HYPRE}   \\ 
-\cline{2-7}
-                    & Time  & \# Iter.  & Time  & \# Iter. & Time  & \# Iter.   \\\hline \hline
-   512              & 5.54      & 685    & 2.5 &       570  & 128.9 & 9     \\
-   2048             & 14.95     & 1,560  &  5.2 &     746   & 335.7 & 9 \\
-   4096             & 25.13    & 2,369   & 5.61 &   859  &   >1000  & -- \\
-   8192             & 44.35   & 3,197   &  7.6  &  1083 & >1000 &  --   \\
-
-\hline
-
-\end{tabular}
-\caption{Comparison of FGMRES  and TSIRM for ex45 of PETSc with two preconditioner (ASM and HYPRE)  having 25,000 components per core on Curie ($\epsilon_{tsirm}=1e-10$, $max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$, $\epsilon_{ls}=1e-40$),  time is expressed in seconds.}
-\label{tab:06}
-\end{center}
-\end{table*}
 
 %%ENDNEW
 
+
 %%%*********************************************************
 %%%*********************************************************
 \section{Conclusion}
@@ -837,28 +1013,20 @@ Concerning the  experiments some  other remarks are  interesting.
 %%%*********************************************************
 %%%*********************************************************
 
-A new two-stage iterative  algorithm TSIRM has been proposed in this article,
-in order to accelerate the convergence of Krylov iterative  methods.
-Our TSIRM proposal acts as a merger between Krylov based solvers and
-a least-squares minimization step.
-The convergence of the method has been proven in some situations, while 
-experiments up to 16,394 cores have been led to verify that TSIRM runs
-5 or  7 times  faster than GMRES.
+%%NEW
+In this paper a new two-stage algorithm TSIRM has been described. This method allows us to improve the convergence of  Krylov iterative  methods. It is based
+on a least-squares minimization step which uses the  Krylov residuals.
+
 
+We have implemented our code in PETSc in order to show that it is efficient and scalable. Some experiments with classical examples of PETSc for linear and nonlinear problems have been performed. We observed that TSIRM outperforms GMRES variants when the number of iterations is large. TSIRM is also scalable since we made some experiments with up to 16,394 cores.
 
-For  future  work, the  authors'  intention is  to  investigate  other kinds  of
-matrices, problems, and  inner solvers. In particular, the possibility 
-to obtain a convergence of TSIRM in situations where the GMRES is divergent will be
-investigated. The influence of  all parameters must be
-tested too, while other methods to minimize the residuals must be regarded.  The
-number of outer  iterations to minimize should become  adaptive to improve the
-overall performances of the proposal.   Finally, this solver will be implemented
-inside PETSc, which would be of interest as it would  allows us to test
-all the non-linear  examples and compare our algorithm  with the other algorithm
-implemented in PETSc.
+We also observed that TSIRM is efficient with different preconditioners. The hypre preconditioner that is globally very efficient for many problems is also very time consuming. Consequently, sometimes using a less performent preconditioners may be a better solution. In that case, TSIRM is also more efficient than traditional Krylov methods.
 
+{\bf A CHECKER !!}
+The influence of some important parameters of TSIRM have been studied. It can be noticed that they have a strong influence on the convergence speed
 
-% conference papers do not normally have an appendix
+In future works, we plan to study other problems coming from different research areas. Other efficient Krylov optimisation methods as communication avoiding technique may be interesting to be investigated
+%%ENDNEW
 
 
 
