\section{Experiments using PETSc}\r
\label{sec:05}\r
\r
+%%NEW\r
+In this section four kinds of experiments have been performed. First, some experiments on real matrices issued from the sparse matrix florida have been achieved out. Second, some experiments in parallel with some linear problems are reported and analyzed. Third, some experiments in parallèle with som nonlinear problems are illustrated. Finally some parameters of TSIRM are studied in order to understand their influences.\r
+\r
+\r
+\subsection{Real matrices}\r
+%%ENDNEW\r
+\r
\r
In order to see the behavior of our approach when considering only one processor,\r
a first comparison with GMRES or FGMRES and the new algorithm detailed\r
\end{table*}\r
\r
\r
-\r
-\r
+%%NEW\r
+\subsection{Parallel linear problems}\r
+%%ENDNEW\r
\r
In order to perform larger experiments, we have tested some example applications\r
of PETSc. These applications are available in the \emph{ksp} part, which is\r
are not available. In our experiments, we have tested different kinds of\r
preconditioners, but as it is not the subject of this paper, we will not\r
present results with many preconditioners. In practice, we have chosen to use a\r
-multigrid (mg) and successive over-relaxation (sor). For further details on the\r
+multigrid (MG) and successive over-relaxation (SOR). For further details on the\r
preconditioners in PETSc, readers are referred to~\cite{petsc-web-page}.\r
\r
\r
nb. cores & precond & \multicolumn{2}{c|}{FGMRES} & \multicolumn{2}{c|}{TSIRM CGLS} & \multicolumn{2}{c|}{TSIRM LSQR} & best gain \\ \r
\cline{3-8}\r
& & Time & \# Iter. & Time & \# Iter. & Time & \# Iter. & \\\hline \hline\r
- 2,048 & mg & 403.49 & 18,210 & 73.89 & 3,060 & 77.84 & 3,270 & 5.46 \\\r
- 2,048 & sor & 745.37 & 57,060 & 87.31 & 6,150 & 104.21 & 7,230 & 8.53 \\\r
- 4,096 & mg & 562.25 & 25,170 & 97.23 & 3,990 & 89.71 & 3,630 & 6.27 \\\r
- 4,096 & sor & 912.12 & 70,194 & 145.57 & 9,750 & 168.97 & 10,980 & 6.26 \\\r
- 8,192 & mg & 917.02 & 40,290 & 148.81 & 5,730 & 143.03 & 5,280 & 6.41 \\\r
- 8,192 & sor & 1,404.53 & 106,530 & 212.55 & 12,990 & 180.97 & 10,470 & 7.76 \\\r
- 16,384 & mg & 1,430.56 & 63,930 & 237.17 & 8,310 & 244.26 & 7,950 & 6.03 \\\r
- 16,384 & sor & 2,852.14 & 216,240 & 418.46 & 21,690 & 505.26 & 23,970 & 6.82 \\\r
+ 2,048 & MG & 403.49 & 18,210 & 73.89 & 3,060 & 77.84 & 3,270 & 5.46 \\\r
+ 2,048 & SOR & 745.37 & 57,060 & 87.31 & 6,150 & 104.21 & 7,230 & 8.53 \\\r
+ 4,096 & MG & 562.25 & 25,170 & 97.23 & 3,990 & 89.71 & 3,630 & 6.27 \\\r
+ 4,096 & SOR & 912.12 & 70,194 & 145.57 & 9,750 & 168.97 & 10,980 & 6.26 \\\r
+ 8,192 & MG & 917.02 & 40,290 & 148.81 & 5,730 & 143.03 & 5,280 & 6.41 \\\r
+ 8,192 & SOR & 1,404.53 & 106,530 & 212.55 & 12,990 & 180.97 & 10,470 & 7.76 \\\r
+ 16,384 & MG & 1,430.56 & 63,930 & 237.17 & 8,310 & 244.26 & 7,950 & 6.03 \\\r
+ 16,384 & SOR & 2,852.14 & 216,240 & 418.46 & 21,690 & 505.26 & 23,970 & 6.82 \\\r
\hline\r
\r
\end{tabular}\r
-\caption{Comparison of FGMRES and TSIRM with FGMRES for example ex15 of PETSc/KSP with two preconditioners (mg and sor) having 25,000 components per core on Juqueen ($\epsilon_{tsirm}=1e-3$, $max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$, $\epsilon_{ls}=1e-40$), time is expressed in seconds.}\r
+\caption{Comparison of FGMRES and TSIRM with FGMRES for example ex15 of PETSc/KSP with two preconditioners (MG and SOR) having 25,000 components per core on Juqueen ($\epsilon_{tsirm}=1e-3$, $max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$, $\epsilon_{ls}=1e-40$), time is expressed in seconds.}\r
\label{tab:03}\r
\end{center}\r
\end{table*}\r
Table~\ref{tab:03} shows the execution times and the number of iterations of\r
example ex15 of PETSc on the Juqueen architecture. Different numbers of cores\r
are studied ranging from 2,048 up-to 16,383 with the two preconditioners {\it\r
- mg} and {\it sor}. For those experiments, the number of components (or\r
+ MG} and {\it SOR}. For those experiments, the number of components (or\r
unknowns of the problems) per core is fixed at 25,000, also called weak\r
scaling. This number can seem relatively small. In fact, for some applications\r
that need a lot of memory, the number of components per processor requires\r
\end{figure}\r
\r
\r
-Concerning the experiments some other remarks are interesting.\r
-\begin{itemize}\r
-\item We have tested other examples of PETSc/KSP (ex29, ex45, ex49). For all these\r
- examples, we have also obtained similar gains between GMRES and TSIRM but\r
- those examples are not scalable with many cores. In general, we had some\r
- problems with more than $4,096$ cores.\r
-\item We have tested many iterative solvers available in PETSc. In fact, it is\r
- possible to use most of them with TSIRM. From our point of view, the condition\r
- to use a solver inside TSIRM is that the solver must have a restart\r
- feature. More precisely, the solver must support to be stopped and restarted\r
- without decreasing its convergence. That is why with GMRES we stop it when it\r
- is naturally restarted (\emph{i.e.} with $m$ the restart parameter). The\r
- Conjugate Gradient (CG) and all its variants do not have ``restarted'' version\r
- in PETSc, so they are not efficient. They will converge with TSIRM but not\r
- quickly because if we compare a normal CG with a CG which is stopped and\r
- restarted every 16 iterations (for example), the normal CG will be far more\r
- efficient. Some restarted CG or CG variant versions exist and may be\r
- interesting to study in future works.\r
-\end{itemize}\r
-%%%*********************************************************\r
-%%%*********************************************************\r
+\begin{figure}[htbp]\r
+\centering\r
+ \includegraphics[width=0.5\textwidth]{nb_iter_sec_ex45_curie}\r
+\caption{Number of iterations per second with ex45 and the same parameters as in Table~\ref{tab:06} (weak scaling)}\r
+\label{fig:03}\r
+\end{figure}\r
\r
\r
%%NEW\r
+\r
+{\bf example ex45/ksp à décrire et commenter en montrant que hypre est pourri avec cet exemple}\r
+\r
\begin{table*}[htbp]\r
\begin{center}\r
\begin{tabular}{|r|r|r|r|r|r|r|r|} \r
\hline\r
\r
\end{tabular}\r
-\caption{Comparison of FGMRES and TSIRM for ex45 of PETSc/KSP with two preconditioner (ASM and HYPRE) having 25,000 components per core on Curie ($\epsilon_{tsirm}=1e-10$, $max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$, $\epsilon_{ls}=1e-40$), time is expressed in seconds.}\r
+\caption{Comparison of FGMRES and TSIRM for ex45 of PETSc/KSP with two preconditioner (ASM and HYPRE) having 5,000 components per core on Curie ($\epsilon_{tsirm}=1e-10$, $max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$,$\epsilon_{ls}=1e-40$), time is expressed in seconds.}\r
\label{tab:06}\r
\end{center}\r
\end{table*}\r
\r
\r
-\begin{figure}[htbp]\r
-\centering\r
- \includegraphics[width=0.5\textwidth]{nb_iter_sec_ex45_curie}\r
-\caption{Number of iterations per second with ex45 and the same parameters as in Table~\ref{tab:06} (weak scaling)}\r
-\label{fig:03}\r
-\end{figure}\r
-\r
-\r
+\subsection{Parallel nonlinear problems}\r
+\r
+With PETSc, linear solvers are used inside nonlinear solvers. The SNES\r
+(Scalable Nonlinear Equations Solvers) module in PETSc implements easy to use\r
+methods, like Newton-type, quasi-Newton or full approximation scheme (FAS)\r
+multigrid to solve systems of nonlinears equations. As SNES is based on the\r
+Krylov methods of PETSc, it is interesting to investigate if the TSIRM method is\r
+also efficient and scalable with non linear problems. In PETSc, some examples\r
+are provided. An important criteria is the scalability of the initial code with\r
+classical solvers. Consequently, we have chosen two of these examples: ex14 and\r
+ex20. In ex14, the code solves the Bratu (SFI - solid fuel ignition) nonlinear\r
+partial difference equations in 3 dimension. In ex20, the code solves a 3\r
+dimension radiative transport test problem. For more details on these examples,\r
+interested readers are invited to see the code in the PETSc examples. For both\r
+these examples, a weak scaling case is chosen where processors have\r
+approximately a number of components equals to 100,000.\r
+\r
+In Table~\ref{tab:07} we report the result of our experiments for the example\r
+ex14 with the block Jacobi preconditioner. For TSIRM the CGLS algorithm is used\r
+to solve the minimization step. In this table, we can see that the number of\r
+iterations used by the linear solver is smaller with TSIRM compared with FGMRES.\r
+Consequently the execution times are smaller with TSIRM. The gain between TSIRM\r
+and FGMRES is around 6 and 7. The parameters of TSIRM are expressed in the\r
+caption of the table.\r
\r
\begin{table*}[htbp]\r
\begin{center}\r
\begin{tabular}{|r|r|r|r|r|r|} \r
\hline\r
\r
- nb. cores & \multicolumn{2}{c|}{FGMRES/BJAC} & \multicolumn{2}{c|}{TSIRM CGLS/BJAC} & gain \\ \r
+ nb. cores & \multicolumn{2}{c|}{FGMRES/BJAC} & \multicolumn{2}{c|}{TSIRM CGLS/BJAC} & gain \\ \r
\cline{2-5}\r
- & Time & \# Iter. & Time & \# Iter. & \\\hline \hline\r
- 1024 & 667.92 & 48,732 & 81.65 & 5,087 & 8.18 \\\r
- 2048 & 966.87 & 77,177 & 90.34 & 5,716 & 10.70\\\r
- 4096 & 1,742.31 & 124,411 & 119.21 & 6,905 & 14.61\\\r
- 8192 & 2,739.21 & 187,626 & 168.9 & 9,000 & 16.22\\\r
+ & Time & \# Iter. & Time & \# Iter. & \\\hline \hline\r
+ 1,024 & 159.52 & 11,584 & 26.34 & 1,563 & 6.06 \\\r
+ 2,048 & 226.24 & 16,459 & 37.23 & 2,248 & 6.08\\\r
+ 4,096 & 391.21 & 27,794 & 50.93 & 2,911 & 7.69\\\r
+ 8,192 & 543.23 & 37,770 & 79.21 & 4,324 & 6.86 \\\r
\r
\hline\r
\r
\end{tabular}\r
-\caption{Comparison of FGMRES and TSIRM for ex20 of PETSc/SNES with a Block Jacobi preconditioner having 100,000 components per core on Curie ($\epsilon_{tsirm}=1e-10$, $max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$, $\epsilon_{ls}=1e-40$), time is expressed in seconds.}\r
+\caption{Comparison of FGMRES and TSIRM for ex14 of PETSc/SNES with a Block Jacobi preconditioner having 100,000 components per core on Curie ($\epsilon_{tsirm}=1e-10$, $max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$, $\epsilon_{ls}=1e-40$), time is expressed in seconds.}\r
\label{tab:07}\r
\end{center}\r
\end{table*}\r
\r
+In Table~\cite{tab:08}, the results of the experiments with the example ex20 are\r
+reported. The block Jacobi preconditioner has also been used and CGLS to solve\r
+the minimization step for TSIRM. For this example, we can observ that the number\r
+of iterations for FMGRES increase drastically when the number of cores\r
+increases. With TSIRM, we can see that the number of iterations is initially\r
+very small compared to the FGMRES ones and when the number of cores increase,\r
+the number of iterations increases slighther with TSIRM than with FGMRES. For\r
+this example, the gain between TSIRM and FGMRES ranges between 8 with 1,024\r
+cores to more than 16 with 8,192 cores.\r
+\r
\begin{table*}[htbp]\r
\begin{center}\r
\begin{tabular}{|r|r|r|r|r|r|} \r
\hline\r
\r
- nb. cores & \multicolumn{2}{c|}{FGMRES/BJAC} & \multicolumn{2}{c|}{TSIRM CGLS/BJAC} & gain \\ \r
+ nb. cores & \multicolumn{2}{c|}{FGMRES/BJAC} & \multicolumn{2}{c|}{TSIRM CGLS/BJAC} & gain \\ \r
\cline{2-5}\r
- & Time & \# Iter. & Time & \# Iter. & \\\hline \hline\r
- 1024 & 159.52 & 11,584 & 26.34 & 1,563 & 6.06 \\\r
- 2048 & 226.24 & 16,459 & 37.23 & 2,248 & 6.08\\\r
- 4096 & 391.21 & 27,794 & 50.93 & 2,911 & 7.69\\\r
- 8192 & 543.23 & 37,770 & 79.21 & 4,324 & 6.86 \\\r
+ & Time & \# Iter. & Time & \# Iter. & \\\hline \hline\r
+ 1,024 & 667.92 & 48,732 & 81.65 & 5,087 & 8.18 \\\r
+ 2,048 & 966.87 & 77,177 & 90.34 & 5,716 & 10.70\\\r
+ 4,096 & 1,742.31 & 124,411 & 119.21 & 6,905 & 14.61\\\r
+ 8,192 & 2,739.21 & 187,626 & 168.9 & 9,000 & 16.22\\\r
\r
\hline\r
\r
\end{tabular}\r
-\caption{Comparison of FGMRES and TSIRM for ex14 of PETSc/SNES with a Block Jacobi preconditioner having 100,000 components per core on Curie ($\epsilon_{tsirm}=1e-10$, $max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$, $\epsilon_{ls}=1e-40$), time is expressed in seconds.}\r
+\caption{Comparison of FGMRES and TSIRM for ex20 of PETSc/SNES with a Block Jacobi preconditioner having 100,000 components per core on Curie ($\epsilon_{tsirm}=1e-10$, $max\_iter_{kryl}=30$, $s=12$, $max\_iter_{ls}=15$, $\epsilon_{ls}=1e-40$), time is expressed in seconds.}\r
\label{tab:08}\r
\end{center}\r
\end{table*}\r
\r
\r
+\r
+\r
+\r
+\r
+\r
+%%NEW\r
+\subsection{Influence of parameters for TSIRM}\r
+\r
+\begin{figure}[htbp]\r
+\centering\r
+ \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_cgls_iter_total}\r
+\caption{Number of total iterations using two different methods for the minimization: LSQR and CGLS.}\r
+\label{fig:cgls-iter} \r
+\end{figure}\r
+\r
+\begin{figure}[htbp]\r
+\centering\r
+ \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_cgls_time}\r
+\caption{Execution time in seconds using two different methods for the minimization: LSQR and CGLS.}\r
+\label{fig:cgls-time} \r
+\end{figure}\r
+\r
+\begin{figure}[htbp]\r
+\centering\r
+ \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_inner_restarts_iter_total}\r
+\caption{Number of total iterations with variation of restarts in the inner solver FGMRES.}\r
+\label{fig:inner_restarts_iter_total} \r
+\end{figure}\r
+\r
+\begin{figure}[htbp]\r
+\centering\r
+ \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_inner_restarts_time}\r
+\caption{Execution time in seconds with variation of restarts in the inner solver FGMRES.}\r
+\label{fig:inner_restarts_time} \r
+\end{figure}\r
+\r
+\begin{figure}[htbp]\r
+\centering\r
+ \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_max_inner_iter}\r
+\caption{Number of total iterations with variation of number of inner iterations.}\r
+\label{fig:max_inner_iter} \r
+\end{figure}\r
+\r
+\begin{figure}[htbp]\r
+\centering\r
+ \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_max_inner_time}\r
+\caption{Execution time in seconds with variation of number of inner iterations.}\r
+\label{fig:max_inner_time} \r
+\end{figure}\r
+\r
+\begin{figure}[htbp]\r
+\centering\r
+ \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_maxiter_ls_iter}\r
+\caption{Number of total iterations with variation of number of iterations in the minimization process.}\r
+\label{fig:maxiter_ls_iter} \r
+\end{figure}\r
+\r
+\begin{figure}[htbp]\r
+\centering\r
+ \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_maxiter_ls_time}\r
+\caption{Execution time in seconds with variation of number of iterations in the minimization process.}\r
+\label{fig:maxiter_ls_time} \r
+\end{figure}\r
+\r
+\begin{figure}[htbp]\r
+\centering\r
+ \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_size_ls_iter}\r
+\caption{Number of total iterations with variation of the size of the least-squares problem in the minimization process.}\r
+\label{fig:size_ls_iter} \r
+\end{figure}\r
+\r
+\begin{figure}[htbp]\r
+\centering\r
+ \includegraphics[angle=-90,width=0.5\textwidth]{ksp_tsirm_size_ls_time}\r
+\caption{Execution time in seconds with variation of the size of the least-squares problem in the minimization process.}\r
+\label{fig:size_ls_time} \r
+\end{figure}\r
+\r
%%ENDNEW\r
\r
+\r
+\r
+\r
+\r
+\subsection{Experiments conclusions }\r
+\r
+{\bf A refaire}\r
+\r
+Concerning the experiments some other remarks are interesting.\r
+\begin{itemize}\r
+\item We have tested other examples of PETSc/KSP (ex29, ex45, ex49). For all these\r
+ examples, we have also obtained similar gains between GMRES and TSIRM but\r
+ those examples are not scalable with many cores. In general, we had some\r
+ problems with more than $4,096$ cores.\r
+\item We have tested many iterative solvers available in PETSc. In fact, it is\r
+ possible to use most of them with TSIRM. From our point of view, the condition\r
+ to use a solver inside TSIRM is that the solver must have a restart\r
+ feature. More precisely, the solver must support to be stopped and restarted\r
+ without decreasing its convergence. That is why with GMRES we stop it when it\r
+ is naturally restarted (\emph{i.e.} with $m$ the restart parameter). The\r
+ Conjugate Gradient (CG) and all its variants do not have ``restarted'' version\r
+ in PETSc, so they are not efficient. They will converge with TSIRM but not\r
+ quickly because if we compare a normal CG with a CG which is stopped and\r
+ restarted every 16 iterations (for example), the normal CG will be far more\r
+ efficient. Some restarted CG or CG variant versions exist and may be\r
+ interesting to study in future works.\r
+\end{itemize}\r
+\r
+%%ENDNEW\r
+\r
+\r
%%%*********************************************************\r
%%%*********************************************************\r
\section{Conclusion}\r
