X-Git-Url: https://bilbo.iut-bm.univ-fcomte.fr/and/gitweb/mpi-energy2.git/blobdiff_plain/9332f8adfd1259270ef033e480ca4613b4f52257..e0b09048908ae554159451863da90b3b877816f9:/mpi-energy2-extension/review/review.tex diff --git a/mpi-energy2-extension/review/review.tex b/mpi-energy2-extension/review/review.tex index 7302d62..d42e804 100644 --- a/mpi-energy2-extension/review/review.tex +++ b/mpi-energy2-extension/review/review.tex @@ -1,11 +1,11 @@ -\documentclass[12pt,a4paper]{report} +\documentclass[12pt,a4paper]{journal} \usepackage[utf8]{inputenc} \usepackage{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{graphicx} \usepackage{color} -\title{Reviewers' comments} +%\title{Reviewers' comments} \newcommand{\AG}[2][inline]{% \todo[color=green!50,#1]{\sffamily\textbf{AG:} #2}\xspace} @@ -52,34 +52,10 @@ \begin{document} +\title{Answers to the questions of the reviewers} +\maketitle +\section{Questions and remarks of the first reviewer} - -\section{Reviewer 1} -This work tries to reduce energy consumption of regular applications, -with no dynamic load balancing, that execute in heterogeneous -platforms with different machine configurations. A energy and -execution time model are given for such configuration that allows the -reader to easily understand the context. The proposed energy reduction -strategy slows down (using DVFS) the faster processes, reducing their -slack time. The technique could be seen as "load balancing" through -de-acceleration. Of course the objective here is to reduce energy -consumption, and the proposed technique is indeed interesting. I -suggest you to read the following paper that has a similar strategy -(slack time is called residual imbalances in this paper) applied in a -slightly different scenario: - -+ Padoin et. al. Saving energy by exploiting residual imbalances on - iterative applications. 21st International Conference on High - Performance Computing (HiPC), HIPC 2014. - -It is very hard to optimize energy consumption and performance. One -affects the other, very few workarounds for that. I found the -discussion in Sec. 4 very interesting since it details a possible -workaround by exploring the fact that the application is distributed -in the platform and we know that the overall execution time is -dominated by the critical path. - -Remarks \begin{enumerate} @@ -92,15 +68,27 @@ measuring the computation time and energy consumption for one iteration only. Let's suppose something went bad in this first iteration. The scaling factors will not be the best tradeoff because variability has been ignored. What would be the solution for that? -Consider variability in the model. Another point is that you mention +Consider variability in the model. + +\textbf{Answer:} In this paper we have considered that the application executes regular iterations over stable computers computing only this application. Therefore, we have assumed that the execution times of all the iterations of the application executed on the same computing node should be almost the same. For this reason we did not take into consideration the variability of the computer system. Moreover, applying the frequency scaling algorithm after many iterations would reduce its impact on the energy consumption especially for applications executing a relatively low number of iterations. + +However, the variability of the computing system can be taken into consideration in a future work. For example, the proposed algorithm can be executed twice: after the first iteration the frequencies are scaled down according to the execution times measured in the first iteration, then after a fixed number of iterations, the frequencies are adjusted according to the execution times measured during the fixed number of iterations. If the computing power of the system is constantly changing, it would be interesting to implement a mechanism that detects this change and adjusts the frequencies according to the variability of the system. + + Taking account of the variability of the system has been added as a perspective at the end of the paper. + + +\item Another point is that you mention in the abstract and introduction that your solution has low overhead, but it is a centralized solution. Probably it won't scale when we reach hundreds or thousands of computer nodes: take one of that large machines for example. In this paper experiments, only 16 and 32 nodes where considered. -\textcolor{blue}{Answer: We plan to take the variability in the proposed algorithm as a future works in two steps. In the first step, the algorithm selects the best frequencies at the end of the first iterations and apply them to the system. In the second step, after some iterations (e.g. 5 iterations) the algorithm recomputes the frequencies depending on the average of the communication and computation times for all previous iterations. It will change the frequency of each node if the new frequency is different from the old one. Otherwise, it keeps the old frequency. We have added this to our perspectives at the paper. - The algorithm overhead is very small, for example in the simulation results [6], it takes 0.15 ms on average for 144 nodes to compute the best scaling factors vector for a heterogeneous cluster. On Grid'5000 it is very hard to book a lot of nodes that allow DVFS operations and have an energy measurement tools. } +\textbf{Answer:} +We agree with the reviewer that the algorithm is centralized and might be a bottleneck if it was applied to an application running on many thousands of nodes. However, up to 144 nodes in a heterogeneous cluster, the overhead of the algorithm was very small, 0.15 ms, as presented in the simulation results of [6]. We did not execute experiments with more than 32 nodes on Grid'5000 because it does not have many nodes that allow DVFS operations and have energy measurement tools. + +On the other hand, the scalability of the proposed algorithm can be improved if we use asynchronous computations or if the algorithm was distributed in a hierarchical manner where a leader is chosen for each cluster or a group of nodes to compute their scaled frequencies. Improving the scalability of the algorithm is beyond the scope of this paper. + \item In Fig 6, you draw lines between the points. Lines here mean nothing @@ -109,7 +97,7 @@ instance a non-stacked bar plot with four colors (one site/16, one site/32, two sites/16, two sites/32). I believe it would be much easier to compare and avoid the problem of lines. - \textcolor{blue}{Answer: we agree with reviewer. We have changed figures 6 and 8 in the paper.} + \textbf{Answer:} We agree with the reviewer. The curves in Figures 6 and 8 in the paper were replaced by histograms. @@ -124,15 +112,17 @@ instance). You say that on scale this would produce less energy savings, but your arguments for providing a solution for this was based that today's supercomputers are achieving massive scale. -\textcolor{blue}{Answer: In grid, the cost of communications between distinct sites is the main factor. The NAS benchmarks are significantly affected by the number of nodes and the increase in the communications between them. So, the instance is too small to be executed over 32 nodes and the computation to communication ratio is very small. Therefore, bigger instances should be executed on -much number of nodes. } +\textbf{Answer:} In Figure 7, the energy consumption of the benchmarks solving the class D and running on many scenarios are presented. The number of used nodes varies between 16 and 32 in the scenarios while the size of the problem is not modified. Therefore, the computations to communications times ratio is lower when 32 nodes are used instead of 16. When this ratio is small, it means there are not enough computations when compared to the communications times and the impact of scaling down the frequency of the CPU on its energy consumption is reduced. To solve this problem, the problem +should be solved on a number of nodes adequate to its size. For example, for the NAS benchmarks, the class E should have been solved on 32 nodes to have a good computations to communications times ratio. + \item In Sec 6.3, why did you choose to keep 32 processes for the evaluation with multi-core clusters? How did you configure MPI for the results -\textcolor{blue}{Answer: We keep choosing 32 nodes in both scenarios -to compare them while one core per node scenario has distributed communications (one network link for each node) and multi-core scenario uses shared network link communications and thus comparing their impact on the results. -We configure MPI on one core per node scenario by choosing one core per nodes (e.g in machine file we did: node1, node2 ,node3, node4). While in multi-core scenario we choose one machine with four cores (e.g. node1 slots=4).} +\textbf{Answer:} In section 6.3, we wanted to evaluate how much energy can be saved when applying the proposed algorithm to message passing applications with iterations running over a grid composed of multi-core nodes. Therefore, the same experiments as in section 6.2 were conducted on the new multi-core platform. Instead of running one process per node as in the previous section, 3 or 4 processes were executed on each multi-core node. The total number of processes, 32 processes, was not modified in order to fairly compare the single core and the multi-core versions. + +Only the architecture file was modified between the single and the multi-core architectures. For the single core architecture, the architecture file contains the name of 32 different nodes. For the multi-core architecture, the architecture file contains less nodes and for every node 3 or 4 slots (cores) are used. The total number of slots is equal to 32. + \item shown in Fig 8a? Some MPI implementations have an option to use shared @@ -141,8 +131,8 @@ the explanation of the network card utilization, but this shared-memory optimization is possible (sometimes automatically detected by MPI if you pin processes to cores). -\textcolor{blue}{Answer: We didn't manually pin processes to cores and since the communication times -increased. We guess that the shared memory wasn't used.} +\textbf{Answer:} We did not manually pin processes to cores. Since the communication times +increased, we think that the shared memory was not used when two processes, running on the same node, exchange data. \item In P33, Sec 6.5, you mention that the proposed algorithm outperforms EDP because the former considers both metrics (time, energy) and the @@ -150,8 +140,7 @@ same time. EDP does also, but using a single metric which you have defined: energy x execution time. I think this is only a matter of phrasing. -\textcolor{blue}{Answer: we use the delay in execution time not the execution time. Then, the equation that we used is EDP= energy x (Tnew-Told). The experiments shows that our objective function -is better than the EDP objective.} + \textbf{Answer:} We agree with the reviewer, EDP also uses two metrics in the objective function: energy and delay. The sentence in the paper was modified to clarify this misunderstanding. The main difference between our algorithm and the EDP method is the used objective function. For EDP, the product of energy and delay must be minimized, while for our algorithm, the difference between the normalized performance and the normalized energy should be maximized. This new formulation of the objective function allows our algorithm to select the set of frequencies that gives the best tradeoff between the energy consumption and the performance. The objective function of EDP does not give the same frequencies as our algorithm and thus it is outperformed by our method. The results of the experiments confirm that the objective function used by our algorithm is more efficient than the one used by EDP. \item Other complementary points to consider: @@ -170,7 +159,7 @@ is better than the EDP objective.} + Same for Fig 7. -\item \textcolor{blue}{ Answer: We have considered these points in the paper.} + \textbf{Answer:} Answer: We have taken in consideration all these remarks and the paper was modified accordingly. \item From the design of experiments, did you consider using replications? There is no variability metric in your results. Have you run multiple @@ -178,65 +167,58 @@ times and got the average (execution time and energy consumption)? I feel that such variability needs to be accounted for, otherwise it is very hard to affirm anything about measurements. - \textcolor{blue}{Answer: Each experiment has been executed many times and the results presented in the - figures are the average values of many executions.} + \textbf{Answer:} Each experiment has been executed many times and the results presented in the + figures are the average values of many executions. Since we have deployed the same operating system on the booked machines and we were the only users executing processes on them during the experiments, no significant variability in the execution time of the applications was noticed. \item In summary, I think this is a very interesting work but the experimental evaluation lacks variability measurements, consider larger experiments (1K nodes for instance) to see how everything scales, and there is no overhead measurements although authors stress that in abstract/introduction. -\textcolor{blue}{Answer: We will expand the experimental over a large number of nodes in the future work while increasing the problem size and considering the variability issues. We have discussed the algorithm overhead and its complexity in section 6.5.} -\end{enumerate} +\textbf{Answer:} For the time being, we do not have the resources nor the time to evaluate the proposed algorithm over large platforms composed of more than 1K nodes. However, as said in the perspectives of the paper, the evaluation of the scalability of the algorithm will be in a conducted in a future work as soon as we have access to larger resources. We have discussed the overhead of the algorithm and its complexity in section 6.5 and given in the answer to question 2 some solutions to improve its scalability and reduce its overhead. +For the variability issue, please refer to the answer to question 1. -\section{Reviewer 2} -This paper presents detailed performance and energy model for iterative message passing applications. Further a method is proposed to select the frequencies of heterogeneous cpus online. The selection method itself is not difficult. But I like the systematic modeling for energy consumption and performance. This paper is well written in general. The technical contents are presented in a logical way overall. The experiments are conducted in real platform, which shows the practicality of this work and also makes the work have more impact on the field. However, I have the following comments and concerns for this paper. The authors should clarify them in the revised version. + +\end{enumerate} --move the contributions from related work to introduction +\section{Questions and remarks of the second reviewer} \begin{enumerate} - +\item Move the contributions from related work to introduction \item why emphasize it is a grid platform? the presentation of related work follows the logic of heterogeneous CPUs. Grid is only a type of platform with heterogeneous CPUs -\textcolor{blue}{Answer: The proposed algorithm was adapted to the grid platform which is composed -of homogeneous clusters and interconnected by a wide area network which is slower than the local -network in each cluster. The algorithm can also work on other heterogeneous platforms.} +\textbf{Answer:} We agree with the reviewer that a grid is a type of heterogeneous architecture and the proposed algorithm can also work on any heterogeneous architecture. + In \cite{4}, we have proposed a frequency selection algorithm for distributed applications running on heterogeneous clusters, while in this work, the proposed algorithm was adapted to the grid architecture which is composed +of homogeneous clusters interconnected by a wide area network which is slower than the local +network in each cluster. \item Define what iterative message passing applications are and give exemplar applications of them targeted by this method. -\textcolor{blue}{Answer: The message passing applications with iterations compute -the same block of operations several times, starting from the initial solution until reaching -the acceptable approximation of the exact solution (we have added this sentence to the paper page 21). -There are many example for these applications -such as JACOBI, GAUSS-SEIDE, Successive over-relaxation and conjugate gradient (CG) method and etc. -Similarly, an offline method uses DVFS and applied to these applications is in \cite{1} } +\textbf{Answer:} In order to clarify things, we have replaced in the paper the sentence ``the iterative message passing applications'' with ``the message passing applications with iterations''. Therefore, the proposed algorithm can be applied to any application that executes the same block of instructions many times and it is not limited to iterative methods that repeat the same block of instructions until convergence. + +Many problems are solved by methods with iterations such as solving a linear system of equations with Jacobi and Gauss-Seidel methods, Image processing with methods that apply the same instructions for each pixel or block of pixels, etc. \item Figure 1 is not clearly explained. Where is the slack time in figure 1 and why slack time =0 for task 1? -\textcolor{blue}{Answer: We agree with the reviewer, this figure was re-plotted to show the slack time. - In the figure, we assumed that task 1 is the slower task. So, there are no slack time (waiting time) - in the slower task because it is not wait for the others while other tasks wait for it.} + +\textbf{Answer:} Figure 1 was redrawn, the white space before the barrier is the slack time. Slack times occur when a node has to wait for another node to finish its computation to synchronously communicate with it. In Figure 1, task 1 was assumed to be the slowest task. All the other tasks will finish their computations before the slowest task and wait until it finishes its computation before being able to synchronously communicate with it. This waiting time is the slack time and since the slowest task do not have to wait for the other tasks it has almost no slack time. \item define the parameters in eq. 1. -\textcolor{blue}{Answer: We have defined Fmax and Fnew in the text.} +\textbf{Answer:} Fmax and Fnew have been defined as follows in the paper: ``$\Fmax$ is the maximum frequency before applying any DVFS and $\Fnew$ is the new frequency after applying DVFS''. \item eq. 2: are you assuming each cluster has the same number of nodes? - - - -\textcolor{blue}{Answer: No, we assume each cluster has different number of node, so in the equation we -have replaced $M_i$ instead of $M$ and the same for $F$ replaced with $F_j$ in the all equations of the paper.} +\textbf{Answer:} No, each cluster can have a different number of nodes. Therefore, in the paper, $M$, the number of nodes in a cluster, was replaced by $M_i$, the number of nodes in cluster $i$, in all the equations. \item Eq.2 implicitly assumes that there is no overlapping between computation and communication. Is it reasonable? -\textcolor{blue}{Answer: The communications between the computing nodes are synchronized, where each node -need to wait for the others to finished their jobs before computing the next iteration. So, there is no -overlapping between computations and communications for a node. The overlapping happens when the communications are asynchronous or the computations are not depend on the data sent by the neighbouring nodes. } +\textbf{Answer:} In this paper, only message passing applications with synchronous and blocking communications were considered. In such applications, there is no overlapping between computations and communications on a node. + +Asynchronous applications are beyond the scope of this paper and will be considered in a future work. \item eq. 2 is not clear: @@ -301,14 +283,7 @@ and with or without message passing. Indeed, the proposed method can be applied The proposed method can be applied to a heterogeneous platform composed from GPUs and CPUs, since modern GPUs like CPUs allow the use of DVFS operation.} \end{enumerate} -\section{Reviewer 3} -In this paper, a new online frequency selecting algorithm for grids, composed of heterogeneous clusters, is presented. It selects the frequencies and tries to give the best trade-off between -energy saving and performance degradation, for each node computing the message passing iterative application. The algorithm has a small overhead and works without training or profiling. It uses a new energy model for message -passing iterative applications running on a grid. The proposed algorithm is evaluated on a real grid, the Grid'5000 platform, while running the NAS parallel benchmarks. The experiments on 16 nodes, distributed on three clusters, show -that it reduces on average the energy consumption by 30\% while the performance is on average only degraded by 3.2\%. Finally, the algorithm is compared to an existing method. The comparison results show that it outperforms the -latter in terms of energy consumption reduction and performance. - -this paper is quite interesting and solid. But before acceptance, I suggest to have the following major revisions: +\section{Questions and remarks of the third reviewer} \begin{enumerate} \item suggest the authors to use much larger size of nodes, instead of on 16 nodes, distributed on three clusters, to see the scalability of the energy saving