section IV corrected

[mpi-energy2.git] / Heter_paper.tex

diff --git a/Heter_paper.tex b/Heter_paper.tex
index c5f885adc134a9dec61972503f2d68b4945ace20..8c329a8adc36e134fdd364eee4cdfed79f04fd4d 100644 (file)
--- a/Heter_paper.tex
+++ b/Heter_paper.tex
@@ -104,8 +104,8 @@ passing distributed iterative synchronous applications running over
 heterogeneous platforms. We define a heterogeneous platform as a collection of
 heterogeneous computing nodes interconnected via a high speed homogeneous
 network. Therefore, each node has different characteristics such as computing
 heterogeneous platforms. We define a heterogeneous platform as a collection of
 heterogeneous computing nodes interconnected via a high speed homogeneous
 network. Therefore, each node has different characteristics such as computing

-power (FLOPS), energy consumption, CPU's frequency range, ... but they all have
-the same network bandwidth and latency.
+power (FLOPS), energy consumption, CPU's frequency range, \dots{} but they all
+have the same network bandwidth and latency.

 
 
 \begin{figure}[t]
 
 
 \begin{figure}[t]


@@ -129,16 +129,21 @@ of a CPU by scaling down its voltage and frequency.  Since DVFS lowers the frequ
  The execution time of a compute bound sequential program is linearly proportional to the frequency scaling factor $S$. 
  On the other hand,  message passing distributed applications consist of two parts: computation and communication. The execution time of the computation part is linearly proportional to the frequency scaling factor $S$ but  the communication time is not affected by the scaling factor because  the processors involved remain idle during the  communications~\cite{17}. The communication time for a task is the summation of periods of time that begin with an MPI call for sending or receiving   a message till the message is synchronously sent or received.
 
  The execution time of a compute bound sequential program is linearly proportional to the frequency scaling factor $S$. 
  On the other hand,  message passing distributed applications consist of two parts: computation and communication. The execution time of the computation part is linearly proportional to the frequency scaling factor $S$ but  the communication time is not affected by the scaling factor because  the processors involved remain idle during the  communications~\cite{17}. The communication time for a task is the summation of periods of time that begin with an MPI call for sending or receiving   a message till the message is synchronously sent or received.
 

-Since in a heterogeneous platform, each node has different characteristics, especially different frequency gears, when applying DVFS operations on these nodes, they may get different scaling factors represented by a scaling vector: $(S_1, S_2,..., S_N)$ where $S_i$ is the scaling factor of processor $i$. To be able to predict the execution time of message passing synchronous iterative applications running over a heterogeneous platform, for different vectors of scaling factors, the  communication time and the computation time  for all the 
- tasks must be measured during the first iteration before applying any DVFS operation. Then the execution time for one iteration of the application with any vector of scaling factors  can be  predicted using EQ (\ref{eq:perf}).
- 
-  
-  
-\begin{multline}
+Since in a heterogeneous platform, each node has different characteristics,
+especially different frequency gears, when applying DVFS operations on these
+nodes, they may get different scaling factors represented by a scaling vector:
+$(S_1, S_2,\dots, S_N)$ where $S_i$ is the scaling factor of processor $i$. To
+be able to predict the execution time of message passing synchronous iterative
+applications running over a heterogeneous platform, for different vectors of
+scaling factors, the communication time and the computation time for all the
+tasks must be measured during the first iteration before applying any DVFS
+operation. Then the execution time for one iteration of the application with any
+vector of scaling factors can be predicted using EQ (\ref{eq:perf}).
+\begin{equation}

   \label{eq:perf}
  \textit  T_\textit{new} = 
   \label{eq:perf}
  \textit  T_\textit{new} = 

- {} \max_{i=1,2,\dots,N} (TcpOld_{i} \cdot S_{i}) +  TcmOld_{j}
-\end{multline}
+ \max_{i=1,2,\dots,N} (TcpOld_{i} \cdot S_{i}) +  TcmOld_{j}
+\end{equation}

 where $TcpOld_i$ is the computation time  of processor $i$ during the first iteration and $TcmOld_j$ is the communication time of the slowest processor $j$.
   The model computes the maximum computation time 
  with scaling factor from each node  added to the communication time of the slowest node, it means  only the
 where $TcpOld_i$ is the computation time  of processor $i$ during the first iteration and $TcmOld_j$ is the communication time of the slowest processor $j$.
   The model computes the maximum computation time 
  with scaling factor from each node  added to the communication time of the slowest node, it means  only the


@@ -188,14 +193,14 @@ system's kernel to lower a core's frequency. we can calculate the new frequency
 $F_{new}$ from EQ(\ref{eq:s}) as follow:
 \begin{equation}
   \label{eq:fnew}
 $F_{new}$ from EQ(\ref{eq:s}) as follow:
 \begin{equation}
   \label{eq:fnew}

-   F_\textit{new} = S^{-1} .  F_\textit{max}
+   F_\textit{new} = S^{-1} \cdot F_\textit{max}

 \end{equation}
 Replacing $F_{new}$ in EQ(\ref{eq:pd}) as in EQ(\ref{eq:fnew}) gives the following equation for dynamic 
 power consumption:
 \begin{multline}
   \label{eq:pdnew}
    {P}_\textit{dNew} = \alpha \cdot C_L \cdot V^2 \cdot F_{new} = \alpha \cdot C_L \cdot \beta^2 \cdot F_{new}^3 \\
 \end{equation}
 Replacing $F_{new}$ in EQ(\ref{eq:pd}) as in EQ(\ref{eq:fnew}) gives the following equation for dynamic 
 power consumption:
 \begin{multline}
   \label{eq:pdnew}
    {P}_\textit{dNew} = \alpha \cdot C_L \cdot V^2 \cdot F_{new} = \alpha \cdot C_L \cdot \beta^2 \cdot F_{new}^3 \\

-   = \alpha \cdot C_L \cdot V^2 \cdot F \cdot S^{-3} = P_{dOld} \cdot S^{-3}
+   {} = \alpha \cdot C_L \cdot V^2 \cdot F \cdot S^{-3} = P_{dOld} \cdot S^{-3}

 \end{multline}
 where $ {P}_\textit{dNew}$  and $P_{dOld}$ are the  dynamic power consumed with the new frequency and the maximum frequency respectively.
 
 \end{multline}
 where $ {P}_\textit{dNew}$  and $P_{dOld}$ are the  dynamic power consumed with the new frequency and the maximum frequency respectively.
 


@@ -210,7 +215,6 @@ The static energy is the static power multiplied by the execution time of the pr
 the execution time of the program is the summation of the computation and the communication times. The computation time is linearly related  
 to the frequency scaling factor, while this scaling factor does not affect the communication time. The static energy 
 of a processor after scaling its frequency is computed as follows: 
 the execution time of the program is the summation of the computation and the communication times. The computation time is linearly related  
 to the frequency scaling factor, while this scaling factor does not affect the communication time. The static energy 
 of a processor after scaling its frequency is computed as follows: 

-

 \begin{equation}
   \label{eq:Estatic}
  E_\textit{s} = P_\textit{s} \cdot (T_{cp} \cdot S  + T_{cm})
 \begin{equation}
   \label{eq:Estatic}
  E_\textit{s} = P_\textit{s} \cdot (T_{cp} \cdot S  + T_{cm})


@@ -219,48 +223,69 @@ of a processor after scaling its frequency is computed as follows:
 In the considered heterogeneous platform, each processor $i$ might have different dynamic and static powers, noted as $P_{di}$ and $P_{si}$ respectively. Therefore, even if the distributed message passing iterative application is load balanced, the computation time of each CPU $i$ noted $T_{cpi}$ might be different and different frequency  scaling factors might be computed in order to decrease the overall energy consumption of the application and reduce the slack times. The communication time of a processor $i$ is noted as $T_{cmi}$ and could contain slack times if it is communicating with slower nodes, see figure(\ref{fig:heter}). Therefore, all nodes do not have equal communication times. While the dynamic energy is computed according to the  frequency scaling factor and the dynamic power of each node as in EQ(\ref{eq:Edyn}),  the static energy is computed as the sum of the execution time of each processor multiplied by its static power. The overall energy consumption of a message passing  distributed application  executed over a heterogeneous platform is the summation of all dynamic and static energies for each  processor.  It is computed as follows:
 \begin{multline}
   \label{eq:energy}
 In the considered heterogeneous platform, each processor $i$ might have different dynamic and static powers, noted as $P_{di}$ and $P_{si}$ respectively. Therefore, even if the distributed message passing iterative application is load balanced, the computation time of each CPU $i$ noted $T_{cpi}$ might be different and different frequency  scaling factors might be computed in order to decrease the overall energy consumption of the application and reduce the slack times. The communication time of a processor $i$ is noted as $T_{cmi}$ and could contain slack times if it is communicating with slower nodes, see figure(\ref{fig:heter}). Therefore, all nodes do not have equal communication times. While the dynamic energy is computed according to the  frequency scaling factor and the dynamic power of each node as in EQ(\ref{eq:Edyn}),  the static energy is computed as the sum of the execution time of each processor multiplied by its static power. The overall energy consumption of a message passing  distributed application  executed over a heterogeneous platform is the summation of all dynamic and static energies for each  processor.  It is computed as follows:
 \begin{multline}
   \label{eq:energy}

- E = \sum_{i=1}^{N} {(S_i^{-2} \cdot P_{di} \cdot  T_{cpi})} +\\ 
- {}\sum_{i=1}^{N} {(P_{si} \cdot (\max_{i=1,2,\dots,N} (T_{cpi} \cdot S_{i}) +} 
- {}\min_{i=1,2,\dots,N} {T_{cmi}))} 
+ E = \sum_{i=1}^{N} {(S_i^{-2} \cdot P_{di} \cdot  T_{cpi})} + {} \\
+ \sum_{i=1}^{N} (P_{si} \cdot (\max_{i=1,2,\dots,N} (T_{cpi} \cdot S_{i}) +
+ \min_{i=1,2,\dots,N} {T_{cmi}))}

  \end{multline}
  
  \end{multline}
  

-Reducing the the frequencies of the processors according to the vector of scaling factors $(S_1, S_2,..., S_N)$  may degrade the  performance of the application and thus,
-increase the static energy because the execution time is increased~\cite{36}.
+Reducing the the frequencies of the processors according to the vector of
+scaling factors $(S_1, S_2,\dots, S_N)$ may degrade the performance of the
+application and thus, increase the static energy because the execution time is
+increased~\cite{36}.

 
 \section{Optimization of both energy consumption and performance}
 \label{sec.compet}
 
 \section{Optimization of both energy consumption and performance}
 \label{sec.compet}

-Applying DVFS to lower level  not surly reducing the energy consumption to minimum level. Also, a big scaling for the frequency produces high performance degradation percent. Moreover, by considering the  drastically  increase in execution time of parallel program, the static energy is related to this time 
-and it also increased by the same ratio. Thus, the opportunity for gaining more energy reduction is restricted. For that choosing frequency scaling factors is very important process to taking into account both energy and performance. In our previous work~\cite{45}, we are proposed a method that selects the optimal frequency scaling factor for an homogeneous cluster, depending on the trade-off relation between the energy and performance. In this work we have an  heterogeneous cluster, at each node there is different scaling factors, so our goal is to selects the optimal set of frequency scaling factors, $Sopt_1,Sopt_2,...,Sopt_N$, that gives the best trade-off  between energy consumption and performance. The relation between the energy and the execution time is complex and nonlinear, Thus, unlike the relation between the performance and  the scaling factor,  the relation of the energy with the frequency scaling factors is nonlinear, for more details refer to~\cite{17}.  Moreover, they are not measured using the same metric.  To solve this problem, we normalize the execution time by calculating the ratio between the new execution time (the scaled execution time) and the old one as follow:
+
+Using the lowest frequency for each processor does not necessarily gives the most energy efficient execution of an application. Indeed, even though the dynamic power is reduced while scaling down the frequency of a processor, its computation power is proportionally decreased and thus the execution time might be drastically increased during which dynamic and static powers are being consumed. Therefore,  it might cancel any gains achieved by scaling down the frequency of all nodes to the minimum  and the overall energy consumption of the application might not be the optimal one. It is not trivial to select the appropriate frequency scaling factor for each processor while considering the characteristics of each processor (computation power, range of frequencies, dynamic and static powers) and the task executed (computation/communication ratio) in order to reduce the overall energy consumption and not significantly increase the execution time. In
+our previous work~\cite{45}, we  proposed a method that selects the optimal
+frequency scaling factor for a homogeneous cluster executing a message passing iterative synchronous application while giving the best trade-off
+ between the energy consumption and the performance for such applications. In this work we are interested in 
+heterogeneous clusters as described above. Due to the heterogeneity of the processors, not one but a  set of scaling factors should be selected and it must  give the best trade-off between energy
+consumption and performance. 
+
+The relation between the energy consumption and the execution
+time for an application is complex and nonlinear, Thus, unlike the relation between the performance
+and the scaling factor, the relation of the energy with the frequency scaling
+factors is nonlinear, for more details refer to~\cite{17}.  Moreover, they are
+not measured using the same metric.  To solve this problem, we normalize the
+execution time by computing the ratio between the new execution time (after scaling down the frequencies of some processors) and the initial one (with maximum frequency for all nodes,) as follows:

 \begin{multline}
   \label{eq:pnorm}
   P_\textit{Norm} = \frac{T_\textit{New}}{T_\textit{Old}}\\
 \begin{multline}
   \label{eq:pnorm}
   P_\textit{Norm} = \frac{T_\textit{New}}{T_\textit{Old}}\\

-          = \frac{ \max_{i=1,2,\dots,N} (Tcp_{i} \cdot S_{i}) +\min_{i=1,2,\dots,N} {Tcm_{i}}} 
+       {} = \frac{ \max_{i=1,2,\dots,N} (Tcp_{i} \cdot S_{i}) +\min_{i=1,2,\dots,N} {Tcm_{i}}}

            {\max_{i=1,2,\dots,N}{(Tcp_i+Tcm_i)}}
 \end{multline}
 
 
            {\max_{i=1,2,\dots,N}{(Tcp_i+Tcm_i)}}
 \end{multline}
 
 

-By the same way, we are normalize the energy by calculating the ratio between the consumed energy with scaled frequency and the consumed energy without scaled frequency:
+In the same way, we normalize the energy by computing the ratio between the consumed energy while scaling down the frequency and the consumed energy with maximum frequency for all nodes:

 \begin{multline}
   \label{eq:enorm}
   E_\textit{Norm} = \frac{E_\textit{Reduced}}{E_\textit{Original}} \\
 \begin{multline}
   \label{eq:enorm}
   E_\textit{Norm} = \frac{E_\textit{Reduced}}{E_\textit{Original}} \\

-   = \frac{ \sum_{i=1}^{N}{(S_i^{-2} \cdot Pd_i \cdot  Tcp_i)} +
+  {} = \frac{ \sum_{i=1}^{N}{(S_i^{-2} \cdot Pd_i \cdot  Tcp_i)} +

  \sum_{i=1}^{N} {(Ps_i \cdot T_{New})}}{\sum_{i=1}^{N}{( Pd_i \cdot  Tcp_i)} +
  \sum_{i=1}^{N} {(Ps_i \cdot T_{Old})}}
 \end{multline} 
  \sum_{i=1}^{N} {(Ps_i \cdot T_{New})}}{\sum_{i=1}^{N}{( Pd_i \cdot  Tcp_i)} +
  \sum_{i=1}^{N} {(Ps_i \cdot T_{Old})}}
 \end{multline} 

-Where $T_{New}$ and $T_{Old}$ is computed as in EQ(\ref{eq:pnorm}). The second problem 
-is that the optimization operation for both energy and performance is not in the same direction.  
-In other words, the normalized energy and the normalized execution time curves are not at the same direction.  
-While the main goal is to optimize the energy and execution time in the same time.  According to the 
-equations~(\ref{eq:enorm}) and~(\ref{eq:pnorm}), the set of frequency scaling factors $S_1,S_2,...,S_N$ reduce both the energy and the
-execution time simultaneously.  But the main objective is to produce maximum energy
-reduction with minimum execution time reduction.  Many researchers used different
-strategies to solve this nonlinear problem for example see~\cite{19,42}, their
-methods add big overheads to the algorithm to select the suitable frequency.
-In this paper we are present a method to find the optimal set of frequency scaling factors to optimize both energy and execution time simultaneously
-without adding a big overhead.  Our solution for this problem is to make the optimization process
-for energy and execution time follow the same direction.  Therefore, we inverse the equation of the normalized
-execution time, the normalized performance,  as follows:
-
+Where $T_{New}$ and $T_{Old}$ are computed as in EQ(\ref{eq:pnorm}).
+
+ While the main
+goal is to optimize the energy and execution time at the same time, the normalized energy and execution time curves are not in the same direction.     According
+to the equations~(\ref{eq:enorm}) and~(\ref{eq:pnorm}), the set of frequency
+scaling factors $S_1,S_2,\dots,S_N$ reduce both the energy and the execution
+time simultaneously.  But the main objective is to produce maximum energy
+reduction with minimum execution time reduction.  
+
+Many researchers used
+different strategies to solve this nonlinear problem for example
+in~\cite{19,42}, their methods add big overheads to the algorithm to select the
+suitable frequency.  In this paper we  present a method to find the optimal
+set of frequency scaling factors to simultaneously optimize both energy and execution time
+ without adding a big overhead. \textbf{put the last two phrases in the related work section}
+ 
+  
+    Our solution for this problem is
+to make the optimization process for energy and execution time follow the same
+direction.  Therefore, we inverse the equation of the normalized execution time which gives 
+the normalized performance equation, as follows:

 \begin{multline}
   \label{eq:pnorm_inv}
   P_\textit{Norm} = \frac{T_\textit{Old}}{T_\textit{New}}\\
 \begin{multline}
   \label{eq:pnorm_inv}
   P_\textit{Norm} = \frac{T_\textit{Old}}{T_\textit{New}}\\


@@ -286,18 +311,17 @@ curve EQ~(\ref{eq:pnorm_inv}) over all available sets of scaling factors.  This
 represents the minimum energy consumption with minimum execution time (better
 performance) at the same time, see figure~(\ref{fig:r1}) or figure~(\ref{fig:r2}) .  Then our objective
 function has the following form:
 represents the minimum energy consumption with minimum execution time (better
 performance) at the same time, see figure~(\ref{fig:r1}) or figure~(\ref{fig:r2}) .  Then our objective
 function has the following form:

-\begin{multline}
+\begin{equation}

   \label{eq:max}
   Max Dist = 
   \max_{i=1,\dots F, j=1,\dots,N}
       (\overbrace{P_\textit{Norm}(S_{ij})}^{\text{Maximize}} -
        \overbrace{E_\textit{Norm}(S_{ij})}^{\text{Minimize}} )
   \label{eq:max}
   Max Dist = 
   \max_{i=1,\dots F, j=1,\dots,N}
       (\overbrace{P_\textit{Norm}(S_{ij})}^{\text{Maximize}} -
        \overbrace{E_\textit{Norm}(S_{ij})}^{\text{Minimize}} )

-\end{multline}
+\end{equation}

 where $N$ is the number of nodes and $F$ is the  number of available frequencies for each nodes. 
 Then we can select the optimal set of scaling factors that satisfies EQ~(\ref{eq:max}).  Our objective function can
 where $N$ is the number of nodes and $F$ is the  number of available frequencies for each nodes. 
 Then we can select the optimal set of scaling factors that satisfies EQ~(\ref{eq:max}).  Our objective function can

-work with any energy model or energy values stored in a data file.
-Moreover, this function works in optimal way when the energy curve has a convex
-form over the available frequency scaling factors as shown in~\cite{15,3,19}.
+work with any energy model or any power values for each node (static and dynamic powers).
+However, the most energy reduction gain can be achieved the energy curve has a convex form as shown in~\cite{15,3,19}.

 
 \section{The heterogeneous scaling algorithm }
 \label{sec.optim}
 
 \section{The heterogeneous scaling algorithm }
 \label{sec.optim}


@@ -354,22 +378,22 @@ maximum frequency of node $i$  and the computation scaling factor $Scp_i$ as fol
     \item[$Ps_i$] array of the static powers for all nodes.
     \item[$Fdiff_i$] array of the difference between two successive frequencies for all nodes.
     \end{description}
     \item[$Ps_i$] array of the static powers for all nodes.
     \item[$Fdiff_i$] array of the difference between two successive frequencies for all nodes.
     \end{description}

-    \Ensure $Sopt_1, \dots ,Sopt_N$ is a set of optimal scaling factors 
+    \Ensure $Sopt_1, \dots, Sopt_N$ is a set of optimal scaling factors

 
     \State $ Scp_i \gets \frac{\max_{i=1,2,\dots,N}(Tcp_i)}{Tcp_i} $
     \State $F_{i} \gets  \frac{Fmax_i}{Scp_i},~{i=1,2,\cdots,N}$
     \State Round the computed initial frequencies $F_i$ to the closest one available in each node.
     \If{(not the first frequency)}
 
     \State $ Scp_i \gets \frac{\max_{i=1,2,\dots,N}(Tcp_i)}{Tcp_i} $
     \State $F_{i} \gets  \frac{Fmax_i}{Scp_i},~{i=1,2,\cdots,N}$
     \State Round the computed initial frequencies $F_i$ to the closest one available in each node.
     \If{(not the first frequency)}

-          \State $F_i \gets F_i+Fdiff_i,~i=1,...,N.$
+          \State $F_i \gets F_i+Fdiff_i,~i=1,\dots,N.$

     \EndIf 
     \EndIf 

-    \State $T_\textit{Old} \gets max_{~i=1,...,N } (Tcp_i+Tcm_i)$
+    \State $T_\textit{Old} \gets max_{~i=1,\dots,N } (Tcp_i+Tcm_i)$

     \State $E_\textit{Original} \gets \sum_{i=1}^{N}{( Pd_i \cdot  Tcp_i)} +\sum_{i=1}^{N} {(Ps_i \cdot T_{Old})}$
     \State $Dist \gets 0$
     \State $E_\textit{Original} \gets \sum_{i=1}^{N}{( Pd_i \cdot  Tcp_i)} +\sum_{i=1}^{N} {(Ps_i \cdot T_{Old})}$
     \State $Dist \gets 0$

-    \State  $Sopt_{i} \gets 1,~i=1,...,N. $
+    \State  $Sopt_{i} \gets 1,~i=1,\dots,N. $

     \While {(all nodes not reach their  minimum  frequency)}
         \If{(not the last freq. \textbf{and} not the slowest node)}
     \While {(all nodes not reach their  minimum  frequency)}
         \If{(not the last freq. \textbf{and} not the slowest node)}

-        \State $F_i \gets F_i - Fdiff_i,~i=1,...,N.$ 
-        \State $S_i \gets \frac{Fmax_i}{F_i},~i=1,...,N.$ 
+        \State $F_i \gets F_i - Fdiff_i,~i=1,\dots,N.$
+        \State $S_i \gets \frac{Fmax_i}{F_i},~i=1,\dots,N.$

         \EndIf
        \State $T_{New} \gets max_\textit{~i=1,\dots,N} (Tcp_{i} \cdot S_{i}) + min_\textit{~i=1,\dots,N}(Tcm_i) $
        \State $E_\textit{Reduced} \gets \sum_{i=1}^{N}{(S_i^{-2} \cdot Pd_i \cdot  Tcp_i)} + $  \hspace*{43 mm} 
         \EndIf
        \State $T_{New} \gets max_\textit{~i=1,\dots,N} (Tcp_{i} \cdot S_{i}) + min_\textit{~i=1,\dots,N}(Tcm_i) $
        \State $E_\textit{Reduced} \gets \sum_{i=1}^{N}{(S_i^{-2} \cdot Pd_i \cdot  Tcp_i)} + $  \hspace*{43 mm} 


@@ -377,11 +401,11 @@ maximum frequency of node $i$  and the computation scaling factor $Scp_i$ as fol
        \State $ P_\textit{Norm} \gets \frac{T_\textit{Old}}{T_\textit{New}}$
        \State $E_\textit{Norm}\gets \frac{E_\textit{Reduced}}{E_\textit{Original}}$
       \If{$(\Pnorm - \Enorm > \Dist)$}
        \State $ P_\textit{Norm} \gets \frac{T_\textit{Old}}{T_\textit{New}}$
        \State $E_\textit{Norm}\gets \frac{E_\textit{Reduced}}{E_\textit{Original}}$
       \If{$(\Pnorm - \Enorm > \Dist)$}

-        \State $Sopt_{i} \gets S_{i},~i=1,...,N. $
+        \State $Sopt_{i} \gets S_{i},~i=1,\dots,N. $

         \State $\Dist \gets \Pnorm - \Enorm$
       \EndIf
     \EndWhile
         \State $\Dist \gets \Pnorm - \Enorm$
       \EndIf
     \EndWhile

-    \State  Return $Sopt_1,Sopt_2, \dots ,Sopt_N$
+    \State  Return $Sopt_1,Sopt_2,\dots,Sopt_N$

   \end{algorithmic}
   \caption{Heterogeneous scaling algorithm}
   \label{HSA}
   \end{algorithmic}
   \caption{Heterogeneous scaling algorithm}
   \label{HSA}


@@ -651,16 +675,31 @@ The results of applying the proposed scaling algorithm to NAS benchmarks is demo
 
 \begin{figure}
   \centering
 
 \begin{figure}
   \centering

-  \subfloat[Balanced nodes type scenario]{%
+  \subfloat[CG, MG, LU and FT Benchmarks]{%

     \includegraphics[width=.23185\textwidth]{fig/avg_eq}\label{fig:avg_eq}}%
   \quad%
     \includegraphics[width=.23185\textwidth]{fig/avg_eq}\label{fig:avg_eq}}%
   \quad%

-  \subfloat[Imbalanced nodes type scenario]{%
+  \subfloat[BT and SP benchmarks]{%

     \includegraphics[width=.23185\textwidth]{fig/avg_neq}\label{fig:avg_neq}}
   \label{fig:avg}
   \caption{The average of energy and performance for all NAS benchmarks running with difference number of nodes}
 \end{figure}
 
     \includegraphics[width=.23185\textwidth]{fig/avg_neq}\label{fig:avg_neq}}
   \label{fig:avg}
   \caption{The average of energy and performance for all NAS benchmarks running with difference number of nodes}
 \end{figure}
 

-In the NAS benchmarks there are some programs executed on different number of nodes. The benchmarks CG, MG, LU and FT executed on 2 to a power of (1, 2, 4, 8, ...) of nodes. The other benchmarks such as BT and SP executed on 2 to a power of (1, 2, 4, 9, ...) of nodes. We are take the average of energy saving, performance degradation and distances for all results of NAS benchmarks. The average of these three objectives are plotted to the number of nodes as in plots (\ref{fig:avg_eq} and \ref{fig:avg_neq}).  In CG, MG, LU, and FT benchmarks the average of energy saving is decreased when the number of nodes is increased due to the increasing in the communication times as mentioned before. Thus, the average of distances (our objective function) is decreased linearly with energy saving while keeping the average of performance degradation the same. In BT and SP benchmarks, the average of energy saving is not decreased significantly compare to other benchmarks when the number of nodes is increased. Nevertheless, the average of performance  degradation approximately still the same ratio. This difference is depends on the characteristics of the benchmarks such as the computation to communication ratio that has. 
+In the NAS benchmarks there are some programs executed on different number of
+nodes. The benchmarks CG, MG, LU and FT executed on 2 to a power of (1, 2, 4, 8,
+\dots{}) of nodes. The other benchmarks such as BT and SP executed on 2 to a
+power of (1, 2, 4, 9, \dots{}) of nodes. We are take the average of energy
+saving, performance degradation and distances for all results of NAS
+benchmarks. The average of these three objectives are plotted to the number of
+nodes as in plots (\ref{fig:avg_eq} and \ref{fig:avg_neq}).  In CG, MG, LU, and
+FT benchmarks the average of energy saving is decreased when the number of nodes
+is increased due to the increasing in the communication times as mentioned
+before. Thus, the average of distances (our objective function) is decreased
+linearly with energy saving while keeping the average of performance degradation
+the same. In BT and SP benchmarks, the average of energy saving is not decreased
+significantly compare to other benchmarks when the number of nodes is
+increased. Nevertheless, the average of performance degradation approximately
+still the same ratio. This difference is depends on the characteristics of the
+benchmarks such as the computation to communication ratio that has.

 
 \subsection{The results for different powers scenarios}
 
 
 \subsection{The results for different powers scenarios}