added some sentences to 5th sec.

diff --git a/Heter_paper.tex b/Heter_paper.tex
index 66dab369171cb1d48e8caf2e3b1c2f5b29e17610..20ca2291dd9f5302e88ff6bf90883d423fc5b6ce 100644 (file)
--- a/Heter_paper.tex
+++ b/Heter_paper.tex
@@ -141,7 +141,7 @@ vector of scaling factors can be predicted using EQ (\ref{eq:perf}).
 \begin{equation}
   \label{eq:perf}
  \textit  T_\textit{new} = 
 \begin{equation}
   \label{eq:perf}
  \textit  T_\textit{new} = 

- \max_{i=1,2,\dots,N} (TcpOld_{i} \cdot S_{i}) +  MinTcm
+ \max_{i=1,2,\dots,N} ({TcpOld_{i}} \cdot S_{i}) +  MinTcm

 \end{equation}
 where $TcpOld_i$ is the computation time  of processor $i$ during the first iteration and $MinTcm$ is the communication time of the slowest processor from the first iteration.  The model computes the maximum computation time 
  with scaling factor from each node  added to the communication time of the slowest node, it means  only the
 \end{equation}
 where $TcpOld_i$ is the computation time  of processor $i$ during the first iteration and $MinTcm$ is the communication time of the slowest processor from the first iteration.  The model computes the maximum computation time 
  with scaling factor from each node  added to the communication time of the slowest node, it means  only the


@@ -262,7 +262,7 @@ In the same way, we normalize the energy by computing the ratio between the cons
   E_\textit{Norm} = \frac{E_\textit{Reduced}}{E_\textit{Original}} \\
   {} = \frac{ \sum_{i=1}^{N}{(S_i^{-2} \cdot Pd_i \cdot  Tcp_i)} +
  \sum_{i=1}^{N} {(Ps_i \cdot T_{New})}}{\sum_{i=1}^{N}{( Pd_i \cdot  Tcp_i)} +
   E_\textit{Norm} = \frac{E_\textit{Reduced}}{E_\textit{Original}} \\
   {} = \frac{ \sum_{i=1}^{N}{(S_i^{-2} \cdot Pd_i \cdot  Tcp_i)} +
  \sum_{i=1}^{N} {(Ps_i \cdot T_{New})}}{\sum_{i=1}^{N}{( Pd_i \cdot  Tcp_i)} +

- \sum_{i=1}^{N} {(Ps_i \cdot T_{Old})}}
+ \sum_{i=1}^{N} {(Ps_i@+eYd162 \cdot T_{Old})}}

 \end{multline} 
 Where $T_{New}$ and $T_{Old}$ are computed as in EQ(\ref{eq:pnorm}).
 
 \end{multline} 
 Where $T_{New}$ and $T_{Old}$ are computed as in EQ(\ref{eq:pnorm}).
 


@@ -323,8 +323,8 @@ However, the most energy reduction gain can be achieved when the energy curve ha
 \section{The scaling factors selection algorithm for heterogeneous platforms }
 \label{sec.optim}
 
 \section{The scaling factors selection algorithm for heterogeneous platforms }
 \label{sec.optim}
 

-In this section we  propose algorithm~\ref{HSA}) which selects the frequency scaling factors vector that gives the best trade-off between minimizing the energy consumption  and maximizing the performance of a message passing synchronous iterative application executed on a heterogeneous platform.  
-IT works online during the execution time of the iterative message passing program.  It uses information gathered during the first iteration such as the computation time and the communication time in one iteration for each node. The algorithm is executed  after the first iteration and returns a vector of optimal frequency scaling factors   that satisfies the objective function EQ(\ref{eq:max}). The program apply DVFS operations to change the frequencies of the CPUs according to the computed scaling factors.  This algorithm is called just once during the execution of the program. Algorithm~(\ref{dvfs}) shows where and when the proposed scaling algorithm is called in the iterative MPI program.
+In this section we  propose algorithm~(\ref{HSA}) which selects the frequency scaling factors vector that gives the best trade-off between minimizing the energy consumption  and maximizing the performance of a message passing synchronous iterative application executed on a heterogeneous platform.  
+It works online during the execution time of the iterative message passing program.  It uses information gathered during the first iteration such as the computation time and the communication time in one iteration for each node. The algorithm is executed  after the first iteration and returns a vector of optimal frequency scaling factors   that satisfies the objective function EQ(\ref{eq:max}). The program apply DVFS operations to change the frequencies of the CPUs according to the computed scaling factors.  This algorithm is called just once during the execution of the program. Algorithm~(\ref{dvfs}) shows where and when the proposed scaling algorithm is called in the iterative MPI program.

 
 
 The nodes in a heterogeneous platform have different computing powers, thus while executing message passing iterative synchronous applications, fast nodes have to wait for the slower ones to finish their computations before being able to synchronously communicate with them as in figure (\ref{fig:heter}). These periods are called idle or slack times.
 
 
 The nodes in a heterogeneous platform have different computing powers, thus while executing message passing iterative synchronous applications, fast nodes have to wait for the slower ones to finish their computations before being able to synchronously communicate with them as in figure (\ref{fig:heter}). These periods are called idle or slack times.


@@ -340,7 +340,7 @@ maximum frequency of node $i$  and the computation scaling factor $Scp_i$ as fol
  F_{i} = \frac{Fmax_i}{Scp_i},~{i=1,2,\cdots,N}
 \end{equation}
 If the computed initial frequency for a node is not available in the gears of that node, the computed initial frequency is replaced by the nearest available frequency.
  F_{i} = \frac{Fmax_i}{Scp_i},~{i=1,2,\cdots,N}
 \end{equation}
 If the computed initial frequency for a node is not available in the gears of that node, the computed initial frequency is replaced by the nearest available frequency.

-In  figure (\ref{fig:st_freq}), the nodes are  sorted by their computing powers in ascending order and the frequencies of the faster nodes are scaled down according to the computed initial frequency scaling factors. The resulting new frequencies are coloured in blue in  figure (\ref{fig:st_freq}). This set of frequencies can be considered as a higher bound for the search space of the optimal set of frequencies because selecting frequency scaling factors higher than the higher bound will not improve the performance of the application and it will increase its overall energy consumption. Therefore the frequency selecting factors algorithm starts its search method from these initial frequencies and takes a downward search direction. The algorithm iterates on all left frequencies, from the higher bound until all nodes reach their minimum frequencies, to compute their overall energy consumption and performance, and select the optimal frequency scaling factors vector. At each iteration the algorithm determines the slowest node according to EQ(\ref{eq:perf}) and keeps its frequency unchanged, while it lowers the frequency of  all other nodes by one gear. The new overall energy consumption and execution time are computed according to the new scaling factors. The optimal set of frequency scaling factors is the set that gives the highest distance according to  the objective function EQ(\ref{eq:max}).
+In  figure (\ref{fig:st_freq}), the nodes are  sorted by their computing powers in ascending order and the frequencies of the faster nodes are scaled down according to the computed initial frequency scaling factors. The resulting new frequencies are coloured in blue in  figure (\ref{fig:st_freq}). This set of frequencies can be considered as a higher bound for the search space of the optimal vector of frequencies because selecting frequency scaling factors higher than the higher bound will not improve the performance of the application and it will increase its overall energy consumption. Therefore the frequency selecting factors algorithm starts its search method from these initial frequencies and takes a downward search direction. If the algorithm starts to search from the first frequencies of all nodes, regardless the higher bound frequencies, at each step the predicted performance and energy  are degreased  together, then the best distance be unreachable. This case is similar to homogeneous scaling algorithm when all nodes in the cluster has the same computing power, therefore there is a smaller distance between the performance and the energy curves, while in a heterogeneous cluster the distance is bigger and the energy saving against smaller execution time is higher, as an example see figure~(\ref{fig:r1} and \ref{fig:r2}). The algorithm iterates on all left frequencies, from the higher bound until all nodes reach their minimum frequencies, to compute their overall energy consumption and performance, and select the optimal frequency scaling factors vector. At each iteration the algorithm determines the slowest node according to EQ(\ref{eq:perf}) and keeps its frequency unchanged, while it lowers the frequency of  all other nodes by one gear. The new overall energy consumption and execution time are computed according to the new scaling factors. The optimal set of frequency scaling factors is the set that gives the highest distance according to  the objective function EQ(\ref{eq:max}).

 \begin{figure}[t]
   \centering
     \includegraphics[scale=0.5]{fig/start_freq}
 \begin{figure}[t]
   \centering
     \includegraphics[scale=0.5]{fig/start_freq}


@@ -349,16 +349,6 @@ In  figure (\ref{fig:st_freq}), the nodes are  sorted by their computing powers
 \end{figure}
 
 
 \end{figure}
 
 

-This algorithm has a small
-execution time: for a heterogeneous cluster composed of four different types of
-nodes having the characteristics presented in table~(\ref{table:platform}), it  
-takes \np[ms]{0.04} on average for 4 nodes and \np[ms]{0.15} on average for 144
-nodes.  The algorithm complexity is $O(F\cdot (N \cdot4) )$, where $F$ is the
-number of iterations and $N$ is the number of computing nodes. The algorithm
-needs on average from 12 to 20 iterations to selects the best vector of frequency scaling factors that give the results of the next section. \textbf{put the lst paragraph in experiments}
-
-
-

 
 
 
 
 
 


@@ -439,15 +429,21 @@ available frequencies and dynamic and static powers values, see table
 (\ref{table:platform}). These different types of processing nodes are simulate some
 real Intel processors. The maximum number of nodes that supported by the cluster
 is 144 nodes according to characteristics of some MPI programs of the NAS
 (\ref{table:platform}). These different types of processing nodes are simulate some
 real Intel processors. The maximum number of nodes that supported by the cluster
 is 144 nodes according to characteristics of some MPI programs of the NAS

-benchmarks that used. We are use the same number from each type of nodes when we
+benchmarks v3.3 \cite{44} that used. We are use the same number from each type of nodes when we

 run the iterative MPI programs, for example if  we are execute the program on 8 node, there
 are 2 nodes from each type participating in the computation. The dynamic and
 static power values is different from one type to other. Each node has a dynamic
 run the iterative MPI programs, for example if  we are execute the program on 8 node, there
 are 2 nodes from each type participating in the computation. The dynamic and
 static power values is different from one type to other. Each node has a dynamic

-and static power values proportional to their computing power (FLOPS), for more
+and static power values proportionally increased to their computing power (FLOPS), for more

 details see the Intel data sheets in \cite{47}.  Each node has a percentage of
 80\% for dynamic power and 20\% for static power of the total power
 consumption of a CPU, the same assumption is made in \cite{45,3}. These nodes are
 details see the Intel data sheets in \cite{47}.  Each node has a percentage of
 80\% for dynamic power and 20\% for static power of the total power
 consumption of a CPU, the same assumption is made in \cite{45,3}. These nodes are

-connected via an ethernet network with 1 Gbit/s bandwidth.
+connected via an ethernet network with 1 Gbit/s bandwidth. The proposed scaling algorithm has a small
+execution time: for a heterogeneous cluster composed of four different types of
+nodes having the characteristics presented in table~(\ref{table:platform}), it  
+takes \np[ms]{0.04} on average for 4 nodes and \np[ms]{0.15} on average for 144
+nodes.  The algorithm complexity is $O(F\cdot (N \cdot4) )$, where $F$ is the
+number of iterations and $N$ is the number of computing nodes. The algorithm
+needs on average from 12 to 20 iterations to selects the best vector of frequency scaling factors that give the results of the next section.

 \begin{table}[htb]
   \caption{Heterogeneous nodes characteristics}
   % title of Table
 \begin{table}[htb]
   \caption{Heterogeneous nodes characteristics}
   % title of Table


@@ -455,7 +451,7 @@ connected via an ethernet network with 1 Gbit/s bandwidth.
   \begin{tabular}{|*{7}{l|}}
     \hline
     Node     & Similar     & Max        & Min          & Diff.          & Dynamic      & Static \\
   \begin{tabular}{|*{7}{l|}}
     \hline
     Node     & Similar     & Max        & Min          & Diff.          & Dynamic      & Static \\

-    type     & to          & Freq. GHz  & Freq. GHz    & Freq GHz       & power        & power \\
+    type     & to          & Freq. GHz  & Freq. GHz    & Freq. GHz       & power        & power \\

     \hline
     1       & core-i3       & 2.5         & 1.2          & 0.1           & 20~w         &4~w    \\
             &  2100T        &             &              &               &              &  \\
     \hline
     1       & core-i3       & 2.5         & 1.2          & 0.1           & 20~w         &4~w    \\
             &  2100T        &             &              &               &              &  \\


@@ -480,11 +476,10 @@ connected via an ethernet network with 1 Gbit/s bandwidth.
 \subsection{The experimental results of the scaling algorithm}
 \label{sec.res}
 
 \subsection{The experimental results of the scaling algorithm}
 \label{sec.res}
 

-The proposed algorithm was applied to seven MPI programs of the NAS benchmarks (EP, CG, MG, FT, BT, LU and SP) NPB v3.3 
-\cite{44},  which were run with three classes (A, B and C).
+The proposed algorithm was applied to seven MPI programs of the NAS benchmarks (EP, CG, MG, FT, BT, LU and SP) NPB v3.3,  which were run with three classes (A, B and C).

 In this experiments we are interested to run the class C, the biggest class compared to A and B, on different number of 
 In this experiments we are interested to run the class C, the biggest class compared to A and B, on different number of 

-nodes, from 4 to 128 or 144 nodes according to the type of the iterative MPI program.  Depending on the proposed energy consumption model EQ(\ref{eq:energy}),
- we are measure the energy consumption for all NAS MPI programs. The dynamic and static power values are used under the same assumption used by \cite{45,3}, we are used a percentage of 80\% for dynamic power and 20\% for static of the total power consumption of a CPU. The heterogeneous nodes in table (\ref{table:platform}) have different  simulated computing power (FLOPS), ranked from the node of type 1 with smaller computing power (FLOPS) to the highest computing power (FLOPS) for node of type 4. Therefore, the power values are used proportionally increased from nodes of type 1 to nodes of type 4 that with highest computing power. Then, we are used an assumption that the power consumption is increased linearly when the computing power (FLOPS) of the processor is increased, see \cite{48}.   
+nodes, from 4 to 144 nodes according to the type of the iterative MPI program.  Depending on the proposed energy consumption model EQ(\ref{eq:energy}),
+ we are measure the energy consumption for all the NAS MPI programs. The dynamic and static power values are used under the same assumption used by \cite{45,3}, we are used a percentage of 80\% for dynamic power and 20\% for static of the total power consumption of a CPU. The heterogeneous nodes in table (\ref{table:platform}) have different  simulated computing power (FLOPS), ranked from the node of type 1 with smaller computing power (FLOPS) to the highest computing power (FLOPS) for node of type 4. Therefore, the power values are used proportionally increased from nodes of type 1 to nodes of type 4 that with highest computing power. Then, we are used an assumption that the power consumption is increased linearly when the computing power (FLOPS) of the processor is increased, see \cite{48}.   

  
 \begin{table}[htb]
   \caption{Running NAS benchmarks on 4 nodes }
  
 \begin{table}[htb]
   \caption{Running NAS benchmarks on 4 nodes }


@@ -677,7 +672,7 @@ the same. In BT and SP benchmarks, the average of the  energy saving is not decr
 significantly compare to other benchmarks when the number of nodes is
 increased. Nevertheless, the average of performance degradation approximately
 still the same ratio. This difference is depends on the characteristics of the
 significantly compare to other benchmarks when the number of nodes is
 increased. Nevertheless, the average of performance degradation approximately
 still the same ratio. This difference is depends on the characteristics of the

-benchmarks such as the computation to communication ratio that has.
+benchmark such as the computations to communications ratio that has.

 
 \subsection{The results for different power consumption scenarios}
 
 
 \subsection{The results for different power consumption scenarios}
 


@@ -758,14 +753,14 @@ running the NAS benchmarks class C on 8 or 9 nodes are place in the tables
     \includegraphics[width=.24\textwidth]{fig/three_scenarios}\label{fig:scales_comp}}
   \label{fig:comp}
   \caption{The comparison of the three power scenarios}
     \includegraphics[width=.24\textwidth]{fig/three_scenarios}\label{fig:scales_comp}}
   \label{fig:comp}
   \caption{The comparison of the three power scenarios}

-\end{figure}
+\end{figure}  

 
 To compare the results of these three powers scenarios, we are take the average of the performance degradation, the energy saving and the distances for all NAS benchmarks running on 8 or 9 nodes of class C, as in figure (\ref{fig:sen_comp}). Thus, according to the average of these results, the energy saving ratio is increased when using a higher percentage for dynamic power (e.g. 90\%-10\% scenario), due to increase in dynamic energy. While the average of energy saving is decreased in 70\%-30\% scenario. Because the static energy consumption is increase. Moreover, the average of distances is more related to energy saving changes. The average of the performance degradation is decreased when using a higher ratio for static power (e.g. 70\%-30\% scenario and 80\%-20\% scenario). The raison behind these relations, that the proposed algorithm optimize both energy consumption and performance in the same time. Therefore, when using a higher ratio for dynamic power the algorithm selecting bigger frequency scaling factors values, more energy saving versus more performance degradation, for example see the figure (\ref{fig:scales_comp}). The inverse happen when using a higher ratio for  static  power, the algorithm proportionally  selects a smaller scaling values, less energy saving versus less performance degradation. This is because the  
 algorithm is optimizes the  static energy consumption that is always related to the execution time. 
 
 \subsection{The verifications of the proposed method}
 \label{sec.verif}
 
 To compare the results of these three powers scenarios, we are take the average of the performance degradation, the energy saving and the distances for all NAS benchmarks running on 8 or 9 nodes of class C, as in figure (\ref{fig:sen_comp}). Thus, according to the average of these results, the energy saving ratio is increased when using a higher percentage for dynamic power (e.g. 90\%-10\% scenario), due to increase in dynamic energy. While the average of energy saving is decreased in 70\%-30\% scenario. Because the static energy consumption is increase. Moreover, the average of distances is more related to energy saving changes. The average of the performance degradation is decreased when using a higher ratio for static power (e.g. 70\%-30\% scenario and 80\%-20\% scenario). The raison behind these relations, that the proposed algorithm optimize both energy consumption and performance in the same time. Therefore, when using a higher ratio for dynamic power the algorithm selecting bigger frequency scaling factors values, more energy saving versus more performance degradation, for example see the figure (\ref{fig:scales_comp}). The inverse happen when using a higher ratio for  static  power, the algorithm proportionally  selects a smaller scaling values, less energy saving versus less performance degradation. This is because the  
 algorithm is optimizes the  static energy consumption that is always related to the execution time. 
 
 \subsection{The verifications of the proposed method}
 \label{sec.verif}

-The precision of the proposed algorithm mainly depends on the execution time prediction model EQ(\ref{eq:perf}) and the energy model EQ(\ref{eq:energy}). The energy model is significantly depends on the execution time model, that the static energy is related linearly. So, our work is depends mainly on execution time model. To verifying thid model, we are compare the predicted execution time with the real execution time (Simgrid time) values that gathered  offline from the NAS benchmarks class B executed on 8 or 9 nodes. The execution time model can predicts the real execution time by maximum normalized error equal to 0.03 for all the NAS benchmarks. The second verification that we are made is for the proposed scaling algorithm to prove its ability to selects the best vector of the frequency scaling factors. Therefore, we are expand the algorithm to test at each iteration the frequency scaling factor of the slowest node with the all available scaling factors of the other nodes, all possible solutions. This version of the algorithm is applied to different NAS benchmarks classes with different number of nodes. The results from the expanded algorithms and the proposed algorithm are identical. While the proposed algorithm is runs  by 10 times faster on average compare to the expanded algorithm.
+The precision of the proposed algorithm mainly depends on the execution time prediction model EQ(\ref{eq:perf}) and the energy model EQ(\ref{eq:energy}). The energy model is significantly depends on the execution time model, that the static energy is related linearly. So, our work is depends mainly on execution time model. To verifying this model, we are compared the predicted execution time with the real execution time (Simgrid time) values that gathered  offline from the NAS benchmarks class B executed on 8 or 9 nodes. The execution time model can predicts the real execution time by maximum normalized error equal to 0.03 for all the NAS benchmarks. The second verification that we are made is for the proposed scaling algorithm to prove its ability to selects the best vector of the frequency scaling factors. Therefore, we are expand the algorithm to test at each iteration the frequency scaling factor of the slowest node with the all available scaling factors of the other nodes, all possible solutions. This version of the algorithm is applied to different NAS benchmarks classes with different number of nodes. The results from the expanded algorithms and the proposed algorithm are identical. While the proposed algorithm is runs  by 10 times faster on average compare to the expanded algorithm.

 
 \section{Conclusion}
 \label{sec.concl}
 
 \section{Conclusion}
 \label{sec.concl}