Merge branch 'master' of ssh://info.iut-bm.univ-fcomte.fr/mpi-energy2

[mpi-energy2.git] / Heter_paper.tex

diff --git a/Heter_paper.tex b/Heter_paper.tex
index e107a6e3e5c39d336dd43df911439cfeb06c7c76..1ac7dcf6e3b99c6dffa29b39343c896a63f5e104 100644 (file)
--- a/Heter_paper.tex
+++ b/Heter_paper.tex
@@ -87,23 +87,23 @@
   number of nodes composing them.  To minimize the operating costs of these
   platforms many techniques have been used. Dynamic voltage and frequency
   scaling (DVFS) is one of them. It reduces the frequency of a CPU to lower its
   number of nodes composing them.  To minimize the operating costs of these
   platforms many techniques have been used. Dynamic voltage and frequency
   scaling (DVFS) is one of them. It reduces the frequency of a CPU to lower its

-  energy consumption.  However, lowering the frequency of a CPU might increase
+  energy consumption.  However, lowering the frequency of a CPU may increase

   the execution time of an application running on that processor.  Therefore,
   the frequency that gives the best trade-off between the energy consumption and
   the performance of an application must be selected.
 
   In this paper, a new online frequency selecting algorithm for heterogeneous
   the execution time of an application running on that processor.  Therefore,
   the frequency that gives the best trade-off between the energy consumption and
   the performance of an application must be selected.
 
   In this paper, a new online frequency selecting algorithm for heterogeneous

-  platforms is presented.  It selects the frequencies and tries to give the best
+  platforms (heterogeneous CPUs) is presented.  It selects the frequencies and tries to give the best

   trade-off between energy saving and performance degradation, for each node
   computing the message passing iterative application. The algorithm has a small
   overhead and works without training or profiling. It uses a new energy model
   for message passing iterative applications running on a heterogeneous
   platform. The proposed algorithm is evaluated on the SimGrid simulator while
   running the NAS parallel benchmarks.  The experiments show that it reduces the
   trade-off between energy saving and performance degradation, for each node
   computing the message passing iterative application. The algorithm has a small
   overhead and works without training or profiling. It uses a new energy model
   for message passing iterative applications running on a heterogeneous
   platform. The proposed algorithm is evaluated on the SimGrid simulator while
   running the NAS parallel benchmarks.  The experiments show that it reduces the

-  energy consumption by up to \np[\%]{35} while limiting the performance
+  energy consumption by up to \np[\%]{34} while limiting the performance

   degradation as much as possible.  Finally, the algorithm is compared to an
   degradation as much as possible.  Finally, the algorithm is compared to an

-  existing method, the comparison results showing that it outperforms the
-  latter.
+  existing method, the comparison results show that it outperforms the
+  latter, on average it saves \np[\%]{4} more energy while keeping the same performance. 

 
 \end{abstract}
 
 
 \end{abstract}
 


@@ -112,7 +112,7 @@
 
 The need for more computing power is continually increasing. To partially
 satisfy this need, most supercomputers constructors just put more computing
 
 The need for more computing power is continually increasing. To partially
 satisfy this need, most supercomputers constructors just put more computing

-nodes in their platform. The resulting platforms might achieve higher floating
+nodes in their platform. The resulting platforms may achieve higher floating

 point operations per second (FLOPS), but the energy consumption and the heat
 dissipation are also increased.  As an example, the Chinese supercomputer
 Tianhe-2 had the highest FLOPS in November 2014 according to the Top500 list
 point operations per second (FLOPS), but the energy consumption and the heat
 dissipation are also increased.  As an example, the Chinese supercomputer
 Tianhe-2 had the highest FLOPS in November 2014 according to the Top500 list


@@ -133,7 +133,7 @@ to lower the energy consumption of these platforms, such as scheduling, DVFS,
 \dots{} DVFS is a widely used process to reduce the energy consumption of a
 processor by lowering its frequency
 \cite{Rizvandi_Some.Observations.on.Optimal.Frequency}. However, it also reduces
 \dots{} DVFS is a widely used process to reduce the energy consumption of a
 processor by lowering its frequency
 \cite{Rizvandi_Some.Observations.on.Optimal.Frequency}. However, it also reduces

-the number of FLOPS executed by the processor which might increase the execution
+the number of FLOPS executed by the processor which may increase the execution

 time of the application running over that processor.  Therefore, researchers use
 different optimization strategies to select the frequency that gives the best
 trade-off between the energy reduction and performance degradation ratio. In
 time of the application running over that processor.  Therefore, researchers use
 different optimization strategies to select the frequency that gives the best
 trade-off between the energy reduction and performance degradation ratio. In


@@ -168,15 +168,15 @@ in Section~\ref{sec.concl} the paper ends with a summary and some future works.
 DVFS is a technique used in modern processors to scale down both the voltage and
 the frequency of the CPU while computing, in order to reduce the energy
 consumption of the processor. DVFS is also allowed in GPUs to achieve the same
 DVFS is a technique used in modern processors to scale down both the voltage and
 the frequency of the CPU while computing, in order to reduce the energy
 consumption of the processor. DVFS is also allowed in GPUs to achieve the same

-goal. Reducing the frequency of a processor lowers its number of FLOPS and might
+goal. Reducing the frequency of a processor lowers its number of FLOPS and may

 degrade the performance of the application running on that processor, especially
 if it is compute bound. Therefore selecting the appropriate frequency for a
 degrade the performance of the application running on that processor, especially
 if it is compute bound. Therefore selecting the appropriate frequency for a

-processor to satisfy some objectives while taking into account all the
+processor to satisfy some objectives, while taking into account all the

 constraints, is not a trivial operation.  Many researchers used different
 strategies to tackle this problem. Some of them developed online methods that
 compute the new frequency while executing the application, such
 as~\cite{Hao_Learning.based.DVFS,Spiliopoulos_Green.governors.Adaptive.DVFS}.
 constraints, is not a trivial operation.  Many researchers used different
 strategies to tackle this problem. Some of them developed online methods that
 compute the new frequency while executing the application, such
 as~\cite{Hao_Learning.based.DVFS,Spiliopoulos_Green.governors.Adaptive.DVFS}.

-Others used offline methods that might need to run the application and profile
+Others used offline methods that may need to run the application and profile

 it before selecting the new frequency, such
 as~\cite{Rountree_Bounding.energy.consumption.in.MPI,Cochran_Pack_and_Cap_Adaptive_DVFS}.
 The methods could be heuristics, exact or brute force methods that satisfy
 it before selecting the new frequency, such
 as~\cite{Rountree_Bounding.energy.consumption.in.MPI,Cochran_Pack_and_Cap_Adaptive_DVFS}.
 The methods could be heuristics, exact or brute force methods that satisfy


@@ -270,7 +270,7 @@ have the same network bandwidth and latency.
 The overall execution time of a distributed iterative synchronous application
 over a heterogeneous platform consists of the sum of the computation time and
 the communication time for every iteration on a node. However, due to the
 The overall execution time of a distributed iterative synchronous application
 over a heterogeneous platform consists of the sum of the computation time and
 the communication time for every iteration on a node. However, due to the

-heterogeneous computation power of the computing nodes, slack times might occur
+heterogeneous computation power of the computing nodes, slack times may occur

 when fast nodes have to wait, during synchronous communications, for the slower
 nodes to finish their computations (see Figure~\ref{fig:heter}).  Therefore, the
 overall execution time of the program is the execution time of the slowest task
 when fast nodes have to wait, during synchronous communications, for the slower
 nodes to finish their computations (see Figure~\ref{fig:heter}).  Therefore, the
 overall execution time of the program is the execution time of the slowest task


@@ -280,7 +280,7 @@ Dynamic Voltage and Frequency Scaling (DVFS) is a process, implemented in
 modern processors, that reduces the energy consumption of a CPU by scaling
 down its voltage and frequency.  Since DVFS lowers the frequency of a CPU
 and consequently its computing power, the execution time of a program running
 modern processors, that reduces the energy consumption of a CPU by scaling
 down its voltage and frequency.  Since DVFS lowers the frequency of a CPU
 and consequently its computing power, the execution time of a program running

-over that scaled down processor might increase, especially if the program is
+over that scaled down processor may increase, especially if the program is

 compute bound.  The frequency reduction process can be  expressed by the scaling
 factor S which is the ratio between  the maximum and the new frequency of a CPU
 as in (\ref{eq:s}).
 compute bound.  The frequency reduction process can be  expressed by the scaling
 factor S which is the ratio between  the maximum and the new frequency of a CPU
 as in (\ref{eq:s}).


@@ -415,11 +415,11 @@ processor after scaling its frequency is computed as follows:
   \Es = \Ps \cdot (\Tcp \cdot S  + \Tcm)
 \end{equation}
 
   \Es = \Ps \cdot (\Tcp \cdot S  + \Tcm)
 \end{equation}
 

-In the considered heterogeneous platform, each processor $i$ might have
+In the considered heterogeneous platform, each processor $i$ may have

 different dynamic and static powers, noted as $\Pd[i]$ and $\Ps[i]$
 respectively.  Therefore, even if the distributed message passing iterative
 application is load balanced, the computation time of each CPU $i$ noted
 different dynamic and static powers, noted as $\Pd[i]$ and $\Ps[i]$
 respectively.  Therefore, even if the distributed message passing iterative
 application is load balanced, the computation time of each CPU $i$ noted

-$\Tcp[i]$ might be different and different frequency scaling factors might be
+$\Tcp[i]$ may be different and different frequency scaling factors may be

 computed in order to decrease the overall energy consumption of the application
 and reduce slack times.  The communication time of a processor $i$ is noted as
 $\Tcm[i]$ and could contain slack times when communicating with slower nodes,
 computed in order to decrease the overall energy consumption of the application
 and reduce slack times.  The communication time of a processor $i$ is noted as
 $\Tcm[i]$ and could contain slack times when communicating with slower nodes,


@@ -500,14 +500,13 @@ Where $\Ereduced$ and $\Eoriginal$ are computed using (\ref{eq:energy}) and
 $\Tnew$ and $\Told$ are computed as in (\ref{eq:pnorm}).
 
 While the main goal is to optimize the energy and execution time at the same
 $\Tnew$ and $\Told$ are computed as in (\ref{eq:pnorm}).
 
 While the main goal is to optimize the energy and execution time at the same

-time, the normalized energy and execution time curves are not in the same
-direction. According to the equations~(\ref{eq:pnorm}) and (\ref{eq:enorm}), the
+time, the normalized energy and execution time curves do not evolve (increase/decrease) in the same way. According to the equations~(\ref{eq:pnorm}) and (\ref{eq:enorm}), the

 vector of frequency scaling factors $S_1,S_2,\dots,S_N$ reduce both the energy
 and the execution time simultaneously.  But the main objective is to produce
 maximum energy reduction with minimum execution time reduction.
 
 This problem can be solved by making the optimization process for energy and
 vector of frequency scaling factors $S_1,S_2,\dots,S_N$ reduce both the energy
 and the execution time simultaneously.  But the main objective is to produce
 maximum energy reduction with minimum execution time reduction.
 
 This problem can be solved by making the optimization process for energy and

-execution time following the same direction.  Therefore, the equation of the
+execution time follow the same evolution according to the vector of scaling factors. Therefore, the equation of the

 normalized execution time is inverted which gives the normalized performance
 equation, as follows:
 \begin{multline}
 normalized execution time is inverted which gives the normalized performance
 equation, as follows:
 \begin{multline}


@@ -562,7 +561,7 @@ in~\cite{Zhuo_Energy.efficient.Dynamic.Task.Scheduling,Rauber_Analytical.Modelin
     \item[{$\Fmax[i]$}] array of the maximum frequencies for all nodes.
     \item[{$\Pd[i]$}] array of the dynamic powers for all nodes.
     \item[{$\Ps[i]$}] array of the static powers for all nodes.
     \item[{$\Fmax[i]$}] array of the maximum frequencies for all nodes.
     \item[{$\Pd[i]$}] array of the dynamic powers for all nodes.
     \item[{$\Ps[i]$}] array of the static powers for all nodes.

-    \item[{$\Fdiff[i]$}] array of the difference between two successive frequencies for all nodes.
+    \item[{$\Fdiff[i]$}] array of the differences between two successive frequencies for all nodes.

     \end{description}
     \Ensure $\Sopt[1],\Sopt[2] \dots, \Sopt[N]$ is a vector of optimal scaling factors
 
     \end{description}
     \Ensure $\Sopt[1],\Sopt[2] \dots, \Sopt[N]$ is a vector of optimal scaling factors
 


@@ -673,12 +672,12 @@ ascending order and the frequencies of the faster nodes are scaled down
 according to the computed initial frequency scaling factors.  The resulting new
 frequencies are highlighted in Figure~\ref{fig:st_freq}.  This set of
 frequencies can be considered as a higher bound for the search space of the
 according to the computed initial frequency scaling factors.  The resulting new
 frequencies are highlighted in Figure~\ref{fig:st_freq}.  This set of
 frequencies can be considered as a higher bound for the search space of the

-optimal vector of frequencies because selecting frequency scaling factors higher
+optimal vector of frequencies because selecting scaling factors higher

 than the higher bound will not improve the performance of the application and it
 will increase its overall energy consumption.  Therefore the algorithm that
 selects the frequency scaling factors starts the search method from these
 initial frequencies and takes a downward search direction toward lower
 than the higher bound will not improve the performance of the application and it
 will increase its overall energy consumption.  Therefore the algorithm that
 selects the frequency scaling factors starts the search method from these
 initial frequencies and takes a downward search direction toward lower

-frequencies. The algorithm iterates on all left frequencies, from the higher
+frequencies. The algorithm iterates on all remaining frequencies, from the higher

 bound until all nodes reach their minimum frequencies, to compute their overall
 energy consumption and performance, and select the optimal frequency scaling
 factors vector. At each iteration the algorithm determines the slowest node
 bound until all nodes reach their minimum frequencies, to compute their overall
 energy consumption and performance, and select the optimal frequency scaling
 factors vector. At each iteration the algorithm determines the slowest node


@@ -700,7 +699,9 @@ power of scaled down nodes are lower than the slowest node. In other words,
 until they reach the higher bound. It can also be noticed that the higher the
 difference between the faster nodes and the slower nodes is, the bigger the
 maximum distance between the energy curve and the performance curve is while the
 until they reach the higher bound. It can also be noticed that the higher the
 difference between the faster nodes and the slower nodes is, the bigger the
 maximum distance between the energy curve and the performance curve is while the

-scaling factors are varying which results in bigger energy savings.
+scaling factors are varying which results in bigger energy savings. 
+Finally, in a homogeneous platform the energy consumption is increased when the scaling factor is very high. 
+Indeed, the dynamic energy  saved  by reducing the frequency of the processor is compensated by the significant increase of the execution time and thus the increased of the static energy. On the other hand, in a heterogeneous platform this is not the case.

 
 \subsection{The evaluation of the proposed algorithm}
 \label{sec.verif.algo}
 
 \subsection{The evaluation of the proposed algorithm}
 \label{sec.verif.algo}


@@ -719,7 +720,7 @@ parallel benchmarks NPB v3.3 \cite{NAS.Parallel.Benchmarks}, running class B on
 very precise, the maximum normalized difference between the predicted execution
 time and the real execution time is equal to 0.03 for all the NAS benchmarks.
 
 very precise, the maximum normalized difference between the predicted execution
 time and the real execution time is equal to 0.03 for all the NAS benchmarks.
 

-Since the proposed algorithm is not an exact method it does not test all the
+Since the proposed algorithm is not an exact method, it does not test all the

 possible solutions (vectors of scaling factors) in the search space. To prove
 its efficiency, it was compared on small instances to a brute force search
 algorithm that tests all the possible solutions. The brute force algorithm was
 possible solutions (vectors of scaling factors) in the search space. To prove
 its efficiency, it was compared on small instances to a brute force search
 algorithm that tests all the possible solutions. The brute force algorithm was


@@ -731,9 +732,9 @@ cluster composed of four different types of nodes having the characteristics
 presented in Table~\ref{table:platform}, it takes on average \np[ms]{0.04} for 4
 nodes and \np[ms]{0.15} on average for 144 nodes to compute the best scaling
 factors vector.  The algorithm complexity is $O(F\cdot N)$, where $F$ is the
 presented in Table~\ref{table:platform}, it takes on average \np[ms]{0.04} for 4
 nodes and \np[ms]{0.15} on average for 144 nodes to compute the best scaling
 factors vector.  The algorithm complexity is $O(F\cdot N)$, where $F$ is the

-number of iterations and $N$ is the number of computing nodes. The algorithm
-needs from 12 to 20 iterations to select the best vector of frequency scaling
-factors that gives the results of the next sections.
+maximum number of available frequencies, and $N$ is the number of computing
+nodes. The algorithm needs from 12 to 20 iterations to select the best vector of
+frequency scaling factors that gives the results of the next sections.

 
 \begin{table}[!t]
   \caption{Heterogeneous nodes characteristics}
 
 \begin{table}[!t]
   \caption{Heterogeneous nodes characteristics}


@@ -761,7 +762,8 @@ factors that gives the results of the next sections.
 \label{sec.expe}
 
 To evaluate the efficiency and the overall energy consumption reduction of
 \label{sec.expe}
 
 To evaluate the efficiency and the overall energy consumption reduction of

-Algorithm~\ref{HSA}, it was applied to the NAS parallel benchmarks NPB v3.3. The
+Algorithm~\ref{HSA}, it was applied to the NAS parallel benchmarks NPB v3.3 which 
+is composed of synchronous message passing applications. The 

 experiments were executed on the simulator SimGrid/SMPI which offers easy tools
 to create a heterogeneous platform and run message passing applications over it.
 The heterogeneous platform that was used in the experiments, had one core per
 experiments were executed on the simulator SimGrid/SMPI which offers easy tools
 to create a heterogeneous platform and run message passing applications over it.
 The heterogeneous platform that was used in the experiments, had one core per


@@ -790,40 +792,16 @@ The proposed algorithm was applied to the seven parallel NAS benchmarks (EP, CG,
 MG, FT, BT, LU and SP) and the benchmarks were executed with the three classes:
 A, B and C. However, due to the lack of space in this paper, only the results of
 the biggest class, C, are presented while being run on different number of
 MG, FT, BT, LU and SP) and the benchmarks were executed with the three classes:
 A, B and C. However, due to the lack of space in this paper, only the results of
 the biggest class, C, are presented while being run on different number of

-nodes, ranging from 4 to 128 or 144 nodes depending on the benchmark being
+nodes, ranging from 8 to 128 or 144 nodes depending on the benchmark being

 executed. Indeed, the benchmarks CG, MG, LU, EP and FT had to be executed on 1,
 2, 4, 8, 16, 32, 64, or 128 nodes.  The other benchmarks such as BT and SP had
 to be executed on 1, 4, 9, 16, 36, 64, or 144 nodes.
 
 \begin{table}[!t]
 executed. Indeed, the benchmarks CG, MG, LU, EP and FT had to be executed on 1,
 2, 4, 8, 16, 32, 64, or 128 nodes.  The other benchmarks such as BT and SP had
 to be executed on 1, 4, 9, 16, 36, 64, or 144 nodes.
 
 \begin{table}[!t]

-  \caption{Running NAS benchmarks on 4 nodes }
-  % title of Table
-  \centering
-  \begin{tabular}{|*{7}{r|}}
-    \hline
-    \hspace{-2.2084pt}%
-    Program & Execution & Energy        & Energy   & Performance   & Distance \\
-    name    & time/s    & consumption/J & saving\% & degradation\% &          \\
-    \hline
-    CG      &  64.64    &  3560.39      & 34.16    & 6.72          & 27.44    \\
-    \hline
-    MG      &  18.89    &  1074.87      & 35.37    & 4.34          & 31.03    \\
-    \hline
-    EP      &  79.73    &  5521.04      & 26.83    & 3.04          & 23.79    \\
-    \hline
-    LU      & 308.65    & 21126.00      & 34.00    & 6.16          & 27.84    \\
-    \hline
-    BT      & 360.12    & 21505.55      & 35.36    & 8.49          & 26.87    \\
-    \hline
-    SP      & 234.24    & 13572.16      & 35.22    & 5.70          & 29.52    \\
-    \hline
-    FT      &  81.58    &  4151.48      & 35.58    & 0.99          & 34.59    \\
-    \hline
-  \end{tabular}
-  \label{table:res_4n}
+  

 % \end{table}
 
 % \end{table}
 

-  \medskip
+  

 % \begin{table}[!t]
   \caption{Running NAS benchmarks on 8 and 9 nodes }
   % title of Table
 % \begin{table}[!t]
   \caption{Running NAS benchmarks on 8 and 9 nodes }
   % title of Table


@@ -983,13 +961,13 @@ The overall energy consumption was computed for each instance according to the
 energy consumption model (\ref{eq:energy}), with and without applying the
 algorithm. The execution time was also measured for all these experiments. Then,
 the energy saving and performance degradation percentages were computed for each
 energy consumption model (\ref{eq:energy}), with and without applying the
 algorithm. The execution time was also measured for all these experiments. Then,
 the energy saving and performance degradation percentages were computed for each

-instance.  The results are presented in Tables~\ref{table:res_4n},
+instance.  The results are presented in Tables 

 \ref{table:res_8n}, \ref{table:res_16n}, \ref{table:res_32n},
 \ref{table:res_64n} and \ref{table:res_128n}. All these results are the average
 values from many experiments for energy savings and performance degradation.
 The tables show the experimental results for running the NAS parallel benchmarks
 \ref{table:res_8n}, \ref{table:res_16n}, \ref{table:res_32n},
 \ref{table:res_64n} and \ref{table:res_128n}. All these results are the average
 values from many experiments for energy savings and performance degradation.
 The tables show the experimental results for running the NAS parallel benchmarks

-on different number of nodes.  The experiments show that the algorithm
-significantly reduces the energy consumption (up to \np[\%]{35}) and tries to
+on different numbers of nodes.  The experiments show that the algorithm
+significantly reduces the energy consumption (up to \np[\%]{34}) and tries to

 limit the performance degradation.  They also show that the energy saving
 percentage decreases when the number of computing nodes increases.  This
 reduction is due to the increase of the communication times compared to the
 limit the performance degradation.  They also show that the energy saving
 percentage decreases when the number of computing nodes increases.  This
 reduction is due to the increase of the communication times compared to the


@@ -1019,7 +997,7 @@ of the benchmarks MG, LU, BT and FT decrease linearly when the number of nodes
 increase. While for the EP and SP benchmarks, the energy saving percentage is
 not affected by the increase of the number of computing nodes, because in these
 benchmarks there are little or no communications. Finally, the energy saving of
 increase. While for the EP and SP benchmarks, the energy saving percentage is
 not affected by the increase of the number of computing nodes, because in these
 benchmarks there are little or no communications. Finally, the energy saving of

-the GC benchmark significantly decrease when the number of nodes increase
+the CG benchmark significantly decreases when the number of nodes increase

 because this benchmark has more communications than the others. The second plot
 shows that the performance degradation percentages of most of the benchmarks
 decrease when they run on a big number of nodes because they spend more time
 because this benchmark has more communications than the others. The second plot
 shows that the performance degradation percentages of most of the benchmarks
 decrease when they run on a big number of nodes because they spend more time


@@ -1228,7 +1206,7 @@ new energy model for measuring and predicting the energy of distributed
 iterative applications running over heterogeneous platforms. To evaluate the
 proposed method, it was applied on the NAS parallel benchmarks and executed over
 a heterogeneous platform simulated by SimGrid. The results of the experiments
 iterative applications running over heterogeneous platforms. To evaluate the
 proposed method, it was applied on the NAS parallel benchmarks and executed over
 a heterogeneous platform simulated by SimGrid. The results of the experiments

-showed that the algorithm reduces up to \np[\%]{35} the energy consumption of a
+showed that the algorithm reduces up to \np[\%]{34} the energy consumption of a

 message passing iterative method while limiting the degradation of the
 performance. The algorithm also selects different scaling factors according to
 the percentage of the computing and communication times, and according to the
 message passing iterative method while limiting the degradation of the
 performance. The algorithm also selects different scaling factors according to
 the percentage of the computing and communication times, and according to the


@@ -1248,11 +1226,11 @@ the iterative system.
 
 \section*{Acknowledgment}
 
 
 \section*{Acknowledgment}
 

-This work has been partially supported by the Labex
-ACTION project (contract ``ANR-11-LABX-01-01''). As a PhD student,
-Mr. Ahmed Fanfakh, would like to thank the University of
-Babylon (Iraq) for supporting his work.
-
+This work  has been  partially supported by  the Labex ACTION  project (contract
+``ANR-11-LABX-01-01'').  Computations  have been performed  on the supercomputer
+facilities  of the  Mésocentre de  calcul de  Franche-Comté. As  a  PhD student,
+Mr. Ahmed  Fanfakh, would  like to  thank the University  of Babylon  (Iraq) for
+supporting his work.

 
 % trigger a \newpage just before the given reference
 % number - used to balance the columns on the last page
 
 % trigger a \newpage just before the given reference
 % number - used to balance the columns on the last page