+\subsection{The experimental results of multicores clusters}
+\label{sec.res-mc}
+The grid'5000 clusters have different number of cores embedded in their nodes
+as in the Table \ref{table:grid5000}. Moreover, the cores of each node are
+connected via shared memory model, the data transfer between cores' local
+memories achieved via the global memory \cite{rauber_book}. Therefore, in
+this section the proposed scaling algorithm is implemented over the grid'5000
+clusters which are included multicores in the selected nodes as same as the
+two previous platform scenarios that mentioned in the section \ref{sec.res}.
+The two platform scenarios, the two sites and one site scenarios, with 32
+nodes are reconfigured to used multicores for each node. For example if
+the participating number of nodes from a certain cluster is equal to 12 nodes,
+in the multicores scenario the selected nodes is equal to 3 nodes with using
+4 cores for each of them to produced 12 cores. These scenarios with one
+core and multicores are demonstrated in Table \ref{table:sen-mc}.
+The energy consumptions and execution times of running the NAS parallel
+benchmarks, class D, over these four different scenarios are represented
+in the figures \ref{fig:eng-cons-mc} and \ref{fig:time-mc} respectively.
+The execution times of NAS benchmarks over the one site multicores scenario
+is higher than the execution time of those running over one site multicores scenario.
+This because in the one site multicores scenario the communication is increased significantly,
+and all node's cores share the same node network link which increased
+the communication times. While, the execution times of the NAS benchmarks over
+the two site multicores scenario is less than those executed over the two
+sites one core scenario. This because using multicores decrease the communications,
+while the cores shared same nodes' link but the communications between the cores
+are less than the communication times between the nodes over the long distance
+networks, and thus the over all execution time decreased. Generally, executing
+the NAS benchmarks over the one site one core gives smaller execution times
+comparing to other scenarios. This because each node in this scenario has it's
+dedicated network link that used independently by one core, while in the other
+scenarios the communication times are higher when using long distance communication
+link or using the shared link communications between cores of each node.
+On the other hand, the energy consumptions of the NAS benchmarks over the
+one site one cores is less than the one site multicores scenario because
+this scenario had less execution time as mentioned before. Also, in the
+one site one core scenario the computations to communications ratio is
+higher, then the new scaled frequencies are decreased the dynamic energy
+consumption, because the dynamic power consumption are decreased exponentially
+with the new frequency scaling factors. These experiments also showed, the energy
+consumption and the execution times of EP and MG benchmarks over these four
+scenarios are not change a lot, because there are no or small communications
+ which are increase or decrease the static power consumptions.
+The other benchmarks were showed that their energy consumptions and execution times
+are changed according to the decreasing or increasing in the communication
+times that are different from scenario to other or due to the amount of
+communications in each of them.
+
+The energy saving percentages of all NAS benchmarks, as in figure
+\ref{fig:eng-s-mc}, running over these four scenarios are presented. The figure
+showed the energy saving percentages of NAS benchmarks over two sites multicores scenario is higher
+than two sites once core scenario, this because the the computation
+times in the two sites multicores scenario is higher than the computation times
+of the two sites one core scenario, then the more reduction in the
+dynamic energy can be obtained as mentioned previously. In contrast, in the one site one
+core and one site multicores scenarios the energy saving percentages
+are approximately equivalent, on average they are up to 25\%. This
+because in the both scenarios there are a small difference in the
+computations to communications ratio, leading the proposed scaling algorithm
+to selects the frequencies proportionally to these ratios and keeping
+as much as possible the energy saving percentages the same. The
+performance degradation percentages of NAS benchmarks are presented in
+figure \ref{fig:per-d-mc}. This figure indicates that performance
+degradation percentages of running NAS benchmarks over two sites
+multocores, on average is equal to 7\%, gives more performance degradation percentage
+than two sites one core scenario, which on average is equal to 4\%.
+This because when using the two sites multicores scenario increased
+the computations to communications ratio, which may be increased the effect
+on the overall execution time when the proposed scaling algorithm is applied and scaling down the frequencies.
+The inverse was happened when the benchmarks are executed over one
+site one core scenario their performance degradation percentages, on average
+is equal to 10\%, are higher than those executed over one sit one core,
+which on average is equal to 7\%. This because in one site
+multicores scenario the computations to communications ratio is decreased
+as mentioned before, thus selecting new frequencies are less effect
+on the overall execution time. The tradeoff distances of all NAS
+benchmarks over all scenarios are presented in the figure \ref{fig:dist-mc}.
+These tradeoff distances are used to verified which scenario is the best in term of
+energy and performance ratio. The one sites multicores scenario is the best scenario in term of
+energy and performance tradeoff, on average is equal to 17.6\%, when comparing to the one site one core
+scenario, one average is equal to 15.3\%. This because the one site multicores scenario
+has the same energy saving percentages of the one site one core scenario but
+with less performance degradation. The two sites multicores scenario is gives better
+energy and performance tradeoff, one average is equal to 14.7\%, than the two sites
+one core, on average is equal to 13.3\%.
+Finally, using multicore in both scenarios increased the energy and performance tradeoff
+distance. This is because using multicores are increased the computations to communications
+ratio in two sites scenario and thus the energy saving increased over the performance degradation, whereas decreased this ratio
+in one site scenario causing the performance degradation decreased over the energy saving.
+
+
+
+
+
+\begin{table}[]
+\centering
+\caption{The multicores scenarios}
+
+\begin{tabular}{|*{4}{c|}}
+\hline
+Scenario name & Cluster name & \begin{tabular}[c]{@{}c@{}}No. of nodes\\ in each cluster\end{tabular} &
+ \begin{tabular}[c]{@{}c@{}}No. of cores\\ for each node\end{tabular} \\ \hline
+\multirow{3}{*}{Two sites/ one core} & Taurus & 10 & 1 \\ \cline{2-4}
+ & Graphene & 10 & 1 \\ \cline{2-4}
+ & Griffon & 12 & 1 \\ \hline
+\multirow{3}{*}{Two sites/ multicores} & Taurus & 3 & 3 or 4 \\ \cline{2-4}
+ & Graphene & 3 & 3 or 4 \\ \cline{2-4}
+ & Griffon & 3 & 4 \\ \hline
+\multirow{3}{*}{One site/ one core} & Graphite & 4 & 1 \\ \cline{2-4}
+ & Graphene & 12 & 1 \\ \cline{2-4}
+ & Griffon & 12 & 1 \\ \hline
+\multirow{3}{*}{One site/ multicores} & Graphite & 3 & 3 or 4 \\ \cline{2-4}
+ & Graphene & 3 & 3 or 4 \\ \cline{2-4}
+ & Griffon & 3 & 4 \\ \hline
+\end{tabular}
+\label{table:sen-mc}
+\end{table}
+
+\begin{figure}
+ \centering
+ \includegraphics[scale=0.5]{fig/eng_con.eps}
+ \caption{Comparing the energy consumptions of running NAS benchmarks over one core and multicores scenarios }
+ \label{fig:eng-cons-mc}
+\end{figure}
+
+
+ \begin{figure}
+ \centering
+ \includegraphics[scale=0.5]{fig/time.eps}
+ \caption{Comparing the execution times of running NAS benchmarks over one core and multicores scenarios }
+ \label{fig:time-mc}
+\end{figure}
+
+ \begin{figure}
+ \centering
+ \includegraphics[scale=0.5]{fig/eng_s_mc.eps}
+ \caption{The energy saving of running NAS benchmarks over one core and multicores scenarios }
+ \label{fig:eng-s-mc}
+\end{figure}
+
+\begin{figure}
+ \centering
+ \includegraphics[scale=0.5]{fig/per_d_mc.eps}
+ \caption{The performance degradation of running NAS benchmarks over one core and multicores scenarios }
+ \label{fig:per-d-mc}
+\end{figure}
+
+\begin{figure}
+ \centering
+ \includegraphics[scale=0.5]{fig/dist_mc.eps}
+ \caption{The tradeoff distance of running NAS benchmarks over one core and multicores scenarios }
+ \label{fig:dist-mc}
+\end{figure}
