adding the first chapter

[ThesisAhmed.git] / CHAPITRE_01.tex

diff --git a/CHAPITRE_01.tex b/CHAPITRE_01.tex
index 76ebf55b90d8c6a54e8b53d3b06d6ab94eb7601c..bc1a5458ff32f1b7306c31b69f2d1714460e2ec5 100644 (file)
--- a/CHAPITRE_01.tex
+++ b/CHAPITRE_01.tex
@@ -11,28 +11,29 @@
 
 \section{Introduction}
 \label{ch1:1}
 
 \section{Introduction}
 \label{ch1:1}

-Traditionally, almost of the software applications are  programmed as a sequential programs according to the Von Neumann report in 1993 \cite{ref50}. The structure of the 
-program code is understandable by the human brain as a series of instructions that execute one after the other. From many years until a short time, the users of the sequential applications had moved their thinking   towards that these applications must run faster with each new generation of microprocessors. This idea is no longer valid nowadays, because the recent release of the microprocessors have many computing units embedded in one chip and these programs are only run over one computing unit sequentially.
-Consequently, the traditional applications not have improved their performance a lot over the new architectures, whereas the new applications run faster over them in parallel. The parallel application is executed over  all the available  computing  units at the same time to improve its performance. Furthermore, the concurrency revolution has been referred to the drastically improvement in the performance of the new applications side by side to  the new parallel architectures \cite{ref51}. Therefore, parallel applications and parallel architectures are closely tied together. It is hard to think about any of  parallel applications without thinking of the parallel hardware that executing them.
-For example, the energy consumption of the parallel system mainly depends on both of the parallel application and the parallel architecture executing this application. Indeed, the energy consumption model or any measurement system depends on many specifications, some of them are concerting  the parallel hardware architecture such as the frequency of the processor, power consumption of the processor and  communication model.  The others are concerting the parallel application such as the computation and communication times of the application. 
+Traditionally, most of the software applications are structured as sequential programs according to the Von Neumann report in 1993 \cite{ref50}. The structure of the program code is understandable by the human brain as a series of instructions that executed one after the other. From many years until a short time,
+with each new generation of microprocessors the users of the sequential applications have believed that these applications run faster over them.
+Nowadays, this idea is no longer valid  because the recent release of the microprocessors have many computing units embedded in one chip and these programs are only run over one computing unit sequentially.
+Consequently,  traditional applications  have not improved their performance a lot over the new architectures, whereas the new applications run faster over them in a parallel. The parallel application is executed over all  available computing units at the same time to improve its performance. Furthermore, the concurrency revolution has been referred to the drastically improvement in the performance of  new applications side by side to  new parallel architectures \cite{ref51}. Therefore, parallel applications and parallel architectures are closely tied together. It is hard to think about any of parallel applications without thinking of the parallel hardware executed them.
+For example, the energy consumption of the parallel system mainly depends on both of the parallel application and the parallel architecture executes this application. Indeed, an energy consumption model or any measurement system depends on many specifications, some of them are concerning  parallel hardware features such as the frequency of the processor, the power consumption of the processor and the communication model.  The others are concerning the parallel application such as the computation and communication times of the application. 

  
  

-In this work, the iterative parallel applications, which is the most popular type of the parallel applications, are interested and running them over different parallel architectures to optimize their energy consumptions is the main goal. 
-As a result, this chapter is aimed to give a brief overview for parallel hardware architectures,  parallel iterative applications and the energy model from the other authors used to measure the energy consumption of these applications. 
+In this work, the iterative parallel applications are interested and running them over different parallel architectures to optimize their energy consumptions is the main goal. 
+As a result, this chapter is aimed to give a brief overview of parallel hardware architectures,  parallel iterative applications and an energy model from the other authors used to measure the energy consumption of these applications. 

 The reminder of this chapter is organized as follows: section \ref{ch1:2}  is  devoted 
 The reminder of this chapter is organized as follows: section \ref{ch1:2}  is  devoted 

-to describe the types of  parallelism and the types of the parallel platforms. It is also gives some information about the parallel programming models.  Section \ref{ch1:3} explains both the synchronous and asynchronous parallel iterative methods and comparing them. Section \ref{ch1:4}, presents the well accepted energy model from the state of the art that can be used to measure the energy consumption of the parallel iterative applications when changing the frequency of the processor. Finally,  section \ref{ch1:5} summaries this chapter.
+to describing  types of  parallelism and  types of  parallel platforms. It also gives some information about  parallel programming models.  Section \ref{ch1:3} explains both of a synchronous and asynchronous parallel iterative methods and comparing them. Section \ref{ch1:4}, presents a well accepted energy model from the state of the art that can be used to measure the energy consumption of parallel iterative applications when changing the frequency of the processor. Finally, section \ref{ch1:5} summaries this chapter.

 
 
 \section{Parallel Computing  Architectures} 
 \label{ch1:2}
 
 
 \section{Parallel Computing  Architectures} 
 \label{ch1:2}

-The process of the simultaneous execution of the calculations is called the parallel computing.
-It has main principle refer to the ability of dividing the large problem into smaller sub-problems that can be solved  at the same time \cite{ref2}. 
-Mainly, solving the sub-problems of the main problem in a parallel computing are carried out on multiple parallel processors.
-Indeed, the parallel processors architecture is a computer system composed of many processing elements connected via network model in addition to the software tools required to make the processing units work together \cite{ref1}.
-Consequently, parallel computing architecture consist of software and hardware resources. 
-The hardware resources are the processing units and memory model in addition to the network system connecting them. The software resources include the specific operating system, the programming language and the compiler, or the runtime libraries. Furthermore, parallel computing can have different levels of parallelism, which can perform in software or hardware. There are five types of parallelism as follows:
+The process of the simultaneous execution of calculations is called the parallel computing.
+Its main principle refer to the ability of dividing the large problem into smaller sub-problems that can be solved  at the same time \cite{ref2}. 
+Mainly, solving  sub-problems of the main problem in a parallel computing are carried out on multiple parallel processors.
+Indeed, the  parallel processors architecture is a computer system composed of many processing elements connected via network model in addition to  software tools required to make the processing units work together \cite{ref1}.
+Consequently, the parallel computing architecture consist of a software and hardware resources. 
+Hardware resources are  processing units and the memory model in addition to the network system connecting them. Software resources include the specific operating system, the programming language and the compiler, or the runtime libraries. Furthermore, the parallel computing can have different levels of parallelism that can be performed in a software or a hardware level. There are five types of parallelism as follows:

 \begin{itemize}
 
 \begin{itemize}
 

-\item \textbf{Bit-level parallelism (BLP)}: The appearance of very-large-scale integration (VLSI) in 1970s has been considered the first approach towards the parallel computing. It is used to increase the number of bits  in word size  being processed by a processor as in the figure~\ref{fig:ch1:1}. For many successive years, the number of bits is increased starting from 4-bit microprocessors reaching until 64 bit microprocessors . For example, the recent  x86-64 architecture becomes the most familiar architecture nowadays. Therefore, the biggest word size gives more parallelism level and  thus less instructions to be executed by the processor at the same time.
+\item \textbf{Bit-level parallelism (BLP)}: The appearance of the very-large-scale integration (VLSI) in 1970s has been considered the first step towards the parallel computing. It is used to increase the number of bits  in the word size  being processed by a processor as in the figure~\ref{fig:ch1:1}. For many successive years, the number of bits have increased starting from 4-bit microprocessors reaching until 64 bit microprocessors. For example, the recent  x86-64 architecture becomes the most familiar architecture nowadays. Therefore, the biggest word size gives more parallelism level and  thus less instructions to be executed by a processor at the same time.

  
 \begin{figure}[h!]
 \centering
  
 \begin{figure}[h!]
 \centering


@@ -41,7 +42,7 @@ The hardware resources are the processing units and memory model in addition to
 \label{fig:ch1:1}
 \end{figure}
 
 \label{fig:ch1:1}
 \end{figure}
 

-\item \textbf{Data-level parallelism (DLP)}:Data parallelism is a process of distributing the data vector between different parallel processors and each one performs the same operations on its data sub-vector. Therefore, many arithmetic operations can be performed on the same data vector in a simultaneous manner.  This type of parallelism can be used in many programs, especially from the area of scientific computing. Usually, data-parallel operations are only provided for arrays operations, for example see figure \ref{fig:ch1:2}. As an example about the applications  of this type of parallelism are vectors multiplication, image and signal processing. 
+\item \textbf{Data-level parallelism (DLP)}: Data parallelism is a process of distributing the data vector between different parallel processors and each one performs same operations on its data sub-vector. Therefore, many arithmetic operations can be performed on the same data vector in a simultaneous manner. This type of parallelism can be used in many programs, especially from the area of scientific computing. Usually, data-parallel operations are only provided for arrays operations, for example see figure \ref{fig:ch1:2}. As an example about the applications use this type of parallelism are vectors multiplication, image and signal processing. 

 
 \begin{figure}[h!]
 \centering
 
 \begin{figure}[h!]
 \centering


@@ -50,9 +51,9 @@ The hardware resources are the processing units and memory model in addition to
 \label{fig:ch1:2}
 \end{figure}
 
 \label{fig:ch1:2}
 \end{figure}
 

-\item \textbf{Instruction-level parallelism (ILP)}: Generally, the sequential program composed of many instructions. These instructions can be executed in a parallel at the same time, if each of them is independent from the others. In particular, parallelism can be achieved in the instruction level by using  pipeline. It means all the independent instructions of the program are overlapped the execution of each  others. For example, if we have two instruction $I_1$ and $I_2$, they are independent if there is no control and data dependency between them.
- In pipeline stages, the execution of each instruction is divided into multiple steps that can be overlapped with the steps of other instructions by the pipeline hardware unit.
-Figure~\ref{fig:ch1:3} demonstrates four instructions each one has four steps denoted as  fetch, decode, execute and write, which are implemented in a hardware units by pipeline. 
+\item \textbf{Instruction-level parallelism (ILP)}: Generally, the sequential program composed of many instructions. These instructions can be executed in a parallel at the same time, if each of them is independent from the others. In particular, the parallelism can be achieved in the instruction level by using  pipeline. It means all the independent instructions of the program are overlapped the execution of each  others. For example, if we have two instructions $I_1$ and $I_2$, they are independent if there is no control and data dependency between them.
+In pipeline stages, the execution of each instruction is divided into multiple steps that can be overlapped with the steps of other instructions by the pipeline hardware unit.
+Figure~\ref{fig:ch1:3} demonstrates four instructions each one has four steps denoted as  fetch, decode, execute and write, which are implemented in hardware units by pipeline. 

 
 \begin{figure}[h!]
 \centering
 
 \begin{figure}[h!]
 \centering


@@ -64,9 +65,9 @@ Figure~\ref{fig:ch1:3} demonstrates four instructions each one has four steps de
 
 
 \item \textbf{Thread-level parallelism (TLP)}: It is also known as a task-level parallelism.
 
 
 \item \textbf{Thread-level parallelism (TLP)}: It is also known as a task-level parallelism.

-According to the Moore’s law \cite{ref9}, the processor can have a number of transistors by a double 
+According to the Moore’s law \cite{ref9}, the processor can have number of transistors by a double 

 each two years to increase the frequency of the processor and thus its performance.  Besides, cache and main memories sizes are must increased together to satisfy this increased. 
 each two years to increase the frequency of the processor and thus its performance.  Besides, cache and main memories sizes are must increased together to satisfy this increased. 

-But, this leads to some limits come from two main reasons, the first one is when the cache size is drastically increased leading to a larger access time. The second  is related to the big increase in the number of the transistors per CPU that can be increased significantly the heat dissipation. As a result, the programmers subdivided their programs into multiple tasks which can be executed in parallel over distributed processors or shared multi-cores processors to improve the performance of the program, see figure~\ref{fig:ch1:4}. Each processor can has a multiple or individual thread dedicated for each task. A thread can be defined as a part of a parallel program which shares processor resources with other threads. 
+But, this leads to some limits come from two main reasons, the first one is when the cache size is drastically increased leading to a larger access time. The second  is related to the big increase in the number of the transistors per CPU that can be increased significantly the heat dissipation. As a result,  programmers subdivided their programs into multiple tasks which can be executed in parallel over distributed processors or shared multi-cores processors to improve the performance of the program, see figure~\ref{fig:ch1:4}. Each processor can has a multiple or an individual thread dedicated for each task. A thread can be defined as a part of the parallel program which shares processor resources with other threads. 

 
 \begin{figure}[h!]
 \centering
 
 \begin{figure}[h!]
 \centering


@@ -76,25 +77,24 @@ But, this leads to some limits come from two main reasons, the first one is when
 \end{figure}
 
 Therefore, we can consider the execution time of a sequential program composed of 
 \end{figure}
 
 Therefore, we can consider the execution time of a sequential program composed of 

-$N$ tasks as the sum of the execution times of all tasks as follows:
+$N$ tasks as sum of the execution times of all tasks as follows:

 
 \begin{equation}
 \label{ch1:eq1}
     Sequential~execution~time = \sum_{i=1}^{N} T_i
 \end{equation}
 
 
 \begin{equation}
 \label{ch1:eq1}
     Sequential~execution~time = \sum_{i=1}^{N} T_i
 \end{equation}
 

-Whereas, if the tasks are executed synchronously over multiple processing units in parallel, the execution time of the program is the execution time of the task that has  maximum execution time (the slowest task) as follows:
+Whereas, if tasks are executed synchronously over multiple processing units in a parallel, the execution time of the program is the execution time of the task that has  maximum execution time (the slowest task) as follows:

 
 

- 

 \begin{equation}
 \label{ch1:eq2}
     Parallel~execution~time = \max_{i=1,\dots,N} T_i
 \end{equation}
 
 \begin{equation}
 \label{ch1:eq2}
     Parallel~execution~time = \max_{i=1,\dots,N} T_i
 \end{equation}
 

-

 \item \textbf{Loop-level parallelism (LLP)}:
 \item \textbf{Loop-level parallelism (LLP)}:

-The numerical algorithms and many other algorithms are executed iteratively the same program portion, the computations, using different forms of the loop statements allowed in the programming languages. At each iteration,  the program need to scan a large data structure such as an array structure to make the arithmetic calculations. Inside the loop structure there are many instructions, which are independent or dependent. In a sequential loop execution the $i$ iteration must be executed after the completion of $(i-1)$ iteration. 
-While, if each iteration is independent from the others, then all the iterations are distributed over many  processors to be executed in a parallel,
+The numerical algorithms and many other algorithms are executed iteratively the same program portion,  computation, using different forms of the loop statements allowed in the programming languages. At each iteration, the program need to scan a large data structure such as an array structure to make the arithmetic calculations. Inside the loop structure there are many instructions, which are independent or dependent. In a sequential loop execution the $i$ iteration must be executed after the completion of 
+$(i-1)$ iteration. 
+Whereas, if each iteration is independent from the others, then all the iterations are distributed over many  processors to be executed in a parallel,

 for example see figure\ref{fig:ch1:5}. In the parallel programming languages this type of a loop is  called $parallel~loop$.
 
 \begin{figure}[h!]
 for example see figure\ref{fig:ch1:5}. In the parallel programming languages this type of a loop is  called $parallel~loop$.
 
 \begin{figure}[h!]


@@ -135,10 +135,10 @@ streams. In 1966, Michel Flynn has been proposed a simple model of categorizing
 \end{figure}
  
 \item \textbf{Single instruction, multiple data (SIMD) stream}: All the processors execute the same instructions on different data. 
 \end{figure}
  
 \item \textbf{Single instruction, multiple data (SIMD) stream}: All the processors execute the same instructions on different data. 

-Each processor stores the data in its local memory, the processor communicates with each others typically via simple communication model, see figure \ref{fig:ch1:7}. Many scientific and engineering
+Each processor stores the data in its local memory, the processor communicates with each others typically via a simple communication model, see figure \ref{fig:ch1:7}. Many scientific and engineering

 applications are suitable to this type of parallel scheme.
 Vector and array processors are well know  examples of this type. 
 applications are suitable to this type of parallel scheme.
 Vector and array processors are well know  examples of this type. 

-As an example about the applications executed over this architecture are the graphics processing, video compression and medical image analysis applications.
+As an example about the applications executed over this architecture are  graphics processing, video compression and medical image analysis applications.

 
 \begin{figure}[h!]
 \centering
 
 \begin{figure}[h!]
 \centering


@@ -157,10 +157,10 @@ As an example about the applications executed over this architecture are the gra
 \end{figure}
 
 
 \end{figure}
 
 

-\item \textbf{Multiple instruction, Multiple data (MIMD) stream}: There are multiple processing elements each of which has a separate instruction  and data local memories.
-At any time, different processing elements may be executing different instructions on different data fragment, see figure \ref{fig:ch1:9}. There are two types of the MIMD machines: the share memory and massage passing MIMD machines. 
-In the share memory architectures, a processors are communicated via a share memory model, while in the message passing architecture each processor has its own local memory and all processors communicate via communication network model. The  multi-core processors, local
-clusters and grid systems are an examples for the MIMD machine.
+\item \textbf{Multiple instruction, Multiple data (MIMD) stream}: There are multiple processing elements each of which has a separate instructions  and  local data memories.
+At any time, different processing elements may be executing different instructions on different data fragment, see figure \ref{fig:ch1:9}. There are two types of MIMD machines: the share memory and the massage passing MIMD machines. 
+In the share memory architectures, a processors are communicated via a share memory model, while in the message passing architecture each processor has its own local memory and all processors communicate via a communication network model. The  multi-core processors, local
+clusters and grid systems are an examples for  MIMD machine.

 Many applications have been conducted over this architecture 
 such as computer-aided design, computer-aided manufacturing, simulation, modeling, iterative applications and so on.
 
 Many applications have been conducted over this architecture 
 such as computer-aided design, computer-aided manufacturing, simulation, modeling, iterative applications and so on.
 


@@ -175,13 +175,13 @@ such as computer-aided design, computer-aided manufacturing, simulation, modelin
 
 The work of this thesis is dedicated to MIMD machines architecture. Therefore, we discuss in
 this chapter some of the commonly used parallel architectures that belong to MIMD machines.
 
 The work of this thesis is dedicated to MIMD machines architecture. Therefore, we discuss in
 this chapter some of the commonly used parallel architectures that belong to MIMD machines.

-As explained before, the MIMD architectures can be classified into two types, the shared memory and the distributed message passing ones. Furthermore, these classifications are based on 
+As explained before,  MIMD architectures can be classified into two types, the shared memory and the distributed message passing ones. Furthermore, these classifications are based on 

 how MIMD processors access the memory model. The shared MIMD machines communication topology can be bus-based, extended or hierarchical type. Whereas, the distributed memory MIMD machines may have hypercube or mesh inter connected networks. In the following are some well known MIMD parallel computing platforms:
 
 \begin{itemize}
 \item \textbf{Multi-core processors}:
 The multi-core processor is a single chip component with two or more processing units.
 how MIMD processors access the memory model. The shared MIMD machines communication topology can be bus-based, extended or hierarchical type. Whereas, the distributed memory MIMD machines may have hypercube or mesh inter connected networks. In the following are some well known MIMD parallel computing platforms:
 
 \begin{itemize}
 \item \textbf{Multi-core processors}:
 The multi-core processor is a single chip component with two or more processing units.

-These processing units are called cores, which connected with each other via main memory model as in the figure \ref{fig:ch1:10}. Each individual core has its cache memory to store its data and execute different data or instructions stream in parallel. Moreover, each core can have  one or more threads to execute a specific programming task as shown in the thread-level parallelism. Historically, the multi-cores of the CPU began as two-core processors, with increase in the number of cores approximately by double with each semiconductor process generation \cite{ref12}. The very quick improvements in the performance and thus the increase in the number of cores  is devoted in graphical processing unit (GPU). A current exemplar of GPU  is the NVIDIA  GeForce TITAN Z with 5700 cores in 2015 \cite{ref17}. While the general-purpose microprocessors (CPU) has less number of the cores, for example the TILE-MX processor from Tilera has 100 cores in the same year \cite{ref16}.
+These processing units are called cores, which connected with each other via main memory model as in the figure \ref{fig:ch1:10}. Each individual core has its cache memory to store its data and execute different data or instructions stream in parallel. Moreover, each core can have  one or more threads to execute a specific programming task as shown in the thread-level parallelism. Historically, the multi-cores of the CPU began as two-core processors, with increase in the number of cores approximately by double with each semiconductor process generation \cite{ref12}. The very quick improvements in the performance and thus the increase in the number of cores  is devoted in the graphical processing unit (GPU). A current exemplar of GPU  is the NVIDIA  GeForce TITAN Z with 5700 cores in year of 2015 \cite{ref17}. While the general-purpose microprocessors (CPU) has less number of the cores, for example the TILE-MX processor from Tilera had 100 cores in the same year \cite{ref16}.

 For more details about the multi-core processors see \cite{ref15}.
 
 \begin{figure}[h!]
 For more details about the multi-core processors see \cite{ref15}.
 
 \begin{figure}[h!]


@@ -207,7 +207,7 @@ local area networks (LAN) with low latency and big bandwidth. Moreover, each nod
 
 \item \textbf{Grid (Distributed clusters)}:
 Grid is a collection of local computing clusters from different sites connected via wide area network (WAN), which can be appeared virtually to the benefit users as a complete computing system  \cite{ref20}. 
 
 \item \textbf{Grid (Distributed clusters)}:
 Grid is a collection of local computing clusters from different sites connected via wide area network (WAN), which can be appeared virtually to the benefit users as a complete computing system  \cite{ref20}. 

-In particular, different local clusters composing the grid are geographically far away from each others. Usually, each local cluster composed from homogeneous nodes, which are different from the nodes of the others cluster located in different sites. These nodes can be different in the hardware and software  specifications such as the computing power, memory, operating system, local network latency and bandwidth. Figure \ref{fig:ch1:12} presents an example of the grid composed of three heterogeneous local clusters located in  different sites which are connected throw wide area network.  Furthermore, the grid can be referred to an infrastructure  applies the integration and the collaboration by using a collection  of different computers, networks, databases servers  and scientific devices belong to  many companies and universities. Therefore, wide heterogeneous computing resources are allowed to many users simultaneously. While the only bottleneck of the grid is the high latency communications between the nodes from different sites. The grid is also called the loosely-coupled fashion platform. However, the fault tolerance  is required to guarantee the process of sending and receiving  the messages between the computing nodes and thus keeps all the messages from the lost.
+In particular, different local clusters composing the grid are geographically faraway from each others. Usually, each local cluster composed from homogeneous nodes, which are different from the nodes of the others cluster located in different sites. These nodes can be different in the hardware and software  specifications such as the computing power, memory size, operating system, local network latency and bandwidth. Figure \ref{fig:ch1:12} presents an example of the grid composed of three heterogeneous local clusters located in  different sites which are connected throw a wide area network.  Furthermore, the grid can be referred to an infrastructure  applies the integration and the collaboration by using  collection  of different computers, networks, databases servers  and scientific devices belong to  many companies and universities. Therefore, wide heterogeneous computing resources are allowed to many users simultaneously. While the only bottleneck of the grid is the high latency communications between the nodes from different sites. The grid is also called the loosely-coupled fashion platform. However, the fault tolerance  is required to guarantee the process of sending and receiving  the messages between the computing nodes and thus keeps all the messages from the lost.

 
 \begin{figure}[h!]
 \centering
 
 \begin{figure}[h!]
 \centering


@@ -276,11 +276,11 @@ some examples for each type of the parallel programming models:
                       send and receive messages, but it allowed many others collective 
                       operations such as gathering and reduction operations. MPI user feel
                       free form the network topology, synchronization and communication
                       send and receive messages, but it allowed many others collective 
                       operations such as gathering and reduction operations. MPI user feel
                       free form the network topology, synchronization and communication

-                      functionality between  group of processes. Furthermore, it has 
+                      functionality between  group of processes. Furthermore, it has an

                       asynchronous point to point operations, which make the computations 
                       to overlap with communications. While MPI is not devoted to a grid,
                        \textbf{MPICH}  is one of the most 
                       asynchronous point to point operations, which make the computations 
                       to overlap with communications. While MPI is not devoted to a grid,
                        \textbf{MPICH}  is one of the most 

-                      popular implementations of MPI dedicated for grid computing. It is used               
+                      popular implementations of MPI dedicated for grid computing. It uses

                       as an extended version for MPI, which implements a fault tolerance  
                       \cite{ref52}. In this work, both of MPI and MPICH programming libraries 
                       have used for programming our algorithms and applications which called 
                       as an extended version for MPI, which implements a fault tolerance  
                       \cite{ref52}. In this work, both of MPI and MPICH programming libraries 
                       have used for programming our algorithms and applications which called 


@@ -293,25 +293,24 @@ some examples for each type of the parallel programming models:
                       distributed and heterogeneous in the operating system environments. 
                       The PVM system is elementarily for parallel programming to be used with 
                       C, C++, and Fortran  languages. 
                       distributed and heterogeneous in the operating system environments. 
                       The PVM system is elementarily for parallel programming to be used with 
                       C, C++, and Fortran  languages. 

-                      It is considered more robust in fault tolerance 
-                      than MPI, easier to add or delete the crashed nodes in the host pool 
+                      It is considered more robust in fault tolerance than MPI, easier to 
+                      add or delete the crashed nodes in the host pool 

                       \cite{ref26}. While MPI has more communication messages support and asynchronous 
                       operations which are not allowed in PVM. 
                       
                       
    \item \textbf{BLACS} \cite{ref27} is for Basic Linear Algebra Communication Subprograms.                 
                       \cite{ref26}. While MPI has more communication messages support and asynchronous 
                       operations which are not allowed in PVM. 
                       
                       
    \item \textbf{BLACS} \cite{ref27} is for Basic Linear Algebra Communication Subprograms.                 

-                 It has a collection of libraries that used to built linear algebra messages 
+                 It has a collection of libraries that used to built a linear algebra messages 

                  communication model which is applied effectively over distributed memory architectures.
                  communication model which is applied effectively over distributed memory architectures.

-                 The primary goal of using 
-                 BLACS is mapping a liner set or processors or any distributed machines into
-                 a two dimensional array or grid, which offers an easy tool for building a
+                 The primary goal of using BLACS is mapping a liner set or processors or any distributed 
+                 machines into two dimensional array or grid, which offers an easy tool for building a

                  linear algebra applications. 
                  
   \end{itemize} 
   
 \item \textbf{Grid programming models}
   \begin{itemize}
                  linear algebra applications. 
                  
   \end{itemize} 
   
 \item \textbf{Grid programming models}
   \begin{itemize}

-   \item \textbf{Gridsolve} \cite{ref28}  is the first middleware for grid  and 
+   \item \textbf{Gridsolve} \cite{ref28}  is the first middleware for a grid  and the

                 high performance computing that offers a good tool to solve a complex
                 scientific applications using distinct distributed machines. It applies the
                 fault tolerance and load balancing features to ensure the reliability of the
                 high performance computing that offers a good tool to solve a complex
                 scientific applications using distinct distributed machines. It applies the
                 fault tolerance and load balancing features to ensure the reliability of the


@@ -319,8 +318,8 @@ some examples for each type of the parallel programming models:
                 It works with different programming languages such as C,C++, Java and Fortran. 
                 
    \item \textbf{GLOBAS} \cite{ref29,ref30} is the most widely standardization tool kit
                 It works with different programming languages such as C,C++, Java and Fortran. 
                 
    \item \textbf{GLOBAS} \cite{ref29,ref30} is the most widely standardization tool kit

-                for grid computing. It permits the users to share their computing resources securely. 
-                While the GLOBAS toolkit is allowed to work with grid, it offers a fault 
+                for a grid computing. It permits the users to share their computing resources securely. 
+                While the GLOBAS toolkit is allowed to work with a grid, it offers a fault 

                 detection mechanism to ensure the delivery of the messages. 
                 The first version of Globus toolkit appeared
                 in 1998 and now the sixth version is available \cite{ref31}. 
                 detection mechanism to ensure the delivery of the messages. 
                 The first version of Globus toolkit appeared
                 in 1998 and now the sixth version is available \cite{ref31}. 


@@ -337,21 +336,21 @@ some examples for each type of the parallel programming models:
  
 \item \textbf{Multi-core CPU programming models} 
   \begin{itemize}
  
 \item \textbf{Multi-core CPU programming models} 
   \begin{itemize}

-   \item \textbf{OpenMP}  \cite{ref34} is parallel programming tool for shared memory 
+   \item \textbf{OpenMP}  \cite{ref34} is a  parallel programming tool for shared memory 

                 architectures. The main goal of using this programming model is to provide 
                 a standard and portable API (application programming interface) for writing 
                 shared memory parallel programs. It can be used with programming languages such 
                 as C, C++ and Fortran to support different types of shared memory platforms 
                 such as multi-core processors.
                 OpenMP uses multi-threading, which is a method of parallel programming 
                 architectures. The main goal of using this programming model is to provide 
                 a standard and portable API (application programming interface) for writing 
                 shared memory parallel programs. It can be used with programming languages such 
                 as C, C++ and Fortran to support different types of shared memory platforms 
                 such as multi-core processors.
                 OpenMP uses multi-threading, which is a method of parallel programming 

-                that organized by using master thread to control a set of slave threads. Each
-                thread can be executed in parallel by allocating it to a processor.  
+                that uses a master thread to control set of slave threads. Each
+                thread can be executed in a parallel by allocating it to a processor.  

                 Moreover, OpenMP can be  used with MPI to support hybrid platforms that have 
                 Moreover, OpenMP can be  used with MPI to support hybrid platforms that have 

-                shared and distributed memory models in the same time.
+                shared and distributed memory models at the same time.

                 
   \item \textbf{Cilk} \cite{ref13,ref35} is a linguistic and runtime technology for algorithmic 
                 multi-threaded programming originally developed at MIT. 
                 
   \item \textbf{Cilk} \cite{ref13,ref35} is a linguistic and runtime technology for algorithmic 
                 multi-threaded programming originally developed at MIT. 

-                It is allowed the programmer to focus on building the program in a structural way 
+                It allows the programmer to focus on building the program in a structured way 

                 to discover the inherent parallelism. Many specifications are used in Cilk 
                 such as the load balancing, synchronization and communication protocols. 
                 
                 to discover the inherent parallelism. Many specifications are used in Cilk 
                 such as the load balancing, synchronization and communication protocols. 
                 


@@ -362,14 +361,14 @@ some examples for each type of the parallel programming models:
    \item \textbf{TBB} \cite{ref36} is for Threading Building Blocks, is a software library used with 
                 C++ programming language for multi-core parallel programming developed by Intel.
                 It woks on the principle of dividing the computations into many tasks that can be 
    \item \textbf{TBB} \cite{ref36} is for Threading Building Blocks, is a software library used with 
                 C++ programming language for multi-core parallel programming developed by Intel.
                 It woks on the principle of dividing the computations into many tasks that can be 

-                executed in  
-                parallel. It also has a management library to schedule the parallel task execution. 
+                executed in a parallel. 
+                It also has a management library to schedule the parallel task execution. 

                 The difference between OpenMP and TBB, is the latter uses a task-based scheduling 
                 mechanism. Furthermore, TBB is more popular with C++ programming language than 
                 others languages. It is designed to work with any compiler environments, and thus 
                 The difference between OpenMP and TBB, is the latter uses a task-based scheduling 
                 mechanism. Furthermore, TBB is more popular with C++ programming language than 
                 others languages. It is designed to work with any compiler environments, and thus 

-                it is easily ported to a new platform. Hence, TBB has been ported to a 
+                it is easily ported to a new platform. Hence, TBB has been ported to 

                 different types of operating systems and processors. While, it has limited
                 different types of operating systems and processors. While, it has limited

-                support to vector processing architecture and then it connected with OpenMP  
+                support to vector processing architecture and then it is connected with OpenMP  

                 and Cilk to support this  platform.
   \end{itemize}
   
                 and Cilk to support this  platform.
   \end{itemize}
   


@@ -377,13 +376,13 @@ some examples for each type of the parallel programming models:
   
 \item \textbf{GPU programming models} 
   \begin{itemize}
   
 \item \textbf{GPU programming models} 
   \begin{itemize}

-   \item \textbf{CUDA} \cite{ref37} Modern graphical processing units (GPUs) have been increasing  chip-level  
+   \item \textbf{CUDA} \cite{ref37} Modern graphical processing units (GPUs) have increased its chip-level  

                        parallelism.  Current  NVIDIA  GPUs  are  many-cores processor having thousands
                        of core.  According to this massively cores parallelism, the NVIDIA in 2007 developed
                        a parallel programming  language called CUDA , which is for Compute Unified Device  
                        Architecture.  A CUDA program has two parts, the first one is called a host which is a
                        parallelism.  Current  NVIDIA  GPUs  are  many-cores processor having thousands
                        of core.  According to this massively cores parallelism, the NVIDIA in 2007 developed
                        a parallel programming  language called CUDA , which is for Compute Unified Device  
                        Architecture.  A CUDA program has two parts, the first one is called a host which is a

-                       set of threads that execute sequentially over the CPU. The second part is called the
-                       kernels, which are a set of threads that can be executed in parallel over the GPU.
+                       set of threads that executed sequentially over the CPU. The second part is called the
+                       kernels, which are a set of threads that can be executed in a parallel over the GPUs.

  
    \item \textbf{OpenCL}\cite{ref38} is for Open Computing Language. It is a parallel 
                        programming language dedicated for heterogeneous platform composed 
  
    \item \textbf{OpenCL}\cite{ref38} is for Open Computing Language. It is a parallel 
                        programming language dedicated for heterogeneous platform composed 


@@ -396,12 +395,12 @@ some examples for each type of the parallel programming models:
                          
    \item \textbf{HLSL} \cite{ref39} is for High Level Shading Language, is the shader 
                        programming language for Direct3D, which is a part of
                          
    \item \textbf{HLSL} \cite{ref39} is for High Level Shading Language, is the shader 
                        programming language for Direct3D, which is a part of

-                       Microsoft’s DirectX API. It supports the shader construction with 
-                       C-like syntax, types, expressions, statements, and functions and it 
+                       Microsoft’s DirectX API. It supports the shader design with 
+                       C language syntax, types, expressions, statements, and functions and it 

                        provides a graphical pipeline parallelism.
                        provides a graphical pipeline parallelism.

-                       The last version of HLSL is v5.0 for DirectX 11, which adds a new general-purpose GPU
-                       functions like CUDA. Recently, the new OpenCL version starts to replace CUDA 
-                       as a multi-platform GPU language.
+                       The last version of HLSL is version 5.0 for DirectX 11, which adds a new 
+                       general-purpose GPU functions like CUDA. Recently, the new OpenCL 
+                       version starts to replace CUDA as a multi-platform GPU language.

                                    
   \end{itemize}
  
                                    
   \end{itemize}
  


@@ -630,7 +629,7 @@ iterative methods for solving different problems over local homogeneous cluster,
 \label{ch1:4}
 
 Many researchers~\cite{ref46,ref47,ref48,ref49} divide the power consumed by a processor into
 \label{ch1:4}
 
 Many researchers~\cite{ref46,ref47,ref48,ref49} divide the power consumed by a processor into

-two power metrics: the static and the dynamic power.  While the first one is
+two power metrics: the static and the dynamic power.  The first one is

 consumed as long as the computing unit is on, the latter is only consumed during
 computation times.  The dynamic power $P_{dyn}$ is related to the switching
 activity $\alpha$, load capacitance $C_L$, the supply voltage $V$ and
 consumed as long as the computing unit is on, the latter is only consumed during
 computation times.  The dynamic power $P_{dyn}$ is related to the switching
 activity $\alpha$, load capacitance $C_L$, the supply voltage $V$ and


@@ -765,14 +764,15 @@ There are two drawbacks of this energy model as follows:
 \end{itemize}
 
 Therefore, one of the more important goals of this work is to develop an energy models that 
 \end{itemize}
 
 Therefore, one of the more important goals of this work is to develop an energy models that 

-take into account the communication times in addition to computation times to modelize and measure the energy consumptions of the parallel iterative methods. These models are dedicated to all parallel architectures such as the homogeneous and heterogeneous platforms, which may be local or distributed computing clusters.  
+has be taken into consideration the communication times in addition to computation times to modelize and measure the energy consumptions of the parallel iterative methods. These models are dedicated to all parallel architectures such as the homogeneous and heterogeneous platforms, which may be local or distributed computing clusters.  

 
 \section{Conclusion}
 \label{ch1:5}
 
 \section{Conclusion}
 \label{ch1:5}

-In this chapter, we have presented three sections for describing the parallel hardware architectures, parallel iterative applications and the energy consumption model used to measure the energies of these applications. 
-In the first section, different types of parallelism levels that can be implemented in  software and hardware techniques have explained. Furthermore, the types of the parallel architectures are demonstrated and classified according to how the computing units are connected to a memory model.  
-Both of the shared and distributed platforms are demonstrated and depending on them we have categorized the parallel programming models. 
-In the second section, we have described the two types of parallel iterative methods, the synchronous and asynchronous iterative methods. The synchronous iterative methods are well implemented over local homogeneous cluster with a high speed network link, while the asynchronous iterative methods are more conventional to implement over the distributed heterogeneous clusters. 
-Finally in the third section, the energy consumption model used for measuring the energy consumption of the parallel applications from the related literature is  described. This model  cannot be used for all types of parallel architectures. Indeed, it assumes  measuring the dynamic power during both of the communication and computation times, while the processor involved remains idle during the communication times and  only consumes the static power. Moreover, it is not well adapted to the heterogeneous architectures when there are different types of the processors, which are consumed different dynamic and static powers.
-
-However, in the next chapters of this thesis a new energy consumption models are developed, which they use for modeling and measuring the energy consumptions by a parallel iterative methods running on both homogeneous and heterogeneous architectures.  Furthermore, these energies model use in a methods for optimizing  both of the energy consumption and the performance of the iterative message passing applications. 
\ No newline at end of file
+In this chapter, three sections have been presented for describing the parallel hardware architectures, parallel iterative applications and the energy consumption model used to measure the energies of these applications. 
+In the first section, different types of parallelism levels that can be implemented in a software and hardware techniques have explained. Furthermore, the types of the parallel architectures are demonstrated and classified according to how the computing units are connected to a memory model.  
+Both of the shared and distributed platforms are demonstrated and depending on them the parallel programming models have categorized. 
+In the second section, the two types of parallel iterative methods are described as synchronous and asynchronous iterative methods. The synchronous iterative methods are well implemented over local homogeneous cluster with a high speed network link, while the asynchronous iterative methods are more conventional to implement over the distributed heterogeneous clusters. 
+Finally in the third section, the energy consumption model used for measuring the energy consumption of the parallel applications from the related literature is  described. This model  cannot be used for all types of parallel architectures. Indeed, it assumes  measuring the dynamic power during both of the communication and computation times, while the processor involved remains idle during the communication times and  only consumes the static power. Moreover, it is not well adapted to  heterogeneous architectures when there are different types of the processors, which are consumed different dynamic and static powers at the same time.
+
+However, in the next chapters of this thesis a new energy consumption models are developed, and how these
+energy models are used for modeling and measuring the energy consumptions by parallel iterative methods running on both homogeneous and heterogeneous architectures.  Furthermore, these energy models use in a methods for optimizing  both of the energy consumption and the performance of the iterative message passing applications. 
\ No newline at end of file