\label{chapter9}
\section{Introduction}
This chapter presents GPU-based parallel
-metaheuristics\index{Metaheuristics!parallel~metaheuristics},
+metaheuristics\index{metaheuristics!parallel metaheuristics},
challenges, and issues related to the particularities of the GPU
architecture and a synthesis on the different implementation
strategies used in the literature. The implementation of parallel
-metaheuristics\index{Metaheuristics!parallel~metaheuristics} on
+metaheuristics on
GPUs is not straightforward. The traditional models used in CPUs
must be rethought to meet the new requirements of GPU
architectures. This chapter is organized as follows. Combinatorial
-optimization\index{Combinatorial~optimization} and resolution
+optimization\index{combinatorial optimization} and resolution
methods are introduced in Section~\ref{ch8:sec:optim}. The main
traditional parallel models used for metaheuristics are recalled
in Section~\ref{ch8:sec:paraMeta}.
Section~\ref{ch8:sec:challenges} highlights the main challenges
related to the GPU implementation of metaheuristics. A
state-of-the-art of GPU-based parallel
-metaheuristics\index{Metaheuristics!parallel~metaheuristics} is
+metaheuristics is
summarized in Section~\ref{ch8:sec:state}. In Section~\ref{ch8:sec:frameworks}, the main developed GPU-based
frameworks for metaheuristics are described. Finally, a case study
is presented in Section~\ref{ch8:sec:case} and some concluding
\section{Combinatorial optimization}
\label{ch8:sec:optim}
-Combinatorial optimization\index{Combinatorial~optimization} (CO) is a branch of applied and discrete mathematics.
+Combinatorial optimization (CO) is a branch of applied and discrete mathematics.
It consists in finding optimal configuration(s) among a finite set of possible configurations
-(or solutions) of a given combinatorial optimization problem (COP). The set of all possible solutions noted $S$ is called solution space or search space. Each solution in $S$ is defined by its real cost calculated by an objective function. COPs are generally defined as follows~\cite{blumMeta}:\\ %(Definition~\ref{def:cops})
+(or solutions) of a given combinatorial optimization problem (COP). The set of all possible solutions noted $S$ is called solution space or search space. Each solution in $S$ is defined by its real cost calculated by an objective function. COPs are generally defined as follows~\cite{blumMeta}: %(Definition~\ref{def:cops})
%\begin{minipage}{0.5\linewidth}
\begin{itemize}
\item a set of decision variables $X$,
\item an objective function $f$ to optimize (minimize or maximize) over the set $S$,
-\item subject to constraints on the decision variables.\\
+\item subject to constraints on the decision variables.
\end{itemize}
%\end{minipage}
methods but with no guarantee to find optimal or even bounded
solutions. This is due to the nature of the search process adopted
by these approaches which consists of performing a
-search in a subset of the whole search space.\\
+search in a subset of the whole search space.
Regarding the number of solutions considered at each iteration in
the search process, two classes of metaheuristics can be
and iteratively improves it by exploring its neighborhood in the
search space. The most known methods in this class are local
search methods that include \emph{simulated
-annealing}\index{Metaheuristics!simulated~annealing}~\cite{Kirkpatrick1983SA},
+annealing}\index{metaheuristics!simulated annealing}~\cite{Kirkpatrick1983SA},
\emph{tabu search}~\cite{Glover1989TS}, \emph{iterated local
-search\index{Metaheuristics!iterated local
+search\index{metaheuristics!iterated local
search}}~\cite{stutzle2006ILSforQAP}, and \emph{variable
-neighborhood search}~\cite{HansenMladenovic1997VNS}.\\
+neighborhood search}~\cite{HansenMladenovic1997VNS}.
Unlike s-metaheuristics, p-metaheuristics start with a population
of solutions and implement an iterative process that evolves the
often NP-hard and CPU time and/or memory consuming. Metaheuristics
allow the significant reduction of the computational time of the search
process but remain time-consuming particularly when it comes
-dealing with large-sized problems. \\
+dealing with large-sized problems.
The use of parallelism in the design of metaheuristics is a relevant
approach that is widely adopted by the combinatorial optimization
\end{itemize}
From the granularity of parallelism point of view, three major parallel
-models for metaheuristics can be distinguished~\cite{talbi2009mfdti}: \emph{algorithmic-level}\index{Metaheuristics!algorithmic-level parallelism},
-\emph{iteration-level}\index{Metaheuristics!iteration-level parallelism}, and \emph{solution-level} as illustrated in Figure~\ref{ch8:fig:paraMeta}. \\
+models for metaheuristics can be distinguished~\cite{talbi2009mfdti}: \emph{algorithmic-level}\index{metaheuristics!algorithmic-level parallelism},
+\emph{iteration-level}\index{metaheuristics!iteration-level parallelism}, and \emph{solution-level} as illustrated in Figure~\ref{ch8:fig:paraMeta}.
\begin{figure}[h!]
\centerline{\includegraphics[width=0.6\textwidth]{Chapters/chapter9/figures/paraMeta.pdf}}
\begin{itemize}
\item{In the
-algorithmic-level\index{Metaheuristics!algorithmic-level
-parallelism} parallel model, several self-contained metaheuristics
+algorithmic-level parallel model, several self-contained metaheuristics
are launched in parallel. The parallel
-metaheuristics\index{Metaheuristics!parallel~metaheuristics} may
+metaheuristics\index{metaheuristics!parallel metaheuristics} may
start with identical or different solutions (s-metaheuristics
case) or populations (p-metaheuristics case). Their parameter
settings such as the size of tabu list for tabu
-search\index{Metaheuristics!tabu~search}, transition probabilities
+search\index{metaheuristics!tabu search}, transition probabilities
for ant colonies, mutation and crossover probabilities for
evolutionary
-algorithm\index{Metaheuristics!evolutionary~algorithm}s may be the
+algorithm\index{metaheuristics!evolutionary algorithm}s may be the
same or different. The parallel processes may evolve independently
or in a cooperative manner. In cooperative parallel models, the
algorithms exchange information related to the search during
evolution in order to find better and more robust solutions.}
-\item{In the iteration-level\index{Metaheuristics!iteration-level
-parallelism} parallel model, the focus is on the parallelization
+\item{In the iteration-level parallel model, the focus is on the parallelization
of each iteration of the metaheuristic. Indeed, metaheuristics are
generally iterative search processes. Moreover, the most
resource-consuming part of a metaheuristic is the evaluation of
the generated solutions at each iteration. For s-metaheuristics
-(e.g., tabu search\index{Metaheuristics!tabu~search}, simulated
+(e.g., tabu search\index{metaheuristics!tabu search}, simulated
annealing, variable neighborhood search), the evaluation and
generation of the neighborhood is the most time-consuming step of
the algorithm particularly when it comes to dealing with large
neighborhood sets. In this parallel model, the neighborhood is
decomposed into partitions, and each partition is evaluated in a
parallel way. For p-metaheuristics (e.g., evolutionary
-algorithm\index{Metaheuristics!evolutionary~algorithm}s, ant
+algorithms, ant
colonies, swarm optimization), the
-iteration-level\index{Metaheuristics!iteration-level parallelism}
+iteration-level
parallel model arises naturally since these metaheuristics deal
with a population of independent solutions. In evolutionary
-algorithm\index{Metaheuristics!evolutionary~algorithm}s, for
-instance, the iteration-level\index{Metaheuristics!iteration-level
-parallelism} model consists of decomposing the whole population
+algorithms, for
+instance, the iteration-level model consists of decomposing the whole population
into several partitions where each partition is evaluated in
parallel.}
\item{In the
-solution-level\index{Metaheuristics!solution-level~parallelism}
+solution-level
parallel model, the focus is on the parallelization of the
evaluation of a single solution. This model is useful when the
objective function and/or the constraints are time and/or memory
consuming. Unlike the two previous parallel models, the
-solution-level\index{Metaheuristics!solution-level~parallelism}
+solution-level\index{metaheuristics!solution-level parallelism}
parallel model is problem-dependent.}
\end{itemize}
-
+\clearpage
\section{Challenges for the design of GPU-based metaheuristics}
\label{ch8:sec:challenges}
Developing GPU-based parallel
-metaheuristics\index{Metaheuristics!parallel~metaheuristics} is
+metaheuristics\index{metaheuristics!parallel metaheuristics} is
not straightforward. The parallel models have to be rethought to
meet the new requirements of the GPU architecture. Several major
issues have to be taken into account both at design and
GPU~\cite{luongMultiStart}.
\subsection{Data placement on a hierarchical memory}
-\index{GPU Challenges!data~placement} During the execution of
+\index{GPU!data placement} During the execution of
metaheuristics on GPU, the different threads may access multiple
data structures from multiple memory spaces. These memories have
different sizes and access latencies. Nevertheless, faster
per thread, one individual per thread, single population per
threads block, one ant per thread, etc.) must be defined to ensure
a maximum occupancy of the GPU and
-to cover CPU/GPU communication\index{GPU Challenges!CPU/GPU~communication} and memory access times.\\
+to cover CPU/GPU communication\index{GPU!CPU/GPU communication} and memory access times.
According to the used metaheuristic and to the handled problem, the data
values may have different types and different ranges of their values. The data
types should then be chosen carefully and the ranges of the data values should
-be analyzed to minimize the amount of occupied memory space.\\
+be analyzed to minimize the amount of occupied memory space.
In addition to the size and latency of GPU memory issues, the
memory access pattern is another important issue to be dealt with
block) on the shared memory.
\subsection{Threads synchronization}
-\index{GPU Challenges!threads~synchronization} The thread
+\index{GPU!threads synchronization} The thread
synchronization issue is caused by both the GPU architecture and
the synchronization requirements of the implemented method.
Indeed, GPUs are based on a multicore architecture organized into
\subsection{Thread divergence}
-Thread divergence\index{GPU Challenges!thread~divergence} is
+Thread divergence\index{GPU!thread divergence} is
another challenging issue in GPU-based
metaheuristics~\cite{cecilia, pugace, audreyANT}. Generally,
metaheuristics contain irregular loops and conditional
(s-metaheuristics), and the population (p-metaheuristics) in the
same block. In addition, the decision to apply a crossover or a
mutation on an individual in a genetic
-algorithm\index{Metaheuristics!genetic~algorithm} and the
+algorithm and the
exploration of different paths using an ant
-colony\index{Metaheuristics!ant~colony~optimization} are random
+colony\index{metaheuristics!ant colony optimization} are random
operations. Threads of the same warp have to execute
instructions simultaneously leading to different branches whereas
in an SIMD model the threads of a same warp execute the same
in terms of execution time. The challenge here is then to revisit
the traditional irregular metaheuristic codes to eliminate these
divergences.
-
+\clearpage
\subsection{Task distribution and CPU/GPU communication}
The performance of GPU-based metaheuristics in terms of execution
time could be improved by choosing the most appropriate parallel
-model (algorithmic-level\index{Metaheuristics!algorithmic-level
-parallelism}, instruction-level,
-solution-level\index{Metaheuristics!solution-level~parallelism}).
+model (algorithmic-level, instruction-level,
+solution-level).
Moreover, an efficient decomposition of the metaheuristic and an
efficient assignment of code portions between the CPU and GPU
should be adopted. The objective is to take benefit from the GPU
computing power without affecting the efficiency and the behavior
of the metaheuristic and without losing performance in CPU/GPU
-communication\index{GPU Challenges!CPU/GPU~communication} and
+communication\index{GPU!CPU/GPU communication} and
memory accesses. In order to decide which part of the
metaheuristic will be executed on which component, one should
perform a careful analysis on the serial code of the
metaheuristic, identify the compute-intensive tasks (e.g.,
exploration of the neighborhood, evaluation of individuals), and
then offload them to the GPU, while the remaining
-tasks still run on the CPU in a serial way. \\
+tasks still run on the CPU in a serial way.
-The CPU/GPU communication\index{GPU
-Challenges!CPU/GPU~communication} is done through the global
+The CPU/GPU communication is done through the global
memory which is a slow memory making the memory transfer between
the CPU and GPU time-consuming which can significantly degrade the
performance of the application. Accesses to this memory should be
\section{State-of-the-art parallel metaheuristics on GPUs}
\label{ch8:sec:state}
After more than two decades of research by the combinatorial optimisation
-community devoted to developing adequate parallel metaheuristics\index{Metaheuristics!parallel~metaheuristics} for different types of
+community devoted to developing adequate parallel metaheuristics for different types of
parallel architectures (clusters, supercomputers and grids), the actual developement
-of General Perpose GPU (GPGPU) brings new challenges for parallel metaheuristics\index{Metaheuristics!parallel~metaheuristics} on SIMD architectures.\\
+of General Perpose GPU (GPGPU) brings new challenges for parallel metaheuristics on SIMD architectures.
The first works on metaheuristic algorithms implemented on GPUs
started on old graphics cards before the appearance of modern GPUs
equipped with high-level programming interfaces such as CUDA and
OpenCL. Among these pioneering works we cite the work of Wong et al.~\cite{wongOldGPU2006} dealing with the
implementation
-of EAs on graphics processing cards and the work by Catala et al. in~\cite{catala2007} where the ACO\index{Metaheuristics!ant~colony~optimization} algorithm
+of EAs on graphics processing cards and the work by Catala et al. in~\cite{catala2007} where the ACO\index{metaheuristics!ant colony optimization} algorithm
is implemented on old GPU architectures. Yu et al.~\cite{yu2005} and
Li et al.~\cite{li2007} proposed a full parallelization of genetic
-algorithm\index{Metaheuristics!genetic~algorithm}s on old GPU architectures using
-shader libraries based on Direct3D and OpenGL.\\
+algorithms on old GPU architectures using
+shader libraries based on Direct3D and OpenGL.
Such architectures are based on preconfigured pipelined stages
used to accelerate the transformation of 3D geometric primitives
occupation of the CUDA threads which may lead in turn to an
overhead due to the communication and memory latencies. Therefore,
large neighborhoods are necessary for efficient implementation of
-local searches on GPUs.\\
+local searches on GPUs.
Luong et al.~\cite{luong2010large} proposed efficient
mappings for large neighborhood structures on GPUs. In this work,
more explicitly, how to calculate the memory index of each
solution associated to each CUDA thread's \textit{id}.
%For 1-Hamming neighborhoods, as there is exactly n solutions in the neighborhood, the mapping of this neighborhood to CUDA threads is obvious: the CPU host offloads to GPU exactly $n$ threads, and each thread id is associated to one index in the binary vector. In the case of 2-Hamming and 3-Hamming neighborhoods, each thread id should be mapped respectively to two and three indexes in the candidate vector.
-The three neighborhoods are implemented and experimented on the Permuted Perceptron Problem (PPP) using a tabu search\index{Metaheuristics!tabu~search} algorithm (TS). Accelerations from $9.9 \times$ to $18.5 \times$ are obtained on different problem sizes.\\ % The experiments are performed on an Intel Xeon 8 cores 3GHz coupled with an NVIDIA GTX 280 card.\\
+The three neighborhoods are implemented and experimented on the Permuted Perceptron Problem (PPP) using a tabu search\index{metaheuristics!tabu search} algorithm (TS). Accelerations from $9.9 \times$ to $18.5 \times$ are obtained on different problem sizes. % The experiments are performed on an Intel Xeon 8 cores 3GHz coupled with an NVIDIA GTX 280 card.\\
In the same context, Deevacq et al.~\cite{audreyANT}
-proposed two parallelization strategies inspired by the multi-walk
+proposed two parallelization strategies inspired by the multiwalk
parallelization strategy, of a 3-opt iterated local
-search\index{Metaheuristics!iterated local search} algorithm (ILS)
+search algorithm (ILS)
over a CPU/GPU architecture. In the first strategy, each Local
Search (LS) is associated to a unique CUDA thread and improves a
unique solution by generating its neighborhood. The second
solution at a time in the second strategy allows the use of the
shared memory to store the related data structures. The two
strategies have been experimented on standard benchmarks of
-the Traveling Salesman Problem (TSP) with sizes varying from $100$ to $3038$ cities. The results indicate that increasing the number of solutions to be explored simultaneously improves the speedup in the two strategies, but at the same time it decreases final solution quality. The greater speedup factor reached by the second strategy is $6 \times$.\\
+the Traveling Salesman Problem (TSP) with sizes varying from $100$ to $3038$ cities. The results indicate that increasing the number of solutions to be explored simultaneously improves the speedup in the two strategies, but at the same time it decreases final solution quality. The greater speedup factor reached by the second strategy is $6 \times$.
The same strategy is used by Luong et al.
in~\cite{luongMultiStart} to implement multistart parallel local
search algorithms (a special case of the
-algorithmic-level\index{Metaheuristics!algorithmic-level
-parallelism} parallel model where several homogeneous LS
+algorithmic-level parallel model where several homogeneous LS
algorithms are used). The multistart model is combined with
-iteration-level\index{Metaheuristics!iteration-level parallelism}
+iteration-level
parallelism: several LS algorithms are managed by the CPU and the
neighborhood evaluation step of each algorithm is parallelized on
the GPU (each GPU thread is associated with one neighbor and
executes the same evaluation function kernel). The advantage of
such a model is that it allows a high occupancy of the GPU
-threads. Nevertheless, memory management\index{GPU
-Challenges!memory~management} causes new issues due to the
+threads. Nevertheless, memory management causes new issues due to the
quantity of data to store and to communicate between CPU and
GPU. A second proposition for implementing the same model on GPU
consists of implementing the whole LS processes on GPU with each
GPU thread being associated to a unique LS algorithm. This solves
the communication issue encountered in the first model. In
-addition, a memory management\index{GPU
-Challenges!memory~management} strategy is proposed to improve the
+addition, a memory management strategy is proposed to improve the
efficiency of the
-algorithmic-level\index{Metaheuristics!algorithmic-level
-parallelism} model: texture memory is used to avoid memory latency
+algorithmic-level model: texture memory is used to avoid memory latency
due to uncoalesced memory accesses. The proposed approaches are
implemented on the quadratic assignment problem (QAP) using CUDA.
The acceleration rates obtained for the
-algorithmic-level\index{Metaheuristics!algorithmic-level
-parallelism} with usage of texture memory rise from $7.8\times$ to
+algorithmic-level with usage of texture memory rise from $7.8\times$ to
$12\times$ (for different
-QAP benchmark sizes). \\
+QAP benchmark sizes).
Janiak et al.~\cite{Janiak_et_al_2008} implemented two
algorithms for TSP and the flow-shop scheduling problem (FSP).
These algorithms are based on a multistart tabu
-search\index{Metaheuristics!tabu~search} model. Both of the
+search model. Both of the
algorithms exploit multicore CPU and GPU. A full parallelization
on GPU is adopted using shader libraries where each thread is
-mapped with one tabu search\index{Metaheuristics!tabu~search}.
+mapped with one tabu search.
However, even though their experiments report that the use of GPU
speedups the serial execution almost $16 \times$, the mapping of
-one thread with one tabu search\index{Metaheuristics!tabu~search}
+one thread with one tabu search
requires a large number of local search algorithms to
-cover the memory access latency. The same mapping policy is adopted by Zhu et al. in~\cite{zhu_et_al_2008} (one thread is associated to one local search) solving the quadratic assignment problem but using the CUDA toolkit instead of shader libraries.\\
+cover the memory access latency. The same mapping policy is adopted by Zhu et al. in~\cite{zhu_et_al_2008} (one thread is associated to one local search) solving the quadratic assignment problem but using the CUDA toolkit instead of shader libraries.
Luong et al.~\cite{luong2012ppsn} proposed a GPU-based
implementation of hybrid metaheuristics on heterogeneous parallel
of using such a heterogeneous architecture is how to distribute
tasks between the CPU cores and the GPU in such a way to have
optimal performances. Among the three traditional parallel models
-(solution-level\index{Metaheuristics!solution-level~parallelism},
-iteration-level\index{Metaheuristics!iteration-level parallelism},
-and algorithmic-level\index{Metaheuristics!algorithmic-level
-parallelism}), the authors point out that the most convenient
+(solution-level,
+iteration-level
+and algorithmic-level), the authors point out that the most convenient
model for the considered heterogeneous architecture is a hybrid
model combining
-iteration-level\index{Metaheuristics!iteration-level parallelism}
-and algorithmic-level\index{Metaheuristics!algorithmic-level
-parallelism} models. Several CPU threads execute several instances
+iteration-level
+and algorithmic-level models. Several CPU threads execute several instances
of the same S-metaheuristic in parallel while the GPU device is
associated to one CPU core and used to accelerate the
neighborhood calculation of several S-metaheuristics at the same
time.
In order to efficiently exploit the remaining CPU cores, a load-balancing heuristic is also proposed in order to decide on the number of additional
-S-metaheuristics to launch on the remaining CPU cores relative to the efficiency of the GPU calculations. The proposed approach is applied to the QAP using several instances of the Fast Ant Colony Algorithm (FANT)~\cite{taillardFant}. \\
+S-metaheuristics to launch on the remaining CPU cores relative to the efficiency of the GPU calculations. The proposed approach is applied to the QAP using several instances of the Fast Ant Colony Algorithm (FANT)~\cite{taillardFant}.
All the previously noted works exploit the same parallel models
used on CPUs based on the task parallelism. A different
implementation approach is proposed by Paul in~\cite{gerald2012}
to implement a simulated
-annealing\index{Metaheuristics!simulated~annealing} (SA) algorithm
+annealing (SA) algorithm
for the QAP on GPUs. Indeed, the author used a preinitialized
matrix \emph{delta} in which the incremental evaluation of simple
swap moves are calculated and stored relative to the initial
done on standard QAP instances from the
QAPLIB~\cite{burkard1991qaplib}. Speedups up to $10 \times$ are
achieved by the GPU implementation compared
-to the same sequential implementation on CPU using SA-matrix.\\
+to the same sequential implementation on CPU using SA-matrix.
-\subsection{Implementing population-based metaheuristics on GPUs}
+\subsection[Implementing population-based metaheuristics\hfill\break on GPUs]{Implementing population-based metaheuristics on GPUs}
State-of-the-art works dealing with the implementation of
p-metaheuristics on GPUs generally rely on parallel models and
supercomputers, clusters, and computational grids. Three main
classes of p-metaheuristics are considered in this section:
evolutionary
-algorithm\index{Metaheuristics!evolutionary~algorithm}s (EAs), ant
-colony\index{Metaheuristics!ant~colony~optimization} optimization
-(ACO\index{Metaheuristics!ant~colony~optimization}), and particle
-swarm optimization\index{Metaheuristics!particle swarm
+algorithms (EAs), ant
+colony optimization
+(ACO), and particle
+swarm optimization\index{metaheuristics!particle swarm
optimization} (PSO).
-
+\clearpage
\subsubsection*{Evolutionary algorithms}
Traditional parallel models for EAs are classified in three main
a toroidal grid.
The three traditional models have been implemented on GPUs by several researchers for different
-optimization problems. The main chalenges to be raised when implementing the traditional models on GPUs concern (1) the saturation of the GPU in order to cover memory latency by calculations, and (2) efficent usage of the hierarchical GPU memories.\\
+optimization problems. The main chalenges to be raised when implementing the traditional models on GPUs concern (1) the saturation of the GPU in order to cover memory latency by calculations, and (2) efficent usage of the hierarchical GPU memories.
In~\cite{kannan}, Kannan and Ganji present a CUDA implementation
of the drug discovery application Autodock (molecular docking
application). Autodock uses a genetic
-algorithm\index{Metaheuristics!genetic~algorithm} to find optimal
+algorithm to find optimal
docking positions of a ligand to a protein. The most
time-consuming task in Autodock is the fitness function
evaluation. The fitness function used for a docking problem
memory instead of global memory to store all the information
related to each individual. The obtained speedups range from $10
\times$ to $47 \times$ for population sizes
-ranging from $50$ to $10000$. \\
+ranging from $50$ to $10000$.
Maitre et al.~\cite{maitre2009} also exploited the
master-slave parallelism of EAs on GPUs using EASEA. EASEA is a
description of the problem-specific components (fitness function,
problem representation, etc) in EASEA. The code is then compiled
to obtain a ready-to-use evolutionary
-algorithm\index{Metaheuristics!evolutionary~algorithm}. The EASEA
+algorithm. The EASEA
compiler uses genetic
-algorithm\index{Metaheuristics!genetic~algorithm} LIB and EO
+algorithm LIB and EO
Libraries to produce C++ or JAVA written EA codes.
In~\cite{maitre2009}, the authors proposed an extension of EASEA
to produce CUDA code from the EASEA files. This extension has been
real problem (molecular structure prediction). In order to
maximize the GPU occupation, very large populations are used (from
$2000$ to $20000$). Even though transferring such large
-populations from the CPU to the GPU device memory at every generation is very costly, the authors report important speedups on the two problems on a GTX260 card: $105 \times$ is reported for the benchmark function while for the real problem the reported speedup is $60 \times$. This may be best explained by the complexity of the fitness functions. Nevertheless, there is no indication in the paper about the memory management\index{GPU Challenges!memory~management} of the populations on GPU.\\
+populations from the CPU to the GPU device memory at every generation is very costly, the authors report important speedups on the two problems on a GTX260 card: $105 \times$ is reported for the benchmark function while for the real problem the reported speedup is $60 \times$. This may be best explained by the complexity of the fitness functions. Nevertheless, there is no indication in the paper about the memory management of the populations on GPU.
The master-slave model is efficient when the fitness function is
highly time intensive. Nevertheless, it requires the use of
of the fitness function itself is possible. The use of many
subpopulations of medium sizes is another way to obtain a maximum
occupation of the GPU. This is coarse-grained parallelism (island
-model).\\
+model).
The coarse-grained model is used by Pospichal et al.
in~\cite{pospichal10} to implement a parallel genetic
-algorithm\index{Metaheuristics!genetic~algorithm} on GPU. In this
+algorithm on GPU. In this
work the entire genetic
-algorithm\index{Metaheuristics!genetic~algorithm} is implemented
+algorithm is implemented
on GPU. This choice is motivated by the overhead engendered by the
-CPU/GPU communication\index{GPU Challenges!CPU/GPU~communication}
+CPU/GPU communication
when only population evaluation is performed on GPU. Each
population island is mapped with a CUDA thread block and each
thread is responsible for a unique individual. Subpopulations are
interblock communications are not possible on the CUDA
architecture, the islands evolve independently in each block, and
migrations are performed
-asynchronously through the global memory. That is, after a given number of generations, selected individuals for migration from each island are copied to the GPU global memory part of the neighbor island and then to its shared memory to replace the worst individuals in the local population. The experiments are performed on three benchmark mathematical functions. During these experiments, the island sizes are varied from $2$ to $256$ individuals and island numbers from $1$ to $1024$. The maximum performance is achieved for high number of islands and increasing population sizes.\\
+asynchronously through the global memory. That is, after a given number of generations, selected individuals for migration from each island are copied to the GPU global memory part of the neighbor island and then to its shared memory to replace the worst individuals in the local population. The experiments are performed on three benchmark mathematical functions. During these experiments, the island sizes are varied from $2$ to $256$ individuals and island numbers from $1$ to $1024$. The maximum performance is achieved for high number of islands and increasing population sizes.
The same strategy is also adopted by Tsutsui and Fujimoto
in~\cite{tsutsuiGAQAP} to implement a coarse-grained genetic
-algorithm\index{Metaheuristics!genetic~algorithm} on GPU to solve
+algorithm on GPU to solve
the QAP. Initially, several subpopulations are created on CPU and
transferred to the global memory. The subpopulations are organized
in the global memory into blocks of $8$ individuals in such a way
QAP benchmarks from the QAPLIB~\cite{burkard1991qaplib}. The GPU
implementation reached speedups of $2.9\times$ to $12.6 \times$
compared to a single core implementation of a coarse-grained
-genetic algorithm\index{Metaheuristics!genetic~algorithm} on a
-Intel i7 processor.\\
+genetic algorithm on a
+Intel i7 processor.
Nowotniak and Kucharski~\cite{nowotniak} proposed a GPU-based
-implementation of a Quantum Inspired Genetic Algorithm\index{Metaheuristics!genetic~algorithm} (QIGA). The
+implementation of a Quantum Inspired Genetic Algorithm (QIGA). The
parallel model used is a hierarchical model based on two levels: each
thread in a block transforms a unique individual and a different
population is assigned to each block. The algorithm is run
populations are stored in the shared memory, the data matrix used
for fitness evaluation is placed in read only constant memory, and
finally seeds for random numbers generated on the GPU are stored
-in the global memory.\\
+in the global memory.
In coarse-grained parallelism, the use of the per-block approach
to implement the islands (one subpopulation per thread block) is
the subpopulations. Fine-grained parallel models for EAs (cellular
EAs) have been used by many authors to implement parallel EAs on
GPUs. Indeed, the fine-grained parallelism of EAs fits perfectly
-to the SIMD architecture of the GPU. \\
+to the SIMD architecture of the GPU.
Pinel et al. in~\cite{pinel2012JPDC} developed a highly
parallel synchronous cellular genetic
-algorithm\index{Metaheuristics!genetic~algorithm} (CGA), called
+algorithm (CGA), called
GraphCell, to solve the independent task scheduling problem on GPU
architectures. In CGAs, the population is arranged into a
two-dimensional toroidal grid where only neighboring solutions are
when dealing with large instances of the problem. In addition to
the recombination operators, the rest of the CGA steps are also
parallelized on GPU (fitness evaluation, mutation, and
-replacement).\\
+replacement).
A similar work is proposed by Vidal and Alba in~\cite{albaCGAGPU}
where a CGA using a toroidal grid is completely implemented on
population is set to $32^2$. When the size of the population is
too small, there are not enough computations to cover the overhead
created by the call of kernel functions, CPU/GPU
-communication\index{GPU Challenges!CPU/GPU~communication}s,
+communications,
synchronization, and access to global memory. Finally, an
interesting review on GPU parallel computation in bio-inspired
algorithms is proposed by Arenas et al.
-in~\cite{arenas2011}. \\
+in~\cite{arenas2011}.
\subsubsection*{Ant colony optimization}
Ant colony optimization
-(ACO\index{Metaheuristics!ant~colony~optimization}) is another
+(ACO) is another
p-metaheuristic subject to parallelization on GPUs.
State-of-the-art works on parallelizing
-ACO\index{Metaheuristics!ant~colony~optimization} focus on
+ACO focus on
accelerating the tour construction step performed by each ant by
taking a task-based parallelism approach, with pheromone
-deposition on the CPU.\\
+deposition on the CPU.
In~\cite{cecilia}, Cecilia et al. present a GPU-based
implementation of
-ACO\index{Metaheuristics!ant~colony~optimization} for TSP where
+ACO for TSP where
the two steps (tour construction and pheromone update) are
parallelized on the GPU. A data parallelism approach is used to
enhance the performance of the tour construction step. The
bandwidth of the application mainly by the use of precalculated
matrices that are easily updated by several threads (one thread
per matrix entry). The achieved speedups are $21 \times$ for tour
-construction and $20 \times$ for pheromone updates.\\
+construction and $20 \times$ for pheromone updates.
In another work, Tsutsui and Fujimoto~\cite{tsutsui} propose a
hybrid algorithm combining
-ACO\index{Metaheuristics!ant~colony~optimization} metaheuristic
-and Tabu Search (TS)\index{Metaheuristics!tabu~search} implemented
+ACO metaheuristic
+and Tabu Search (TS) implemented
on GPU to solve the QAP. A solution of QAP is represented as a
permutation of ${1,2,..,n}$ with $n$ being the size of the
problem. The TS algorithm is based on the 2-opt neighborhood
(swapping of two elements $(i,j)$ in the permutation). The authors
point out that the move cost of each neighbor depends on the
couple $(i,j)$. Two groups of moves are formed according to the
-move cost. In order to avoid thread divergence\index{GPU
-Challenges!thread~divergence} within the same warp, the
+move cost. In order to avoid thread divergence\index{GPU!thread divergence} within the same warp, the
neighborhood evaluation is parallelized in such a way to assign
only moves of the same cost to each thread warp. This strategy is
called MATA for Move-cost Adjusted Thread Assignment. Concerning
-the memory management\index{GPU Challenges!memory~management}, all
-the data of the ACO\index{Metaheuristics! ant~colony~optimization}
+the memory management\index{GPU!memory management}, all
+the data of the ACO\index{metaheuristics!ant colony optimization}
(population, pheromone matrix), QAP matrices, and tabu list are
placed on the global memory of the GPU. Shared memory is used only
for working data common to all threads in a given block.
All the
steps of the hybrid algorithm
-ACO\index{Metaheuristics!ant~colony~optimization}-TS
-(ACO\index{Metaheuristics!ant~colony~optimization} initialization,
+ACO-TS
+(ACO initialization,
pheromone update, construct solutions, applying TS) are
implemented as kernel functions on the GPU. The GPU/CPU
communications are only used to transfer the best-so-far solution
implementation using MATA obtained a speedup of $19 \times$
compared to the CPU implementation, compared with a speedup of
only $5 \times$
-when MATA is not used. \\
+when MATA is not used.
\subsubsection*{Particle swarm optimization}
In~\cite{zhou2009} Zhou and Tan propose a full GPU implementation
particles of the swarm one to one. Experiments done on four
benchmark functions show speedups ranging from $3.7 \times$ to
$9.0 \times$ for swarm sizes
-ranging from $400$ to $2800$.\\
+ranging from $400$ to $2800$.
\subsection{Synthesis of the implementation strategies}
\label{ch8:sec:synthesis} After reviewing some works dealing with
for GPUs are derived from the traditional parallel models of each
metaheuristic (on CPU), their implementation could take a
different way and sometimes it may result in new parallel models
-customized for GPUs.\\
+customized for GPUs.
Traditional parallel models for metaheuristics are based on an
intuitive task parallelism: the independent tasks of the
the GPU architecture, some authors have used new implementation
techniques to enhance the data parallelism in the sequential
algorithms in order to increase the data throughput of the
-application.\\
+application.
From this observation we propose the following classification
based on 2 levels: design level and implementation level as
illustrated in Figure~\ref{ch8:fig:classification}. The design
level regroups the three classes of parallel models used in
metaheuristics
-(solution-level\index{Metaheuristics!solution-level~parallelism},
-iteration-level\index{Metaheuristics!iteration-level parallelism},
-algorithmic-level\index{Metaheuristics!algorithmic-level
-parallelism}) with examples for s-metaheuristics, EAs,
-ACO\index{Metaheuristics!ant~colony~optimization} and PCO. This
+(solution-level,
+iteration-level,
+algorithmic-level) with examples for s-metaheuristics, EAs,
+ACO and PCO. This
classification is principally built from the reviewed
state-of-the-art works in the previous section. The
implementation level refers to the way these parallel design
the mapping strategies between the GPU threads and the
parallelized tasks (neighborhood evaluation, solution
construction, and so on). The different implementation strategies
-are explained in the following sections.\\
+are explained in the following sections.
\begin{figure}[h]
\centerline{\includegraphics[width=1\textwidth]{Chapters/chapter9/figures/classification.pdf}}
\subsubsection*{GPU thread mapping for solution-level parallelism}
-\index{GPU-based Metaheuristics!GPU-thread mapping} Parallel
+\index{GPU!thread mapping} Parallel
models at solution level consist of parallelizing a time intensive
atomic task of the algorithm. Generally, it consists of the
fitness evaluation~\cite{kannan}. Nevertheless, crossover
used inside each block to parallelize the fitness evaluation of a
single solution. This kind of mapping allows the use of shared
memory to store the data structures of the solution and does not
-require the use of very large neighborhoods or populations.\\
+require the use of very large neighborhoods or populations.
%data parallelism in SA-matrix to parallelize
\subsubsection*{GPU thread mapping for iteration-level parallelism}
-\index{GPU-based Metaheuristics!GPU-thread mapping}
+\index{GPU!thread mapping}
Iteration-level parallelism consists of parallelizing the tasks
performed independently on different solutions. Different mapping
strategies are adopted in the reviewed works to implement these
-models.\\
+models.
In Figure \ref{ch8:fig:classification}, the first example of
-iteration-level\index{Metaheuristics!iteration-level parallelism}
+iteration-level
parallelism is the parallel evaluation of neighborhoods in
s-metaheuristics. In most of the reviewed works, a per-thread
mapping approach is used: each solution of the neighborhood is
%pheromone update data parallelism matrices
\subsubsection*{GPU thread mapping for algorithmic-level parallelism}
-\index{GPU-based Metaheuristics!GPU-thread mapping}
-Algorithmic-level parallelism consists of launching several self-contained algorithms in parallel. In the previously reviewed works two algorithmic-level\index{Metaheuristics!algorithmic-level parallelism} models have been used: the multistart model and the island model (parallel EAs).\\
+Algorithmic-level parallelism consists of launching several self-contained algorithms in parallel. In the previously reviewed works two algorithmic-level models have been used: the multistart model and the island model (parallel EAs).
The implementation of the multistart model is based on two
different mapping strategies~\cite{luongMultiStart, audreyANT}:
are placed on CPU and the neighborhood evaluation of each LS is
parallelized on GPU using per-thread mapping strategy (one thread
per solution). This consists of a hierarchical parallel model
-combining algorithmic-level\index{Metaheuristics!algorithmic-level
-parallelism} parallelism (multistart) with
-iteration-level\index{Metaheuristics!iteration-level parallelism}
-parallelism (master-worker).\\
+combining algorithmic-level parallelism (multistart) with
+iteration-level
+parallelism (master-worker).
In the island model, the same mapping is used in all the reviewed
works~\cite{tsutsuiGAQAP, nowotniak, maitre2009}: each
the occupation of a large number of threads even for medium
population sizes. The second advantage consists of using shared
memory to store subpopulations to benefit from the low latency of
-shared memory.\\
+shared memory.
\section{Frameworks for metaheuristics on GPUs}
\label{ch8:sec:frameworks}
do not cover the main metaheuristic algorithms, we will present
the only three works to our knowledge, which propose open source
frameworks for developing
-metaheuristics on GPUs.\\
+metaheuristics on GPUs.
The three frameworks reviewed in this section are
PUGACE\index{GPU-based frameworks!PUGACE}~\cite{pugace},
implementation of p-metaheuristics on GPU. The three frameworks
are presented in more detail in the following.
-\subsection{PUGACE\index{GPU-based frameworks!PUGACE}: framework for implementing evolutionary computation on GPUs}
-PUGACE\index{GPU-based frameworks!PUGACE} is a generic framework
+\subsection{PUGACE: framework for implementing evolutionary computation on GPUs}
+PUGACE is a generic framework
for easy implementation of cellular evolutionary algorithms on
GPUs implemented using C and CUDA. It is based on the frameworks
MALLBA and JCell (a framework for cellular algorithms). The
authors justified the choice of cellular evolutionary
-algorithm\index{Metaheuristics!evolutionary~algorithm} by the good
+algorithm by the good
feedback found in the literature concerning its efficient
implementation on GPUs compared to other parallel models for EAs
(island, master-slave). The main standard evolutionary operators
-are already implemented in PUGACE\index{GPU-based
-frameworks!PUGACE}: different selection strategies, standard
+are already implemented in PUGACE: different selection strategies, standard
crossover, and mutation operators (PMX, swap, 2-exchange,
etc.). Different problem encoding is also supported. The framework
is organized as a set of modules in which the different
functionalities are separated as much as possible in order to
facilitate the extension of the framework. All
-the functions and procedures that execute on GPU are implemented in the same file kernel.cu. \\
+the functions and procedures that execute on GPU are implemented in the same file kernel.cu.
The implementation strategy adopted on the GPU is as follows.
Population initialization is done on the CPU side and the
individual is associated to a unique CUDA thread. The function
evaluation and mutation are done on the GPU while selection and
replacement are maintained on the CPU. In order to avoid thread
-divergence\index{GPU Challenges!thread~divergence} appearing in
+divergence\index{GPU!thread divergence} appearing in
the same CUDA thread block at the crossover step (because of the
probability of application which may give different results from
one thread to the other), the decision of whether to apply a
crossover is taken at the block level and applied to all the
individuals within the block. It is the decision on the choose
-of the cutting point for the crossover.\\
+of the cutting point for the crossover.
The framework is validated on standard benchmarks of the QAP. Speedups of $15.44 \times$ to $18.41
\times$ are achieved compared to a CPU implementation of a cEA
framework ParadisEO-MO-GPU\index{GPU-based
frameworks!ParadisEO-MO-GPU} for parallel local search
metaheuristics (s-metaheuristics) on GPUs. It focuses on the
-iteration-level\index{Metaheuristics!iteration-level parallelism}
+iteration-level
parallel model of s-metaheuristics which consists of exploring in
parallel the neighborhood of a problem solution on GPU. The
framework, implemented using C++ and CUDA, is an extension of the
allows one to efficiently manage the hierarchical organization of
the memories (latencies and sizes) of the GPU and its
communication with the CPU as well as the minimizing of the user
-involvement in its management.\\
+involvement in its management.
\begin{figure}[h]
\centerline{\includegraphics[width=0.8\textwidth]{Chapters/chapter9/figures/paradiseoGPU.pdf}}
for the design and implementation of metaheuristics on GPUs. Until
now, the metaheuristics supported by LibCUDAOptimize are: scatter
search, differential evolution, and particle swarm
-optimization\index{Metaheuristics!particle swarm optimization}.
+optimization\index{metaheuristics!particle swarm optimization}.
Nevertheless, the library is designed in such a way to allow
further extensions for other metaheuristics and it is still in
development phase by the authors. The parallelization strategy
\label{ch8:sec:case}
In this case study, a large neighborhood GPU-based local
-search\index{GPU-based Metaheuristics!GPU-based~local~search}
+search\index{GPU!based local search}
method for solving the Quadratic 3-dimensional Assignment Problem
(Q3AP) will be presented. The local search method is an Iterated
-Local Search\index{Metaheuristics!iterated local search}
+Local Search\index{metaheuristics!iterated local search}
(ILS)~\cite{stutzle2006ILSforQAP} using an embedded
-TS\index{Metaheuristics!tabu~search} algorithm. The ILS principle
+TS algorithm. The ILS principle
consists of executing iteratively the embedded local search, each
iteration which starts from a disrupted local optima reached by
the previous local search process. The disruption heuristic is a
performance parameter of an ILS algorithm and should be
judiciously defined. A template of an
-ILS algorithm is given by the Algorithm~\ref{ch8:ils_algorithm_template}.\\
+ILS algorithm is given by the Algorithm~\ref{ch8:ils_algorithm_template}.
\begin{algorithm}[H]
\SetAlgoLined
applications (site planning, schedule problem, computer-aided
design, etc.) and to its computational challenges since it is
considered as one of the most computationally difficult
-optimization problems.\\
+optimization problems.
The Q3AP was first introduced by William P. Pierskalla in
1967~\cite{Pierskalla1967Q3AP} and, unlike the QAP, the Q3AP is a
$\left\lbrace 0,1,\ldots,n-1\right\rbrace$. According to this
formulation, minimizing the Q3AP consists of finding a double
permutation $(\pi_1,\pi_2)$ which minimizes the objective function
-(\ref{Q3APPermutationBasedFormulation}).\\
+(\ref{Q3APPermutationBasedFormulation}).
The Q3AP is proven to be an NP-hard problem since it is an
extension of the quadratic assignment problem and of the axial
\subsection{Iterated tabu search algorithm for the Q3AP}
To tackle large-sized instances of the Q3AP and speed up the
-search process, a parallel ILS\index{Metaheuristics!iterated local
-search} algorithm has been designed. The local search embedded in
-the ILS is a TS\index{Metaheuristics!tabu~search}. A TS
+search process, a parallel ILS algorithm has been designed. The local search embedded in
+the ILS is a TS. A TS
procedure~\cite{Glover1989TS} starts from an initial feasible
solution and tries, at each step, to move to a neighboring
solution that minimizes the fitness (for a minimization case). If
escape local optima. However, this strategy may generate cycles,
i.e., previous moves can be selected again. To avoid cycles, the
TS manages a short-term memory that contains the moves that have
-been recently performed. A TS\index{Metaheuristics!tabu~search}
-template is given by Algorithm \ref{TS_pseudo_code}.\\
+been recently performed. A TS
+template is given by Algorithm \ref{TS_pseudo_code}.
%
% \begin{algorithm}
% \label{TS_pseudo_code}
search algorithm. Indeed, if the neighborhood function is not
adequate to the problem and/or does not consider the targeted
computing framework, any local search algorithm will fail to reach
-good quality solutions of the search space.\\
+good quality solutions of the search space.
Regarding the Q3AP, many neighborhood structures can be
considered. A basic neighborhood was proposed and investigated in
such a neighborhood, the generation/evaluation step of an LS
becomes a time-consuming task and may dramatically increase the
computational time of the LS process. This
-justifies the use of intensive data-parallelism provided by GPUs where all neighboring solutions may be concurrently evaluated. \\
+justifies the use of intensive data-parallelism provided by GPUs where all neighboring solutions may be concurrently evaluated.
The considered large-sized neighborhood consists of swapping two
positions in both permutations $\pi_1$ and $\pi_2$. This
local search algorithm are the efficient distribution of the
search process between the CPU and the GPU minimizing the data
transfer between them, the hierarchical memory
-management\index{GPU Challenges!memory~management} and the
+management\index{GPU!memory management} and the
capacity constraints of GPU memories, and the thread
synchronization. All these issues must be regarded when designing
parallel LS models to allow
-solving of large scale optimization problems on GPU architectures.\\
+solving of large scale optimization problems on GPU architectures.
To go back to our problem (i.e., Q3AP), we propose in
Algorithm~\ref{ch8:algoITS} an iterated tabu
-search\index{Metaheuristics!tabu~search} on GPU (GPU-ITS). The
+search on GPU (GPU-ITS). The
parallel model is in agreement with the
-iteration-level\index{Metaheuristics!iteration-level parallelism}
+iteration-level
parallel model of LS methods presented in Section
\ref{ch8:sec:paraMeta} (Fig. \ref{ch8:fig:paraMeta}). This
algorithm can be seen as a cooperative model between the CPU and
not change during all the execution of the LS algorithm.
Therefore, their associated memory is copied only once during all
the execution. Third, comes the parallel
-iteration-level\index{Metaheuristics!iteration-level parallelism},
+iteration-level,
in which each neighboring solution is generated, evaluated, and
copied into the neighborhood fitnesses structure (from lines 10 to
14). Fourth, since the order in which candidate neighbors are
In this section, some experimental results related to the approach
presented in Section \ref{ch8:ITS-Q3APSection} are reported. We
recall that the approach is a GPU-based iterated tabu
-search\index{Metaheuristics!tabu~search} (GPU-ITS) method
+search (GPU-ITS) method
consisting in an iterated local search (ILS) embedding a tabu
-search\index{Metaheuristics!tabu~search} (TS) and where the
+search (TS) and where the
generation/evaluation step of the TS process is executed on GPU.
The ILS is used to improve the quality of successive local optima
provided by TS methods. This is achieved by perturbing the local
initial solution of the following TS process. Regarding our
algorithm, the applied perturbation is a random number $\mu $ of
swaps in either the first or the second permutation where $\mu \in
-[2:n]$ ($n$ is the instance size).\\
+[2:n]$ ($n$ is the instance size).
Experiments have been carried out on a node of the Chirloute
cluster of the Lille site. This is one of the 10 sites that
cores) GPU type. The number of ILS iterations and the number of TS
iterations were set respectively to 20 and 500. The tabu list size
has been initalized to $\frac{m}{4}$, $m$ being the size of the
-neighborhood set.\\
+neighborhood set.
Table \ref{ch8:ITSQ3APResults} reports the obtained results for
the GPU-ITS using our large-sized neighborhood structure. The
measured. The number of successful tries (hits) and the average
number of ILS iterations to converge to the optimal/best known
value are also represented. The associated standard deviation for
-each average measurement is shown in small type characters.\\
+each average measurement is shown in small type characters.
\begin{table}
%\tiny
& & \tiny{$529.6$} & & & \tiny{$341.1$} & \tiny{$6.6$} & & \tiny{$4.0$} \\ \hline
\end{tabular}
\caption{Results of the GPU-based iterated tabu search for
-different Q3AP instances.} \label{ch8:ITSQ3APResults} \center
+different Q3AP instances.} \label{ch8:ITSQ3APResults} % \center %Shashi
\end{table}
%\begin{table}
\label{ch8:conclusion}
This chapter has presented state-of-the-art GPU-based parallel
-metaheuristics\index{Metaheuristics!parallel~metaheuristics} and
+metaheuristics and
a case study on implementing large neighborhood local search
methods on GPUs for solving large benchmarks of the quadratic
-three-dimensional assignment problem (Q3AP). \\
+three-dimensional assignment problem (Q3AP).
Implementing parallel
-metaheuristics\index{Metaheuristics!parallel~metaheuristics} on
+metaheuristics on
GPU architectures poses new issues and challenges such as memory
-management\index{GPU Challenges!memory~management}; finding
+management; finding
efficient mapping strategies between tasks to parallelize; and the
-GPU threads, thread divergence\index{GPU
-Challenges!thread~divergence}, and synchronization. Actually, most
+GPU threads, thread divergence, and synchronization. Actually, most
of metaheuristics have been implemented on GPU using different
implementation strategies. In this chapter, a two-level
classification of the reviewed works has been proposed: design
metaheuristic tasks to parallelize and the GPU threads. Indeed,
the choice of a given mapping strategy strongly influences the
other challenges (memory usage, communication, thread
-divergence\index{GPU Challenges!thread~divergence}).
+divergence).
\putbib[Chapters/chapter9/biblio9]
