[hdrcouchot.git] / chaosANN.tex

% \section{Introduction}
% \label{S1}

Les réseaux de neurones chaotiques ont été étudiés à de maintes reprises 
par le passé en raison notamment de leurs applications potentielles:
%les mémoires associatives~\cite{Crook2007267}  
les  composants utils à la  sécurité comme les fonctions de 
hachage~\cite{Xiao10},
le tatouage numérique~\cite{1309431,Zhang2005759}
ou les schémas de chiffrement~\cite{Lian20091296}.  
Dans tous ces cas, l'emploi de fonctions chaotiques est motivé par 
leur comportement imprévisibile et proche de l'aléa. 


Les réseaux de neurones chaotiques peuvent être conçus selon plusieurs 
principes. Des neurones modifiant leur état en suivant une fonction non 
linéaire son par exemple appelés neurones chaotiques~\cite{Crook2007267}.
L'architecture de réseaux de neurones de type Perceptron multi-couches
(MLP) n'iterent quant à eux, pas nécesssairement de fonctions chaotiques.
Il a cependant été démontré  que ce sont des approximateurs 
universels~\cite{Cybenko89,DBLP:journals/nn/HornikSW89}.   
Ils permettent, dans certains cas, de simuler des comportements 
physiques chaotiques comme le  circuit de Chua~\cite{dalkiran10}.  
Parfois~\cite{springerlink:10.1007/s00521-010-0432-2},
la fonction de transfert de cette famille de réseau celle 
d'initialisation sont toutes les deux définies à l'aide de 
fonctions chaotiques. 





Ces réseaux de neurones partagent le fait qu'ils sont qualifiés de 
``chaotiques'' sous prétexte qu'ils embarquent une fonction de ce type 
et ce sans aucune preuve rigoureuse. Ce chapitre caractérise la 
classe des réseaux de neurones MLP chaotiques. Il  
s'intéresse ensuite à l'étude de prévisibilité de systèmes dynamiques
discrets chaotiques par cette famille de MLP.

\JFC{revoir plan}

The remainder of this research  work is organized as follows. The next
section is devoted to the basics of Devaney's chaos.  Section~\ref{S2}
formally  describes  how  to  build  a neural  network  that  operates
chaotically.  Section~\ref{S3} is devoted to the dual case of checking
whether  an existing neural  network is  chaotic or  not.  Topological
properties of chaotic neural networks are discussed in Sect.~\ref{S4}.
The  Section~\ref{section:translation}  shows  how to  translate  such
iterations  into  an Artificial  Neural  Network  (ANN),  in order  to
evaluate the  capability for this  latter to learn  chaotic behaviors.
This  ability  is  studied in  Sect.~\ref{section:experiments},  where
various ANNs try to learn two  sets of data: the first one is obtained
by chaotic iterations while the  second one results from a non-chaotic
system.  Prediction success rates are  given and discussed for the two
sets.  The paper ends with a conclusion section where our contribution
is summed up and intended future work is exposed.


\section{Un réseau de neurones chaotique au sens de Devaney}
\label{S2}

On considère une fonction 
$f:\Bool^n\to\Bool^n$ telle que  $\textsc{giu}(f)$ est fortement connexe.
Ainsi $G_{f_u}$ est chaotique d'après le théorème~\ref{Th:CaracIC}.

On considère ici le schéma unaire défini par l'équation (\ref{eq:asyn}).
On construit un perceptron multi-couche associé à la fonction  
$F_{f_u}$. 
Plus précisément, pour chaque entrée 
 $(x,s)   \in  \mathds{B}^n \times [n]$,
la couche de sortie doit générer $F_{f_u}(x,s)$.   
On peut ainsi lier la couche de sortie avec celle d'entrée pour représenter 
les dépendance entre deux itérations successives.
On obtient une réseau de neurones dont le comportement est le suivant 
(voir Figure.~\ref{Fig:perceptron}):

\begin{itemize}
\item   Le réseau est initialisé avec le vecteur d'entrée
  $\left(x^0,S^0\right)  \mathds{B}^n \times [n]$      
  et calcule le  vecteur de sortie 
  $x^1=F_{f_u}\left(x^0,S^0\right)$. 
  Ce vecteur est publié comme une sortie et est aussi retournée sur la couche d'entrée 
  à travers les liens de retours.
\item Lorsque le réseau est  activé à la $t^{th}$  itération, l'état du 
  système $x^t  \in \mathds{B}^n$ reçu depuis la couche de sortie ainsi que le 
  premier terme de la  sequence $(S^t)^{t  \in \Nats}$
  (\textit{i.e.},  $S^0  \in [n]$)  servent à construire le nouveau vecteur de sortie.
  Ce nouveau vecteur, qui représente le nouvel état du système dynamique, satisfait:
  \begin{equation}
    x^{t+1}=F_{f_u}(x^t,S^0) \in \mathds{B}^n \enspace .
  \end{equation}
\end{itemize}

\begin{figure}
  \centering
  \includegraphics[scale=0.625]{images/perceptron}
  \caption{Un perceptron équivalent  aux itérations unitaires}
  \label{Fig:perceptron}
\end{figure}

Le comportement de ce réseau de neurones est tel que lorsque l'état 
initial est composé de $x^0~\in~\mathds{B}^n$ et d'une séquence
 $(S^t)^{t  \in \Nats}$, alors la séquence  contenant les vecteurs successifs 
publiés $\left(x^t\right)^{t \in \mathds{N}^{\ast}}$ est exactement celle produite 
par les itérations unaires décrites à la section~\ref{sec:TIPE12}.
Mathématiquement, cela signifie que si on utilise les mêmes vecteurs d'entrées
les deux approches génèrent successivement les mêmes sorties.
En d'autres termes ce réseau de neurones modélise le comportement de  
$G_{f_u}$,  dont les itérations sont chaotiques sur $\mathcal{X}_u$.
On peut donc le  qualifier de chaotique au sens de Devaney.

\section{Vérifier si un réseau de neurones est chaotique}
\label{S3}
On s'intéresse maintenant au cas où l'on dispose d'un
réseau de neurones de type perceptron multi-couches
dont on cherche à savoir s'il est chaotique (parce qu'il a par exemple été 
déclaré comme tel) au sens de Devaney. 
On considère de plus que sa topologie est la suivante:
l'entrée est constituée de  $n$ bits et un entier, la sortie est constituée de $n$ bits
et chaque sortie est liée à une entrée par une boucle.

\begin{itemize}
\item Le réseau est initialisé avec  $n$~bits
   $\left(x^0_1,\dots,x^0_n\right)$ et une valeur entière $S^0 \in [n]$.
\item  A l'itération~$t$,     le vecteur 
  $\left(x^t_1,\dots,x^t_n\right)$  permet de construire les $n$~bits 
  servant de sortie $\left(x^{t+1}_1,\dots,x^{t+1}_n\right)$.
\end{itemize}

Le comportement de ce type de réseau de neurones peut être prouvé comme 
étant chaotique en suivant la démarche énoncée maintenant.
On nomme tout d'abord $F:    \mathds{B}^n  \times [n] \rightarrow
\mathds{B}^n$     la      fonction qui associe  
au vecteur 
$\left(\left(x_1,\dots,x_n\right),s\right)    \in   \mathds{B}^n \times[n]$ 
le vecteur 
$\left(y_1,\dots,y_n\right)       \in       \mathds{B}^n$, où
$\left(y_1,\dots,y_n\right)$  sont les sorties du réseau neuronal
àaprès l'initialisation de la couche d'entrée avec 
$\left(s,\left(x_1,\dots, x_n\right)\right)$.  Ensuite, on définie $f:
\mathds{B}^n       \rightarrow      \mathds{B}^n$  telle que 
$f\left(x_1,x_2,\dots,x_n\right)$ est égal à 
\begin{equation}
\left(F\left(\left(x_1,x_2,\dots,x_n\right),1\right),\dots,
  F\left(\left(x_1,x_2,\dots,x_n\right)\right),n\right) \enspace .
\end{equation}
Ainsi pour chaque $j$, $1 \le j \le n$, on a 
$f_j\left(x_1,x_2,\dots,x_n\right) = 
F\left(\left(x_1,x_2,\dots,x_n\right),j\right)$.
Si ce réseau de neurones est initialisé avec 
$\left(x_1^0,\dots,x_n^0\right)$   et $S   \in    [n]^{\mathds{N}}$, 
il produit exactement les même sorties que les itérations de  $F_{f_u}$ avec une 
condition initiale $\left((x_1^0,\dots,  x_n^0),S\right)  \in  \mathds{B}^n \times [n]^{\mathds{N}}$.
Les itérations de $F_{f_u}$ 
sont donc un modèle formel de ce genre de réseau de neurones.
Pour vérifier s'il est chaotique, il suffit ainsi 
de vérifier si le graphe d'itérations
$\textsc{giu}(f)$ est fortement connexe.


\section{Suitability of Feedforward Neural Networks 
for Predicting Chaotic and Non-chaotic Behaviors}

In  the context  of computer  science  different topic  areas have  an
interest       in       chaos,       as       for       steganographic
techniques~\cite{1309431,Zhang2005759}.    Steganography  consists  in
embedding a  secret message within  an ordinary one, while  the secret
extraction takes place once  at destination.  The reverse ({\it i.e.},
automatically detecting the presence  of hidden messages inside media)
is  called   steganalysis.   Among  the   deployed  strategies  inside
detectors,          there         are          support         vectors
machines~\cite{Qiao:2009:SM:1704555.1704664},                   neural
networks~\cite{10.1109/ICME.2003.1221665,10.1109/CIMSiM.2010.36},  and
Markov   chains~\cite{Sullivan06steganalysisfor}.    Most   of   these
detectors  give quite  good results  and are  rather  competitive when
facing steganographic  tools.  However, to  the best of  our knowledge
none of the considered information hiding schemes fulfills the Devaney
definition  of chaos~\cite{Devaney}.  Indeed,  one can  wonder whether
detectors  continue to  give good  results when  facing  truly chaotic
schemes.  More  generally, there remains the open  problem of deciding
whether artificial intelligence is suitable for predicting topological
chaotic behaviors.

\subsection{Representing Chaotic Iterations for Neural Networks} 
\label{section:translation}

The  problem  of  deciding  whether  classical  feedforward  ANNs  are
suitable  to approximate  topological chaotic  iterations may  then be
reduced to  evaluate such neural  networks on iterations  of functions
with  Strongly  Connected  Component  (SCC)~graph of  iterations.   To
compare with  non-chaotic iterations, the experiments  detailed in the
following  sections  are carried  out  using  both  kinds of  function
(chaotic and non-chaotic). Let  us emphasize on the difference between
this  kind  of  neural  networks  and  the  Chaotic  Iterations  based
multilayer peceptron.

We are  then left to compute  two disjoint function  sets that contain
either functions  with topological chaos properties  or not, depending
on  the strong  connectivity of  their iterations graph.  This  can be
achieved for  instance by removing a  set of edges  from the iteration
graph $\Gamma(f_0)$ of the vectorial negation function~$f_0$.  One can
deduce whether  a function verifies the topological  chaos property or
not  by checking  the strong  connectivity of  the resulting  graph of
iterations.

For instance let us consider  the functions $f$ and $g$ from $\Bool^4$
to $\Bool^4$ respectively defined by the following lists:
$$[0,  0,  2,   3,  13,  13,  6,   3,  8,  9,  10,  11,   8,  13,  14,
  15]$$ $$\mbox{and } [11, 14, 13, 14, 11, 10, 1, 8, 7, 6, 5, 4, 3, 2,
  1, 0]  \enspace.$$ In  other words,  the image of  $0011$ by  $g$ is
$1110$: it  is obtained as the  binary value of the  fourth element in
the  second  list  (namely~14).   It   is  not  hard  to  verify  that
$\Gamma(f)$ is  not SCC  (\textit{e.g.}, $f(1111)$ is  $1111$) whereas
$\Gamma(g)$ is. The  remaining of this section shows  how to translate
iterations of such functions into a model amenable to be learned by an
ANN.   Formally, input  and  output vectors  are pairs~$((S^t)^{t  \in
  \Nats},x)$          and          $\left(\sigma((S^t)^{t          \in
  \Nats}),F_{f}(S^0,x)\right)$ as defined in~Eq.~(\ref{eq:Gf}).

Firstly, let us focus on how to memorize configurations.  Two distinct
translations are  proposed.  In the first  case, we take  one input in
$\Bool$  per  component;  in   the  second  case,  configurations  are
memorized  as   natural  numbers.    A  coarse  attempt   to  memorize
configuration  as  natural  number  could  consist  in  labeling  each
configuration  with  its  translation  into  decimal  numeral  system.
However,  such a  representation induces  too many  changes  between a
configuration  labeled by  a  power  of two  and  its direct  previous
configuration: for instance, 16~(10000)  and 15~(01111) are close in a
decimal ordering, but  their Hamming distance is 5.   This is why Gray
codes~\cite{Gray47} have been preferred.

Secondly, let us detail how to deal with strategies.  Obviously, it is
not possible to  translate in a finite way  an infinite strategy, even
if both $(S^t)^{t \in \Nats}$ and $\sigma((S^t)^{t \in \Nats})$ belong
to  $\{1,\ldots,n\}^{\Nats}$.  Input  strategies are  then  reduced to
have a length of size $l \in \llbracket 2,k\rrbracket$, where $k$ is a
parameter of the evaluation. Notice  that $l$ is greater than or equal
to $2$ since  we do not want the shift  $\sigma$~function to return an
empty strategy.  Strategies are memorized as natural numbers expressed
in base  $n+1$.  At  each iteration, either  none or one  component is
modified  (among the  $n$ components)  leading to  a radix  with $n+1$
entries.  Finally,  we give an  other input, namely $m  \in \llbracket
1,l-1\rrbracket$, which  is the  number of successive  iterations that
are applied starting  from $x$.  Outputs are translated  with the same
rules.

To address  the complexity  issue of the  problem, let us  compute the
size of the data set an ANN has to deal with.  Each input vector of an
input-output pair  is composed of a configuration~$x$,  an excerpt $S$
of the strategy to iterate  of size $l \in \llbracket 2, k\rrbracket$,
and a  number $m \in  \llbracket 1, l-1\rrbracket$ of  iterations that
are executed.

Firstly, there are $2^n$  configurations $x$, with $n^l$ strategies of
size $l$ for  each of them. Secondly, for  a given configuration there
are $\omega = 1 \times n^2 +  2 \times n^3 + \ldots+ (k-1) \times n^k$
ways  of writing  the pair  $(m,S)$. Furthermore,  it is  not  hard to
establish that
\begin{equation}
\displaystyle{(n-1) \times \omega = (k-1)\times n^{k+1} - \sum_{i=2}^k n^i} \nonumber
\end{equation}
then
\begin{equation}
\omega =
\dfrac{(k-1)\times n^{k+1}}{n-1} - \dfrac{n^{k+1}-n^2}{(n-1)^2} \enspace . \nonumber
\end{equation}
\noindent And then, finally, the number of  input-output pairs for our 
ANNs is 
$$
2^n \times \left(\dfrac{(k-1)\times n^{k+1}}{n-1} - \dfrac{n^{k+1}-n^2}{(n-1)^2}\right) \enspace .
$$
For  instance, for $4$  binary components  and a  strategy of  at most
$3$~terms we obtain 2304~input-output pairs.

\subsection{Experiments}
\label{section:experiments}

To study  if chaotic iterations can  be predicted, we  choose to train
the multilayer perceptron.  As stated  before, this kind of network is
in  particular  well-known for  its  universal approximation  property
\cite{Cybenko89,DBLP:journals/nn/HornikSW89}.  Furthermore,  MLPs have
been  already  considered for  chaotic  time  series prediction.   For
example,   in~\cite{dalkiran10}  the   authors  have   shown   that  a
feedforward  MLP with  two hidden  layers, and  trained  with Bayesian
Regulation  back-propagation, can learn  successfully the  dynamics of
Chua's circuit.

In  these experiments  we consider  MLPs  having one  hidden layer  of
sigmoidal  neurons  and  output   neurons  with  a  linear  activation
function.     They    are    trained    using    the    Limited-memory
Broyden-Fletcher-Goldfarb-Shanno quasi-newton algorithm in combination
with the Wolfe linear search.  The training process is performed until
a maximum number of epochs  is reached.  To prevent overfitting and to
estimate the  generalization performance we use  holdout validation by
splitting the  data set into  learning, validation, and  test subsets.
These subsets  are obtained through  random selection such  that their
respective size represents 65\%, 10\%, and 25\% of the whole data set.

Several  neural  networks  are  trained  for  both  iterations  coding
schemes.   In  both  cases   iterations  have  the  following  layout:
configurations of  four components and  strategies with at  most three
terms. Thus, for  the first coding scheme a data  set pair is composed
of 6~inputs and 5~outputs, while for the second one it is respectively
3~inputs and 2~outputs. As noticed at the end of the previous section,
this  leads to  data sets  that  consist of  2304~pairs. The  networks
differ  in the  size of  the hidden  layer and  the maximum  number of
training epochs.  We remember that  to evaluate the ability  of neural
networks to  predict a  chaotic behavior for  each coding  scheme, the
trainings of two data sets, one of them describing chaotic iterations,
are compared.

Thereafter we give,  for the different learning setups  and data sets,
the mean prediction success rate obtained for each output. Such a rate
represents the percentage of  input-output pairs belonging to the test
subset  for  which  the   corresponding  output  value  was  correctly
predicted.   These values are  computed considering  10~trainings with
random  subsets  construction,   weights  and  biases  initialization.
Firstly, neural networks having  10 and 25~hidden neurons are trained,
with   a  maximum   number  of   epochs  that   takes  its   value  in
$\{125,250,500\}$  (see Tables~\ref{tab1} and  \ref{tab2}).  Secondly,
we refine the second coding scheme by splitting the output vector such
that   each  output   is  learned   by  a   specific   neural  network
(Table~\ref{tab3}). In  this last  case, we increase  the size  of the
hidden layer up to 40~neurons and we consider larger number of epochs.

\begin{table}[htbp!]
\caption{Prediction success rates for configurations expressed as boolean vectors.}
\label{tab1}
\centering {\small
\begin{tabular}{|c|c||c|c|c|}
\hline 
\multicolumn{5}{|c|}{Networks topology: 6~inputs, 5~outputs, and one hidden layer} \\
\hline
\hline
\multicolumn{2}{|c||}{Hidden neurons} & \multicolumn{3}{c|}{10 neurons} \\
\cline{3-5} 
\multicolumn{2}{|c||}{Epochs} & 125 & 250 & 500 \\ 
\hline
\multirow{6}{*}{\rotatebox{90}{Chaotic}}&Output~(1) & 90.92\% & 91.75\% & 91.82\% \\ 
& Output~(2) & 69.32\% & 78.46\% & 82.15\% \\
& Output~(3) & 68.47\% & 78.49\% & 82.22\% \\
& Output~(4) & 91.53\% & 92.37\% & 93.4\% \\
& Config. & 36.10\% & 51.35\% & 56.85\% \\
& Strategy~(5) & 1.91\% & 3.38\% & 2.43\% \\
\hline
\multirow{6}{*}{\rotatebox{90}{Non-chaotic}}&Output~(1) & 97.64\% & 98.10\% & 98.20\% \\
& Output~(2) & 95.15\% & 95.39\% & 95.46\% \\
& Output~(3) & 100\% & 100\% & 100\% \\
& Output~(4) & 97.47\% & 97.90\% & 97.99\% \\
& Config. & 90.52\% & 91.59\% & 91.73\% \\
& Strategy~(5) & 3.41\% & 3.40\% & 3.47\% \\
\hline
\hline
\multicolumn{2}{|c||}{Hidden neurons} & \multicolumn{3}{c|}{25 neurons} \\ %& \multicolumn{3}{|c|}{40 neurons} \\
\cline{3-5} 
\multicolumn{2}{|c||}{Epochs} & 125 & 250 & 500 \\ %& 125 & 250 & 500 \\ 
\hline
\multirow{6}{*}{\rotatebox{90}{Chaotic}}&Output~(1) & 91.65\% & 92.69\% & 93.93\% \\ %& 91.94\% & 92.89\% & 94.00\% \\
& Output~(2) & 72.06\% & 88.46\% & 90.5\% \\ %& 74.97\% & 89.83\% & 91.14\% \\
& Output~(3) & 79.19\% & 89.83\% & 91.59\% \\ %& 76.69\% & 89.58\% & 91.84\% \\
& Output~(4) & 91.61\% & 92.34\% & 93.47\% \\% & 82.77\% & 92.93\% & 93.48\% \\
& Config. & 48.82\% & 67.80\% & 70.97\% \\%& 49.46\% & 68.94\% & 71.11\% \\
& Strategy~(5) & 2.62\% & 3.43\% & 3.78\% \\% & 3.10\% & 3.10\% & 3.03\% \\
\hline
\multirow{6}{*}{\rotatebox{90}{Non-chaotic}}&Output~(1) & 97.87\% & 97.99\% & 98.03\% \\ %& 98.16\% \\
& Output~(2) & 95.46\% & 95.84\% & 96.75\% \\ % & 97.4\% \\
& Output~(3) & 100\% & 100\% & 100\% \\%& 100\% \\
& Output~(4) & 97.77\% & 97.82\% & 98.06\% \\%& 98.31\% \\
& Config. & 91.36\% & 91.99\% & 93.03\% \\%& 93.98\% \\
& Strategy~(5) & 3.37\% & 3.44\% & 3.29\% \\%& 3.23\% \\
\hline
\end{tabular}
}
\end{table}

Table~\ref{tab1}  presents the  rates  obtained for  the first  coding
scheme.   For  the chaotic  data,  it can  be  seen  that as  expected
configuration  prediction becomes  better  when the  number of  hidden
neurons and maximum  epochs increases: an improvement by  a factor two
is observed (from 36.10\% for 10~neurons and 125~epochs to 70.97\% for
25~neurons  and  500~epochs). We  also  notice  that  the learning  of
outputs~(2)   and~(3)  is   more  difficult.    Conversely,   for  the
non-chaotic  case the  simplest training  setup is  enough  to predict
configurations.  For all these  feedforward network topologies and all
outputs the  obtained results for the non-chaotic  case outperform the
chaotic  ones. Finally,  the rates  for the  strategies show  that the
different feedforward networks are unable to learn them.

For  the  second  coding   scheme  (\textit{i.e.},  with  Gray  Codes)
Table~\ref{tab2} shows  that any network learns about  five times more
non-chaotic  configurations than  chaotic  ones.  As  in the  previous
scheme,       the      strategies      cannot       be      predicted.
Figures~\ref{Fig:chaotic_predictions}                              and
\ref{Fig:non-chaotic_predictions} present the predictions given by two
feedforward multilayer  perceptrons that were  respectively trained to
learn chaotic  and non-chaotic data,  using the second  coding scheme.
Each figure  shows for  each sample of  the test  subset (577~samples,
representing 25\%  of the 2304~samples) the  configuration that should
have been predicted and the one given by the multilayer perceptron. It
can be  seen that for  the chaotic data  the predictions are  far away
from the  expected configurations.  Obviously,  the better predictions
for the non-chaotic data reflect their regularity.

Let us now compare the  two coding schemes. Firstly, the second scheme
disturbs  the   learning  process.   In   fact  in  this   scheme  the
configuration is always expressed as  a natural number, whereas in the
first one  the number  of inputs follows  the increase of  the Boolean
vectors coding configurations. In this latter case, the coding gives a
finer information on configuration evolution.
\begin{table}[b]
\caption{Prediction success rates for configurations expressed with Gray code}
\label{tab2}
\centering
\begin{tabular}{|c|c||c|c|c|}
\hline 
\multicolumn{5}{|c|}{Networks topology: 3~inputs, 2~outputs, and one hidden layer} \\
\hline
\hline
& Hidden neurons & \multicolumn{3}{c|}{10 neurons} \\
\cline{2-5}
& Epochs & 125 & 250 & 500 \\ %& 1000 
\hline
\multirow{2}{*}{Chaotic}& Config.~(1) & 13.29\% & 13.55\% & 13.08\% \\ %& 12.5\%
& Strategy~(2) & 0.50\% & 0.52\% & 1.32\% \\ %& 1.42\%
\hline
\multirow{2}{*}{Non-Chaotic}&Config.~(1) & 77.12\% & 74.00\% & 72.60\% \\ %& 75.81\% 
& Strategy~(2) & 0.42\% & 0.80\% & 1.16\% \\ %& 1.42\% 
\hline
\hline
& Hidden neurons & \multicolumn{3}{c|}{25 neurons} \\
\cline{2-5}
& Epochs & 125 & 250 & 500 \\ %& 1000 
\hline
\multirow{2}{*}{Chaotic}& Config.~(1) & 12.27\% & 13.15\% & 13.05\% \\ %& 15.44\%
& Strategy~(2) & 0.71\% & 0.66\% & 0.88\% \\ %& 1.73\%
\hline
\multirow{2}{*}{Non-Chaotic}&Config.~(1) & 73.60\% & 74.70\% & 75.89\% \\ %& 68.32\%
& Strategy~(2) & 0.64\% & 0.97\% & 1.23\% \\ %& 1.80\%
\hline
\end{tabular}
\end{table}

\begin{figure}
  \centering
  \includegraphics[scale=0.5]{images/chaotic_trace2}
  \caption {Second coding scheme - Predictions obtained for a chaotic test subset.}
  \label{Fig:chaotic_predictions}
\end{figure}

\begin{figure}
  \centering
  \includegraphics[scale=0.5]{images/non-chaotic_trace2} 
  \caption{Second coding scheme - Predictions obtained for a non-chaotic test subset.}
  \label{Fig:non-chaotic_predictions}
\end{figure}

Unfortunately, in  practical applications the number  of components is
usually  unknown.   Hence, the  first  coding  scheme  cannot be  used
systematically.   Therefore, we  provide  a refinement  of the  second
scheme: each  output is learned  by a different  ANN. Table~\ref{tab3}
presents the  results for  this approach.  In  any case,  whatever the
considered feedforward  network topologies, the  maximum epoch number,
and the kind of iterations, the configuration success rate is slightly
improved.   Moreover, the  strategies predictions  rates  reach almost
12\%, whereas in Table~\ref{tab2} they never exceed 1.5\%.  Despite of
this improvement,  a long term prediction of  chaotic iterations still
appear to be an open issue.

\begin{table}
\caption{Prediction success rates for split outputs.}
\label{tab3}
\centering
\begin{tabular}{|c||c|c|c|}
\hline 
\multicolumn{4}{|c|}{Networks topology: 3~inputs, 1~output, and one hidden layer} \\
\hline
\hline
Epochs & 125 & 250 & 500 \\ 
\hline
\hline
Chaotic & \multicolumn{3}{c|}{Output = Configuration} \\
\hline
10~neurons & 12.39\% & 14.06\% & 14.32\% \\
25~neurons & 13.00\% & 14.28\% & 14.58\% \\
40~neurons & 11.58\% & 13.47\% & 14.23\% \\
\hline
\hline
Non chaotic & \multicolumn{3}{c|}{Output = Configuration} \\
\cline{2-4}
%Epochs & 125 & 250 & 500 \\ 
\hline
10~neurons & 76.01\% & 74.04\% & 78.16\% \\
25~neurons & 76.60\% & 72.13\% & 75.96\% \\
40~neurons & 76.34\% & 75.63\% & 77.50\% \\
\hline
\hline
Chaotic/non chaotic & \multicolumn{3}{c|}{Output = Strategy} \\
\cline{2-4}
%Epochs & 125 & 250 & 500 \\ 
\hline
10~neurons & 0.76\% & 0.97\% & 1.21\% \\
25~neurons & 1.09\% & 0.73\% & 1.79\% \\
40~neurons & 0.90\% & 1.02\% & 2.15\% \\
\hline
\multicolumn{4}{c}{} \\
\hline
Epochs & 1000 & 2500 & 5000 \\ 
\hline
\hline
Chaotic & \multicolumn{3}{c|}{Output = Configuration} \\
\hline
10~neurons & 14.51\% & 15.22\% & 15.22\% \\
25~neurons & 16.95\% & 17.57\% & 18.46\% \\
40~neurons & 17.73\% & 20.75\% & 22.62\% \\
\hline
\hline
Non chaotic & \multicolumn{3}{c|}{Output = Configuration} \\
\cline{2-4}
%Epochs & 1000 & 2500 & 5000 \\ 
\hline
10~neurons & 78.98\% & 80.02\% & 79.97\% \\
25~neurons & 79.19\% & 81.59\% & 81.53\% \\
40~neurons & 79.64\% & 81.37\% & 81.37\% \\
\hline
\hline
Chaotic/non chaotic & \multicolumn{3}{c|}{Output = Strategy} \\
\cline{2-4}
%Epochs & 1000 & 2500 & 5000 \\ 
\hline
10~neurons & 3.47\% & 9.98\% & 11.66\% \\
25~neurons & 3.92\% & 8.63\% & 10.09\% \\
40~neurons & 3.29\% & 7.19\% & 7.18\% \\
\hline
\end{tabular}
\end{table}

\section{Conclusion}

In  this paper,  we have  established an  equivalence  between chaotic
iterations,  according to  the Devaney's  definition of  chaos,  and a
class  of multilayer  perceptron  neural networks.   Firstly, we  have
described how to build a neural network that can be trained to learn a
given chaotic map function. Secondly,  we found a condition that allow
to check whether  the iterations induced by a  function are chaotic or
not, and thus  if a chaotic map is obtained.  Thanks to this condition
our  approach is not  limited to  a particular  function. In  the dual
case, we show that checking if a neural network is chaotic consists in
verifying  a property  on an  associated  graph, called  the graph  of
iterations.   These results  are valid  for recurrent  neural networks
with a  particular architecture.  However,  we believe that  a similar
work can be done for  other neural network architectures.  Finally, we
have  discovered at  least one  family of  problems with  a reasonable
size, such  that artificial neural  networks should not be  applied in
the  presence  of chaos,  due  to  their  inability to  learn  chaotic
behaviors in this  context.  Such a consideration is  not reduced to a
theoretical detail:  this family of discrete  iterations is concretely
implemented  in a  new steganographic  method  \cite{guyeux10ter}.  As
steganographic   detectors  embed  tools   like  neural   networks  to
distinguish between  original and stego contents, our  studies tend to
prove that such  detectors might be unable to  tackle with chaos-based
information  hiding  schemes.

In  future  work we  intend  to  enlarge  the comparison  between  the
learning   of  truly   chaotic  and   non-chaotic   behaviors.   Other
computational intelligence tools such  as support vector machines will
be investigated  too, to  discover which tools  are the  most relevant
when facing a truly chaotic phenomenon.  A comparison between learning
rate  success  and  prediction  quality will  be  realized.   Concrete
consequences in biology, physics, and computer science security fields
will then be stated.

% \appendix{}

% \begin{Def} \label{def2}
% A continuous function $f$  on a topological space $(\mathcal{X},\tau)$
% is defined  to be  {\emph{topologically transitive}}  if for any  pair of
% open  sets $U$,  $V  \in  \mathcal{X}$ there  exists  
% $k \in
% \mathds{N}^{\ast}$
%  such  that
% $f^k(U) \cap V \neq \emptyset$.
% \end{Def}

%\bibliography{chaos-paper}% Produces the bibliography via BibTeX.

%\end{document}
%
% ****** End of file chaos-paper.tex ******