update

[chloroplast13.git] / annotated.tex

diff --git a/annotated.tex b/annotated.tex
index 9d98987f93da1dcd098484538e1d31921123625b..dd57e11dd760cf0fbef6fcfa783cfa26c7678ebb 100644 (file)
--- a/annotated.tex
+++ b/annotated.tex
@@ -190,9 +190,9 @@ to extract core genes, as explained in Algorithm \ref{Alg3:thirdM}.
 \STATE $geneList=\text{empty list}$
 \STATE $common=set(dir(NCBI\_Genes)) \cap set(dir(Dogma\_Genes))$
 \FOR{$\text{gene in common}$}
 \STATE $geneList=\text{empty list}$
 \STATE $common=set(dir(NCBI\_Genes)) \cap set(dir(Dogma\_Genes))$
 \FOR{$\text{gene in common}$}

-       \STATE $gen1 \leftarrow open(NCBI\_Genes(gene)).read()$         
-       \STATE $gen2 \leftarrow open(Dogma\_Genes(gene)).read()$
-       \STATE $score \leftarrow geneChk(gen1,gen2)$
+       \STATE $g1 \leftarrow open(NCBI\_Genes(gene)).read()$   
+       \STATE $g2 \leftarrow open(Dogma\_Genes(gene)).read()$
+       \STATE $score \leftarrow geneChk(g1,g2)$

        \IF {$score > Threshold$}
                \STATE $geneList \leftarrow gene$
        \ENDIF 
        \IF {$score > Threshold$}
                \STATE $geneList \leftarrow gene$
        \ENDIF 


@@ -207,16 +207,16 @@ and {\it reverse complement}. Algorithm~\ref{Alg3:genechk} gives the outline of
 geneChk subroutine.
 
 \begin{algorithm}[H]
 geneChk subroutine.
 
 \begin{algorithm}[H]

-\caption{Find the maximum similarity score between two sequences}
+\caption{Find the Maximum Similarity Score between two sequences}

 \label{Alg3:genechk}
 \begin{algorithmic} 
 \label{Alg3:genechk}
 \begin{algorithmic} 

-\REQUIRE $gen1,gen2 \leftarrow \text{NCBI gene sequence, Dogma gene sequence}$
+\REQUIRE $g1,g2 \leftarrow \text{NCBI gene sequence, Dogma gene sequence}$

 \ENSURE $\text{Maximum similarity score}$
 \ENSURE $\text{Maximum similarity score}$

-\STATE $Score1 \leftarrow needle(gen1,gen2)$
-\STATE $Score2 \leftarrow needle(gen1,Reverse(gen2))$
-\STATE $Score3 \leftarrow needle(gen1,Complement(gen2))$
-\STATE $Score4 \leftarrow needle(gen1,Reverse(Complement(gen2)))$
-\RETURN $max(Score1, Score2, Score3, Score4)$
+\STATE $score1 \leftarrow needle(g1,g2)$
+\STATE $score2 \leftarrow needle(g1,Reverse(g2))$
+\STATE $score3 \leftarrow needle(g1,Complement(g2))$
+\STATE $score4 \leftarrow needle(g1,Reverse(Complement(g2)))$
+\RETURN $max(score1,score2,score3,score4)$

 \end{algorithmic}
 \end{algorithm}  
 
 \end{algorithmic}
 \end{algorithm}  
 


@@ -230,168 +230,141 @@ to   one   genome.   Hence,   an   element   of   the  matrix   stores
 the  \textit{Intersection Score}  (IS):  the cardinality  of the  core
 genes   set  obtained   by  intersecting   one  genome   with  another
 one. Maximum  cardinality results in selecting the  two genomes having
 the  \textit{Intersection Score}  (IS):  the cardinality  of the  core
 genes   set  obtained   by  intersecting   one  genome   with  another
 one. Maximum  cardinality results in selecting the  two genomes having

-the maximum score. Mathematically speaking, if we have an $n \times n$
-matrix where $n$ is the number  of genomes in local database, then let
-us consider:
-
+the maximum score. Mathematically speaking, if we have $n$ genomes in
+local database, the ICM is an $n \times n$ matrix whose elements
+satisfy: 

 \begin{equation}
 \begin{equation}

-Score=\max_{i<j}\vert x_i \cap x_j\vert
+score_{ij}=\vert g_i \cap g_j\vert

 \label{Eq1}
 \end{equation}
 \label{Eq1}
 \end{equation}

-
-% TO BE CONTINUED
-
-\noindent where $x_i, x_j$ are elements in the matrix. The generation of 
-a new core genes is depending on the cardinality value of intersection 
-scores, we call it \textit{Score}:
-$$
-\text{New Core} = 
-\begin{cases}
-\text{Ignored} & \text{if $\textit{Score}=0$;} \\
-\text{new Core id} & \text{if $\textit{Score}>0$.}
-\end{cases}
-$$
-
-if     $\textit{Score}=0$     then     we    have     \textit{disjoint
-relation} \emph{i.e.},  no common genes between two  genomes.  In this
-case  the  system  ignores  the   genome  that  annul  the  core  gene
-size. Otherwise, The system removes these two genomes from ICM and add
-new  core  genome  with a  \textit{coreID}  of  them  to ICM  for  the
-calculation in  next iteration. This  process reduces the size  of ICM
-and repeats until all genomes  are treated \emph{i.e.} ICM has no more
-genomes.  We observe  that ICM is very large because  of the amount of
-data that it stores. This results  to be time and memory consuming for
-calculating  the  intersection  scores.   To  increase  the  speed  of
-calculations, it  is sufficient to  only calculate the  upper triangle
-scores. The time complexity for this process after enhancement is thus
-$O(\frac{n.(n-1)}{2})$.   Algorithm   \ref{Alg1:ICM}  illustrates  the
-construction of  the ICM matrix and  the extraction of  the core genes
-where \textit{GenomeList},  represents the database  where all genomes
-data are stored. At each iteration, it computes the maximum core genes
-with its two genomes parents.
+\noindent where $1 \leq i \leq n$, $1 \leq j \leq n$, and $g_i, g_j$ are 
+genomes. The  generation of a new  core gene depends  obviously on the
+value  of the  intersection scores  $score_{ij}$. More  precisely, the
+idea is  to consider a  pair of genomes  such that their score  is the
+largest element in ICM. These two genomes are then removed from matrix
+and the  resulting new  core genome is  added for the  next iteration.
+The ICM is then updated to take into account the new core gene: new IS
+values are computed for it. This process is repeated until no new core
+gene can be obtained.
+
+We  can observe  that  the ICM  is very  large  due to  the amount  of
+data. As a consequence, the  computation of the intersection scores is
+both  time and  memory consuming.  However,  since ICM  is a  symetric
+matrix we can reduce the  computation overhead by considering only its
+triangular  upper part.  The  time complexity  for this  process after
+enhancement is thus $O(\frac{n.(n-1)}{2})$.  Algorithm ~\ref{Alg1:ICM}
+illustrates the construction  of the ICM matrix and  the extraction of
+the  core  genes, where  \textit{GenomeList}  represents the  database
+storing all genomes  data. At each iteration, it  computes the maximum
+core genes with its two genomes parents.
+
+% ALGORITHM HAS BEEN REWRITTEN

 
 \begin{algorithm}[H]
 \caption{Extract Maximum Intersection Score}
 \label{Alg1:ICM}
 \begin{algorithmic} 
 
 \begin{algorithm}[H]
 \caption{Extract Maximum Intersection Score}
 \label{Alg1:ICM}
 \begin{algorithmic} 

-\REQUIRE $L \leftarrow \text{genomes vectors}$
-\ENSURE $B1 \leftarrow Max core vector$ 
+\REQUIRE $L \leftarrow \text{genomes sets}$
+\ENSURE $B1 \leftarrow \text{Max Core set}$ 

 \FOR{$i \leftarrow 0:len(L)-1$}
 \FOR{$i \leftarrow 0:len(L)-1$}

+        \STATE $score \leftarrow 0$

        \STATE $core1 \leftarrow set(GenomeList[L[i]])$
        \STATE $core1 \leftarrow set(GenomeList[L[i]])$

-       \STATE $score1 \leftarrow 0$
-       \STATE $g1,g2 \leftarrow$ " "
+       \STATE $g1 \leftarrow L[i]$

        \FOR{$j \leftarrow i+1:len(L)$}
        \FOR{$j \leftarrow i+1:len(L)$}

-               \STATE $core2 \leftarrow set(GenomeList[L[i]])$
-               \IF{$i < j$}
-                       \STATE $Core \leftarrow core1 \cap core2$
-                       \IF{$len(Core) > score1$}
-                               \STATE $g1 \leftarrow L[i]$
-                               \STATE $g2 \leftarrow L[j]$
-                               \STATE $Score \leftarrow len(Core)$
-                       \ELSIF{$len(Core) == 0$}
-                               \STATE $g1 \leftarrow L[i]$
-                               \STATE $g2 \leftarrow L[j]$
-                               \STATE $Score \leftarrow -1$
-                       \ENDIF
-               \ENDIF
+               \STATE $core2 \leftarrow set(GenomeList[L[j]])$
+               \STATE $Core \leftarrow core1 \cap core2$
+               \IF{$len(Core) > score$}
+                  \STATE $score \leftarrow len(Core)$
+                 \STATE $g2 \leftarrow L[j]$
+                \ENDIF

        \ENDFOR
        \ENDFOR

-       \STATE $B1[score1] \leftarrow (g1,g2)$
+       \STATE $B1[score] \leftarrow (g1,g2)$

 \ENDFOR
 \RETURN $max(B1)$
 \end{algorithmic}
 \end{algorithm}
 
 \ENDFOR
 \RETURN $max(B1)$
 \end{algorithmic}
 \end{algorithm}
 

-\subsection{Features Visualization}
+\subsection{Features visualization}

 
 The goal is to visualize results  by building a tree of evolution. All
 
 The goal is to visualize results  by building a tree of evolution. All

-core  genes generated  represent  important information  in the  tree,
-because they  provide information about  the ancestors of two  or more
+core  genes generated  represent  an important information  in the  tree,
+because they  provide ancestor information of two  or more

 genomes. Each  node in the  tree represents one chloroplast  genome or
 genomes. Each  node in the  tree represents one chloroplast  genome or

-one predicted core called \textit{(Genes count:Family name\_Scientific
-names\_Accession number)},  while an edge  is labeled with  the number
-genes lost from a leaf genome or an intermediate core gene.
-
-
-The number of lost genes here can represent an important factor
-for evolution:  it represents how much  is the lost of  genes from the
-species belongs  to same  or different families.  By the  principle of
-classification, a  small number of  gene lost among  species indicates
-that those species are close to  each other and belong to same family,
-while big genes  lost means that we have  an evolutionary relationship
-between species  from different families. To see  the picture clearly,
-Phylogenetic  tree  is an  evolutionary  tree  generated  also by  the
-system. Generating  this tree  is based on  the distances  among genes
-sequences. There are  many resources to build such  tree (for example:
+one predicted core and labelled as \textit{(Genes count:Family name\_Scientific
+names\_Accession number)}. While an edge is labelled with the number of
+lost  genes from  a leaf  genome or  an intermediate  core  gene. Such
+numbers are  very interesting because  they give an  information about
+the evolution:  how many genes  were lost between two  species whether
+they  belong  to  the  same  lineage  or not. Phylogenetic relationships are mainly built by comparison of sets of coding and non-coding sequences. Phylogenies of photosynthetic plants are important to assess the origin of chloroplasts (REF) and the modalities of gene loss among lineages. These phylogenies are usually done using less than ten chloroplastic genes (REF), and some of them may not be conserved by evolution process for every taxa. As phylogenetic relationships inferred from data matrices complete for each species included and with the same evolution history are better assumptions, we selected core genomes for a new investivation of photosynthetic plants phylogeny. To depict  the links between
+species   clearly,  we   built   a  phylogenetic   tree  showing   the
+relationships based on the distances among genes sequences. Many tools
+are    available   to    obtain    a   such    tree,   for    example:

 PHYML\cite{guindon2005phyml},
 PHYML\cite{guindon2005phyml},

-RAxML{\cite{stamatakis2008raxml,stamatakis2005raxml},   BioNJ,  and
-TNT\cite{goloboff2008tnt}}.       
-
-We use
-RAxML\cite{stamatakis2008raxml,stamatakis2005raxml} because it is fast
-and  accurate  for  build  large  trees for  large  count  of  genomes
-sequences. 
-
-The  procedure of constructing phylogenetic  tree stated in
-the following steps:
+RAxML{\cite{stamatakis2008raxml,stamatakis2005raxml},    BioNJ,    and
+TNT\cite{goloboff2008tnt}}.    In   this  work,   we   chose  to   use
+RAxML\cite{stamatakis2008raxml,stamatakis2005raxml}   because   it  is
+fast, accurate,  and can build large  trees when dealing  with a large
+number of genomic sequences.

 
 

+The procedure used to built a phylogenetic tree is as follows:

 \begin{enumerate}
 \begin{enumerate}

-\item Extract gene sequence for all gene in all core genes, store it in database.
-\item Use multiple alignment tool such as (****to be write after see christophe****) to align these sequences with each others.
-\item aligned genomes sequences then submitted to RAxML program to compute the distances and draw phylogenetic tree.
+\item For each gene in a core gene, extract its sequence and store it in the database.
+\item Use multiple alignment tools such as (****to be write after see christophe****) 
+to align these sequences with each others.
+\item we use an outer-group genome from cyanobacteria to calculate distances.
+\item Submit the resulting aligned sequences to RAxML program to compute the distances and finally draw the phylogenetic tree.

 \end{enumerate} 
 
 \begin{figure}[H]
 \end{enumerate} 
 
 \begin{figure}[H]

-  \centering
-    \includegraphics[width=0.7\textwidth]{Whole_system}
-    \caption{Total overview of the system pipeline}\label{wholesystem}
+  \centering \includegraphics[width=0.75\textwidth]{Whole_system}
+  \caption{Overview of the pipeline}\label{wholesystem}

 \end{figure}
 
 \end{figure}
 

-% STOP HERE
-

 \section{Implementation}
 \section{Implementation}

-We implemented four algorithms to extract maximum core genes from large amount of chloroplast genomes. Two algorithms used to extract core genes based on NCBI annotation, and the others based on dogma annotation tool. Evolutionary tree generated as a result from each method implementation. In this section, we will present the four methods, and how they can extract maximum core genes?, and how the developed code will generate the evolutionary tree.
-
-\subsection{Extract Core Genes based on Gene Contents}
+We implemented the three algorithms using dell laptop model latitude E6430 with 6 GB of memory, and Intel core i5 processor of 2.5 Ghz$\times 4$ with 3 MB of CPU cash. We built the code using python version 2.7 under ubuntu 12.04 LTS. We also used python packages such as os, Biopython, memory\_profile, re, numpy, time, shutil, and xlsxwriter to extract core genes from large amount of chloroplast genomes. Table \ref{Etime}, show the annotation type, execution time, and the number of core genes for each method:
+
+\begin{center}
+\begin{tiny}
+\begin{table}[H]
+\caption{Type of Annotation, Execution Time, and core genes for each method}\label{Etime}
+\begin{tabular}{p{2.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.5cm}p{0.2cm}}
+\hline\hline
+ & \multicolumn{2}{c}{Annotation} & \multicolumn{2}{c}{Features} & \multicolumn{2}{c}{E. Time} & \multicolumn{2}{c}{C. genes} & \multicolumn{2}{c}{Bad Gen.} \\
+~ & N & D & Name & Seq & N & D & N & D & N & D \\
+\hline
+Gene prediction & $\surd$ & - & - & $\surd$ & 1.7 & - & ? & - & 0 & -\\[0.5ex]
+Gene Features & $\surd$ & $\surd$ & $\surd$ & - & 4.98 & 1.52 & 28 & 10 & 1 & 0\\[0.5ex]
+Gene Quality & $\surd$ & $\surd$ & $\surd$ & $\surd$ & \multicolumn{2}{c}{$\simeq$3 days + 1.29} & \multicolumn{2}{c}{4} & \multicolumn{2}{c}{1}\\[1ex]
+\hline
+\end{tabular}
+\end{table}
+\end{tiny}
+\end{center} 
+
+In table \ref{Etime}, we show that all methods need low execution time to finish extracting core genes from large chloroplast genomes except in gene quality method where we need about 3-4 days for sequence comparisons to construct quality genomes then it takes just 1.29 minute to extract core genes. This low execution time give us a privilage to use these methods to extract core genes on a personal comuters rather than main frames or parallel computers. In the table, \textbf{N} means NCBI, \textbf{D} means DOGMA, and \textbf{Seq} means Sequence. Annotation is represent the type of algorithm used to annotate chloroplast genome. We can see that the two last methods used the same annotation sources. Features means the type of gene feature used to extract core genes, and this is done by extracting gene name, gene sequence, or both of them. The execution time is represented the whole time needed to extract core genes in minutes. We can see in the table that the second method specially with DOGMA annotation has the lowest execution time of 1.52 minute. In last method We needs approxemetly three days (this period is depend on the amount of genomes) to finish the operation of extracting quality genomes only, while the execution time will be 1.29 minute if we have quality genomes. The number of core genes is represents the amount of genes in the last core genome. The main goal is to find the maximum core genes that simulate biological background of chloroplasts. With NCBI we have 28 genes for 96 genomes instead of 10 genes with DOGMA for 97 genomes. But the biological distribution of genomes with NCBI in core tree did not reflect good biological perspective. While in the core tree with DOGMA, the distribution of genomes are biologically good. Bad genomes are the number of genomes that destroy core genes because of the low number of gene intersection. \textit{NC\_012568.1 Micromonas pusilla}, is the only genome that observed to destroy the core genome with NCBI based on the method of gene features and in the third method of gene quality.  \\
+
+The second important factor is the amount of memory usage in each methodology. Table \ref{mem} show the amounts of memory consumption by each method.
+
+\begin{center}
+\begin{tiny}
+\begin{table}[H]
+\caption{Memory usages in (MB) for each methodology}\label{mem}
+\begin{tabular}{p{2.5cm}p{1.5cm}p{1cm}p{1cm}p{1cm}p{1cm}p{1cm}p{1cm}}
+\hline\hline
+Method& & Load Gen. & Conv. gV & Read gV & ICM & Core tree & Core Seq. \\
+\hline
+Gene prediction & NCBI & 100 & - & - & - & 108 & -\\
+\multirow{2}{*}{Gene Features} & NCBI & 15.4 & 18.9 & 17.5 & 18 & 18 & 28.1\\
+              & DOGMA& 15.3 & 15.3 & 16.8 & 17.8 & 17.9 & 31.2\\
+Gene Quality  & ~ & 15.3 & $\le$3G & 16.1 & 17 & 17.1 & 24.4\\ 
+\hline
+\end{tabular}
+\end{table}
+\end{tiny}
+\end{center}  
+
+We used a package from PyPI~(\textit{the Python Package Index}) where located at~ (https://pypi.python.org/pypi) named \textit{Memory\_profile} to extract all the values in table \ref{mem}. In this table, all the values are presented in mega bytes and \textit{gV} means genevision file format. We see that all memory levels in all methods are reletively low and can be available in any personal computer. All memory values shows that the method of gene features based on DOGMA annotation have the more resonable memory values to extract core genome from loading genomes until extracting core sequences. The third method, gives us the lowest values if we already have the quality genomes, but it will consume high memory locations if we do not have them. Also, the amount of memory locations in the third method vary according to the size of each genome.\\

 
 

-\subsubsection{Core Genes based on NCBI Annotation}
-The first idea to construct the core genome is based on the extraction of Genes names (as gene presence or absence). For instant, in this stage neither sequence comparison nor new annotation were made, we just want to extract all genes with counts stored in each chloroplast genome, then find the intersection core genes based on gene names. \\
-The pipeline of extracting core genes can summarize in the following steps according to pre-processing method used:\\
-
-\begin{enumerate}
-\item We downloads already annotated chloroplast genomes in the form of fasta coding genes (\emph{i.e.} \textit{exons}).
-\item Extract genes names and apply to solve gene duplication using first method. 
-\item Convert fasta file format to geneVision file format to generate ICM. 
-\item Calculate ICM matrix to find maximum core \textit{Score}. New core genes for two genomes will generate and a specific \textit{CoreId} will assign to it. This process continue until no elements remain in the matrix.
-\item Evolutionary tree will take place by using all data generated from step 1 and 4. The tree will also display the amount of genes lost from each intersection iteration. A specific excel file will be generated that store all the data in local database.
-\end{enumerate}

 
 

-There main drawback with this method is genes orthography (e.g two different genes sequences with same gene name). In this case, Gene lost is considered by solving gene duplication based on first method to solve gene duplication.

 
 

-\subsubsection{Core Genes based on Dogma Annotation}  
-The main goal is to get as much as possible the core genes of maximum coding genes names. According to NCBI annotation problem based on \cite{Bakke2009}, annotation method like dogma can give us more reliable coding genes than NCBI. This is because NCBI annotation can carry some annotation and gene identification errors. The general overview of whole process of extraction illustrated in figure \ref{wholesystem}.

 
 

-extracting core genes based on genes names and counts summarized in the following steps:\\
-\begin{enumerate}
-\item We apply the genome annotation manually using Dogma annotation tool. 
-\item Analysing genomes to store lists of code genes names (\textit{i.e. exons}). solve gene fragments is done by using first method in solve gene fragments. The output from annotation process with dogma is genomes files in GenVision file format. Sets of genes were stored in the database.
-\item Generate ICM matrix to calculate maximum core genes.
-\item Draw the evolutionary tree by extracted all genes sequences from each core. Then applying multiple alignment process on the sequences to calculate the distance among cores to draw a phylogenetic tree.
-
-\end{enumerate}
-
-
-The main drawback from the method of extracting core genes based on gene names and counts is that we can not depending only on genes names because of three causes: first, the genome may have not totally named (This can be found in early versions of NCBI genomes), so we will have some lost sequences. Second, we may have two genes sharing the same name, while their sequences are different. Third, we need to annotate all the genomes.               
-
-\subsection{Extract Core Genes based on Gene Quality Control}
-The main idea from this method is to focus on genes quality to predict maximum core genes. By comparing only genes names from one annotation tool is not enough. The question here, does the predicted gene from NCBI is the same gene predicted by Dogma based on gene name and gene sequence?. If yes, then we can predict new quality genomes based on quality control test with a specific threshold. Predicted Genomes comes from merging two annotation techniques. While if no, we can not depending neither on NCBI nor Dogma because of annotation error. Core genes can by predicted by using one of the 
-
-\subsubsection{Core genes based on NCBI and Dogma Annotation}
-This method summarized in the following steps:\\
-
-\begin{enumerate}
-\item Retrieve the annotation of all genomes from NCBI and Dogma: in this step, we apply the annotation of all chloroplast genomes in the database using NCBI annotation and Dogma annotation tool.
-\item Convert NCBI genomes to GeneVision file format, then apply the second method of gene defragmentation methods for NCBI and dogma genomes.  
-\item Predict quality genomes: the process is to pick a genome annotation from two sources, extracting all common genes based on genes names, then applying Needle-man wunch algorithm to align the two sequences based on a threshold equal to 65\%. If the alignment score pass the threshold, then this gene will removed from the competition and store it in quality genome by saving its name with the largest gene sequence with respect to start and end codons. All quality genomes will store in the form of GenVision file format.    
-\item Extract Core genes: from the above two steps, we will have new genomes with quality genes, ofcourse, we have some genes lost here, because dogma produced tRNA and rRNA genes while NCBI did not generate rRNA genes and vise-versa. Build ICM to extract core genes will be sufficient because we already check genes sequences.
-\item Display tree: An evolution tree then will be display based on the intersections of quality genomes.   
-\end{enumerate}