1 TAMPERE IMAGE DATABASE 2008 TID2008, version 1.0
\r
3 TID2008 is intended for evaluation of full-reference image visual
\r
4 quality assessment metrics. TID2008 allows estimating how a given
\r
5 metric corresponds to mean human perception. For example, in
\r
6 accordance with TID2008, Spearman correlation between the metric
\r
7 PSNR (Peak Signal to Noise Ratio) and mean human perception (MOS,
\r
8 Mean Opinion Score) is 0.525.
\r
10 Permission to use, copy, or modify this database and its documentation
\r
11 for educational and research purposes only and without fee is hereby
\r
12 granted, provided that this copyright notice and the original authors'
\r
13 names appear on all copies and supporting documentation. This database
\r
14 shall not be modified without first obtaining permission of the authors.
\r
15 The authors make no representations about the suitability of this
\r
16 database for any purpose. It is provided "as is" without express
\r
17 or implied warranty.
\r
19 In case of publishing results obtained by means of TID2008 please refer
\r
20 to one of the following papers (see files mre2009tid.pdf and
\r
21 vpqm2009tid.pdf in the "papers\" direcory):
\r
23 [1] N. Ponomarenko, V. Lukin, A. Zelensky, K. Egiazarian, M. Carli,
\r
24 F. Battisti, "TID2008 - A Database for Evaluation of Full-Reference
\r
25 Visual Quality Assessment Metrics", Advances of Modern
\r
26 Radioelectronics, Vol. 10, pp. 30-45, 2009.
\r
28 [2] N. Ponomarenko, F. Battisti, K. Egiazarian, J. Astola, V. Lukin
\r
29 "Metrics performance comparison for color image database", Fourth
\r
30 international workshop on video processing and quality metrics
\r
31 for consumer electronics, Scottsdale, Arizona, USA. Jan. 14-16, 2009, 6 p.
\r
33 The TID2008 contains 25 reference images and 1700 distorted images
\r
34 (25 reference images x 17 types of distortions x 4 levels of distortions).
\r
35 All images are saved in database in Bitmap format without any compression.
\r
36 File names are organized in such a manner that they indicate a number of
\r
37 the reference image, then a number of distortion's type, and, finally, a
\r
38 number of distortion's level: "iXX_YY_Z.bmp".
\r
40 For example, the name "i03_08_4.bmp" means the 3-rd reference image corrupted
\r
41 by the 8-th type of distortions with the 4-th level of this distortion.
\r
42 Similarly, the name "i12_10_1.bmp" means that this is the 12-th reference
\r
43 image corrupted by the 10-th type of distortion with the first level.
\r
44 "i17.bmp" means that this is non-distorted 17-th reference image.
\r
46 TABLE I. Types of distortion used in TID2008
\r
48 ü Type of distortion
\r
50 1 Additive Gaussian noise
\r
51 2 Additive noise in color components is more intensive than additive noise in the luminance component
\r
52 3 Spatially correlated noise
\r
54 5 High frequency noise
\r
56 7 Quantization noise
\r
60 11 JPEG2000 compression
\r
61 12 JPEG transmission errors
\r
62 13 JPEG2000 transmission errors
\r
63 14 Non eccentricity pattern noise
\r
64 15 Local block-wise distortions of different intensity
\r
65 16 Mean shift (intensity shift)
\r
68 See [1] for a more detailed explanation.
\r
70 The file "mos.txt" contains the Mean Opinion Score for each distorted image.
\r
71 The file "mos_with_names.txt" contains the same information and filenames of
\r
72 the coressponding distorted images.
\r
73 The file "mos_std.txt" contains standard deviation of MOS for each
\r
76 The MOS was obtained from the results of 838 experiments carried out by
\r
77 observers from three countries: Finland, Italy, and Ukraine (251 experiments
\r
78 have been carried out in Finland, 150 in Italy, and 437 in Ukraine). Totally,
\r
79 the 838 observers have performed 256428 comparisons of visual quality of
\r
80 distorted images or 512856 evaluations of relative visual quality in image
\r
83 Higer value of MOS (0 - minimal, 9 - maximal) corresponds to higer visual
\r
84 quality of the image.
\r
86 The following files contain values of some quality metrics calculated for
\r
89 "psnr.txt" - peak signal to noise ratio;
\r
90 "psnry.txt" - peak signal to noise ratio calculated for the luminance component;
\r
91 "snr.txt" - signal to noise ratio [3].
\r
92 "mse.txt" - inverted values of mean square error [3].
\r
93 "dctune.txt" - inverted values of the DCTune metric [4];
\r
94 "uqi.txt" - values of the UQI metric [5];
\r
95 "ssim.txt" - values of the SSIM metric [6];
\r
96 "mssim.txt" - vaules of the MSSIM metric [7,3];
\r
97 "linlab.txt" - inverted values of the LinLab metric [8];
\r
98 "xyz" - inverted values of the YCxCz2XYZ metric [9];
\r
99 "psnrhvs.txt" - values of the PSNR-HVS metric [10];
\r
100 "psnrhvsm.txt" - values of the PSNR-HVS-M metric [11];
\r
101 "vif.txt" - values of the VIF metric [12,3];
\r
102 "vifp.txt" - pixel domain version VIF [12,3];
\r
103 "nqm.txt" - values of the NQM metric [13,3];
\r
104 "wsnr.txt" - values of the WSNR metric [14,3];
\r
105 "ifc.txt" - values of the IFC metric [15,3];
\r
106 "vsnr.txt" - values of the VSNR metric [16,3];
\r
108 [3] Matthew Gaubatz, "Metrix MUX Visual Quality Assessment Package: MSE,
\r
109 PSNR, SSIM, MSSIM, VSNR, VIF, VIFP, UQI, IFC, NQM, WSNR, SNR",
\r
110 http://foulard.ece.cornell.edu/gaubatz/metrix_mux/
\r
111 [4] A. B. Watson, "DCTune: A technique for visual optimization of DCT
\r
112 quantization matrices for individual images," Soc. Inf. Display Dig.
\r
113 Tech. Papers, vol. XXIV, pp. 946-949, 1993.
\r
114 [5] Z. Wang, A. Bovik, "A universal image quality index", IEEE Signal
\r
115 Processing Letters, vol. 9, pp. 81-84, March, 2002.
\r
116 [6] Z. Wang, A. Bovik, H. Sheikh, E. Simoncelli, "Image quality assessment:
\r
117 from error visibility to structural similarity", IEEE Transactions on
\r
118 Image Proc., vol. 13, issue 4, pp. 600-612, April, 2004.
\r
119 [7] Z. Wang, E. P. Simoncelli and A. C. Bovik, "Multi-scale structural
\r
120 similarity for image quality assessment," Invited Paper, IEEE Asilomar
\r
121 Conference on Signals, Systems and Computers, Nov. 2003.
\r
122 [8] B. Kolpatzik and C. Bouman, "Optimized Error Diffusion for High Quality
\r
123 Image Display", Journal Electronic Imaging, pp. 277-292, 1992.
\r
124 [9] B. W. Kolpatzik and C. A. Bouman, "Optimized Universal Color Palette
\r
125 Design for Error Diffusion", Journal Electronic Imaging, vol. 4,
\r
127 [10] K. Egiazarian, J. Astola, N. Ponomarenko, V. Lukin, F. Battisti,
\r
128 M. Carli, "New full-reference quality metrics based on HVS", CD-ROM
\r
129 Proceedings of the Second International Workshop on Video Processing
\r
130 and Quality Metrics, Scottsdale, USA, 2006, 4 p.
\r
131 [11] N. Ponomarenko, F. Silvestri, K. Egiazarian, M. Carli, J. Astola,
\r
132 V. Lukin "On between-coefficient contrast masking of DCT basis
\r
133 functions", CD-ROM Proc. of the Third International Workshop on Video
\r
134 Processing and Quality Metrics. - USA, 2007. - 4 p.
\r
135 [12] H.R. Sheikh.and A.C. Bovik, "Image information and visual quality,"
\r
136 IEEE Transactions on Image Processing, Vol.15, no.2, 2006, pp. 430-444.
\r
137 [13] Damera-Venkata N., Kite T., Geisler W., Evans B. and Bovik A. "Image
\r
138 Quality Assessment Based on a Degradation Model", IEEE Trans. on Image
\r
139 Processing, Vol. 9, 2000, pp. 636-650.
\r
140 [14] T. Mitsa and K. Varkur, "Evaluation of contrast sensitivity functions
\r
141 for the formulation of quality measures incorporated in halftoning
\r
142 algorithms", ICASSP '93-V, pp. 301-304.
\r
143 [15] H.R. Sheikh, A.C. Bovik and G. de Veciana, "An information fidelity
\r
144 criterion for image quality assessment using natural scene statistics",
\r
145 IEEE Transactions on Image Processing, vol.14, no.12, 2005, pp. 2117-2128.
\r
146 [16] D.M. Chandler, S.S. Hemami, "VSNR: A Wavelet-Based Visual Signal-to-Noise
\r
147 Ratio for Natural Images", IEEE Transactions on Image Processing,
\r
148 Vol. 16 (9), pp. 2284-2298, 2007.
\r
150 The programs "spearman.exe" and "kendall.exe" calculate values of Spearman
\r
151 and Kendall rank correlations for entire set of the TID2008 images as well
\r
152 as for particular subsets given in the Table II.
\r
154 TABLE II. Subsets of TID2008 definded by default
\r
156 ü Type of distortion Noise Noise2 Safe Hard Simple Exotic Exotic2 Full
\r
158 1 Additive Gaussian noise + + + - + - - +
\r
159 2 Noise in color comp. - + - - - - - +
\r
160 3 Spatially correl. noise + + + + - - - +
\r
161 4 Masked noise - + - + - - - +
\r
162 5 High frequency noise + + + - - - - +
\r
163 6 Impulse noise + + + - - - - +
\r
164 7 Quantization noise + + - + - - - +
\r
165 8 Gaussian blur + + + + + - - +
\r
166 9 Image denoising + - - + - - - +
\r
167 10 JPEG compression - - + - + - - +
\r
168 11 JPEG2000 compression - - + - + - - +
\r
169 12 JPEG transm. errors - - - + - - + +
\r
170 13 JPEG2000 transm. errors - - - + - - + +
\r
171 14 Non ecc. patt. noise - - - + - + + +
\r
172 15 Local block-wise dist. - - - - - + + +
\r
173 16 Mean shift - - - - - + + +
\r
174 17 Contrast change - - - - - + + +
\r
176 The command line is "spearman <data1> <data2>" or "kendall <data1> <data2>".
\r
178 Command line examples:
\r
180 spearman mos.txt ssim.txt
\r
181 kendall mos.txt dctune.txt
\r
182 spearman linlab.txt xyz.txt
\r
183 kendall psnr.txt psnr-hvs.txt
\r
185 An example of usage:
\r
187 kendall.exe mos.txt uqi.txt
\r
197 TABLE III. Ranking of compared metrics in accordance with
\r
198 Spearman correlation with MOS
\r
199 Rank Measure Spearman correlation
\r
219 TABLE IV. Ranking of compared metrics in accordance with
\r
220 Kendall correlation with MOS
\r
221 Rank Measure Kendall correlation
\r
241 We plan to regularly update the versions of this database. New versions
\r
242 will include new types of distortion and take into account results of
\r
243 additional experiments.
\r
245 We will highly appreciate authors of other metrics if they will inform
\r
246 us (please, mail to karen@cs.tut.fi or nikolay@ponomarenko.info) how to get
\r
247 executable files (e.g., Matlab codes) of their metrics. We guarantee
\r
248 that we will not pass them to other users and will include future
\r
249 results obtained for such metrics in analysis for our database.
\r
