TAMPERE IMAGE DATABASE 2008 TID2008, version 1.0 TID2008 is intended for evaluation of full-reference image visual quality assessment metrics. TID2008 allows estimating how a given metric corresponds to mean human perception. For example, in accordance with TID2008, Spearman correlation between the metric PSNR (Peak Signal to Noise Ratio) and mean human perception (MOS, Mean Opinion Score) is 0.525. Permission to use, copy, or modify this database and its documentation for educational and research purposes only and without fee is hereby granted, provided that this copyright notice and the original authors' names appear on all copies and supporting documentation. This database shall not be modified without first obtaining permission of the authors. The authors make no representations about the suitability of this database for any purpose. It is provided "as is" without express or implied warranty. In case of publishing results obtained by means of TID2008 please refer to one of the following papers (see files mre2009tid.pdf and vpqm2009tid.pdf in the "papers\" direcory): [1] N. Ponomarenko, V. Lukin, A. Zelensky, K. Egiazarian, M. Carli, F. Battisti, "TID2008 - A Database for Evaluation of Full-Reference Visual Quality Assessment Metrics", Advances of Modern Radioelectronics, Vol. 10, pp. 30-45, 2009. [2] N. Ponomarenko, F. Battisti, K. Egiazarian, J. Astola, V. Lukin "Metrics performance comparison for color image database", Fourth international workshop on video processing and quality metrics for consumer electronics, Scottsdale, Arizona, USA. Jan. 14-16, 2009, 6 p. The TID2008 contains 25 reference images and 1700 distorted images (25 reference images x 17 types of distortions x 4 levels of distortions). All images are saved in database in Bitmap format without any compression. File names are organized in such a manner that they indicate a number of the reference image, then a number of distortion's type, and, finally, a number of distortion's level: "iXX_YY_Z.bmp". For example, the name "i03_08_4.bmp" means the 3-rd reference image corrupted by the 8-th type of distortions with the 4-th level of this distortion. Similarly, the name "i12_10_1.bmp" means that this is the 12-th reference image corrupted by the 10-th type of distortion with the first level. "i17.bmp" means that this is non-distorted 17-th reference image. TABLE I. Types of distortion used in TID2008 ü Type of distortion 1 Additive Gaussian noise 2 Additive noise in color components is more intensive than additive noise in the luminance component 3 Spatially correlated noise 4 Masked noise 5 High frequency noise 6 Impulse noise 7 Quantization noise 8 Gaussian blur 9 Image denoising 10 JPEG compression 11 JPEG2000 compression 12 JPEG transmission errors 13 JPEG2000 transmission errors 14 Non eccentricity pattern noise 15 Local block-wise distortions of different intensity 16 Mean shift (intensity shift) 17 Contrast change See [1] for a more detailed explanation. The file "mos.txt" contains the Mean Opinion Score for each distorted image. The file "mos_with_names.txt" contains the same information and filenames of the coressponding distorted images. The file "mos_std.txt" contains standard deviation of MOS for each distorted image. The MOS was obtained from the results of 838 experiments carried out by observers from three countries: Finland, Italy, and Ukraine (251 experiments have been carried out in Finland, 150 in Italy, and 437 in Ukraine). Totally, the 838 observers have performed 256428 comparisons of visual quality of distorted images or 512856 evaluations of relative visual quality in image pairs. Higer value of MOS (0 - minimal, 9 - maximal) corresponds to higer visual quality of the image. The following files contain values of some quality metrics calculated for the TID2008 images: "psnr.txt" - peak signal to noise ratio; "psnry.txt" - peak signal to noise ratio calculated for the luminance component; "snr.txt" - signal to noise ratio [3]. "mse.txt" - inverted values of mean square error [3]. "dctune.txt" - inverted values of the DCTune metric [4]; "uqi.txt" - values of the UQI metric [5]; "ssim.txt" - values of the SSIM metric [6]; "mssim.txt" - vaules of the MSSIM metric [7,3]; "linlab.txt" - inverted values of the LinLab metric [8]; "xyz" - inverted values of the YCxCz2XYZ metric [9]; "psnrhvs.txt" - values of the PSNR-HVS metric [10]; "psnrhvsm.txt" - values of the PSNR-HVS-M metric [11]; "vif.txt" - values of the VIF metric [12,3]; "vifp.txt" - pixel domain version VIF [12,3]; "nqm.txt" - values of the NQM metric [13,3]; "wsnr.txt" - values of the WSNR metric [14,3]; "ifc.txt" - values of the IFC metric [15,3]; "vsnr.txt" - values of the VSNR metric [16,3]; [3] Matthew Gaubatz, "Metrix MUX Visual Quality Assessment Package: MSE, PSNR, SSIM, MSSIM, VSNR, VIF, VIFP, UQI, IFC, NQM, WSNR, SNR", http://foulard.ece.cornell.edu/gaubatz/metrix_mux/ [4] A. B. Watson, "DCTune: A technique for visual optimization of DCT quantization matrices for individual images," Soc. Inf. Display Dig. Tech. Papers, vol. XXIV, pp. 946-949, 1993. [5] Z. Wang, A. Bovik, "A universal image quality index", IEEE Signal Processing Letters, vol. 9, pp. 81-84, March, 2002. [6] Z. Wang, A. Bovik, H. Sheikh, E. Simoncelli, "Image quality assessment: from error visibility to structural similarity", IEEE Transactions on Image Proc., vol. 13, issue 4, pp. 600-612, April, 2004. [7] Z. Wang, E. P. Simoncelli and A. C. Bovik, "Multi-scale structural similarity for image quality assessment," Invited Paper, IEEE Asilomar Conference on Signals, Systems and Computers, Nov. 2003. [8] B. Kolpatzik and C. Bouman, "Optimized Error Diffusion for High Quality Image Display", Journal Electronic Imaging, pp. 277-292, 1992. [9] B. W. Kolpatzik and C. A. Bouman, "Optimized Universal Color Palette Design for Error Diffusion", Journal Electronic Imaging, vol. 4, pp. 131-143, 1995. [10] K. Egiazarian, J. Astola, N. Ponomarenko, V. Lukin, F. Battisti, M. Carli, "New full-reference quality metrics based on HVS", CD-ROM Proceedings of the Second International Workshop on Video Processing and Quality Metrics, Scottsdale, USA, 2006, 4 p. [11] N. Ponomarenko, F. Silvestri, K. Egiazarian, M. Carli, J. Astola, V. Lukin "On between-coefficient contrast masking of DCT basis functions", CD-ROM Proc. of the Third International Workshop on Video Processing and Quality Metrics. - USA, 2007. - 4 p. [12] H.R. Sheikh.and A.C. Bovik, "Image information and visual quality," IEEE Transactions on Image Processing, Vol.15, no.2, 2006, pp. 430-444. [13] Damera-Venkata N., Kite T., Geisler W., Evans B. and Bovik A. "Image Quality Assessment Based on a Degradation Model", IEEE Trans. on Image Processing, Vol. 9, 2000, pp. 636-650. [14] T. Mitsa and K. Varkur, "Evaluation of contrast sensitivity functions for the formulation of quality measures incorporated in halftoning algorithms", ICASSP '93-V, pp. 301-304. [15] H.R. Sheikh, A.C. Bovik and G. de Veciana, "An information fidelity criterion for image quality assessment using natural scene statistics", IEEE Transactions on Image Processing, vol.14, no.12, 2005, pp. 2117-2128. [16] D.M. Chandler, S.S. Hemami, "VSNR: A Wavelet-Based Visual Signal-to-Noise Ratio for Natural Images", IEEE Transactions on Image Processing, Vol. 16 (9), pp. 2284-2298, 2007. The programs "spearman.exe" and "kendall.exe" calculate values of Spearman and Kendall rank correlations for entire set of the TID2008 images as well as for particular subsets given in the Table II. TABLE II. Subsets of TID2008 definded by default ü Type of distortion Noise Noise2 Safe Hard Simple Exotic Exotic2 Full 1 Additive Gaussian noise + + + - + - - + 2 Noise in color comp. - + - - - - - + 3 Spatially correl. noise + + + + - - - + 4 Masked noise - + - + - - - + 5 High frequency noise + + + - - - - + 6 Impulse noise + + + - - - - + 7 Quantization noise + + - + - - - + 8 Gaussian blur + + + + + - - + 9 Image denoising + - - + - - - + 10 JPEG compression - - + - + - - + 11 JPEG2000 compression - - + - + - - + 12 JPEG transm. errors - - - + - - + + 13 JPEG2000 transm. errors - - - + - - + + 14 Non ecc. patt. noise - - - + - + + + 15 Local block-wise dist. - - - - - + + + 16 Mean shift - - - - - + + + 17 Contrast change - - - - - + + + The command line is "spearman " or "kendall ". Command line examples: spearman mos.txt ssim.txt kendall mos.txt dctune.txt spearman linlab.txt xyz.txt kendall psnr.txt psnr-hvs.txt An example of usage: kendall.exe mos.txt uqi.txt Noise : 0.363 Noise2 : 0.419 Safe : 0.454 Hard : 0.568 Simple : 0.586 Exotic : 0.214 Exotic2: 0.405 Full : 0.438 TABLE III. Ranking of compared metrics in accordance with Spearman correlation with MOS Rank Measure Spearman correlation 1 MSSIM 0.853 2 VIF 0.750 3 VSNR 0.705 4 VIFP 0.655 5 SSIM 0.645 6 NQM 0.624 7 UQI 0.600 8 PSNRHVS 0.594 9 XYZ 0.577 10 IFC 0.569 11 PSNRHVSM 0.559 12 PSNRY 0.553 13 PSNR 0.525 14 MSE 0.525 15 SNR 0.523 16 WSNR 0.488 17 LINLAB 0.487 18 DCTUNE 0.476 TABLE IV. Ranking of compared metrics in accordance with Kendall correlation with MOS Rank Measure Kendall correlation 1 MSSIM 0.654 2 VIF 0.586 3 VSNR 0.534 4 VIFP 0.495 5 PSNRHVS 0.476 6 SSIM 0.468 7 NQM 0.461 8 PSNRHVSM 0.449 9 UQI 0.435 10 XYZ 0.434 11 IFC 0.426 12 PSNRY 0.402 13 WSNR 0.393 14 LINLAB 0.381 15 SNR 0.374 16 DCTUNE 0.372 17 PSNR 0.369 18 MSE 0.369 We plan to regularly update the versions of this database. New versions will include new types of distortion and take into account results of additional experiments. We will highly appreciate authors of other metrics if they will inform us (please, mail to karen@cs.tut.fi or nikolay@ponomarenko.info) how to get executable files (e.g., Matlab codes) of their metrics. We guarantee that we will not pass them to other users and will include future results obtained for such metrics in analysis for our database.