In the development of perceptual metrics for texture similarity, Zujovic et al. [59, 60] combined separate estimates of texture similarity in terms of structure and color composition. Taking into consideration the fact that textures can have similar structure and different color composition, as shown in Figure 1(a), or similar color composition and different structure, as shown in Fig. 1(b), they argued that how these separate estimates should be combined should depend on the observer and the application [59, 60].

Zujovic et al. [59, 60] argued that the structure of a texture can be reasonably approximated with the structure of the grayscale component of the image. While the chrominance also contributes to the structure, the case where structure is solely determined by the chrominance is possible [56] but unlikely in natural or synthetic textures. To evaluate the similarity of the structure of grayscale textures, they then used perceptually based structural texture similarity metrics (STSIMs) [61].

To estimate the color composition of a texture, as perceived by a human, rather than as a histogram of the colors of the pixels, Zujovic et al. [59, 60] adopted the representation in terms of dominant colors and their percentages, and used the optimal color composition distance (OCCD) [37], which is closely related to the earth mover’s distance (EMD) [49], to compare the color compositions of two textures. Figure 2(a) shows an original texture image, Fig. 2(b) shows its grayscale component, and Fig. 2(c) shows its dominant colors. The representation of the color composition of a texture in terms of its dominant colors has been well established in the engineering literature [7, 29, 38, 59, 60]. We could not find any related work in the psychophysical literature, except for the work of Kuriki [25] on average-color perception of multicolored patterns. Kimura [22] also considered color averaging of multicolor mosaics into a few color categories, with emphasis on the mean color judgments.

The key assumption in Zujovic’s approach to texture similarity is that texture structure and color composition are independent of each other. The goal of this article is to test whether this assumption is consistent with human perception and, in particular, whether the perception of color composition is veridical and whether it is affected by the texture structure.

We explore the influence of texture structure on the perception of the color composition through a series of empirical studies. While the motivation comes from texture similarity and content-based retrieval, the problem is interesting in its own right, and to the best of our knowledge has not been addressed in the psychophysical literature. Thus, while we will discuss the implications of the conclusions of our studies for texture similarity metrics, the main focus is on understanding the perception of the color composition of visual textures.

To study the effects of texture structure on color composition, we synthesized color textures with the same color composition as that extracted from the original texture but with different structural patterns, and conducted empirical studies to compare the perceived color composition of the synthesized textures.

As in [59, 60], we estimated the color composition of a given original texture image using the adaptive clustering algorithm (ACA) [43] to segment the image in KACA = 4 classes based on color, and “painting” the segments with the average of each class. ACA is an iterative algorithm that can be regarded as a generalization of the K-means clustering algorithm [28, 55] in two respects: it adapts to local variations in image intensity and includes spatial constraints in the form of Markov random fields (MRFs). We will refer to the resulting image, which preserves the structure of the original image, and consolidates the point clouds of each dominant color into points in color space, as the posterized texture. The posterized texture for the original texture image of Fig. 2(a) is shown in Fig. 2(c). The color composition of the texture consists of the (dominant) colors of this posterized texture and their percentages, shown in Fig. 2(d). We then synthesized textures with the same or modified color composition. We used three types of structural patterns for the synthetic textures: isotropic blobs, squares, and rectangles. As in the Julesz experiments [19], the synthesized textures were designed to eliminate familiarity cues, and the pattern shapes were chosen to create different structures (shape, scale) so that we can test their effect on the perceived color composition. An example of a synthetic texture with isotropic blobs is shown in Fig. 2(e).

We conducted three empirical studies with the synthetic textures to determine the effects of structure on the perception of color composition. In the first study, we compared the structure of the posterized texture with isotropic blob synthetic textures that have varying target color percentages. In the other two studies, we used square and rectangular synthetic textures to investigate the effect of scale and shape on the perceived amount of the target color. We conducted the empirical studies in the form of two-alternative forced choice (2AFC) tests to determine which of a pair of patterns is perceived as containing a larger amount of a given target dominant color. The main conclusions of our empirical studies are that (a) participants are able to consistently assess differences in color composition for textures of similar scale and shape, and (b) pattern agglomeration in both scale (larger) and shape (less elongated) has a strong positive influence on the perceived amount of a target color. Thus, the perception of color composition is nonveridical.

The question is then: What are the perceptual mechanisms that can account for the results of our empirical studies? First, the results are consistent with Julesz’s texton theory [20], whereby the size and shape of the textons affect the texture perception. In Section 5, we present a simple model that estimates the contribution of each pixel to the perceived amount of the target color. Pixels near the blob boundaries contribute less than pixels at the center of the blobs. While there is a large literature on texture and color perception, we could not find any perceptual mechanisms that can explain such an edge effect. Here we should clarify that the size of the color blobs in our studies was selected so that the blobs are perceived as distinct patches of color with sharp edges. There is no averaging across the blob boundaries, even though a simple linear filter model based on estimates of the spatial frequency sensitivity of the eye (e.g., in [32]) predicts averaging over a few pixels at the given display resolution and viewing distance. The perception of sharp boundaries can be attributed to adaptation [12] or color spreading [47], but neither provides a solid explanation. In addition, with only a couple of exceptions, the differences in the colors of the blobs are well above threshold in both luminance and chrominance, so that differences in contrast sensitivity of luminance and chrominance edges [24] and changes in contrast sensitivity based on spatial pattern [48] or interactions between chrominance and luminance [53] cannot account for the observed results, and the effects of scale in particular. Another interesting observation is that, as in the case of texture metamers [8], all pixels in the color blobs are clearly visible, yet their contributions to the overall perception of color composition are different.

Webster et al. [57] and Maule and Franklin [36] conducted experiments with ensembles of distinct color blobs, like ours, but in regular formations of color circles. However, the emphasis was on the perception of the average color rather than the color composition.

Objective descriptors of the color composition of an image have been extensively studied in the image retrieval literature over the past decades. The most straightforward way for describing the color content of an image is via a color histogram; two images are then compared using a histogram distance metric [50, 52]. Manjunath et al. [31] review three descriptors based on histogram representation for the MPEG-7 standard: the scalable color descriptor, the dominant color descriptor, and the color layout descriptor. The scalable color descriptor is a color histogram of an image encoded based on the Haar transform. The dominant color descriptor describes the colors of an image using a small number of dominant color values and the related statistical properties. The color layout descriptor captures the spatial distribution of dominant colors based on the discrete cosine transform (DCT).

The use of dominant colors and associated percentages as a compact color representation for image analysis was introduced by Ma et al. [29] and Deng et al. [9] and adopted by Mojsilovic et al. [38]. They argued that, when evaluating the color composition of an image, the visual system discounts local subtle variations and focuses on a few dominant colors.

In addition to an appropriate color representation, the color composition of images must be compared in a way that agrees with human perception. Mojsilovic et al. [37] proposed the OCCD, which is implemented in CIELab space and which is closely related to the EMD [49]. In the paper that introduced EMD, Rubner et al. [49] also emphasized the need for compact color representations, which is consistent with the use of dominant colors.

In line with these findings, and in order to account for the nonuniformity of the statistical characteristics of natural textures, Chen et al. [7] introduced the idea of spatially adaptive dominant colors in the context of color texture segmentation and proposed the use of ACA [43] to estimate them and OCCD [37] to compare them. Zujovic et al. [59, 60] then incorporated the spatially adaptive dominant colors and the OCCD into a structural texture similarity metric that, as we discussed above, assumes that color composition and texture structure can be estimated independently.

As we discussed in the introduction, we estimate the color composition of a texture image by extracting the dominant colors and the associated percentages, using ACA [43] to segment the image into regions of slowly varying colors with rapid changes at the boundaries. As we saw, this results in the posterized images of Fig. 2(c).

For spatially homogeneous textures, a small number of segment classes, typically KACA = 2 to 4, is sufficient to capture the dominant structure and colors of the image. Pappas et al. [44] found that, for natural images, the majority of segments containing perceptually uniform textures can be characterized by just the first two dominant colors for effective texture classification. Similarly, He and Pappas [14, 15] proposed a segmentation algorithm that is based on the fact that natural textures consist of intensity variations of a single hue [41]. However, as we discuss in Section 3, for our experiments we selected stimuli that have a variety of patterns and colors, with KACA = 4.

The feature vector that specifies the color composition, shown in Fig. 2(d), consists of the KACA averages, expressed in CIELab color coordinates Lk∗, ak∗, bk∗ and the associated color percentages pk with k = 1, …, KACA.

Among the extracted dominant colors, we define the most illuminant color as the color with the highest value in the luminance (L∗) channel and with percentage at least 20%. We also define the most distinct color as the color with the largest average ΔE distance from the other dominant colors. The ΔE distance between two colors (L1∗, a1∗, b1∗) and (L2∗, a2∗, b2∗) is determined as follows: where ΔL∗ = L1∗− L2∗, Δa∗ = a1∗− a2∗, and Δb∗ = b1∗− b2∗. The selection of the most illuminant and most distinct colors for the empirical studies we describe below was arbitrary. Initially, we thought that it would make it easier to run our studies. However, we have no evidence of that, and expect that similar outcomes would be observed with any other color choice.

Our goal is to synthesize textures with a given color composition but with structure different from that of the posterized texture. We used two approaches for generating textures, one with isotropic blobs and the other with blocks of different geometries (squares, rectangles, lines) and scales.

The first study was carried out with the posterized textures and the synthetic isotropic blob textures. As we discussed, the goal of this study was to determine whether the difference in structure between the posterized texture and the synthetic isotropic blob textures affects the perception of the target color amount. As we will see, this study also demonstrates that the perceived differences in the amount of target color are consistent with the actual color percentages.

We employed Thurstonian scaling [54] to convert the 2AFC results to preference scores for each texture set. The model assumes that the relative magnitudes of the preferences for the stimuli can be determined by the winning frequencies that one stimulus is selected over another in a paired comparison task. We accumulated the comparative choices across participants for each pair and computed a preference matrix for each texture set with the winning frequencies between 0 and 1. The values in each preference matrix were omitted and treated as missing values when the winning frequency was too small (< 0.02) or too large (> 0.98) to give stable estimates [11]. The diagonal values were set to 0.5. We then applied the Thurstone Case V model to convert pairwise preferences to continuous perception scores. The scores were further normalized to Z-scores to facilitate evaluation across the texture sets. The Z-scores are obtained by subtracting the mean and dividing by the standard deviation. Larger Z-score indicates stronger perception of the target color amount.

Figure 8 displays the Z-scores of each texture set as a function of the physical target color percentage difference. The dotted line in each plot represents the perception of the posterized image among the synthetic images. The overall increasing tendencies in Fig. 8 show that the participants are capable of perceiving the changes in the amount of target color. The fitted linear regression between the Z-scores and the physical target color percentage difference in each texture set suggests that the perceived color amount is linearly related to the physical percentage difference in target color. We counted the correct responses of the 2AFC results between S–S pairs. The probabilities of correct response for 3%, 6%, and 9% target color differences are 0.82, 0.9, and 0.98, with 36, 45, and 36 samples, respectively, which indicates that the just noticeable difference of color amount is below the 3% difference in color.

By comparing the Z-scores of the posterized and the synthetic images, we notice that the perceived target color amount of most texture sets is between the perceived target color amount of the synthetic images with ± 3% actual target color amount difference. The only exceptions are Texture Sets 1 and 5. For Texture Set 1, the posterized texture is perceived as having a much higher percentage of the target color than the synthetic image with 0% target color difference, while the opposite is true for Texture Set 5. Table I lists the perceived target color difference (ΔZ) of each texture set.

We now examine the factors that may have caused the perceived color amount difference between the posterized and the synthetic image (0%). Three possible factors are explored: (a) the scale difference of the target color blobs in the posterized and synthetic images (ΔS); (b) the lightness difference between the posterized and the synthetic images (ΔL); (c) the color distance between the target color and the most similar adjacent color (ΔE).

One obvious difference is the color scale. For Texture Set 1 of Fig. 4, the target color blobs of the posterized texture appear to be larger than those of the synthetic textures, while the opposite is true for Texture Set 5. To quantitatively check the above observations, we calculated the average linear scale of the target color blobs in each image as the geometric mean of the average horizontal and vertical lengths of the blobs, as described in appendix B. Table I lists the scale difference (ΔS) between the texture synthesized with 0% difference in the target color and the posterized texture. To check how the scale difference relates to the perceived target color amount difference, we fitted a linear regression between ΔZ and ΔS (ΔZ = c ⋅ ΔS + interp). The estimated coefficients (c = 0.1, p = 0.002, interp = 0.08) with R2 = 0.778 show that the scale difference has high positive correlation with the perceptual difference.

Another factor that may have influenced the outcome of our experiments is the average lightness of the image. Note that if the target color is lighter or darker than the other colors, then by modifying the amount of target color we are also modifying the average lightness of the image. Thus, one could hypothesize that the participants may have ordered the textures according to overall lightness rather than according to the amount of perceived target color, as instructed. Figure 9 plots the average lightness of each synthetic image for each texture set. Note that the most dramatic changes occur in Texture Sets 3, 4, 6, 7 (increasing slope), and 9 (decreasing slope). However, the placement of the posterized texture does not seem to be related to the change in average lightness. The lightness difference ΔL between the posterized and the synthetic (0%) images for each texture set is listed in Table I. Similar to the analysis of scale difference, we fitted a linear regression between ΔZ and ΔL (ΔZ = c ⋅ ΔL + interp) to check the relation between the perception and lightness. The estimated coefficients (c = 0.36, p = 0.57, interp = −0.198) with R2 = 0.047 show that the lightness does not play a significant role in influencing the perceived difference in color amount.

Apart from the color scale and lightness, the similarity of spatially adjacent colors could also affect the perceived target color amount, especially for Texture Set 1. In Texture Set 1, the most similar colors (two shades of strawberries) are adjacent in the posterized texture but randomly placed in the synthetic texture. ΔE in Table I lists the distance between the target color and the most similar adjacent color for each texture set. Note that the most similar adjacent color (dark red) occurs for Texture Set 1; for all the other sets, the difference between adjacent colors is quite large, and hence does not seem to have any effect on the perceived target color amount.

To investigate the relative contributions of each factor, we fitted a multiple regression with the three factors on the perceived color amount difference ΔZ (ΔZ = c1 ⋅ ΔS + c2 ⋅ ΔL + c3 ⋅ ΔE + interp). The estimated coefficients with the corresponding p-values and standardized coefficients are listed in Table I. Note that ΔS and ΔL contribute positively to ΔZ, and ΔE negatively, but the p-values and the standardized coefficients show that only the ΔS contributions to ΔZ are statistically significant, and that the contributions of the other independent variables are negligible.

In the analysis of Study 2 below, where we vary the scale of the blobs for a fixed color composition (and hence fixed lightness), we further investigate the relationship between the target color scale and the perception of target color amount.

The goal of this study was to consider the direct effects of scale on the perceived amount of the target color. This study was carried out with the posterized textures and textures synthesized with squares of different sizes. We compared the synthetic textures to each other, as well as to the posterized texture. As shown in Fig. 5, each color set has four different scales (generating block sizes of 8 × 8, 16 × 16, 24 × 24, and 32 × 32) with the same color composition, and the same four scales with +3% of the target color.

As in Study 1, we employed Thurstonian scaling [54] to convert the 2AFC results to preference scores for each texture set. Using the nine sets as replication, a repeated-measures analysis of variance (ANOVA) with two within-subject factors (block scale level and percentage) verified that the perceived target color amount is affected by the scale (block size), F(3,24) = 123.4, p < 0.0001, and by the actual color amount change, F(1,24) = 109.3, p < 0.0001. There is no interaction between the block size and actual color amount, F(3,24) = 0.578, p = 0.635.

Figure 10 shows the Z-scores of each texture set. A higher score indicates a higher amount of perceived target color. It is no surprise that the values for the +3% images are higher than those for the 0% images at each scale since the perceived color amount is consistent with the actual color amount, as shown in Study 1. In addition, the increasing scores for each of the texture sets in Fig. 10 demonstrate that pattern agglomeration (increasing scale) results in the perception of increasing target color amount. The score of the posterized texture is shown in dotted black line for each texture set. It is clear that the placement of the posterized images varies considerably across the texture sets.

Figure 11 presents the average linear scale of the target color blobs of the posterized and synthetic textures. A two-way ANOVA was employed to test whether, in addition to the generating block size, the + 3% color difference affected the scale values. The analysis shows that apart from the statistically significant effect of block size (F(3,24) = 73.9, p < 0.0001), the average linear scale is also affected by the actual color percentage difference (F(1,24) = 5.84, p = 0.02). There is no interaction between the block size and actual color amount, F(3,24) = 0.29, p = 0.83.

Note that for Texture Sets 1, 4, 8, and 9, the linear scale of the posterized texture is relatively higher compared to that of the synthetic textures, and the scale of Texture Set 5 relatively lower, which is consistent with the results in Fig. 10. Note, however, that the relative location of the posterized (dotted black line) compared to the synthetic textures in Fig. 10 is generally higher than the corresponding location of the posterized relative to the synthetic textures in Fig. 11. Here we should also point out that even though the images synthesized with a given generating square size are expected to have the same scale, and hence the 0% and + 3% bar heights are expected to coincide, the actual estimated scales differ due to the varying percentages (as we noted above, higher percentages result in higher probability of agglomeration) and the randomness in the block placements.

To summarize the relationship between color scale and color amount perception, Figure 12 shows a combined view of Figs. 10 and 11 across all texture sets (posterized textures excluded). The position of each dot is obtained by averaging the linear scales (x-axis) and the Z-scores (y-axis) of all synthetic textures with the same color block scale (8 × 8, 16 × 16, 24 × 24, 32 × 32) and the same actual color amount difference (0%, +3%). We used linear regression (Z = b ⋅ Scale + δ ⋅ColorAmountDifference + interp) to investigate the influence of the average linear scale on the perceived color amount (Z-score) with the actual color amount difference as a dummy variable. The estimated coefficients (slope: b = 0.102, p < 0.001; δ = 0.709, p < 0.001; interp = −2.48, p < 0.001) demonstrate that there exists a linear positive relationship between the target linear scale and the perceived color amount.

The goal of this study was to investigate the effect of shape on the perception of the amount of the target color. In this study, we only compared synthesized textures (with rectangles of different H/W ratios) to each other. The analysis is similar to that of the second study. Using the nine textures as replication, a repeated-measures ANOVA with two within-subject factors (elongation level and percentage) verified that the perceived target color amount is affected by the H/W ratio, F(3,24) = 16.46, p < 0.0001, and by the actual color amount change, F(1,24) = 190.8, p < 0.0001. The actual color amount and elongation degree in structure are independent with no interaction, F(3,24) = 0.654, p = 0.589.

Figure 13 shows the estimates of the perceived scores within each color set. The labels of the x-axis are the H/W ratios of color blocks used to synthesize each texture. The + 3% synthesized textures are almost always higher than the 0% synthesized textures at each H/W ratio due to the increase in the actual target color amount. Apart from Texture Set 6, the general trend is that the perceived amount of target color decreases as the geometric shape becomes more elongated (H/W ratio increases) for both the 0% and the + 3% textures. One possible explanation of the weak relation between shape and color perception in Texture Set 6 is the influence of pattern scale, which is considerably smaller than in the other sets. While the H/W ratio was changed, the scales of all synthesized textures in this study were designed with scales similar to the posterized texture. When the pattern scale is small, the shape variations are not as noticeable as they are in the larger pattern scales.

Figure 14 shows the bar plots of the average elongation degree of each texture composed of color blocks with the specified H/W ratio. The average elongation degree is defined as max(Lv,Lh)min(Lv,Lh), where Lv and Lh are the average vertical and horizontal lengths of the target color blobs based on the algorithm in Appendix A. As expected, the elongation degree increases as the color block H/W ratio increases. Note, however, that the resulting elongation degree is not equal to the H/W ratio, due to block overlap and random block placement. As in the analysis of the scale study, a two-way ANOVA was employed to test whether, in addition to the generating block H/W ratio, the + 3% color difference affected the elongation degree values. The analysis shows that apart from the statistically significant effect of color block H/W ratio (F(3,24) = 231, p < 0.0001), the influence of actual color percentage difference on the average elongation degree is not statistically significant (F = 0.97, p = 0.33). There is no interaction between the color block H/W ratio and the actual color amount, F(3,24) = 0.5, p = 0.68.

To summarize the relationship between color elongation and color amount perception, Figure 15 shows a combined view of Figs. 13 and 14 across all texture sets. The position of each dot represents the average elongation degree (x-axis) and the Z-score (y-axis) of all synthetic textures with the same color block ratio (1 : 1, 2 : 1, 4 : 1, 8 : 1) and the same actual color amount difference (0%, +3%). We used linear regression (Z = b ⋅Elongation + δ ⋅ColorAmountDifference + interp) to investigate the influence of shape (average elongation degree) on the perceived color amount (Z-score) with the actual color amount difference as a dummy variable. The estimated coefficients (slope: b = −0.38, p < 0.001; δ = 1.366, p < 0.001; interp = 0.217, p = 0.08) demonstrate that there exists a linear negative relationship between the elongation degree and the perceived color amount.