Learning-based Artificial Intelligence Artwork: Methodology Taxonomy and Quality Evaluation

NST is a prototypical style-transform AI artwork method. Figure 2 shows an NST result generated by Gatys et al. [38]. NST works in an image-to-image manner, extracting texture features from a style image and content features from a content image, then fusing them to synthesize a new image. Modeling the style image and extracting its texture features is crucial. The goal is to reconstruct an image with the style textures from the style image while preserving the content of the content image.

The NST method, introduced in the work of Gatys et al. [38], uses convolutional neural networks (CNNs) to transfer style texture to a target image while resolving its content. The Gram matrix models the style image’s representation, and the pre-trained VGG network’s high-level features represent the content image. By minimizing content and style losses, the method synthesizes an image with both input images’ content and style. However, this style representation focuses on texture rather than global arrangement, resulting in unsatisfactory results for long-range symmetric structures. Berger and Memisevic [5] improved this by imposing a Markov structure on high-level features. The StrokePyramid module of Jing et al. [69] considers receptive field and scale, producing variant stroke sizes.

NST-generated images often have hard style features, making them appear unnatural. Careful selection of input-style images is essential to avoid unattractive results.

GANs, introduced by Goodfellow et al. [42], have been widely applied in various research fields. GANs consist of a generator and discriminator, trained in an adversarial manner. The generator learns to produce realistic images, whereas the discriminator aims to distinguish between real and generated images. This minimax optimization process ends at a saddle point, balancing the two networks. GANs generate visually compelling fake images, blending authenticity with novelty.

GAN-based methods have revolutionized AI art, with notable applications like CycleGAN [170], AttentionGAN [132], and Gated-GAN [14]. These models learn the style features from datasets, transforming real photos into artistic styles without harsh style features. However, GAN-based methods have their drawbacks: the difficulty of training, large model size, sometimes poor detailed representation, and even mistakes.

Diffusion model (DM) style transfer represents a major breakthrough in AIGC (Artificial Intelligence Generated Content). It harnesses the power of DMs, which transform random noise into novel data samples through a unique stochastic diffusion process. This technology has fueled the rise of AI drawing platforms like OpenAI’s DALL·E 2 [84, 111] and Google’s Imagen [118], showcasing their remarkable image generation capabilities. In style transfer, DMs apply their generative prowess to imagery, enabling the seamless transformation of any input image into a specified artistic style. Their working mechanism seamlessly integrates noising and denoising processes, gradually degrading and then reconstructing the image with the desired style while preserving its original content.

This approach not only offers exceptional controllability, allowing users to fine-tune generated images with precision, but also guarantees diversity and flexibility. It effortlessly accommodates a wide spectrum of style requirements and reference images, yielding results ranging from photorealistic fakes [8, 49, 113, 118] to artistic interpretations [35, 49, 76, 99, 114, 164]. Furthermore, DMs exhibit remarkable stability and robustness, consistently producing high-quality stylized images even under noisy or varying input conditions. This reliability has sparked interest in research exploring partial image re-editing [51, 80], further underscoring the versatility of this technology.

Line-drawing artworks such as sketches [6, 11, 47, 85, 89, 119, 129, 160], pencil drawings [87], portraits [96, 139], and doodles [105, 169] are created by line strokes. Significant research has been undertaken on line-drawing methods. Many studies have concerned the generation of line-drawing artworks by reconstructing input photos into line drawings. Compared with the input photos, generated line drawings lose much detailed content but retain the key contour of the object. Photo-sketch methods are mainly focused on the approach for capturing the contour information of an object in a photo, then mimicking the human sketching process to present the object. We usually consider photo-to-sketch synthesis as a cross-domain reconstruction issue. For example, Song et al. [129] constructed a generative sequence model with a recurrent neural network (RNN) acting as a neural sketcher. Their neural sketcher reconstructed a photo into a synthesis sketch by learning the noisy photo-sketch pairs dataset. Many methods for reconstructing photos into line drawings have been proposed. Line-drawing methods emphasize extracting the edge features of the object but not paying attention to the image’s color information. In particular, when comparing methods of line drawings, the key point is the line stroke or the shade drawn by line strokes. Portraits and pencil drawings (except with colored pencils) similar to sketches usually have black-and-white color characteristics.

Painting is an important form of visual art. Oil painting and watercolor painting, distinct from line drawings, emphasize color and tone. The essence of painting is color, which is made up of hue, saturation, and value, dispersed over a surface. In generating oil paintings and watercolor paintings, mimicking the color and stroke texture of paintings is a main task for the reconstruction of image to painting. With deep learning coming into widespread use, researchers have conducted studies on training machines to learn to paint like human artists. In particular, Mellor et al. [105] proposed a neural network SPIRAL++ to doodle human portraits. The style of the generated image is close to that of an oil painting, although the results lose detailed content. Jia et al. [68] proposed a self-supervised learning algorithm to achieve painting stroke by stroke, and the results outperformed SPIRAL++ on the presentation of details, although the detailed contents were still not sharp. Huang et al. [64] designed a painting model based on reinforcement learning to mimic the painting process of a human artist. The color strokes rendered onto the digital canvas in a certain order made their generated images similar to oil paintings, although the texture of the strokes was different from human artists’ strokes. Zou et al. [171] proposed an automatic image-to-painting model that generates oil paintings with controllable brushstrokes. The authors reframed the stroke prediction as a parameter searching process so that it mimicked the human painting process. Schaldenbrand and Oh [123] also proposed a model using Content Masked Loss (CML) to generate paintings stroke by stroke, although they lost some detailed contents of the image. For the stroke-based methods, the key point is how to present the detailed contents of the input image when reconstructing it to the painting stroke by stroke. The problem is that retaining as many details as possible will produce a close-to-photo result instead of a painting.

Ink wash painting is a type of Chinese ink brush painting that uses black or colored ink in different concentrations. The stroke texture and character of ink wash painting are so different from that of oil painting and watercolor painting that teaching a machine or computer to do ink wash painting is difficult. Research has been conducted on methods to simulate the special stroke of ink wash painting. For example, in a conventional stroke-based method in the work of Yao and Shao [34], B-spline curves were used to simulate the trajectory of the Chinese brush. This method inspired later researchers to improve the simulation of Chinese brushstrokes for deep neural networks. Xie et al. [151] first modeled the tip of the Chinese brush and then utilized a reinforcement learning algorithm to formulate the automatic stroke generator.

Robotic painting, an intersection of art and robotics, has seen significant advancements. Researchers and interdisciplinary artists have employed various painting techniques and human-machine collaboration models to create visual media on canvas. Although robot paintings differ from the AI artworks discussed in this work, they share some similarities. Robotic painting requires the use of physical robotic arms or robots to complete stroke-by-stroke painting, ultimately resulting in physical paintings. However, the AI paintings discussed in this article are almost exclusively electronic versions and do not require the use of robotic arms or robots. Their similarity lies in the stroke-by-stroke painting algorithm, as most AI models for stroke-by-stroke painting, after processing, can be applied to robotic painting. Nevertheless, since the focus of this article is not an in-depth exploration of algorithms, in Section 4.4.5 we conduct a more comprehensive analysis and discussion on robotic painting.

GAN is a min-max game between two neural networks with different objectives. One network, the generator (\(G\)), aims to trick the other, the discriminator (\(D\)), by generating images that resemble the dataset from a random latent vector \(z\). The objective of \(G\) is to create images closer to the dataset, whereas \(D\) tries to distinguish between real and generated images. Both networks optimize their tasks according to their objective functions. The dataset image is denoted as \(x\), and \(D(x)\) represents the probability that \(x\) is from the dataset. \(G(z)\) denotes the image generated by the generator, and the cost for \(G\) is \(\log (1-D(G(z)))\). The overall loss function isThe discriminator aims to maximize its ability to distinguish between real training data images and those generated by the generator. In the loss function 3, minimizing \(-\log (D(x))\) equates to maximizing the discriminator’s probability. The generator, however, minimizes \(\log (1-D(G(z)))\) to generate images that can trick the discriminator. Training a GAN, being a two-player adversarial game, is complex and challenging.

When Goodfellow et al. [42] first proposed GANs, they were not capable of generating stylized images. As shown in Equation (3), the generator aims to minimize its cost to produce images similar to the real data. Building on the GAN framework, researchers developed image-to-image translation methods [66, 130, 170] to achieve style transfer. CycleGAN, proposed by Zhu et al. [170], transforms photos into paintings that closely resemble the styles of various artists using unpaired data. This method maps a source image data domain \(S\) to a target image domain \(T\), learning the mapping \(G: S \rightarrow T\). It employs an adversarial loss to distinguish between the data distribution of \(T\) and the distribution of images generated by \(G(S)\).

Since the mapping \(G: S \rightarrow T\) lacks constraints, another generator \(\widetilde{G}\) is introduced for the reverse mapping \(\widetilde{G}: T \rightarrow S\) to ensure consistent results. Cycle consistency loss is added to enforce \(\widetilde{G}(G(S)) \approx S\). When \(G\) translates an image from \(S\) to \(T\), \(\widetilde{G}\) should be able to translate it back to \(S\), ensuring the reconstructed image \(\widetilde{G}(G(S))\) closely matches the original image \(S\). Similarly, for each image from \(T\), the reverse should hold. For the mapping \(G: S \rightarrow T\) and its discriminator \(D_T\), the objective function isFor each image \(s\) from the source image domain \(S\), the image reconstruction cycle should be able to bring \(s\) back the original image—that is, \(s~\rightarrow ~G(s)~\rightarrow ~ \widetilde{G}(G(s))~\approx ~s\). This gives the forward cycle consistency. However, for each image \(t\) from the target image domain \(T\), \(G\) and \(\widetilde{G}\) should also finish backward cycle consistency: \(t~\rightarrow ~\widetilde{G}(t)~\rightarrow ~G(\widetilde{G}(t))~\approx ~t\). Therefore, we get the cycle consistency loss function written as follows:

The whole loss function of CycleGAN is

CycleGAN allows the generation of stylized images that contain both the content of input images and the style of the training dataset, controlled by \(\gamma\). It enriches diverse art styles for unpaired image datasets, enabling reconstructions like transforming a modern photo into a Monet or Van Gogh painting. As shown in Figure 3, CycleGAN’s stylized results exhibit harmonious stylized characteristics, closely resembling Monet’s style, compared to NST methods like AAMS [159], ASTSAN [110], and URUST [144], which contain varied features not truly reflective of Monet’s style.

CycleGAN has drawbacks, such as unclear detailed contents. To improve image quality, AttentionGAN [132] incorporates the attention mechanism [140] into CycleGAN. AttentionGAN redesigns the second generator \(\widetilde{G}\) to generate content and attention masks, fusing them with the generated image \(G(s)\) to restore the source image \(s\). This process is formulated as \(\widetilde{G}(G(s)) = C_s * A_s + G(s) * (1-A_s)\). The term of \(\widetilde{G}\) consists of an encoder \(\widetilde{G}_E\), an attention mask module \(\widetilde{G}_A\), and a content mask module \(\widetilde{G}_C\). \(\widetilde{G}_C\) generates content masks, whereas \(\widetilde{G}_A\) generates attention masks for both background and foreground. These masks are fused with \(G(s)\) to restore \(s\), formulated as \(\widetilde{G}(G(s)) = \sum _{f=1}^{n-1}(C_s^f * A_s^f) + G(s) * A_s^b\), where the reconstructed image \(\widetilde{G}(G(s))\) should closely match the input source image \(s\). Similarly, for a target image \(t\), the cycle is formulated, and the reconstructed image should closely match \(t\).

Figure 4 compares CycleGAN and AttentionGAN. The first row shows real photos (small images), and subsequent rows display style-reconstructed results. AttentionGAN generates images with more detailed content than CycleGAN, especially in photo-to-Monet transformations, due to its attention mask mechanism. Different datasets yield distinct styles, enabling diverse AI artwork. For instance, training CycleGAN with a photo-to-anime dataset transforms real photos into anime images. CartoonGAN [15] and MS-CartoonGAN [125] focus on reconstructing photos to anime, emphasizing sharp edges, smooth shading, and abstract textures. CartoonGAN’s edge-promoting adversarial loss is given byThe discriminator \(D\) maximizes the probability of distinguishing the generated image \(G(P_I)\), cartoon images without sharp edges, and real cartoon images. CartoonGAN also introduces a content loss for smooth shading:where \(l\) denotes a specific layer of VGG [126] for feature extraction. This loss uses \(\ell _1\) sparse regularization for better representation and regional characteristic preservation. Mimicking real art styles is crucial for AI artworks; however, diversity is also important. CycleGAN-based methods contribute to vivid art styles but generate only one style per model, which is inconvenient for diverse art style applications.

Gated-GAN, proposed by Chen et al. [14], enables the generation of multiple styles within a single framework. It uses an adversarial gated network, known as the gated transformer, for multi-collection style transfer. The model includes a switching trigger to select the desired style for the output. The gated transformer processes a set of photos \(\lbrace p_i \rbrace ^N_{i=1} \in P\) and multiple painting collections \(Q = \lbrace Q_1, Q_2, \ldots , Q_K \rbrace\), where \(K\) is the number of collections, each containing \(N_c\) images \(\lbrace q_i \rbrace ^{N_c}_{i=1}\). The network generates multiple styles \(G(p, c)\) by applying the style of collection \(c\) to the input photo: \(G(p, c) = Dec(T(Enc(p), c))\). Here, \(T(.)\) is a transformer built with residual networks, and \(Enc(p)\) denotes the encoded feature space. Each style-specific branch in the transformer module contains additional parameters, minimizing the overall model complexity. Inspired by LabelGAN [135], Gated-GAN incorporates an auxiliary classifier to handle multiple style categories, optimizing the entropy to improve classification confidence. This design enables the model to generate diverse styles within a unified framework.

Despite its ability to produce multiple styles, Gated-GAN has limitations, such as occasionally lacking detailed content. Figure 5 shows examples generated by Gated-GAN, highlighting issues like the unnatural color block in the cloud region of the Van Gogh styled image.

Gated-GAN’s per-model multi-style approach contrasts with per-model-per-style methods like CycleGAN and CartoonGAN. Whereas CycleGAN and CartoonGAN generate one style per model, Gated-GAN supports multiple styles, enhancing versatility. However, models like AttentionGAN, which builds on CycleGAN, tend to produce higher-quality images with more detailed content. Gated-GAN’s strength lies in its ability to manage multiple styles efficiently, but it sometimes sacrifices detail. Combining the advantages of these approaches could lead to models that handle multiple styles and maintain high-quality, detailed outputs.

As NST methods achieve sketching directly from images (e.g., APDrawingGAN [161], synthesizing human-like sketches [74]), we analyze line-drawing methods focusing on the drawing process.

Ha and Eck [47] proposed sketch-rnn, an RNN capable of generating stroke-based drawings. A sketch is defined as a point list, where each point is a vector with five elements: (\(\Delta x\), \(\Delta y\), \(st_1\), \(st_2\), \(st_3\)). The sketch-rnn model employs a sequence-to-sequence VAE architecture, similar to those in other works [78, 121]. It encodes a sketch image into a latent vector and decodes it stroke by stroke, guided by the encoded states.

The encoding process involves two RNNs processing the sketch sequence and its reverse, resulting in final hidden states \(\underrightarrow{h}\) and \(\underleftarrow{h}\), combined into \(h_s\). The process can be written as follows:

The sketch-rnn encoder processes the concatenated hidden states \(h_s\) into \(\delta\) and \(\hat{\eta }\) of size \(V_z\). \(\hat{\eta }\) is transformed into the non-negative standard deviation \(\eta\) via exponentiation. Using \(\delta\), \(\eta\), \(\mathcal {N}(0, 1)\), and a vector of 2D Gaussian variables, a random latent vector \(z \in \mathbb {R}^{V_z}\) is constructed, akin to the VAE approach in the work of Kingma and Welling [78]. \(z\) is conditioned on the input sketch, differing from deterministic outputs.

The auto-regressive RNN decoder of sketch-rnn sequentially predicts strokes using the last point, previous sketch sequence \(Sq_{di-1}\), and latent vector \(z\). It iterates through drawing steps to generate simple object sketches and can produce ablation sketches by adjusting the Kullback-Leibler loss weight. However, sketch-rnn struggles with complex images and supports limited sketch styles, allowing human participation only in predicting unfinished sketches.

The Creative Sketch Generation method [41] introduces DoodlerGAN, which leverages StyleGAN2 [41] to sequentially generate sketch parts guided by human observations. Its part selector facilitates a human-in-the-loop sketching process but is currently limited to birds and creative creatures.

An alternative approach [169] uses reinforcement learning (Deep Q-learning) in Doodle-SDQ to train an agent to draw strokes on a virtual canvas, aiming to reconstruct a reference image stroke by stroke. The similarity metric \(\mathbb {S}_k\) evaluates the canvas’s closeness to the input image: \(\mathbb {S}_k = \frac{\sum _{i=1}^L\sum _{j=1}^L(P_{ij}^k - P_{ij}^{\text{ref}})}{L^2}\), where \(P_{ij}^k\) and \(P_{ij}^{\text{ref}}\) are pixel values at position (\(i, j\)) on the canvas and input image, respectively, at step \(k\). The pixel reward \(R_{\text{P}} = \mathbb {S}_k - \mathbb {S}_{k+1}\) optimizes the executing action at each step.

Doodle-SDQ’s line-stroke sketching penalizes slow movements (\(P_{\rm {s}}\) for <5 pixels/step or pen lift) and incorrect color choices (\(P_{\rm {c}}\) with \(\beta\) adjusted for grayscale/color input). The final reward \(R_k = R_{\rm {P}} + P_{\rm {s}} + \beta P_{\rm {c}}\) combines pixel similarity and penalties. Although Doodle-SDQ reproduces reference sketches well, it cannot sketch from real photos and lacks artistic creativity. In the work of Zhou et al. [169], strokes are simulated by a virtual ‘pen,’ with reinforcement learning mapping actions to strokes. This inspires the development of diverse stroke types, potentially mimicking oil paintings and ink wash paintings.

The method in the work of Huang et al. [64] utilizes a model-based DDPG (deep deterministic policy gradient) [91] algorithm to simulate a stroke-by-stroke oil-painting process. Bézier curves mimic brushstroke paths, and a circle represents the brush tip. The control points of the Bézier curves serve as actions, enabling action-to-stroke mapping. Given an input photo \(P_I\) and an initial canvas \(C_0\), the model generates an action sequence \((b_0; b_1, \ldots , b_{n-1})\) to sequentially render strokes onto the canvas, producing the final painting \(C_N\). This task is formulated as a Markov decision process with a state space \(\mathfrak {S}\), action space \(\mathfrak {B}\), transition function trans(\(s_n, b_n\)), and reward function \(R(s_n, b_n)\) designed to minimize the distance between the input image and the canvas at each step: \(R(s_n, b_n) = L_n - L_{n+1}\), where \(L_n\) and \(L_{n+1}\) represent the losses between \(P_I\) and the current/next canvases, respectively. The model aims to maximize the accumulated discounted future reward \(R_n = \sum _{i=n}^T \epsilon ^{(i-n)}R(s_i, b_i)\) with a discount factor \(\epsilon \in (0,1)\).

The original DDPG algorithm is composed of an actor network \(\Phi (s)\) that maps state \(s_n\) to actions \(b_n\) and a critic network \(\Psi (s, b)\) that estimates reward to guide the actor. Both networks are trained using the Bellman equation (12), with an experienced replay buffer storing the latest 800 episodes to enhance data usage:

The MDRL Painter (MDRLP) method in the work of Huang et al. [64] improves upon line drawing approaches by simulating oil-painting brushstrokes using Bézier curves and circles. This method is improved from the line-drawing method of Zhou et al. [169] by designing the brushstroke. Although it can create paintings from various input images, the details are coarse, and the simulated stroke textures lack realism compared to human-made oil paintings.

The Artistic Style in Robotic Painting (ASRP) approach of Bidgoli et al. [7] aimed to mimic human artist styles by generating brushstroke samples with similar textures. It uses Bézier curves to simulate strokes without tuning transparency, ensuring realism. VAEs were trained to capture artist brushstroke features, resulting in stroke textures close to those of human artists, but the final paintings lacked content detail.

Schaldenbrand and Oh [123] improved painting quality by proposing CML, a reinforcement learning model based on the work of Huang et al. [64]. CML emphasizes salient regions using VGG-16 features and \(\ell _2\) distance, mimicking the human painting process. However, even though the model captures the painting process well, it loses detailed content and stroke texture clarity.

Another AI oil-painting model, Stylized Neural Painting (SNP) by Zou et al. [171], contributes to stroke modeling by generating strokes with realistic oil-painting textures. A dual-pathway neural network independently generates stroke colors and textures. The model predicts and renders strokes step by step to optimize the final canvas \(C_N\) to resemble the input image \(I_r\): \(C_N = \phi _{n=1\sim N}(\tilde{s}) \approx I_r\), where \(\phi _{n=1\sim N}(.)\) maps stroke parameters to canvas states. The model optimizes stroke parameters \(\tilde{s} = [s_1, \ldots , s_N]\) using gradient descent to minimize the visual similarity loss \(\mathcal {L}(C_N, I_r)\): \(\tilde{s} \leftarrow \tilde{s} - \theta \frac{\partial \mathcal {L}(C_N, I_r)}{\partial \tilde{s}}\), where \(\theta\) is the learning rate.

The SNP method [171] produces paintings with more details and realistic oil-painting stroke textures compared to other works [7, 64, 123], as shown in Figure 6. ASRP [7] and SNP exhibit clear oil-painting textures; however, SNP’s output size is fixed, requiring input images with the same aspect ratio. This can distort non-conforming images, and some input details may become blurry. Additionally, SNP requires more computation time than MDRLP.

Ink wash painting seems difficult to achieve with learning-based methods, and there are only a few research studies on the topic [151, 152]. For example, the texture of Chinese hair brush is difficult to mimic, although conventional SBR methods make contributions [131] to stroke modelling. Xie et al. [151] proposed using the Markov decision process to imitate drawing a stroke. The authors first used a tip \(V\) and a circle with center \(C_o\) and radius \(r_o\) to model the brush agent. The Markov decision process consists of a tuple (\(\hat{\mathcal {S}},\hat{\mathcal {A}},P_d,P_T,\phi\)), where \(\hat{\mathcal {S}}\) is a set of continuous states of the canvas, \(\hat{\mathcal {A}}\) is a set of continuous actions, and \(P_d\) is the probability density of the initial state. \(P_T(\hat{s}^{\prime }|\hat{s},\hat{a})\) is the transition of the probability density from the current state of the canvas \(\hat{s}\in \hat{\mathcal {S}}\) to the next state \(\hat{s}^{\prime }\in \hat{\mathcal {S}}\) when taking action \(\hat{a}\in \hat{\mathcal {A}}\). The term \(\phi (\hat{s},\hat{a},\hat{s}^{\prime })\) denotes the immediate reward for the transition from \(\hat{s}\) to \(\hat{s}^{\prime }\). Let \(\mathcal {T} = (\hat{s}_1, \hat{a}_1, \hat{s}_L, \hat{a}_L, \hat{s}_{L+1})\) be a trajectory of length \(L\). Then, the return (i.e., the sum of the accumulating discounted future rewards) along \(\mathcal {T}\) is written as \(\phi (\mathcal {T}) =\sum ^L_{l=1}\sigma ^{L-1}\phi (\hat{s}_l, \hat{a}_l,\hat{s}_{l+1})\), where \(\sigma \in [0, 1)\) is the discount value for the future reward. Meanwhile, the authors designed four actions to move the brush agent, and in the reinforcement learning model, the brush agent was trained to generate hair brushstrokes.

Since the algorithm achieves high fidelity of hair brushstroke textures, the reinforcement learning model is, at last, able to use the brush agent to generate ink wash paintings or Chinese paintings. Although the painting results contain textures of hair brushstrokes and characteristics of ink wash paintings, the method does not provide the painting process. Therefore, we do not know what happened during the painting procedure. We are not sure if the paintings are painted stroke by stroke. Moreover, the method description does not explain how the painting agent processes the input reference images and how the agent decomposes the images into strokes.

The method of Neural Painters (NP) in the work of Nakano [109] uses the GAN-based model and VAE-based model to simulate an intrinsic style-transform painting. Since the stroke textures are close to the pastel-painting style, we have called this form of painting pastel-like painting. However, the finished paintings express few characteristics of pastel paintings. The GAN-based and VAE-based models in the method were used to generate pastel-like strokes by training the models on the stroke dataset provided by the MyPaint program. When training the GAN- and VAE-based models, Nakano [109] labeled the dataset for the action space mapping a single action to a single brushstroke. The entire model (a neural painter) then used the GAN- or VAE-based model to generate pastel-like strokes rendering on the canvas. By dividing the canvas into grids with the same size as the stroke image generated by the GAN- or VAE-base model, the neural painter was able to recreate a pastel-like painting based on the given image. However, the paintings generated by NP lost much detailed content and the pastel-painting stroke textures were not clear. As Figure 7 shows, with images from Nakano [109], the stroke samples contained characteristics of pastel-painting stroke textures, but the painting not only lost too much detailed content but also had few pastel-painting characteristics.

Robotic painting has long captivated both artists and robotics experts. Most artistic painting robots use acrylic paints [75], which are nearly as versatile as oil paints but are water soluble, eliminating the need for harsh or toxic thinners and solvents. An example of an acrylic painting robot is the e-David robot [43, 92, 93], developed by Oliver Dessoin, Thomas Lindmeier, Mark Tautzenberger, and Sören Pirk. This system comprises an industrial robot equipped with a paintbrush and a visual feedback system, utilizing a set of pre-mixed colors. Additional color mixing is achieved by applying translucent brushstrokes to the canvas, considering the Kubelka-Munk paint film theory. The e-David robot can also learn to replicate brushstrokes through trial and error. The LETI painting robot [75] introduces a new type of robot capable of precisely metering and mixing acrylic paints, demonstrating high-quality painting results. The robotic system’s capabilities are showcased through four artworks: replicas of landscapes by Claude Monet and Arkhip Kuindzhi, and synthetic images generated by StyleGAN2 and Midjourney neural networks. These results can be applied to computer-generated creativity, art replication and restoration, and color 3D printing.

The work by Bidgoli et al. [7] presents a new approach that integrates artistic style into the process of robotic painting through collaboration with human artists. The method involves collecting brushstroke samples from artists, training a generative model to imitate the artist’s style, and then fine-tuning the brushstroke rendering model to adapt it to robotic painting. Their user studies have shown that this method can effectively apply the artist’s style to robotic painting. The use of a VMS (Visual Measurement System) and an RPS (Robotic Painting System) to simulate brushstrokes is presented by Guo et al. [44]. The specific method involves using VMS to capture the interaction trajectories and environmental state information during the artist’s painting process. Then, RPS mimics human painting actions based on this information, utilizing real-time visual feedback to adjust the robot’s movements, thus achieving precise brushstroke simulation. Through these methods, the proposed ShadowPainter system can simulate brushstroke effects that are close to human levels.

Work Mikalonyté and Kneer [106] explores whether AI-driven robots can be regarded as artists and create real works of art. Two experiments were conduction to investigate people’s perception of the artistic quality of robot paintings and their acceptance of the identity of robot artists. Experimental results show that although people generally believe that robot paintings are not much different from human works in terms of artistic quality, they have reservations about identifying robots as artists.

In conclusion, robotic painting has become a fascinating field that bridges art and technology. Various systems and methods have been developed to mimic and even surpass human artistic abilities. From using acrylic paints to precise metering and mixing techniques, these robots have demonstrated extraordinary painting capability. The integration of artistic styles through human-machine collaboration further enhances the creative possibilities of robotic painting. As technology advances, we can expect more innovative and captivating artworks to emerge from this exciting field, breaking the boundaries of traditional art forms and opening new avenues for artistic expression. However, the debate over whether AI-driven robots can truly be considered artists remains unresolved. Despite the increasing technical proficiency and artistic quality approaching human standards, societal acceptance of robots as genuine creators of art continues to lag. Future research and development in this field may focus on bridging this gap, enhancing the creative capabilities of robots, and addressing the ethical and philosophical issues surrounding AI and art.

We first compare the results generated by the methods of image style transfer. In particular, the stylized images are synthesized by the content image and the style image. Figure 8 shows the sample results generated by methods of AAMS [159], ASTSAN [110], and URUST [144]. The first column contains the content images and style images (small). The remaining columns, from left to right, are the generated images of AAMS [159], ASTSAN [110], and URUST [144], respectively. All of the results present the style features well.

However, as can be seen from the top row (see Figure 8), the style image is a pencil drawing in the top row. Yet, the image generated by ASTSAN [110] still retains the original color features of the content image, indicating incomplete style transfer. Although the image generated by URUST [144] exhibits pencil drawing features, the content of the bird is blurred, indicating imperfect content expression. The image generated by AAMS [159] presents clear content of the target image, and the style features are also harmoniously synthesized into the target image. From a visual aesthetic perspective, considering overall aesthetic beauty, lines, colors, content details, and style, the image generated by AAMS [159] appears more aesthetically pleasing than the others. Therefore, we conclude that the results of image style transfer should contain detailed content of the target image, and the features of the style image should not overshadow the content image.

Figure 9 shows the visual results of new style transfer methods. The visual effects of the images generated by AesPA-Net [60], EFDM [167], AdaIN [63], CAST [168], StyTR2 [23], and AdaAttN [95] are quite impressive. They maintain high clarity and content detail, with good color reproduction and contrast. The stroke and line textures are also well presented. The cat’s image is vivid, and the background environments have their own characteristics, showcasing different artistic styles. However, in terms of style transfer, they do not fully embody the features of the style image, so they are not the best in this aspect.

The images generated by MAST [24] and SID (Style Injection in Diffusion) [21] are slightly inferior in content detail. Although they basically capture the cat’s image and background environment, they are slightly lacking in clarity, color reproduction, and contrast. Some details may be blurry, and the colors may be somewhat distorted, affecting the overall visual effect. The line sense and stroke texture are not very obvious. The content detail expression in images generated by DiffuseIT [80], InST [166], and DiffStyle [67] is very poor. For InST [166] and DiffStyle [67], the cat’s image is almost indistinguishable. On the contrary, InST [166] expresses more content from the style image. Although it is hard to recognize the content of the image generated by DiffStyle [67], its overall color expression creates a fresh and ‘cute’ effect.

In summary, the evaluation of style transfer results across various models highlights several key features necessary for generating high-quality, new-style artistic images. From the perspective of beauty, an ideal artistic image should exhibit a balanced composition of visually pleasing elements, including harmonious color schemes and well-composed subjects. Regarding lines, clarity and sharpness are crucial for defining objects and subjects, contributing to the overall structural readability of the image. In terms of colors, accurate color reproduction and contrast are essential for enhancing visual appeal and reflecting the desired mood and atmosphere. Stroke texture plays a vital role in conveying the sense of artistic technique and traditional medium, providing a tactile experience for the viewer. Content details are important for maintaining the recognizability and realism of the main subject, ensuring that key elements are neither lost nor distorted during the transformation process. Finally, the style itself must be faithfully reproduced, capturing the unique characteristics and nuances of the reference style image. Balancing these elements ensures that the generated artistic image not only adheres to the desired style but also stands out as a cohesive and aesthetically engaging piece of art.

Figure 10 shows the results generated by style transfer methods. Note that the style of the generated images is learned from the training dataset, not synthesized from a style image. The first column shows the input images, and the remaining columns, from left to right, are generated images by GANs N’ Roses [20], U-GAT-IT [77], and WBC [147], respectively. The first row input image is from the dataset provided by U-GAT-IT [77], and the last input image is from the sample image test provided by WBC [147]. When comparing the first three rows of images, we observe that images generated by WBC [147] retain more realistic contents of the input images than the others. The images generated by GANs N’ Roses [20] and U-GAT-IT [77] present more non-realistic cartoon features than WBC [147]. However, when comparing the bottom row images, we observe that the image generated by U-GAT-IT [77] has few cartoon features but blurred contents. Based on the analysis, we conclude that U-GAT-IT [77] has a low generalization.

Figure 11 shows the results generated by line drawings methods. The top row shows the input reference images (small images), and the rest of the rows, from top to bottom, show the results generated by photo-sketching [85] and APDrawingGAN [161], respectively. The images generated by photo-sketching [85] lose so much content that it is difficult to recognize the object in the image. Although results generated by APDrawingGAN [161] contain sufficient image content, the expression of the girl’s hair is not satisfactory.

Figure 12 shows another line drawing results generated by DoodlerGAN [41]. The images are created by the online demo provided by the authors. The model only creates birds or bird-like creatures. The images are generated step by step. The whole image consists of several components of a bird or bird-like creatures. The human or the computer draws a final step in the process to finish a component. Figure 12(a) and (c) are finished by the cooperation of a human and a computer. Figure 12(b) and (d) are generated by the computer only. We observe that all images are like birds but not real birds.

Figure 13 shows the results generated by methods of painting. The results are created stroke by stroke. The left column shows the input images, and the remaining columns from left to right are the results generated by methods of MDRLP [64], SNP [171], Stroke-GAN Painter [145], and NP [109], respectively. The images in the three middle columns have colors closer to the input images than the right-column images. Images generated by SNP [171] present clearer stroke textures than others. Images generated by MDRLP [64] contain more details than others. Images generated by MDRLP [64], Stroke-GAN Painter [145], and SNP [171] look like oil painting, especially the brushstroke texture of SNP [171]. The style of images generated by NP [109] is difficult to recognize since the stroke texture is more like pastel painting than oil painting, but the art style is close to watercolor painting.

To make an objective evaluation of the generated images, we undertake a two-step user study. For a fair comparison, we conduct a blind-trial test among the participants. The participants know neither the authors of the methods used for generating comparison paintings nor the experimenter. The participants are chosen from various backgrounds (69.2% in the art field, and 85.1% know about AI art), age groups (18–60 years), and genders (74 females and 127 males).

We designed the user study as a two-step test for the six-dimensional evaluation index analysis to find suitable items for a certain art style, inspired by the work of Tong et al. [136]. For the first step, we mix all the painting results in the same questionnaire and then ask the participants to score all the paintings according to the six evaluation items. In the second step, we classify the paintings into two categories: style-transform paintings and style-reconstruction paintings (stroke-by-stroke paintings). The style-reconstruction paintings contain the painting process images, and the paintings with the same style are put in the same group. We then ask the participants to score the paintings based on a 5-point Likert scale [90]. The participants finish the user study’s Step 1 and Step 2.

Tables 2 and 3 show the Intraclass Correlation Coefficient (ICC) results of the two-step user study. In analyzing two sets of ICC data, we observed similar trends regarding the reliability of single and average measurements. In both datasets, the single measure ICC (C,1) values, 0.437 and 0.498 respectively, indicate a certain to moderate degree of correlation in single measurements, but not particularly strong. The 95% CIs (confidence intervals) for these single measures show a range of fluctuation, suggesting room for improvement and reflecting the potential impact of random errors or individual differences. However, the average measure ICC (C,K) values exhibit extremely high reliability in both sets, reaching 0.985 and 0.988. The narrow CIs further confirm that averaging multiple measurements significantly enhances measurement accuracy and consistency. These findings underscore the importance of repeated measurements in improving data quality and reliability. In the subsequent data analysis, we mainly took the average score of each question for further analysis.

Table 4 shows the experimental results of Step 1, and Table 5 shows the results of Step 2. Scores in the two tables are marked with different colors for observation. Red indicates the highest scores, blue indicates the lowest scores, and orange represents scores lower than 3 except blue ones.

Table 4 shows the six-dimensional evaluation index scores on mixed artworks. In the beauty column of Table 4, we observe that the results generated by the method of photo-sketching [85] give the lowest scores (2.849). Compared with other paintings, the sketches generated by photo-sketching [85] have little content from the input image, and we cannot readily recognize what the sketches express in some cases (as Figure 11 shows). The score is 2.849, which means that most participants judged the sketches to be poor in terms of beauty. The sketches generated by DoodlerGAN [41] also obtain a lower score (3.000) compared with other paintings. However, when comparing the line smoothness of the paintings, we observe that paintings generated by the method of APDrawingGAN [161] gained higher scores than most. Paintings generated by DiffStyle [67], ASTSAN [110], DiffuseIT [80], and H-SRC [72] obtained scores lower than 3. This means these paintings have poor line expressions. The texture column compares the stroke texture of the test artworks. MAST [24] and H-SRC [72] obtain scores lower than 3; however, AesPA-Net [60], APDrawingGAN [161], StyTR2 [23], CAST [168], and PST [98] obtain scores higher than 3.6. That means these methods express stroke texture well. Methods obtaining high scores, especially PST [98] (3.823), present clear stroke textures in their paintings. In the color column, most of the methods score higher than 3 except photo-sketching [85], MAST [24], DiffStyle [67], H-SRC [72], and DoodlerGAN [41]. For the content comparison, only H-SRC [72], photo-sketching [85], and DoodlerGAN [41] obtain a score lower than 3. Scanning Figure 11, the images generated by photo-sketching [85] lose too much content. Thus, the line drawings or sketches, when compared with other paintings with rich contents, only gain lower scores. When compared in terms of art style recognizability, only the paintings generated by H-SRC [72] obtained low scores (2.940). In other words, most of the participants cannot recognize the art style of the paintings created by H-SRC [72]. Table 5 shows the scores of the six-dimensional evaluation index on the classified artworks. The artworks are divided into four groups: style transfer/transform, photo-to-cartoon, line drawing, and stroke-by-stroke painting. Some of the artworks created stroke by stroke also exhibit the painting process images (Figure 14). In the user study Step 2, the scores were significantly higher than those of Step 1, especially since the number of scores below 3 was much fewer. The reason is that in the second test, users were informed of the style type and the image generation method so that users had a fuller understanding of the object they were evaluating. Therefore, users would be more tolerant and accepting of some less distinguishable options, thus giving higher scores. In the beauty column of Table 5, results of PST [98], AAMS [159], Im2Oil [137], APDrawingGAN [161], and Intelli-paint [127] obtained higher scores than most others. Especially, in the style column, the lowest score is higher than 3, which means that when users are informed of the styles and generation methods, their scores for artworks will be more accurate in the style confirmation item. In addition, it is in line with the principle of fairness to evaluate paintings by classifying them according to their styles and generation methods.

To conduct a more detailed analysis of the user study, we have sorted and classified the scores of the users based on their backgrounds. Figure 15 shows the scores of all artworks by five backgrounds of users: all users, users with artistic backgrounds who understand AI art, users with artistic backgrounds but do not understand AI art, users without artistic backgrounds but understand AI art, and users without artistic backgrounds who also do not understand AI art.

The analysis identified that the average scores of users with artistic backgrounds are higher than those of other users, whether in artworks-mixed or artworks-categorized tests. In the artworks-mixed test, users with an artistic background but no knowledge of AI art gave the highest scores, followed by users with an artistic background and knowledge of AI art. In the artworks-categorized tests, users with an artistic background and knowledge of AI art gave the highest scores except for the color item, followed by users with an artistic background but no knowledge of AI art. Especially in the color item, the latter group gave the highest scores. Interestingly, in the two-step user study, the average scores given by users with an artistic background were higher than the average scores given by all users. Among users without an artistic background, in the artworks-mixed test, the scores given by users who understand AI art are lower than those who do not understand AI art in every category. In the categorized test, only the Beauty and Line items have lower scores from users who understand AI art compared to those who do not. Overall, in both tests, users who understand AI art gave lower scores than those who do not.

Creativity has a profound impact on society [16, 163], especially in art. No matter whether we are considering style-transform AI artworks or art-style-reconstruction artworks, existing methods can ‘almost’ turn a photo into an artwork. Therefore, it is worth discussing the fidelity and creativity [50] of the results. Unfortunately, most painting/drawing methods have difficulty in achieving high fidelity because of the art style representation. For example, some methods (e.g., [7, 94, 123, 171]), although presenting the stroke texture of oil painting well, produce results that lose much detailed content owing to the invariant stroke shape or type. The method in the work of Huang et al. [64] also mimics the oil-painting process and can generate high-fidelity results when giving a large number of strokes, but the high-fidelity result is almost a photo rather than an oil painting because the strokes lack oil-painting stroke textures. In summary, turning a photo into a painting is a creative task requiring the result to not be the same as the photo itself, but the fidelity requiring the preservation of as many details as possible is still a difficult challenge, and we have yet to deliver pleasing results consistently.

Most painting/drawing methods claim that they can mimic the human painting/drawing process. In reality, they model stroke generation to render a large number of strokes onto the canvas to finish the creation of an artwork. However, the generation process is so different from the human painting process that they ignore the creation order that humans follow. In particular, when human artists create artwork, such as an oil painting, they tend to draft the main objects by lines first and then paint the background and the objects progressively. It is worthwhile to teach machines to really mimic the human painting process so as to reveal the mysterious veil of art creation, even though it is difficult to achieve this task. If we make a step to achieve the real human painting process, we make the machine painting more intelligent and closer to the human artist; if we endow the machine or computer with inspiration and motivation for its creation (as pointed out by Hertzmann [56, 57]), then we may claim that the machine or computer can create art.

Existing methods for creating AI artworks usually refer to the input image to re-create the artwork. However, a human artist can create artwork without real referent objects thanks to their human inspiration and imagination. Consequently, teaching a machine or computer to create artworks without reference images is a very challenging task. Although Xu et al. [155] achieved the generation of images from fine-grained text, the result was photorealistic and could not really be called artwork. Elgammal et al. [32] generated abstract artworks with their creative adversarial networks, but the model itself could not name the artwork according to its creation. In other words, this model just generates abstract images but does not know what the image is or what meaning the image represents. However, researchers can obtain inspiration from these two works, since the combination of text-to-image and abstract artworks can prompt areas of consideration and development for future AI art creation.

Chen et al. [14] managed to generate multiple styles of results within an unified framework for image style transfer. It is popular to design a model to address multiple tasks; however, it is difficult to design a model that paints with multiple art styles. Although some works [64, 94, 171] could change the visual representation of the results by replacing different stroke styles, the art style stayed the same, almost close to oil paintings. Can machines or computers create different art styles of artworks within the unified framework? Similar to a human artist who can create a watercolor painting, a pastel painting, and an oil painting, seemingly by changing their painting tools, can a painting system create different art styles of paintings by changing its stroke style? It is an interesting and challenging issue for both artists and computer scientists.

Aesthetic evaluation is a critical issue for AI artworks. Some works [33, 45, 55, 61, 103, 108, 133] argued that aesthetic evaluation is important to develop methods for AI artworks. Especially for such diverse types of AI artworks as mentioned in the work of Rosin and Collomosse [115], a fair and scientific evaluation system is very important. In this article, we propose an evaluation system to cover several types of AI artworks so as to unify the diverse evaluating methods as well as make the evaluation fair when facing different types of AI artworks. However, even the proposed evaluation system is still based on user studies. Can we evaluate AI artwork and its methods via computing indexes? The proposed six-dimensional evaluation index may give some ideas and inspirations for the following research. For the development of AI artworks, fair, objective, and scientific evaluation is still an important and challenging area to be addressed.

The application of AI artworks in the social media community is very popular. In an era of ever-higher aesthetic aspirations and requirements, self-actualization and self-creation are areas of increasing attention and demand resources accordingly. Current techniques and algorithms cannot meet the demand of interaction and creation for everyone. Whether via social application software or on social websites, people are enthusiastic about making their own virtual characters or turning photos they have taken into artworks. However, it is difficult to make high technology and applications accessible universally for all people. First, the operation of creating an artwork based on a photo should be convenient and easy. Second, the method itself should have a small model size and a short inference time. Last but not least, the aesthetic quality should be acceptable to a relevant proportion of people.

If the virtual artworks are visible but untouchable, that reduces subjective feelings: real artworks give a more direct sensory experience. When talking about direct sensory experience, painting artwork by oneself must be the act that gives the most comprehensive sensory experience. However, learning to paint from scratch is so difficult that most people do not know how to start. Not everyone who likes to paint needs or wants to go to school to learn how. Learning to paint by referring to videos or websites is popular; even so, it is not convenient for people who want to paint a certain artwork. Imagining that an application in your mobile phone can generate any artwork process according to your input, is this not more convenient or interesting? Such AI-aided art education can enrich individualized art education [156], which will bring more opportunities and possibilities for art education.

AI technologies bring diversity and possibility for all kinds of art. GAN-based methods in particular have made a visual feast of style transfer or feature texture fusion. In traditional art history, it is always humans who create and present art. In this AI era, can computers really create art and diversify the presentation of art, differentiating from human art? As Hertzmann [57] gave a viewpoint, computers cannot make art because they have no creation, motivation, or emotion, but people do. In addressing the motivation and emotion of computers, we may have a long way to go, and it is not only the issue of AI artworks. Is it impossible for AI to create enriching forms of art and occupy a place in art history? The answer is no! We can, at least, make efforts to apply collaborative intelligence to the creation of digital art. As mentioned in the work of Wilson and Daugherty [149], humans should collaborate with AI so that, when creating a new artwork, we have a clear motivation and emotion, and even create an amazing artwork out of our imagination. Meanwhile, Cécile Paris pointed out that collaborative intelligence is the next scientific frontier of digital transformation [153]. It must be an interesting task to achieve the collaboration of AI and human artists to create a new form of art, and collaborative intelligence must do something wonderful in this task [3].

Since AI artworks can be used in many scenarios, it is necessary to discuss the value of AI artworks. Cetinic and She [12] proposed that the novelty of AI art should be taken into account when we talk about the values of this type of artwork in the context of art history. This type of art, as generative art [30], has been extensively theoretically and practically explored in the past few decades [29]. Recently, Chohan [17] noted that there is a category of blockchain-based virtual assets known as NFTs (non-fungible tokens, attracting an incredible amount of interest from investors in a very recent and short period. Digital artworks can be added to the growing list of uses for the blockchain technology that is now becoming a part of modern life in application such as accounting and auditing, agriculture, AI, business supply chains, and creative and artistic endeavors [138]. Hong and Curran [59] also investigated the price value of machine-made artworks compared with man-made artworks by user studies. The work found that man-made and machine-made artworks are not judged equivalent in their artistic value. The authors pointed out when the participants are told that the artworks are made by machines, then the evaluation is not influenced compared with participants not knowing. We can predict that AI artworks can be traded online and offline in the future, and people have a stable evaluation of artworks. Of course, we should take into account that the sale and subsequent reaction to the work resurrect venerable questions regarding autonomy, authorship, authenticity, and intention in computer-generated art [104].

Inspired by some works [22, 70, 112], we focus on making a unified evaluation system for AI artworks. Note that the unified system contains several items (color, content, stroke texture, style, and beauty), and for a certain type of artwork, certain items should be chosen. For example, line drawings without color design should choose content, stroke texture, style, and beauty without the color item. We conduct a comparable experiment to find out the relation of the six items and different types of artworks. We first design the user study with all the artworks put together, composing the questionnaire. We then compose the second questionnaire by classifying the artworks according to art types. In these two questionnaires, the evaluation items are the same. From the analysis of Section 5, we determine that the six evaluation items are reasonable, and for different types of artworks, certain items gain very low scores, demonstrating that they are inappropriate for that type. We propose a unified evaluation system for AI artworks, where the items are flexible and are to be chosen for a certain type of artwork. This six-dimensional evaluation index is able to cover many types of AI artworks as well as assign the abstract aesthetic evaluation into several concrete dimensions. However, it is still not enough to cover all kinds of AI artworks, and it needs to be developed into a more objective evaluation system based on computational aesthetics in the future.