2025 01 v.55 63-74
SN-CLPGAN:基于谱归一化的中国传统山水画风格迁移方法
基金项目(Foundation):
国家自然科学基金(62471390、62306237);
陕西省重点研发计划(2024GX-YBXM-149);
西北大学研究生创新项目(CX2024204、CX2024206)
邮箱(Email):
pxl@nwu.edu.cn;
DOI:
10.16152/j.cnki.xdxbzr.2025-01-005
中文作者单位:
西北大学信息科学与技术学院;生成式人工智能与混合现实陕西省高等学校重点实验室;西北大学网络与数据中心;西北大学艺术学院;西北大学新型网络智能信息服务国家地方联合工程研究中心;
摘要(Abstract):
中国传统山水画的风格迁移为文化遗产数字化保护与传承提供了新的路径，近年来，深度学习技术已实现了不同图像间的风格迁移，并展现出栩栩如生的效果。中国传统山水画的风格迁移旨在继承中国古代画家独特的绘画技巧，但存在3个缺陷：(1)缺乏高质量的中国传统山水画图像数据集；(2)忽略了中国传统山水画特有的技法和笔墨细节；(3)风格迁移效果与真实山水画有所差距。为了弥补上述缺陷，首先，创建了一个基于风格迁移的中国传统山水画数据集STCLP,包含4 281幅高质量的中国山水画以及自然景观图像，并提出了一种基于谱归一化的中国山水画风格迁移方法SN-CLPGAN。其次，提出了在生成器中使用融合反射填充层的残差稠密块(residual-in-residual dense block, RRDB)学习中国山水画独特的笔触和技法。接着，引入了多尺度结构相似性指数测量(multi-scale structural similarity index measure, MS-SSIM)损失以减少2幅图像之间的像素差异，使生成图像更接近传统绘画的色彩和颜料。最后，采用了融合谱归一化(spectral normalization, SN)的U-Net判别器增强图像纹理细节，并确保了模型训练过程的稳定性。大量实验验证了提出的方法在中国传统山水画风格迁移任务中的有效性和先进性。
关键词(KeyWords):
风格迁移;人工智能艺术;中国传统山水画;生成对抗网络
| 198 | 0 | 33 |
| 下载次数 | 被引频次 | 阅读次数 |
参考文献
[1] 彭进业，余喆，屈书毅，等.基于深度学习的图像修复方法研究综述[J].西北大学学报(自然科学版),2023,53(6):943-963.PENG J Y,YU Z,QU S Y,et al.A survey of image inpainting methods based on deep learning[J].Journal of Northwest University (Natural Science Edition),2023,53(6):943-963.
[2] 胡琦瑶，杨皓文，王佳欣，等.基于弱监督深度学习的图像检索技术研究[J].西北大学学报(自然科学版),2020,50(5):793-801.HU Q Y,YANG H W,WANG J X,et al.Research on image retrieval based on weakly-supervised deep learning[J].Journal of Northwest University(Natural Science Edition),2020,50(5):793-801.
[3] GOODFELLOW I J,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial networks[EB/OL].(2014-06-10)[2024-04-10].http://arxiv.org/abs/1406.2661.
[4] ZHU J Y,PARK T,ISOLA P,et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//2017 IEEE International Conference on Computer Vision.Venice:IEEE,2017:2242-2251.
[5] ISOLA P,ZHU J Y,ZHOU T,et al.Image-to-image translation with conditional adversarial networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition.Honolulu:IEEE,2017:5967-5976.
[6] ZHENG Z T,LIU J Y,ZHENG N N.P2-GAN:Efficient stroke style transfer using single style image[J].IEEE Transactions on Multimedia,2023,25:6000-6012.
[7] WANG Z Z,ZHANG Z J,ZHAO L,et al.AesUST:Towards aesthetic-enhanced universal style transfer[C]//Proceedings of the 30th ACM International Conference on Multimedia.Lisboa:ACM,2022:1095-1106.
[8] WANG X T,YU K,WU S X,et al.ESRGAN:Enhanced super-resolution generative adversarial networks[C]//Computer Vision-ECCV 2018 Workshops.Cham:Springer International Publishing,2019:63-79.
[9] WANG X T,XIE L B,DONG C,et al.Real-ESRGAN:Training real-world blind super-resolution with pure synthetic data[C]//2021 IEEE/CVF International Conference on Computer Vision Workshops.Montreal:IEEE,2021:1905-1914.
[10] RONNEBERGER O,FISCHER P,BROX T.U-Net:Convolutional networks for biomedical image segmentation[M]//Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015.Cham:Springer International Publishing,2015:234-241.
[11] MIYATO T,KATAOKA T,KOYAMA M,et al.Spectral normalization for generative adversarial networks[EB/OL].(2018-02-16)[2024-04-10].https://arxiv.org/abs/1802.05957.
[12] ZHAO H,GALLO O,FROSIO I,et al.Loss functions for image restoration with neural networks[J].IEEE Transactions on Computational Imaging,2017,3(1):47-57.
[13] 孟宪佳，傅利平，刘栋，等.高性能计算发展现状及其在文化遗产保护中的应用展望[J].西北大学学报(自然科学版),2021,51(5):807-815.MENG X J,FU L P,LIU D,et al.Development status of high performance computing and its application prospect in cultural heritage protection[J].Journal of Northwest University (Natural Science Edition),2021,51(5):807-815.
[14] GATYS L A,ECKER A S,BETHGE M,et al.Texture synthesis using convolutional neural networks[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems.Montreal:ACM,2015:262-270.
[15] CHEN R F,HUANG W B,HUANG B H,et al.Reusing discriminators for encoding:Towards unsupervised image-to-image translation[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle:IEEE,2020:8165-8174.
[16] YI Z L,ZHANG H,TAN P,et al.DualGAN:Unsupervised dual learning for image-to-image translation[C]//2017 IEEE International Conference on Computer Vision.Venice:IEEE,2017:2868-2876.
[17] KIM J,KIM M,KANG H,et al.U-GAT-IT:Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation[EB/OL].(2020-04-08)[2024-04-10].https://arxiv.org/abs/1907.10830v4.
[18] 张美玉，刘跃辉，秦绪佳，等.基于拉普拉斯算子抑制伪影的神经风格迁移方法[J].计算机科学，2020,47(S2):209-214.ZHANG M Y,LIU Y H,QIN X J,et al.Neural style transfer method based on Laplacian operator to suppress artifacts [J].Computer Science,2020,47(S2):209-214.
[19] 茹超，周延，陈晓璇，等.一种面向文本图像的颜色迁移算法[J].西北大学学报(自然科学版),2017,47(6):815-821.RU C,ZHOU Y,CHEN X X,et al.A color transfer algorithm for text image[J].Journal of Northwest University(Natural Science Edition),2017,47(6):815-821.
[20] PENG X L,PENG S L,HU Q Y,et al.Contour-enhanced CycleGAN framework for style transfer from scenery photos to Chinese landscape paintings[J].Neural Computing and Applications,2022,34(20):18075-18096.
[21] ZHANG F Q,GAO H M,LAI Y P.Detail-preserving CycleGAN-AdaIN framework for image-to-ink painting translation[J].IEEE Access,2020,8:132002-132011.
[22] IOFFE S,SZEGEDY C,PARANHOS L,et al.Batch normalization:Accelerating deep network training by reducing internal covariate shift[EB/OL].(2015-03-02)[2024-04-10].https://arxiv.org/abs/1502.03167v3.
[23] LIM B,SON S,KIM H,et al.Enhanced deep residual networks for single image super-resolution[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops.Honolulu:IEEE,2017:1132-1140.
[24] NAH S,KIM T H,LEE K M.Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition.Honolulu:IEEE,2017:257-265.
[25] WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality assessment:From error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[26] KINGMA D P,BA J,HAMMAD M M.Adam:A method for stochastic optimization[EB/OL].(2017-01-30)[2024-04-10].https://arxiv.org/abs/1412.6980v9.
[27] CHONG M J,FORSYTH D.Effectively unbiased FID and inception score and where to find them[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle:IEEE,2020:6069-6078.
[28] TALEBI H,MILANFAR P.NIMA:Neural image assessment[J].IEEE Transactions on Image Processing,2018,27(8):3998-4011.
[29] 孙满利，张景科.文物保护学的理论探讨[J].西北大学学报(自然科学版),2022,52(2):192-198.SUN M L,ZHANG J K.Theoretical discussion on conservation of cultural heritages[J].Journal of Northwest University (Natural Science Edition),2022,52(2):192-198.
[30] HO J,JAIN A,ABBEEL P,et al.Denoising diffusion probabilistic models[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems.Vancouver:ACM,2020:6840-6851.
[2] 胡琦瑶，杨皓文，王佳欣，等.基于弱监督深度学习的图像检索技术研究[J].西北大学学报(自然科学版),2020,50(5):793-801.HU Q Y,YANG H W,WANG J X,et al.Research on image retrieval based on weakly-supervised deep learning[J].Journal of Northwest University(Natural Science Edition),2020,50(5):793-801.
[3] GOODFELLOW I J,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial networks[EB/OL].(2014-06-10)[2024-04-10].http://arxiv.org/abs/1406.2661.
[4] ZHU J Y,PARK T,ISOLA P,et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//2017 IEEE International Conference on Computer Vision.Venice:IEEE,2017:2242-2251.
[5] ISOLA P,ZHU J Y,ZHOU T,et al.Image-to-image translation with conditional adversarial networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition.Honolulu:IEEE,2017:5967-5976.
[6] ZHENG Z T,LIU J Y,ZHENG N N.P2-GAN:Efficient stroke style transfer using single style image[J].IEEE Transactions on Multimedia,2023,25:6000-6012.
[7] WANG Z Z,ZHANG Z J,ZHAO L,et al.AesUST:Towards aesthetic-enhanced universal style transfer[C]//Proceedings of the 30th ACM International Conference on Multimedia.Lisboa:ACM,2022:1095-1106.
[8] WANG X T,YU K,WU S X,et al.ESRGAN:Enhanced super-resolution generative adversarial networks[C]//Computer Vision-ECCV 2018 Workshops.Cham:Springer International Publishing,2019:63-79.
[9] WANG X T,XIE L B,DONG C,et al.Real-ESRGAN:Training real-world blind super-resolution with pure synthetic data[C]//2021 IEEE/CVF International Conference on Computer Vision Workshops.Montreal:IEEE,2021:1905-1914.
[10] RONNEBERGER O,FISCHER P,BROX T.U-Net:Convolutional networks for biomedical image segmentation[M]//Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015.Cham:Springer International Publishing,2015:234-241.
[11] MIYATO T,KATAOKA T,KOYAMA M,et al.Spectral normalization for generative adversarial networks[EB/OL].(2018-02-16)[2024-04-10].https://arxiv.org/abs/1802.05957.
[12] ZHAO H,GALLO O,FROSIO I,et al.Loss functions for image restoration with neural networks[J].IEEE Transactions on Computational Imaging,2017,3(1):47-57.
[13] 孟宪佳，傅利平，刘栋，等.高性能计算发展现状及其在文化遗产保护中的应用展望[J].西北大学学报(自然科学版),2021,51(5):807-815.MENG X J,FU L P,LIU D,et al.Development status of high performance computing and its application prospect in cultural heritage protection[J].Journal of Northwest University (Natural Science Edition),2021,51(5):807-815.
[14] GATYS L A,ECKER A S,BETHGE M,et al.Texture synthesis using convolutional neural networks[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems.Montreal:ACM,2015:262-270.
[15] CHEN R F,HUANG W B,HUANG B H,et al.Reusing discriminators for encoding:Towards unsupervised image-to-image translation[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle:IEEE,2020:8165-8174.
[16] YI Z L,ZHANG H,TAN P,et al.DualGAN:Unsupervised dual learning for image-to-image translation[C]//2017 IEEE International Conference on Computer Vision.Venice:IEEE,2017:2868-2876.
[17] KIM J,KIM M,KANG H,et al.U-GAT-IT:Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation[EB/OL].(2020-04-08)[2024-04-10].https://arxiv.org/abs/1907.10830v4.
[18] 张美玉，刘跃辉，秦绪佳，等.基于拉普拉斯算子抑制伪影的神经风格迁移方法[J].计算机科学，2020,47(S2):209-214.ZHANG M Y,LIU Y H,QIN X J,et al.Neural style transfer method based on Laplacian operator to suppress artifacts [J].Computer Science,2020,47(S2):209-214.
[19] 茹超，周延，陈晓璇，等.一种面向文本图像的颜色迁移算法[J].西北大学学报(自然科学版),2017,47(6):815-821.RU C,ZHOU Y,CHEN X X,et al.A color transfer algorithm for text image[J].Journal of Northwest University(Natural Science Edition),2017,47(6):815-821.
[20] PENG X L,PENG S L,HU Q Y,et al.Contour-enhanced CycleGAN framework for style transfer from scenery photos to Chinese landscape paintings[J].Neural Computing and Applications,2022,34(20):18075-18096.
[21] ZHANG F Q,GAO H M,LAI Y P.Detail-preserving CycleGAN-AdaIN framework for image-to-ink painting translation[J].IEEE Access,2020,8:132002-132011.
[22] IOFFE S,SZEGEDY C,PARANHOS L,et al.Batch normalization:Accelerating deep network training by reducing internal covariate shift[EB/OL].(2015-03-02)[2024-04-10].https://arxiv.org/abs/1502.03167v3.
[23] LIM B,SON S,KIM H,et al.Enhanced deep residual networks for single image super-resolution[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops.Honolulu:IEEE,2017:1132-1140.
[24] NAH S,KIM T H,LEE K M.Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition.Honolulu:IEEE,2017:257-265.
[25] WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality assessment:From error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[26] KINGMA D P,BA J,HAMMAD M M.Adam:A method for stochastic optimization[EB/OL].(2017-01-30)[2024-04-10].https://arxiv.org/abs/1412.6980v9.
[27] CHONG M J,FORSYTH D.Effectively unbiased FID and inception score and where to find them[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle:IEEE,2020:6069-6078.
[28] TALEBI H,MILANFAR P.NIMA:Neural image assessment[J].IEEE Transactions on Image Processing,2018,27(8):3998-4011.
[29] 孙满利，张景科.文物保护学的理论探讨[J].西北大学学报(自然科学版),2022,52(2):192-198.SUN M L,ZHANG J K.Theoretical discussion on conservation of cultural heritages[J].Journal of Northwest University (Natural Science Edition),2022,52(2):192-198.
[30] HO J,JAIN A,ABBEEL P,et al.Denoising diffusion probabilistic models[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems.Vancouver:ACM,2020:6840-6851.
基本信息:
DOI：10.16152/j.cnki.xdxbzr.2025-01-005
中图分类号:J212;TP391.41;TP18
引用信息:
[1]胡琦瑶,刘乾龙,彭先霖等.SN-CLPGAN:基于谱归一化的中国传统山水画风格迁移方法[J].西北大学学报(自然科学版),2025,55(01):63-74.DOI:10.16152/j.cnki.xdxbzr.2025-01-005.
基金信息:
国家自然科学基金(62471390、62306237); 陕西省重点研发计划(2024GX-YBXM-149); 西北大学研究生创新项目(CX2024204、CX2024206)