AudioSketch: Controllable Image-to-Audio Generation via Semantic-Temporal Energy Modulation

EUSIPCO 2026 (Accepted)

Authors

Jihyun Lee, Jiahao Li, Woojin Chung, Yan Lu, and Hong-Goo Kang

Abstract

In this paper, we propose AudioSketch, a diffusion-based image-to-audio (I2A) generation model with an effective and user-friendly energy guidance mechanism. Unlike prior methods that rely on either overly simplistic or impractically complex energy controls, AudioSketch introduces an energy modulator that transforms user-provided energy inputs into a refined and expressive representation, enabling the synthesis of high-quality and natural-sounding audio. To identify an optimal energy structure, we conduct an in-depth analysis of how different energy representations influence audio quality, temporal fidelity, and image-audio semantic alignment. Integrated with a lightweight neural network that maps image features to audio-related representations, our framework supports a progressive translation process: from an image and user-provided control to a refined energy representation, and subsequently from these conditions to controllable audio synthesis. Experimental results demonstrate that AudioSketch consistently outperforms existing I2A approaches in terms of audio quality and image-audio semantic consistency, while providing intuitive and perceptually meaningful temporal control over the generated audio. Audio samples are available on our demo page.

Result samples#1: Performance comparison on I2A methods

We verify the vanilla image-to-audio generation capability of AudioSketch without energy control, comparing it to previous I2A methods.

Input image	AudioSketch w/o control	Im2Wav	V2A-Mapper	Seeing and Hearing

Result samples#2: Performance comparison on various energy representations

We conduct ablation studies on different energy representations used as conditions for ControlNet.

RMS-based energy with smoothing and normalization achieves higher audio quality and naturalness compared to others.

Blue curves depict input energy trajectories provided to ControlNet.

Input image	RMS + Smoothing + Z-norm	Timestamp	Logmel mean + Smoothing + Z-norm Caution! Popping noise

Result samples#3: Impact of energy modulator

Given an user-friendly timestamp input, applying energy modulator improves audio quality and naturalness while preserving temporal alignment.

Without a semantic embedding, however, the modulator loses the temporal information of the input timestamp, disabling the overall model from operating as a temporal-controllable audio generator.

Blue curves depict the input timestamps or modulated energy trajectories provided to ControlNet.