Drawing Beyond the Prompt

Exploring Real-Time Generative AI using Drawing and Speech as Multimodal Interactions for Graphic Design

  • Creative Technology
  • Interaction
  • Generative Design

Drawing Beyond the Prompt is a design-led research project that explores how generative AI can better support designers—not by replacing them, but by adapting to how they think and work. Instead of typing static prompts, designers can draw freely while speaking their ideas. A speech-to-text model transcribes their voice into prompt text, allowing for seamless, real-time interaction without breaking creative flow. This research focuses on designing with the human at the centre. By developing functional prototypes and testing them in workshops with design students and professionals, the project examines how sketch-and-speak interaction changes the way designers engage with AI. The goal is to create tools that feel less like machines waiting for instructions, and more like creative partners that respond naturally to human input.

A project by Ma Jiahao Darrick
A project by Ma Jiahao Darrick
A project by Ma Jiahao Darrick
A project by Ma Jiahao Darrick
A project by Ma Jiahao Darrick
A project by Ma Jiahao Darrick
A project by Ma Jiahao Darrick
A project by Ma Jiahao Darrick
A project by Ma Jiahao Darrick
A project by Ma Jiahao Darrick
A project by Ma Jiahao Darrick