A
Andrew Camp
Description:
Introduce a built-in voice assistant that can create and edit templates based on natural language descriptions. The assistant should understand spatial layout requests (“put three loops across the top, faders along the bottom”), widget behavior (“this button should start all drums”), and routing/logic, and then translate the user’s spoken instructions into concrete changes in the template.
The assistant would work in phases:
  • Phase 1:
    The assistant parses spoken instructions and replies with text in a popup window, describing what it would do or suggesting how to achieve the layout/behavior.
  • Phase 2:
    The assistant can actually perform complex edits across multiple widgets and loops, with a confirmation step before applying changes, and optionally reply via on-screen text and/or audible voice.
Problem:
Building and editing complex templates in Loopy can be time-consuming and cognitively demanding, especially for users who:
  • Are new to the app and don’t yet know where every option lives.
  • Think in musical or spatial terms rather than in UI terminology.
  • Want to iterate quickly while keeping their focus on playing rather than on detailed UI editing.
Current workflow challenges include:
  • Manual layout work:
    Placing widgets, sizing them, and arranging them spatially can be fiddly, especially on smaller screens.
  • Steep learning curve for logic and behavior:
    Defining what each widget should do, which loops it controls, and how they interact may require navigating multiple menus and inspectors.
  • High friction for complex editing:
    Editing several widgets and loops at once (e.g. changing multiple parameters, routing schemes, or actions) is possible but involves many separate manual steps.
  • Accessibility and ergonomics:
    Some users may have physical or visual limitations that make drag-and-drop UI editing or intricate menus less comfortable.
A voice-based assistant could dramatically reduce the friction of going from “I have an idea for a layout/behavior” to a working template.
Proposed Solution:
Implement a two-phase voice assistant feature with a strong focus on clarity, safety, and iterative conversation:
  1. Phase 1 – Conversational design assistant (text-only):
- Users can speak instructions such as:
- “Create a template with four loop widgets arranged in a row at the top.”
- “Add a big play/stop button in the center that controls all loops.”
- “Make this knob control the filter cutoff of my main synth.”
- The assistant:
- Interprets the request and displays its proposed actions in a popup text window.
- Asks clarifying questions when unsure, for example:
- “When you say ‘drums’, do you mean the group ‘Drums’ or track 3?”
- “Should this button start all loops on this page or in the whole project?”
- Suggests how to achieve the requested behavior using existing features, even before it has direct editing capabilities.
- This phase focuses on understanding, guiding, and educating the user, building trust and a shared vocabulary.
  1. Phase 2 – Voice-driven editing and execution:
- The assistant gains the ability to actually apply changes to templates:
- Create, move, resize and delete widgets.
- Assign actions and bindings.
- Configure loops, buses, and basic routing.
- Support for
complex multi-step commands
, for example:
- “Create a new performance page with eight loop buttons, each controlling its own track, and a master mute button in the bottom-right corner.”
- “Make all existing faders control the headphone mix instead of the main output.”
- Before executing, the assistant:
- Shows a
confirmation popup
summarizing what will change.
- Lets the user approve or cancel the operation.
- The user can choose how responses are presented:
- Text in a popup window only.
- Text plus audible spoken feedback (text-to-speech), useful for hands-free editing.
  1. Clarification and safety:
- The assistant should:
- Default to asking clarifying questions when intent is ambiguous.
- Avoid destructive changes without clear confirmation.
- Offer undo or a quick “revert” option after applying changes.
- Provide a simple way to inspect what the assistant has just done (e.g. a brief change log or “What did you just change?” query).
  1. Integration with existing workflows:
- Voice commands should work alongside all existing UI and action-based workflows:
- Users can mix voice-driven edits with manual fine-tuning.
- Potential actions to start/stop listening, or to trigger a “repeat last instruction” or “explain current layout” interaction.
- Over time, the assistant could also help explain the current template:
- “Tell me what this button does.”
- “Describe how this page is laid out and what each control is for.”