Bryan Wang

Representative Publications

1. 

   Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models

   Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman

   In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (UIST) 2023

   Abs Bib PDF

   Text-to-image generative models have demonstrated remarkable capabilities in generating high-quality images based on textual prompts. However, crafting prompts that accurately capture the user’s creative intent remains challenging. It often involves laborious trial-and-error procedures to ensure that the model interprets the prompts in alignment with the user’s intention. To address the challenges, we present Promptify, an interactive system that supports prompt exploration and refinement for text-to-image generative models. Promptify utilizes a suggestion engine powered by large language models to help users quickly explore and craft diverse prompts. Our interface allows users to organize the generated images flexibly, and based on their preferences, Promptify suggests potential changes to the original prompt. This feedback loop enables users to iteratively refine their prompts and enhance desired features while avoiding unwanted ones. Our user study shows that Promptify effectively facilitates the text-to-image workflow and outperforms an existing baseline tool widely used for text-to-image generation.

   @inproceedings{brade2023promptify,
     title = {Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models},
     author = {Brade, Stephen and Wang, Bryan and Sousa, Mauricio and Oore, Sageev and Grossman, Tovi},
     year = {2023},
     eprint = {2304.09337},
     archiveprefix = {arXiv},
     primaryclass = {cs.HC},
     booktitle = {Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (UIST)},
     doi = {10.1145/3586183.3606725},
     category = {Human-AI Systems for Creativity}
   }

2. 

   Enabling Conversational Interaction with Mobile UI using Large Language Models

   Bryan Wang, Gang Li, and Yang Li

   In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI) 2023

   Abs Bib PDF Video

   Conversational agents show the promise to allow users to interact with mobile devices using language. However, to perform diverse UI tasks with natural language, developers typically need to create separate datasets and models for each specific task, which is expensive and effort-consuming. Recently, pre-trained large language models (LLMs) have been shown capable of generalizing to various downstream tasks when prompted with a handful of examples from the target task. This paper investigates the feasibility of enabling versatile conversational interactions with mobile UIs using a single LLM. We propose a design space to categorize conversations between the user and the agent when collaboratively accomplishing mobile tasks. We design prompting techniques to adapt an LLM to conversational tasks on mobile UIs. The experiments show that our approach enables various conversational interactions with decent performances, manifesting its feasibility. We discuss the use cases of our work and its implications for language-based mobile interaction.

   @inproceedings{llm4ui,
     video = {https://www.youtube.com/watch?v=_M4ID2XlRpw},
     author = {Wang, Bryan and Li, Gang and Li, Yang},
     title = {Enabling Conversational Interaction with Mobile UI using Large Language Models},
     year = {2023},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     booktitle = {Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI)},
     category = {UI Modeling and Agents}
   }

3. 

   Record Once, Post Everywhere: Automatic Shortening of Audio Stories for Social Media

   Bryan Wang, Zeyu Jin, and Gautham Mysore

   In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (UIST), 2022

   Abs Bib PDF Video Website

   Following the prevalence of short-form video, short-form voice content has emerged on social media platforms like Twitter and Facebook. A challenge that creators face is hard constraints on the content length. If the initial recording is not short enough, they need to re-record or edit their content. Both are time-consuming, and the latter, if supported, can have a learning curve. Moreover, creators need to manually create multiple versions to publish content on platforms with different length constraints. To simplify this process, we present ROPE1 (Record Once, Post Everywhere). Creators can record voice content once, and our system will automatically shorten it to all length limits by removing parts of the recording for each target. We formulate this as a combinatorial optimization problem and propose a novel algorithm that automatically selects optimal sentence combinations from the original content to comply with each length constraint. Creators can customize the algorithmically shortened content by specifying sentences to include or exclude. Our system can also use the user-specified constraints to recompute and provides a new version. We conducted a user study comparing ROPE with a sentence-based manual editing baseline. The results show that ROPE can generate high-quality edits, alleviating the cognitive loads of creators for shortening content. While our system and user study address short-form voice content specifically, we believe that the same concept can also be applied to other media such as video with narration and dialog.

   @inproceedings{wang2022rope,
     video = {https://youtu.be/6F60KCaYGu8},
     author = {Wang, Bryan and Jin, Zeyu and Mysore, Gautham},
     title = {Record Once, Post Everywhere: Automatic Shortening of Audio Stories for Social Media},
     year = {2022},
     isbn = {9781450393201},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3526113.3545680},
     doi = {10.1145/3526113.3545680},
     booktitle = {Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (UIST),},
     articleno = {14},
     numpages = {11},
     keywords = {storytelling, social media, short-form audio, Audio editing},
     location = {Bend, OR, USA},
     series = {UIST '22},
     category = {Human-AI Systems for Creativity}
   }

4. 

   Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning

   Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li

   In Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology (UIST), 2021

   Abs Bib PDF Code Presentation Dataset

   Mobile User Interface Summarization generates succinct language descriptions of mobile screens for conveying important contents and functionalities of the screen, which can be useful for many language-based application scenarios. We present Screen2Words, a novel screen summarization approach that automatically encapsulates essential information of a UI screen into a coherent language phrase. Summarizing mobile screens requires a holistic understanding of the multi-modal data of mobile UIs, including text, image, structures as well as UI semantics, motivating our multi-modal learning approach. We collected and analyzed a large-scale screen summarization dataset annotated by human workers. Our dataset contains more than 112k language summarization across ∼ 22k unique UI screens. We then experimented with a set of deep models with different configurations. Our evaluation of these models with both automatic accuracy metrics and human rating shows that our approach can generate high-quality summaries for mobile screens. We demonstrate potential use cases of Screen2Words and open-source our dataset and model to lay the foundations for further bridging language and user interfaces.

   @inproceedings{10.1145/3472749.3474765,
     dataset = {https://github.com/google-research-datasets/screen2words},
     author = {Wang, Bryan and Li, Gang and Zhou, Xin and Chen, Zhourong and Grossman, Tovi and Li, Yang},
     title = {Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning},
     year = {2021},
     isbn = {9781450386357},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3472749.3474765},
     doi = {10.1145/3472749.3474765},
     booktitle = {Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology (UIST),},
     pages = {498–510},
     numpages = {13},
     keywords = {dataset., Mobile UI summarization, language-based UI, screen understanding, deep learning},
     location = {Virtual Event, USA},
     series = {UIST '21},
     talk = {https://youtu.be/9ftQqNIJpVI},
     category = {UI Modeling and Agents}
   }

5. 

   Soloist: Generating Mixed-Initiative Tutorials from Existing Guitar Instructional Videos Through Audio Processing

   Bryan Wang, Meng Yu Yang, and Tovi Grossman

   In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI), 2021

   Abs Bib PDF Presentation Video

   Learning musical instruments using online instructional videos has become increasingly prevalent. However, pre-recorded videos lack the instantaneous feedback and personal tailoring that human tutors provide. In addition, existing video navigations are not optimized for instrument learning, making the learning experience encumbered. Guided by our formative interviews with guitar players and prior literature, we designed Soloist, a mixed-initiative learning framework that automatically generates customizable curriculums from off-the-shelf guitar video lessons. Soloist takes raw videos as input and leverages deep-learning based audio processing to extract musical information. This back-end processing is used to provide an interactive visualization to support effective video navigation and real-time feedback on the user’s performance, creating a guided learning experience. We demonstrate the capabilities and specific use-cases of Soloist within the domain of learning electric guitar solos using instructional YouTube videos. A remote user study, conducted to gather feedback from guitar players, shows encouraging results as the users unanimously preferred learning with Soloist over unconverted instructional videos.

   @inproceedings{10.1145/3411764.3445162,
     video = {https://youtu.be/JqQCeUPdk2U},
     author = {Wang, Bryan and Yang, Meng Yu and Grossman, Tovi},
     title = {Soloist: Generating Mixed-Initiative Tutorials from Existing Guitar Instructional Videos Through Audio Processing},
     year = {2021},
     isbn = {9781450380966},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3411764.3445162},
     doi = {10.1145/3411764.3445162},
     booktitle = {Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI),},
     articleno = {98},
     numpages = {14},
     keywords = {Music Learning, Intelligent Tutoring System, Waveform Navigation, Soloist, Mixed-Initiative Learning, Audio Processing, Converting Videos to Interactive Tutorials, Video Learning},
     location = {Yokohama, Japan},
     series = {CHI '21},
     talk = {https://youtu.be/joDokLJYLjs},
     category = {Human-AI Systems for Creativity}
   }