Nanxin Chen

Gemini: A Family of Highly Capable Multimodal Models

Gemini Robotics Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac et autres

This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained …

SLM: Bridge the Thin Gap Between Speech and Text Foundation Models

Mingqiu Wang, Wei Han, Izhak Shafran, Zelin Wu et autres

We present a joint Speech and Language Model (SLM), a multitask, multilingual, and dual-modal model that takes advantage of pretrained foundational speech and language models. SLM freezes the pretrained foundation models to maximally preserves their capabilities, and only trains a simple adapter …

us (code pays fourni par la source)

SLM: Bridge the thin gap between speech and text foundation models

Mingqiu Wang, Wei Han, Izhak Shafran, Zelin Wu et autres

We present a joint Speech and Language Model (SLM), a multitask, multilingual, and dual-modal model that takes advantage of pretrained foundational speech and language models. SLM freezes the pretrained foundation models to maximally preserves their capabilities, and only trains a simple adapter …

Efficient Adapters for Giant Speech Models

Nanxin Chen, Izhak Shafran, Yu Zhang, Chung‐Cheng Chiu et autres

Large pre-trained speech models are widely used as the de-facto paradigm, especially in scenarios when there is a limited amount of labeled data available. However, finetuning all parameters from the self-supervised learned model can be computationally expensive, and becomes infeasiable as the …

How to Estimate Model Transferability of Pre-Trained Speech Models?

Zih-Ching Chen, Chao-Han Huck Yang, Bo Li, Yu Zhang et autres

In this work, we introduce a "score-based assessment" framework for estimating the transferability of pre-trained speech models (PSMs) for fine-tuning target tasks. We leverage upon two representation theories, Bayesian likelihood estimation and optimal transport, to generate rank scores for the PSM candidates …

From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition

Chao-Han Huck Yang, Bo Li, Yu Zhang, Nanxin Chen et autres

In this work, we propose a new parameter-efficient learning framework based on neural model reprogramming for cross-lingual speech recognition, which can \textbf{re-purpose} well-trained English automatic speech recognition (ASR) models to recognize the other languages. We design different auxiliary neural architectures focusing on …

A Quantum Kernel Learning Approach to Acoustic Modeling for Spoken Command Recognition

Chao-Han Huck Yang, Bo Li, Yu Zhang, Nanxin Chen et autres

We propose a quantum kernel learning (QKL) framework to address the inherent data sparsity issues often encountered in training large-scare acoustic models in low-resource scenarios. We project acoustic features based on classical-to-quantum feature encoding. Different from existing quantum convolution techniques, we utilize …

Residual Adapters for Few-Shot Text-to-Speech Speaker Adaptation

Nobuyuki Morioka, Heiga Zen, Nanxin Chen, Yu Zhang et autres

Adapting a neural text-to-speech (TTS) model to a target speaker typically involves fine-tuning most if not all of the parameters of a pretrained multi-speaker backbone model. However, serving hundreds of fine-tuned neural TTS models is expensive as each of them requires significant …

Maestro-U: Leveraging joint speech-text representation learning for zero supervised speech ASR

Zhehuai Chen, Ankur Bapna, Andrew E. Rosenberg, Yu Zhang et autres

Training state-of-the-art Automated Speech Recognition (ASR) models typically requires a substantial amount of transcribed speech. In this work, we demonstrate that a modality-matched joint speech and text model can be leveraged to train a massively multilingual ASR model without any supervised (manually …

Research on Mechanical Fault Diagnosis of Vacuum Circuit Breaker Based on Deep Belief Network

Yan Tong, Zhaoyu Ku, Nanxin Chen, Hu Sheng

VCB is an important component to ensure the safe and smooth operation of the power system. As an important driving part of the vacuum circuit breaker, the operating mechanism is prone to mechanical failure, which leads to power grid accidents. This paper …

cn (code pays fourni par la source)

Preprocessed Data and Pretrained Models for Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings

Erica Cooper, Cheng-I Lai, Yusuke Yasuda, Fuming Fang et autres

This is preprocessed data and pretrained models from two of our papers: "Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings," by Erica Cooper, Cheng-I Lai, Yusuke Yasuda, Fuming Fang, Xin Wang, Nanxin Chen, and Junichi Yamagishi. (ICASSP 2020) https://arxiv.org/abs/1910.10838 "Pretraining Strategies, Waveform …

jp, us (code pays fourni par la source)

Preprocessed Data and Pretrained Models for Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings

Erica Cooper, Cheng-I Lai, Yusuke Yasuda, Fuming Fang et autres

This is preprocessed data and pretrained models from two of our papers: "Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings," by Erica Cooper, Cheng-I Lai, Yusuke Yasuda, Fuming Fang, Xin Wang, Nanxin Chen, and Junichi Yamagishi. (ICASSP 2020) https://arxiv.org/abs/1910.10838 "Pretraining Strategies, Waveform …

jp, us (code pays fourni par la source)