Audio Samples from "Assem-VC: Realistic Voice Conversion by Assembling Modern Speech Synthesis Techniques"

Paper: arXiv:2104.00931

Repository: mindslab-ai/assem-vc @ GitHub

Authors: Kang-wook Kim, Seung-won Park, Junhyeok Lee, Myun-chul Joe @MINDsLab Inc., SNU

Abstract: In this paper, we pose the current state-of-the-art voice conversion (VC) systems as two-encoder-one-decoder models. After comparing these models, we combine the best features and propose Assem-VC, a new state-of-the-art any-to-many non-parallel VC system. This paper also introduces the GTA finetuning in VC, which significantly improves the quality and the speaker similarity of the outputs. Assem-VC outperforms the previous state-of-the-art approaches in both the naturalness and the speaker similarity on the VCTK dataset. As an objective result, the degree of speaker disentanglement of features such as phonetic posteriorgrams (PPG) is also explored. Our investigation indicates that many-to-many VC results are no longer distinct from human speech and similar quality can be achieved with any-to-many models.

Contents

Mellotron-VC: VC System based on Mellotron which we modify to normalize the pitch to suit the VC task.

PPG-VC: Our proposed VC system for comparison between Cotatron and PPG. It uses PPG, Normalized F0, and causal decoder.

Cotatron-VC: VC System proposed in Cotatron paper.

Assem-VC (Proposed): Our Proposed VC system using Cotatron, Normalized F0, and causal decoder.

Assem-VC (adv): Assem-VC with adversarial Cotatron. There is a degradation of quality when adopting adversarial training on Cotatron.

GTA Finetuning (Proposed methodology): Finetuning vocoder with the reconstructed mel spectrograms from the acoustic model.

Source speakers: Randomly selected speech from VCTK test split, which consists of 108 English speakers. All the speakers are seen during the training step.

Target speakers: Randomly selected speech from VCTK test split, which consists of 108 English speakers.

Text Content: "it was an empowering journey"

Converted Results

Text Content: "the clarity is vital"

Converted Results

Text Content: "there may be some resistance"

Converted Results

Text Content: "what do we do ?"

Converted Results

Text Content: "england were ahead until two minutes into injury time"

Converted Results

Source speakers: Randomly selected speech from LibriTTS test-clean split. All the speakers are unseen during the training step.

Target speakers: Randomly selected speech from VCTK test split, which consists of 108 English speakers

Text Content: "If the prosecution were withdrawn and the case settled with the victim of the forged check, then the young man would be allowed his freedom. But under the circumstances I doubt if such an arrangement could be made."

Text Content: "My son," said his mamma, "just stop and think how badly you would feel, if you really couldn't see Percy."

Text Content: "Magic must be a very interesting study," said Ojo.

Text Content: Soon another fire sparkled and snapped on the hearth, and there were songs and poems and choruses and Osh Popham's fiddle, to say nothing of the supreme event of the evening, his rendition of "Fly like a youthful hart or roe, over the hills where spices grow," to Mother Carey's accompaniment.

Text Content: This is my schooling, major; and if one neglects the book, there is little chance of learning from the open land of Providence.

Text Content: She makes effort after effort, trembling with eagerness, and when she fails to reproduce what she sees, she works herself into a frenzy of grief and disappointment."

Text Content: I shall return to the consideration of familiarity shortly.

We show some entertaining conversion results from the speech of celebrities.