vtbai

Function in coding ....

• VTS API keyword trigger expressions
• Sentence emotional inference triggers VTS expression
• Azure API TTS
• Link CharacterAI Role-playing Best AI
• Multi link link
• ChatGLM-6B Local 1
• ChatterBot Local 2
• Twitch & Youtube Platform Support

resource

• vits vits source

• MoeGoe vits chinese

• vits_with_chatgpt-gpt3 tts reasoning reference

• blivedm crawl the live broadcast room information of B station

• Demonstration model vits model (commercial use, please refine the legal sound source agreed by the natural person or use Azure)

• Architecture

Grab barrage and gifts from the Bilibili live broadcast room, receive them and send them to the official chatgpt, wait for the gpt callback message and use vits to perform tts inference, and then trigger expressions and play sounds based on the keywords/VTS API. When playing, vts matches the lip shape according to the sound.

Unless the company or the boss, it is very important to rub live2d by hand. First, it is time-consuming, and second, vts and prprlive can be used to trigger expression shortcut keys with keywords, and the VTS API live broadcast effect is still OK.

This live broadcast process is only tested and passed under win. Theoretically, Linux and Mac can also be used in a suitable py environment. The py version is conda 3.10.10

Note: Those who are capable can extend the mouse and oil pipes the same way, py thread coroutine process shark me

Workflow

blivedm (grab live broadcast room information) —>openai (cat girl dialogue) —>vits (tts text to voice) —>vts (voice to lip, shortcut keys trigger expression) —>obs (push stream)

Function

• Support API requests, use extensions, such as linkage, and request after voice conversion to text http://127.0.0.1:8080?text=Hello
• Support priority: api>Captain>Gift>SC>Barc, the capacity of gifts and barrage queues is 5/10, and the last timeline is full. SC and captain have no capacity limit.

My development environment

• CPU: 5700G
• GPU: Core display
• Motherboard: Asus B550m TUF
• Hard drive: Kaixia RC20
• Memory: 16*4 (3133hz Jinbaida)
• Power supply: Delta 300W
• Chassis: X40 5.5L

The main speed is to request openai and cpu inference, and using gpu will be much faster. Since you need to detect homophone sensitive words to prevent explosions, SSE cannot be used.

Obs, VTS, and Quehun AI are fully opened. The CPU load is about 70% when reasoning. Theoretically speaking, the Beggar 5600G with a budget of 2k can also run, the power consumption is not more than 100W, and ITX can run, and what bicycle is needed. In actual situation, please use N card GPU to run better.

Construction process

Please make sure you have installed conda, obs, vts, vscode

Please use conda to create a new python 3.10 environment and launch vscode

git clone -b 1.1-py https://github.com/newreport/vtbai.git
cd vtbai 
start.bat
# configmy_config.ini 填写房间号和 openai key
python main.py
# 弹幕，将 resource 中的 hmtl 拖到 obs 里

]