awesome generative models

awesome-generative-models

A curated list of awesome generative model frameworks, libraries, software and resources for media production

inspired by awesome-python

---

^{My open source work is supported by the community}

• Awesome Generative Mdels
  • Code generation: ML algorithm + Static Analysis
  • Code to GUI: algorithm to user interface
  • Image Synthesis
    • Image Style Transfer
  • Text Generation/NLP
    • Audio and music generation/processing
    • Music-Video Synthesis
  • Video-Synth
  • Procedural Generation for gaming
  • Image-Generation
  • Forum

Code to GUI: algorithm to user interface:

• Screenshoot to Code
• pix2code: Generating Code from a Graphical User Interface Screenshot
• Bred Victor: Machine learning in Engineering inference from enviroment

Code generation: ML algorithm + Static Analysis

• Tabnine: Autocompletion with deep learning
• Kite: Algorithm assisted coder completion
• Insignt: Standard code completion tools often still use alphabetical sorting, while Kite uses ML algorithms to infer what a developer is likely trying to do

Image Synthesis

• SPADE by NVlabs: Synthesizing photorealistic images given an input semantic layout. code

Image Style Transfer

• Artbreeder based on BigGAN models opensource version BigGAN models About

• A cool artistic project: ... DGSpitzer(Eddie Hu) a indie game maker Use Artbreeder based on BigGAN models, StyleGAN-Artm Realistic-Neural_Talking_Head_Modelsm First-Order_Modelsm DAIN and Topaz Lab for some great work on digital video repair, which colorize black and while video.

• A cool project: Visual Novel using ML Style transfer

NLP: Text-Sentiment-Analysis:

• DeepMoji:a sentiment analysis model This model also used in 15.ai, a text-to-speech tool for generating voices of various characters.

Text-to-Image:

• dataset: Google’s “Quick Draw” open source dataset github

Text-to-Video:

• AllenNLP: Imagine This! Scripts to Compositions to Videos

Image Captioning

• Tensorflow core: Image captioning with visual attention

Text-to-Animation

• Generating Animations from Screenplays

Text Generation/NLP

Question Generation:

• Question Generation: generate multiple choice answers from text

Other

• OpenAI 1.5 billion params GPT-2 release
• AIDungeon
• ctrl-gce: CTRL text-generating model on Google Compute Engine with just a few console commands., why google compute engine: The CTRL model is so large (12 GB on disk, 15.5 GB GPU VRAM when loaded, even more system RAM during runtime) that it will currently not fit into a free Colaboratory or Kaggle Notebook.
• Writing with the machine scifi corpus txt dataset Could be useful for gpt2 model fine tune
• GPT2-Chinese-wuxiao-novel
• GPT-2 Chinese
• [Hugging face Transformer]-Pytorch hub

Text Corpus/dataset

• Sci-fi-Script
• Detroit-Becoming-Human Could be source to analyze dialogue tree and decision tree structure.

Audio and music generation/processing

Project:

• Talking like your favorite character: Text-To-Speech audio generation related research:
• Tacotron2
• ForwardTacotron: Tacotron2 without attention
• Voice clone: Real-Time-Voice-Cloning
• Spleeter: sound track seperation sound seperation is under domain of music information retrival.
• Ambient Generative Music by Alex Bainter Although this project is not generated by algorithm, gives much inspiration in the field of music generation. Medium code *NeuralFunk: Sound design with ML

Music-Video Synthesis

• Deep Music Visualizer using BigGan code

Video Synthesis and Generation:

• pix2pix-tensorflow: poweed interative rendereed Virtual world code update: pretrained model added more about colab

  • Using this technique we can colorize black and white photos, convert google maps to google earth, etc.

• CRAFT, which generates cartoons based on text descritpionsa A very creative work involved text to video generation from allen nlp by researcher Project page Video

Audio Systhesis

• nvidia's taco2 pytorch implementation
• real-time-voice-clone

ProceduralGenerationForGaming

Map generation

• AI-Powered Procedural Fantasy Map Generator, reference: @linonetwo's blog

Animation

• deep learning for character animation and control
• DeepMimic: Motion imitation with deep reinforcement learning

Distributed Training:

• MPI Reduce and Allreduce Very useful tutorial to illustrate the concept of MPT. CHECK CODE
• Also check on Tensorflow-core
• Distributed Traning strategy

Computer Vision application in gaming

• E-Sports Talent Scouting Based on Multimodal Twitch Stream Data data acquisition and modeling code Chat Log Twich Stream

Paper with code

• Paper with code

Resources on related course:

• UIUC CS598RK: HCI for ML
• coursera: Sequence Models
• Full Stack Deep Learning
• fast.ai
• UIUC: ECE420 Video processing lab
• GPU free resource:Setting Up a Google Cloud Instance GPU for fast.ai for Free

Cloud computing set up

• AWS
• Google colab

Full Stack Deep Learning tools in data processing pipeline:

• cortex: Deploy machine learning models in production possibly without docker and kubernetes medium

Resources for dev tool:

• Lumen: Video syth software
• Dialogue tree based node editor for unity and example demo

More resources on pretrained model:

• [Pytorch hub]https://pytorch.org/hub/research-models
• Tensorflow hub

MEET mind-linked people in forum:

• reddit-MediaSynthesis

Application:

Generating voices of various characters: Ideally podcast and gaming etc:

• 15.ai

FILM MAKING:

• Black Mirror: Bandersnatch Show case: dialogue tree
• LATE-SHIFT

Design

• Algorithm Driven Design
• Algorithm generated Logo

Reference:

• For details about how to represent animation architecture from software perspective and in math. Check Chapter11-Animation System of Game Engine Architecture,2nd edition by Jason Gregory
• Kite's new AI model