BetterTransformer integration for more models!

BetterTransformer API provides faster inference on CPU & GPU through a simple interface!

Models can benefit from very interesting speedups using a one liner and by making sure to install the latest version of PyTorch. A complete guideline on how to convert a new model has been created on the BetterTransformer documentation!

Here is a list of models that could be potentially supported, pick one of the architecture below and let's discuss about the conversion!

Text models 🖊️ :

• FSMT - FSMTEncoderLayer / @Sumanth077 [BT] Add Bettertransformer support for FSMT #494
• MobileBERT - MobileBertLayer / @raghavanone [BetterTransformer] Add MobileBERT support for BetterTransformer #506
• MBart - MBartEncoderLayer + M2M100EncoderLayer / Add MBart support for BetterTransformer #516 @ravenouse
• ProphetNet - ProphetNetEncoderLayer
• RemBert - RemBertLayer
• RocBert - RocBertLayer
• RoFormer - RoFormerLayer
• Tapas - TapasLayer / Added support for Tapas Model #520

Vision models 📷 :

• Blip - BlipLayer
• Detr - DetrLayer
• Flava - FlavaLayer
• GLPN - GLPNLayer | Cannot be supported
• ViLT - ViLTLayer / [BT] add BetterTransformer support for ViLT architecture #508

Audio models 🔉 :

• Speech2Text - Speech2TextLayer
• NEW: Audio Speech Transformer - ASTLayer

Let us also know if you think that some architectures can be supported that we missed. Note that for encoder-decoder based models below, we expect to convert the encoder only.

Support for decoder-based models coming soon!

cc @michaelbenayoun @fxmarty

huggingface/transformers#20372