This project contains implementation of transformer models being used in NLP research for various tasks.

This repo contains

• Transformer as introduced in Attention is all you need
• Google BERT

• Pytorch == 1.0.0

To create a Transformer

from attention_main import *   
model = create_transformer(5, 5)
model_opt = get_trans_optim(model)
criterion = LabelSmoothing(size=5, padding_idx=0, smoothing=0.0)
fit_transformer(generate_data(5, 30, 20), model, loss_compute(model.generator,criterion,model_opt))