
Keras BERT
==========


.. image:: https://travis-ci.org/CyberZHG/keras-bert.svg
   :target: https://travis-ci.org/CyberZHG/keras-bert
   :alt: Travis


.. image:: https://coveralls.io/repos/github/CyberZHG/keras-bert/badge.svg?branch=master
   :target: https://coveralls.io/github/CyberZHG/keras-bert
   :alt: Coverage


Implementation of the paper: `BERT: Pre-training of Deep Bidirectional Transformers for
Language Understanding <https://arxiv.org/pdf/1810.04805.pdf>`_

Install
-------

.. code-block:: bash

   pip install keras-bert

Usage
-----

.. code-block:: python

   from keras_bert import get_base_dict, get_model, gen_batch_inputs


   sentence_pairs = [
       [['all', 'work', 'and', 'no', 'play'], ['makes', 'jack', 'a', 'dull', 'boy']],
       [['from', 'the', 'day', 'forth'], ['my', 'arm', 'changed']],
       [['and', 'a', 'voice', 'echoed'], ['power', 'give', 'me', 'more', 'power']],
   ]
   token_dict = get_base_dict()
   for pairs in sentence_pairs:
       for token in pairs[0] + pairs[1]:
           if token not in token_dict:
               token_dict[token] = len(token_dict)
   token_list = list(token_dict.keys())
   model = get_model(
       token_num=len(token_dict),
       head_num=5,
       transformer_num=12,
       embed_dim=25,
       feed_forward_dim=100,
       seq_len=20,
       pos_num=20,
       dropout=0.05,
   )
   model.summary()

   def _generator():
       while True:
           yield gen_batch_inputs(
               sentence_pairs,
               token_dict,
               token_list,
               seq_len=20,
               mask_rate=0.3,
               swap_sentence_rate=1.0,
           )

   model.fit_generator(
       generator=_generator(),
       steps_per_epoch=1000,
       epochs=100,
       validation_data=_generator(),
       validation_steps=100,
       callbacks=[
           keras.callbacks.EarlyStopping(monitor='val_loss', patience=5)
       ],
   )
