An Application of Generative Adversarial Networks (GAN)
If you are planning to start your own religion the first thigh you would need is a holy book! Now, writing a Holy book is from scratch is a sedulous task. To end this problem you can train the GANs model and it will do the job for you.
Basically, What I am trying to imply is that you train a GANs model to learn from any text and generate similar txt out of it.
Here in this example, we are going to use the Holly Bible as training data, which is in plain text format and can be downloaded from given link. (https://raw.githubusercontent.com/mxw/grmr/master/src/finaltests/bible.txt)
Here is the output(with start string=’A’) after training a single layer GRU for 20 epochs with the default settings below:
And the third part of the law of the Lord Jesus Christ, the God of Israel, and the son of Joseph, and the son of Joseph were scattered and said, The first called the things which are in the first day, and the son of man, and the heavens and of the earth, and the sons of Joseph was a company of the Lord Jesus Christ, and the son of Joseph said unto him, Lord, what thou hast seen in the LORD our God, and the son of man is not afraid.
1:15 And the son of man, and the son of Joseph, which is in the faith of the Lord Jesus Christ, and the son of Joseph said unto him, Lord, what thou hast seen in the LORD our God, and the son of man is not afraid.
1:15 And the son of man, and the son of Joseph, which is in the faith of the Lord Jesus Christ, and the son of Joseph said unto him, Lord, what thou hast seen in the LORD our God, and the son of man is not afraid.
1:15 And the son of man, and the son of Joseph, which is in the faith of the Lord Jesus Christ, and the son of Joseph said unto him, L
Of course, some of the sentences do not make sense. But, considering they appear to very similar to the Bible itself.
Download File 2 (Codes)
Unicode library is required
We need to convert Unicode to ASCII.
In [0]: !pip install unidecode
Import tensorflow and other libraries.
In [0]: # Import TensorFlow import tensorflow as tf
# Enable eager execution tf.enable_eager_execution() import numpy as np import os import re import random import unidecode import time
Download the dataset
In this example, As already mentioned we will use The Holly Bible.
In [0]:
path_to_file = tf.keras.utils.get_file('bible.txt', 'https://raw.githubusercontent.com/mxw/grmr/master/src/finaltests/bible.txt')
Read the dataset
In [0]:
text = unidecode.unidecode(open(path_to_file).read())
# length of text is the number of characters in it
print (len(text))
Create dictionaries to map from characters to their indices.
In [0]:
# unique contains all the unique characters in the file
unique = sorted(set(text))
# creating a mapping from unique characters to indices
char2idx = {u:i for i, u in enumerate(unique)}
idx2char = {i:u for i, u in enumerate(unique)}
In [0]:
# On average there are 60 characters per line in our text file. set the maximum length sentence we want for a single input in characters
max_length = 60
# length of the vocabulary in chars
vocab_size = len(unique)
# the embedding dimension
embedding_dim = 256
# number of RNN (here GRU) units
units = 1024
# batch size
BATCH_SIZE = 64
# buffer size to shuffle our dataset
BUFFER_SIZE = 10000
Creating the input and output tensors
In [0]: input_text = [] target_text = [] for f in range(0, len(text)-max_length, max_length): inps = text[f:f+max_length] targ = text[f+1:f+1+max_length] input_text.append([char2idx[i] for i in inps]) target_text.append([char2idx[t] for t in targ]) print (np.array(input_text).shape) print (np.array(target_text).shape)
Creating batches and shuffling them using tf.data
In [0]: dataset = tf.data.Dataset.from_tensor_slices((input_text, target_text)).shuffle(BUFFER_SIZE) dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)
Creating the model
In [0]: class Model(tf.keras.Model): def __init__(self, vocab_size, embedding_dim, units, batch_size): super(Model, self).__init__() self.units = units self.batch_sz = batch_size self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim) if tf.test.is_gpu_available(): self.gru = tf.keras.layers.CuDNNGRU(self.units, return_sequences=True, return_state=True, recurrent_initializer='glorot_uniform') else: self.gru = tf.keras.layers.GRU(self.units, return_sequences=True, return_state=True, recurrent_activation='sigmoid', recurrent_initializer='glorot_uniform') self.fc = tf.keras.layers.Dense(vocab_size) def call(self, x, hidden): x = self.embedding(x) # output shape == (batch_size, max_length, hidden_size) # states shape == (batch_size, hidden_size) # states variable to preserve the state of the model # this will be used to pass at every step to the model while training output, states = self.gru(x, initial_state=hidden) # reshaping the output so that we can pass it to the Dense layer # after reshaping the shape is (batch_size * max_length, hidden_size) output = tf.reshape(output, (-1, output.shape[2])) # The dense layer will output predictions for every time_steps(max_length) # output shape after the dense layer == (max_length * batch_size, vocab_size) x = self.fc(output) return x, states
Call the model and set the optimizer and the loss function
In [0]: model = Model(vocab_size, embedding_dim, units, BATCH_SIZE) In [0]: optimizer = tf.train.AdamOptimizer() # using sparse_softmax_cross_entropy so that we don't have to create one-hot vectors def loss_function(real, preds): return tf.losses.sparse_softmax_cross_entropy(labels=real, logits=preds) Checkpoints (Object-based saving)
In [0]: checkpoint_dir = './training_checkpoints' checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt") checkpoint = tf.train.Checkpoint(optimizer=optimizer, model=model)
Train the model
This most time taking the task, and dependents upon the number of epochs for which you want to train your model. For this example, we will set epochs to only 20. For each epoch, it took about 100 seconds for me.
In [0]:
# Training step
EPOCHS = 20
for epoch in range(EPOCHS):
start = time.time()
# initializing the hidden state at the start of every epoch
hidden = model.reset_states()
for (batch, (inp, target)) in enumerate(dataset):
with tf.GradientTape() as tape:
# feeding the hidden state back into the model
# This is the interesting step
predictions, hidden = model(inp, hidden)
# reshaping the target because that's how the
# loss function expects it
target = tf.reshape(target, (-1,))
loss = loss_function(target, predictions)
grads = tape.gradient(loss, model.variables)
optimizer.apply_gradients(zip(grads, model.variables))
if batch % 100 == 0:
print ('Epoch {} Batch {} Loss {:.4f}'.format(epoch+1,
batch,
loss))
# saving (checkpoint) the model every 5 epochs
if (epoch + 1) % 5 == 0:
checkpoint.save(file_prefix = checkpoint_prefix)
print ('Epoch {} Loss {:.4f}'.format(epoch+1, loss))
print('Time taken for 1 epoch {} sec\n'.format(time.time() - start))
Restore the latest checkpoint
In [0]: # restoring the latest checkpoint in checkpoint_dir checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))
Predicting using our trained model
The below code block is used to generated the text In [0]: # Evaluation step(generating text using the model learned) # number of characters to generate num_generate = 1000 # You can change the start string to experiment start_string = 'A' # converting our start string to numbers(vectorizing!) input_eval = [char2idx[s] for s in start_string] input_eval = tf.expand_dims(input_eval, 0) # empty string to store our results text_generated = '' # low temperatures results in more predictable text. # higher temperatures results in more surprising text # experiment to find the best setting temperature = 1.0 # hidden state shape == (batch_size, number of rnn units); here batch size == 1 hidden = [tf.zeros((1, units))] for i in range(num_generate): predictions, hidden = model(input_eval, hidden) # using a multinomial distribution to predict the word returned by the model predictions = predictions / temperature predicted_id = tf.argmax(predictions[0]).numpy() # We pass the predicted word as the next input to the model # along with the previous hidden state input_eval = tf.expand_dims([predicted_id], 0) text_generated += idx2char[predicted_id] print (start_string + text_generated)
Download File 2 (Codes)
Note: This is a guest post, and opinion in this article is of the guest writer. If you have any issues with any of the articles posted at www.marktechpost.com please contact at asif@marktechpost.com
I am Nilesh Kumar, a graduate student at the Department of Biology, UAB under the mentorship of Dr. Shahid Mukhtar. I joined UAB in Spring 2018 and working on Network Biology. My research interests are Network modeling, Mathematical modeling, Game theory, Artificial Intelligence and their application in Systems Biology.
I graduated with master’s degree “Master of Technology, Information Technology (Specialization in Bioinformatics)” in 2015 from Indian Institute of Information Technology Allahabad, India with GATE scholarship. My Master’s thesis was entitled “Mirtron Prediction through machine learning approach”. I worked as a research fellow at The International Centre for Genetic Engineering and Biotechnology, New Delhi for two years.
