训练过程报错:
来源:7-17 LSTM单元内部结构实现
DATUONIAO
2020-05-17
问题描述:学习完本节课程后,我想在Tensorflow 2.0中手动实现一个rnn cell
rnn 层,在训练的过程中出现错误,WARNING:tensorflow:Gradients do not exist for variables
希望知道,这个错误是怎么造成的
class customizeRnn(keras.layers.Layer):
def __init__(self,rnn_units,**kwargs):
self.units = rnn_units
self.dense_layer = keras.layers.Dense(1,activation="softmax")
super().__init__(**kwargs)
def build(self, input_shape):
self.batch_size = input_shape[0]
self.sentense_len = input_shape[1]
self.embed_dim = input_shape[2]
self.S = self.add_weight("status",shape=(self.batch_size,self.units),trainable=False,
initializer=tf.keras.initializers.GlorotUniform(),
dtype=tf.float32)
self.B = self.add_weight("baise",shape=(self.batch_size,1),trainable=False,
initializer=tf.keras.initializers.zeros(),
dtype=tf.float32)
self.U = self.add_weight("weight_u",shape=(self.embed_dim,self.units),
initializer=keras.initializers.GlorotUniform(),
dtype=tf.float32
)
self.W = self.add_weight("weight_w",shape=(self.units,self.units),
initializer=keras.initializers.GlorotUniform(),
dtype=tf.float32
)
self.V = self.add_weight("weight_v",shape=(self.units,self.units),
initializer=keras.initializers.GlorotUniform(),
dtype=tf.float32
)
return super(customizeRnn,self).build(input_shape)
def rnn_node(self,inputs):
for i in range(self.sentense_len):
vocabs = inputs[:,i,:] #(128,1,16)
vocabs = tf.reshape(vocabs,shape=(self.batch_size,self.embed_dim))#(128,16)
self.S.assign(tf.nn.tanh(self.S@self.W+vocabs@self.U))
out_puts =tf.nn.softmax(self.S@self.V+self.B)
return out_puts
def call(self,inputs):
out_put = self.rnn_node(inputs)
return out_put
model:
class cnn_category(keras.models.Model):
def __init__(self,vocabsize,embed_dim):
super(cnn_category,self).__init__()
self.embed_layer = keras.layers.Embedding(vocabsize,embed_dim)
#self.lstm_layer = keras.layers.Bidirectional(keras.layers.LSTM(32,return_sequences=False))
self.lstm_layer = customizeRnn(32)
self.dense_layer = keras.layers.Dense(10,activation="softmax")
def call(self, inputs, training=None, mask=None):
input_embed = self.embed_layer(inputs)
ls_fature = self.lstm_layer(input_embed)
return self.dense_layer(ls_fature)
报错:
Epoch 1/100
WARNING:tensorflow:Gradients do not exist for variables ['embedding_10/embeddings:0', 'customize_rnn_10/weight_u:0', 'customize_rnn_10/weight_w:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['embedding_10/embeddings:0', 'customize_rnn_10/weight_u:0', 'customize_rnn_10/weight_w:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['embedding_10/embeddings:0', 'customize_rnn_10/weight_u:0', 'customize_rnn_10/weight_w:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['embedding_10/embeddings:0', 'customize_rnn_10/weight_u:0', 'customize_rnn_10/weight_w:0'] when minimizing the loss.
390/390 [==============================] - 41s 105ms/step - loss: 2.3501 - accuracy: 0.0918 - val_loss: 2.3028 - val_accuracy: 0.1002
Epoch 2/100
390/390 [==============================] - 29s 75ms/step - loss: 2.3291 - accuracy: 0.0837 - val_loss: 2.3016 - val_accuracy: 0.1109
Epoch 3/100
390/390 [==============================] - 30s 76ms/step - loss: 2.3272 - accuracy: 0.0786 - val_loss: 2.3010 - val_accuracy: 0.1074
Epoch 4/100
390/390 [==============================] - 29s 75ms/step - loss: 2.3265 - accuracy: 0.0634 - val_loss: 2.2998 - val_accuracy: 0.1144
Epoch 5/100
390/390 [==============================] - 30s 77ms/step - loss: 2.3243 - accuracy: 0.0976 - val_loss: 2.2986 - val_accuracy: 0.1109
Epoch 6/100
390/390 [==============================] - 30s 77ms/step - loss: 2.3214 - accuracy: 0.0857 - val_loss: 2.2964 - val_accuracy: 0.1256
Epoch 7/100
390/390 [==============================] - 29s 75ms/step - loss: 2.3187 - accuracy: 0.0885 - val_loss: 2.2936 - val_accuracy: 0.1244
Epoch 8/100
390/390 [==============================] - 30s 76ms/step - loss: 2.3175 - accuracy: 0.0978 - val_loss: 2.2913 - val_accuracy: 0.1383
Epoch 9/100
390/390 [==============================] - 30s 76ms/step - loss: 2.3152 - accuracy: 0.1064 - val_loss: 2.2867 - val_accuracy: 0.1363
Epoch 10/100
390/390 [==============================] - ETA: 0s - loss: 2.3117 - accuracy: 0.0940
经过了10步训练以后,损失没有下降的意思
写回答
1回答
-
是不是因为你的S的trainable是false?
BTW: B的trainable也不应该是false吧。
012020-06-05
相似问题