训练过程报错:

来源:7-17 LSTM单元内部结构实现

DATUONIAO

2020-05-17

问题描述:学习完本节课程后,我想在Tensorflow 2.0中手动实现一个rnn cell
rnn 层,在训练的过程中出现错误,WARNING:tensorflow:Gradients do not exist for variables
希望知道,这个错误是怎么造成的

class customizeRnn(keras.layers.Layer):
    def __init__(self,rnn_units,**kwargs):
        self.units = rnn_units
        self.dense_layer = keras.layers.Dense(1,activation="softmax")
        super().__init__(**kwargs)
    def build(self, input_shape):
        self.batch_size = input_shape[0]
        self.sentense_len = input_shape[1]
        self.embed_dim = input_shape[2]
        self.S = self.add_weight("status",shape=(self.batch_size,self.units),trainable=False,
        initializer=tf.keras.initializers.GlorotUniform(),
        dtype=tf.float32)
        self.B = self.add_weight("baise",shape=(self.batch_size,1),trainable=False,
        initializer=tf.keras.initializers.zeros(),
        dtype=tf.float32)
        self.U = self.add_weight("weight_u",shape=(self.embed_dim,self.units),
        initializer=keras.initializers.GlorotUniform(),
        dtype=tf.float32
        )
        self.W = self.add_weight("weight_w",shape=(self.units,self.units),
        initializer=keras.initializers.GlorotUniform(),
        dtype=tf.float32
        )
        self.V = self.add_weight("weight_v",shape=(self.units,self.units),
        initializer=keras.initializers.GlorotUniform(),
        dtype=tf.float32
        )
        
        return super(customizeRnn,self).build(input_shape)
    
    
    def rnn_node(self,inputs):
        for i in range(self.sentense_len):
            vocabs = inputs[:,i,:] #(128,1,16)
            vocabs = tf.reshape(vocabs,shape=(self.batch_size,self.embed_dim))#(128,16)
            self.S.assign(tf.nn.tanh(self.S@self.W+vocabs@self.U))
            out_puts =tf.nn.softmax(self.S@self.V+self.B)
        return out_puts
    def call(self,inputs):
        out_put = self.rnn_node(inputs)
        return out_put

model:

class cnn_category(keras.models.Model):
    def __init__(self,vocabsize,embed_dim):
        super(cnn_category,self).__init__()
        self.embed_layer = keras.layers.Embedding(vocabsize,embed_dim)
        #self.lstm_layer = keras.layers.Bidirectional(keras.layers.LSTM(32,return_sequences=False))
        self.lstm_layer = customizeRnn(32)
        self.dense_layer = keras.layers.Dense(10,activation="softmax")
    def call(self, inputs, training=None, mask=None):
        input_embed = self.embed_layer(inputs)
        ls_fature = self.lstm_layer(input_embed)
        return self.dense_layer(ls_fature)

报错:

Epoch 1/100
WARNING:tensorflow:Gradients do not exist for variables ['embedding_10/embeddings:0', 'customize_rnn_10/weight_u:0', 'customize_rnn_10/weight_w:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['embedding_10/embeddings:0', 'customize_rnn_10/weight_u:0', 'customize_rnn_10/weight_w:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['embedding_10/embeddings:0', 'customize_rnn_10/weight_u:0', 'customize_rnn_10/weight_w:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['embedding_10/embeddings:0', 'customize_rnn_10/weight_u:0', 'customize_rnn_10/weight_w:0'] when minimizing the loss.
390/390 [==============================] - 41s 105ms/step - loss: 2.3501 - accuracy: 0.0918 - val_loss: 2.3028 - val_accuracy: 0.1002
Epoch 2/100
390/390 [==============================] - 29s 75ms/step - loss: 2.3291 - accuracy: 0.0837 - val_loss: 2.3016 - val_accuracy: 0.1109
Epoch 3/100
390/390 [==============================] - 30s 76ms/step - loss: 2.3272 - accuracy: 0.0786 - val_loss: 2.3010 - val_accuracy: 0.1074
Epoch 4/100
390/390 [==============================] - 29s 75ms/step - loss: 2.3265 - accuracy: 0.0634 - val_loss: 2.2998 - val_accuracy: 0.1144
Epoch 5/100
390/390 [==============================] - 30s 77ms/step - loss: 2.3243 - accuracy: 0.0976 - val_loss: 2.2986 - val_accuracy: 0.1109
Epoch 6/100
390/390 [==============================] - 30s 77ms/step - loss: 2.3214 - accuracy: 0.0857 - val_loss: 2.2964 - val_accuracy: 0.1256
Epoch 7/100
390/390 [==============================] - 29s 75ms/step - loss: 2.3187 - accuracy: 0.0885 - val_loss: 2.2936 - val_accuracy: 0.1244
Epoch 8/100
390/390 [==============================] - 30s 76ms/step - loss: 2.3175 - accuracy: 0.0978 - val_loss: 2.2913 - val_accuracy: 0.1383
Epoch 9/100
390/390 [==============================] - 30s 76ms/step - loss: 2.3152 - accuracy: 0.1064 - val_loss: 2.2867 - val_accuracy: 0.1363
Epoch 10/100
390/390 [==============================] - ETA: 0s - loss: 2.3117 - accuracy: 0.0940

经过了10步训练以后,损失没有下降的意思

写回答

1回答

正十七

2020-05-24

是不是因为你的S的trainable是false?

BTW: B的trainable也不应该是false吧。

0
1
DATUONIAO
非常感谢!
2020-06-05
共1条回复

深度学习之神经网络(CNN/RNN/GAN)算法原理+实战

深度学习算法工程师必学,深入理解深度学习核心算法CNN RNN GAN

2617 学习 · 935 问题

查看课程