バッチnormaliztion - tf.contrib.layers.batch_normが訓練に良い動作しますが、貧弱なテスト/検証は私がMnistデータセットの機能tf.contrib.layers.batch_normでCNNを実装しようと

Question

結果。

私は損失が減少している（良い！）が、テストデータセットの精度がランダムに残っていることがわかりモデル（〜10％）を訓練し、確認します（BAD !!!）

私が使用している場合バッチの正規化をしていない同じモデルでは、テストの精度が期待どおりに向上しています。

以下のコードでは、バッチ正規化機能の使用方法を確認できます。 テストデータセットで使用する場合、is_training = True良い結果が得られます。問題はis_training = Falseバッチ正規化機能のモードです。

私にこれを手伝ってください。事前にすべての人に感謝します。

# BLOCK2 - Layer 1 
    conv1 = tf.nn.conv2d(output, block2_layer1_1_weights, [1, 1, 1, 1], padding='SAME') 
    conv2 = tf.nn.conv2d(output, block2_layer1_2_weights, [1, 1, 1, 1], padding='SAME') 
    conv3 = tf.nn.conv2d(output, block2_layer1_3_weights, [1, 1, 1, 1], padding='SAME') 
    conv4 = tf.nn.conv2d(output, block2_layer1_4_weights, [1, 1, 1, 1], padding='SAME') 

    conv_normed1 = tf.contrib.layers.batch_norm(conv1, scale=True, decay=batch_norm_decay, center=True, is_training=is_training, updates_collections=None) 
    conv_normed2 = tf.contrib.layers.batch_norm(conv2, scale=True, decay=batch_norm_decay, center=True, is_training=is_training, updates_collections=None) 
    conv_normed3 = tf.contrib.layers.batch_norm(conv3, scale=True, decay=batch_norm_decay, center=True, is_training=is_training, updates_collections=None) 
    conv_normed4 = tf.contrib.layers.batch_norm(conv4, scale=True, decay=batch_norm_decay, center=True, is_training=is_training, updates_collections=None) 

    after_stack = tf.stack([conv_normed1, conv_normed2, conv_normed3, conv_normed4]) 

    after_maxout = tf.reduce_max(after_stack, 0) 
    # BLOCK2 - Layer 2 
    conv1 = tf.nn.conv2d(after_maxout, block2_layer2_1_weights, [1, 1, 1, 1], padding='SAME') 
    conv2 = tf.nn.conv2d(after_maxout, block2_layer2_2_weights, [1, 1, 1, 1], padding='SAME') 
    conv_normed1 = tf.contrib.layers.batch_norm(conv1, scale=True, decay=batch_norm_decay, center=True, is_training=is_training, updates_collections=None) 
    conv_normed2 = tf.contrib.layers.batch_norm(conv2, scale=True, decay=batch_norm_decay, center=True, is_training=is_training, updates_collections=None) 

    after_stack = tf.stack([conv_normed1, conv_normed2]) 

    after_maxout = tf.reduce_max(after_stack, 0) 
    # BLOCK2 - Layer 3 
    conv1 = tf.nn.conv2d(after_maxout, block2_layer3_1_weights, [1, 1, 1, 1], padding='SAME') 
    conv2 = tf.nn.conv2d(after_maxout, block2_layer3_2_weights, [1, 1, 1, 1], padding='SAME') 
    conv_normed1 = tf.contrib.layers.batch_norm(conv1 , scale=True, decay=batch_norm_decay, center=True, is_training=is_training, updates_collections=None) 
    conv_normed2 = tf.contrib.layers.batch_norm(conv2 , scale=True, decay=batch_norm_decay, center=True, is_training=is_training, updates_collections=None) 

    after_stack = tf.stack([conv_normed1, conv_normed2]) 

    after_maxout = tf.reduce_max(after_stack, 0) 
    pooled = tf.nn.max_pool(after_maxout, [1, 3, 3, 1], [1, 3, 3, 1], 'SAME') 
    output = tf.nn.dropout(pooled, 0.5) 




# # Training computation. 
logits = model(tf_train_dataset) 
loss = tf.reduce_mean(
    tf.nn.softmax_cross_entropy_with_logits(labels=tf_train_labels, logits=logits)) 

l2_loss = tf.add_n([tf.nn.l2_loss(v) for v in tf.trainable_variables() if 'BatchNorm' not in v.name]) 
loss += LAMBDA * l2_loss 

# 
# # Optimizer. 



tf.train.GradientDescentOptimizer(LEARNING_RATE).minimize(loss) 

# # Predictions for the training, validation, and test data. 
train_prediction = tf.nn.softmax(logits) 
valid_prediction = tf.nn.softmax(model(tf_valid_dataset)) 
#print(valid_prediction.shape) 
test_prediction = tf.nn.softmax(model(tf_test_dataset)) 

num_steps = 6000 
with tf.Session(graph=graph) as session: 
tf.global_variables_initializer().run() 
print('Initialized') 
for step in range(num_steps): 

    offset = (step * batch_size) % (train_labels.shape[0] - batch_size) 
    test_offset = (step * batch_size) % (test_labels.shape[0] - batch_size) 

    batch_data = train_dataset[offset:(offset + batch_size), :, :, :] 
    batch_labels = train_labels[offset:(offset + batch_size), :] 
    feed_dict = {tf_train_dataset: batch_data, tf_train_labels: batch_labels, is_training: True} 

    _, l, predictions = session.run(
     [optimizer, loss, train_prediction], feed_dict=feed_dict) 


    if (step % 50 == 0): 

     print('Minibatch loss at step %d: %f' % (step, l)) 
     print('Minibatch accuracy: %.1f%%' % accuracy(predictions, batch_labels)) 

     for i in range(1, 10001): 
      test_batch = test_dataset[((i - 1) * test_batch_size):(i * test_batch_size), :, :, :] 
      pred = test_prediction.eval(feed_dict={tf_test_dataset: test_batch, is_training: False}) 


      if i == 1: 
       stacked_pred = pred 
      else: 
       stacked_pred = np.vstack((stacked_pred, pred)) 


     print(np.argmax(stacked_pred,1)) 
     print('test accuracy: %.1f%%' % accuracy(stacked_pred, test_labels))` 

Answer 1

トレーニング中、バッチノルムはバッチに基づく統計を使用します。評価/テスト中（is_trainingがFalseの場合）は、母集団統計が使用されます。

内部的には、集計統計は暗黙的に作成された更新操作によって更新され、tf.GraphKeys.UPDATE_OPSコレクションに追加されますが、これらの操作を実行するにはテンソルフローを強制する必要があります。これを行う簡単な方法は、最適化オペレーションにcontrol_dependenciesを導入することです。

update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS) 
with tf.control_dependencies(update_ops): 
    train_op = optimizer.minimize(loss, step)