"PoolAllocator"に長時間いる

Question

こんにちは私はpoolallocatorについて質問があります。トレーニングを始めると、「PoolAllocator」を実行するのに数時間かかりました。いくつかのログを以下に示します。理由をデバッグ/プロファイルする方法はありますか？どうすれば改善できますか？ ありがとう！

tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 9639 get requests, put_count=4341 evicted_count=1000 eviction_rate=0.230362 and unsatisfied allocation rate=0.663762 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 100 to 110 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2013 evicted_count=2000 eviction_rate=0.993542 and unsatisfied allocation rate=0 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 7080 get requests, put_count=6922 evicted_count=5000 eviction_rate=0.722335 and unsatisfied allocation rate=0.730791 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 176 to 193 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2025 evicted_count=2000 eviction_rate=0.987654 and unsatisfied allocation rate=0 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=5030 evicted_count=5000 eviction_rate=0.994036 and unsatisfied allocation rate=0 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2044 evicted_count=2000 eviction_rate=0.978474 and unsatisfied allocation rate=0 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 9617 get requests, put_count=8892 evicted_count=5000 eviction_rate=0.562303 and unsatisfied allocation rate=0.600915 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 596 to 655 


I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2087 evicted_count=2000 eviction_rate=0.958313 and unsatisfied allocation rate=0 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=3095 evicted_count=3000 eviction_rate=0.969305 and unsatisfied allocation rate=0 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=1115 evicted_count=1000 eviction_rate=0.896861 and unsatisfied allocation rate=0 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=1140 evicted_count=1000 eviction_rate=0.877193 and unsatisfied allocation rate=0 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=1169 evicted_count=1000 eviction_rate=0.855432 and unsatisfied allocation rate=0 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=1204 evicted_count=1000 eviction_rate=0.830565 and unsatisfied allocation rate=0 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2247 evicted_count=2000 eviction_rate=0.890076 and unsatisfied allocation rate=0 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=8272 evicted_count=8000 eviction_rate=0.967118 and unsatisfied allocation rate=0 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=2362 evicted_count=2000 eviction_rate=0.84674 and unsatisfied allocation rate=0 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 10614 get requests, put_count=10944 evicted_count=2000 eviction_rate=0.182749 and unsatisfied allocation rate=0.198606 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:256] Raising pool_size_limit_ from 4823 to 5305 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 0 get requests, put_count=3705 evicted_count=3000 eviction_rate=0.809717 and unsatisfied allocation rate=0 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 4204990 get requests, put_count=4204742 evicted_count=3000 eviction_rate=0.00071348 and unsatisfied allocation rate=0.00104257 
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:244] PoolAllocator: After 16377314 get requests, put_count=16374197 evicted_count=13000 eviction_rate=0.000793932 and unsatisfied allocation rate=0.00105347 

Answer 1

実際には正常に動作しています。問題は、Pythonの出力バッファリングです。仕事が終わるまで、トレーニング結果は表示されません。あなたはでバッファリングを無効にすることができます

sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0) 

か、で他の方法を試すことができます。 Disable output buffering

Answer 2

ジョブが正常に動作しています。これらのメッセージは、メモリが不足している場合にのみ懸念の原因となります。

この質問の回答は役に立ちます。 How to interpret Poolallocator messages in tensorflow?