tensorflow-gpuWindows 10 PCに1.4をインストールしました。チェックポイント を使用してアライグマ検出器をトレーニングしようとしました。ssd_mobilenetGoogle Cloud ではこれに成功しましたが、次の出力でローカルでの実行に失敗しました (明らかに、事前トレーニング済みのモデルの読み込みに失敗しています)。
python object_detection/train.py --logtostderr --pipeline_config_path="C://raccoon/training/cfg.config" --train_dir="C://raccoon/training" > out.txt
Instructions for updating:
WARNING:tensorflow:From C:\Users\orona\Dropbox\projects\models\research\object_detection\trainer.py:210: create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Please switch to tf.train.create_global_step
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:Summary name /clone_loss is illegal; using clone_loss instead.
2018-01-22 17:53:34.358048: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2018-01-22 17:53:34.685205: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:01:00.0
totalMemory: 11.00GiB freeMemory: 9.10GiB
2018-01-22 17:53:34.685312: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from C://raccoon/training\model.ckpt
2018-01-22 17:53:41.215719: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/weights/ExponentialMovingAverage not found in checkpoint
2018-01-22 17:53:41.216292: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/weights/RMSProp_1 not found in checkpoint
2018-01-22 17:53:41.217017: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/weights/RMSProp not found in checkpoint
2018-01-22 17:53:41.217942: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/weights/RMSProp_1 not found in checkpoint
2018-01-22 17:53:41.219617: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/weights/RMSProp not found in checkpoint
2018-01-22 17:53:41.220835: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_4_3x3_s2_256/BatchNorm/beta/ExponentialMovingAverage not found in checkpoint
2018-01-22 17:53:41.221508: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_4_3x3_s2_256/BatchNorm/beta/RMSProp not found in checkpoint
2018-01-22 17:53:41.223914: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_9_pointwise/weights/ExponentialMovingAverage not found in checkpoint
2018-01-22 17:53:41.223936: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_4_3x3_s2_256/BatchNorm/beta/RMSProp_1 not found in checkpoint
2018-01-22 17:53:41.225563: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_4_3x3_s2_256/BatchNorm/gamma/ExponentialMovingAverage not found in checkpoint
2018-01-22 17:53:41.229052: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_4_3x3_s2_256/BatchNorm/gamma/RMSProp not found in checkpoint
2018-01-22 17:53:41.229776: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_4_3x3_s2_256/BatchNorm/gamma/RMSProp_1 not found in checkpoint
2018-01-22 17:53:41.234741: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_4_3x3_s2_256/weights/ExponentialMovingAverage not found in checkpoint
.
.
.
.
2018-01-22 17:53:41.726224: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Not found: Key FeatureExtractor/MobilenetV1/Conv2d_4_pointwise/BatchNorm/gamma/RMSProp_1 not found in checkpoint
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.NotFoundError'>, Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/weights/ExponentialMovingAverage not found in checkpoint
[[Node: save_1/RestoreV2_303 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save_1/Const_0_0, save_1/RestoreV2_303/tensor_names, save_1/RestoreV2_303/shape_and_slices)]]
Caused by op 'save_1/RestoreV2_303', defined at:
File "object_detection/train.py", line 164, in <module>
tf.app.run()
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "object_detection/train.py", line 160, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "C:\Users\orona\Dropbox\projects\models\research\object_detection\trainer.py", line 317, in train
keep_checkpoint_every_n_hours=keep_checkpoint_every_n_hours)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1218, in __init__
self.build()
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1227, in build
self._build(self._filename, build_save=True, build_restore=True)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1263, in _build
build_save=build_save, build_restore=build_restore)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 751, in _build_internal
restore_sequentially, reshape)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 427, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 267, in restore_op
[spec.tensor.dtype])[0])
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 1020, in restore_v2
shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2956, in create_op
op_def=op_def)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1470, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
NotFoundError (see above for traceback): Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/weights/ExponentialMovingAverage not found in checkpoint
[[Node: save_1/RestoreV2_303 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save_1/Const_0_0, save_1/RestoreV2_303/tensor_names, save_1/RestoreV2_303/shape_and_slices)]]
Traceback (most recent call last):
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1323, in _do_call
return fn(*args)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1302, in _run_fn
status, run_metadata)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 473, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/weights/ExponentialMovingAverage not found in checkpoint
[[Node: save_1/RestoreV2_303 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save_1/Const_0_0, save_1/RestoreV2_303/tensor_names, save_1/RestoreV2_303/shape_and_slices)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "object_detection/train.py", line 164, in <module>
tf.app.run()
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "object_detection/train.py", line 160, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "C:\Users\orona\Dropbox\projects\models\research\object_detection\trainer.py", line 332, in train
saver=saver)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 742, in train
master, start_standard_services=False, config=session_config) as sess:
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\contextlib.py", line 81, in __enter__
return next(self.gen)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\supervisor.py", line 964, in managed_session
self.stop(close_summary_writer=close_summary_writer)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\supervisor.py", line 792, in stop
stop_grace_period_secs=self._stop_grace_secs)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\six.py", line 693, in reraise
raise value
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\supervisor.py", line 953, in managed_session
start_standard_services=start_standard_services)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\supervisor.py", line 708, in prepare_or_wait_for_session
init_feed_dict=self._init_feed_dict, init_fn=self._init_fn)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\session_manager.py", line 273, in prepare_session
config=config)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\session_manager.py", line 205, in _restore_checkpoint
saver.restore(sess, ckpt.model_checkpoint_path)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1666, in restore
{self.saver_def.filename_tensor_name: save_path})
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 889, in run
run_metadata_ptr)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1317, in _do_run
options, run_metadata)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/weights/ExponentialMovingAverage not found in checkpoint
[[Node: save_1/RestoreV2_303 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save_1/Const_0_0, save_1/RestoreV2_303/tensor_names, save_1/RestoreV2_303/shape_and_slices)]]
Caused by op 'save_1/RestoreV2_303', defined at:
File "object_detection/train.py", line 164, in <module>
tf.app.run()
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "object_detection/train.py", line 160, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "C:\Users\orona\Dropbox\projects\models\research\object_detection\trainer.py", line 317, in train
keep_checkpoint_every_n_hours=keep_checkpoint_every_n_hours)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1218, in __init__
self.build()
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1227, in build
self._build(self._filename, build_save=True, build_restore=True)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1263, in _build
build_save=build_save, build_restore=build_restore)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 751, in _build_internal
restore_sequentially, reshape)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 427, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 267, in restore_op
[spec.tensor.dtype])[0])
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 1020, in restore_v2
shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2956, in create_op
op_def=op_def)
File "C:\Users\orona\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1470, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
NotFoundError (see above for traceback): Key FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_2_Conv2d_3_3x3_s2_256/weights/ExponentialMovingAverage not found in checkpoint
[[Node: save_1/RestoreV2_303 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save_1/Const_0_0, save_1/RestoreV2_303/tensor_names, save_1/RestoreV2_303/shape_and_slices)]]