2021-09-16:17:27:07,185 INFO [iwn.py:43] Loading bengali language synsets... 2021-09-16:17:27:20,93 WARNING [utils.py:429] using automatically assigned random_state=3048715693 2021-09-16:17:27:20,169 INFO [splitting.py:60] done splitting triples to groups of sizes [267005, 38830, 38831] /srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release. from numpy.core.umath_tests import inner1d /srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/gradient_boosting.py:34: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations from ._gradient_boosting import predict_stages /srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/gradient_boosting.py:34: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations from ._gradient_boosting import predict_stages [32m[I 2021-09-16 17:27:20,924][0m A new study created in memory with name: no-name-51080656-f2e2-4ebe-abb4-f172fe38107f[0m 2021-09-16:17:27:20,973 INFO [hpo.py:622] Using model: 2021-09-16:17:27:20,973 INFO [hpo.py:626] Using loss: 2021-09-16:17:27:20,974 INFO [hpo.py:641] Using optimizer: 2021-09-16:17:27:20,974 INFO [hpo.py:645] Using training loop: 2021-09-16:17:27:20,974 INFO [hpo.py:651] Using negative sampler: 2021-09-16:17:27:20,974 INFO [hpo.py:662] Using evaluator: 2021-09-16:17:27:20,974 INFO [hpo.py:666] Attempting to maximize adjusted_arithmetic_mean_rank_index 2021-09-16:17:27:20,974 INFO [hpo.py:668] Filter validation triples when testing: True /srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932]. low=low, old_high=old_high, high=high, step=step 2021-09-16:17:27:20,978 WARNING [api.py:823] No random seed is specified. Setting to 2218594265. Training epochs on cuda: 0%| | 0/10 [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:13<00:00, 14.56s/epoch, loss=0.00134, prev_loss=0.00134] Training epochs on cuda: 100%|██████████| 10/10 [01:13<00:00, 7.32s/epoch, loss=0.00134, prev_loss=0.00134] 2021-09-16:17:28:40,29 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpyjl8c6h3' 2021-09-16:17:28:40,73 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpyjl8c6h3' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 13.91s/epoch, loss=0.000635, prev_loss=0.00064] Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 6.65s/epoch, loss=0.000635, prev_loss=0.00064] 2021-09-16:17:30:21,603 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpptxttqya' 2021-09-16:17:30:21,660 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpptxttqya' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 13.25s/epoch, loss=0.000232, prev_loss=0.000232] Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 5.88s/epoch, loss=0.000232, prev_loss=0.000232] 2021-09-16:17:31:55,423 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpplw8krx1' 2021-09-16:17:31:55,485 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpplw8krx1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 13.19s/epoch, loss=2.37e-5, prev_loss=2.11e-5] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.99s/epoch, loss=2.37e-5, prev_loss=2.11e-5] 2021-09-16:17:33:30,248 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmphbd2i_4l' 2021-09-16:17:33:30,310 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmphbd2i_4l' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 14.67s/epoch, loss=0.00026, prev_loss=0.000265] Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 7.43s/epoch, loss=0.00026, prev_loss=0.000265] 2021-09-16:17:35:19,246 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp55wrbiom' 2021-09-16:17:35:19,305 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp55wrbiom' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:19<00:00, 15.18s/epoch, loss=0.000134, prev_loss=0.000143] Training epochs on cuda: 100%|██████████| 10/10 [01:19<00:00, 7.95s/epoch, loss=0.000134, prev_loss=0.000143] 2021-09-16:17:37:13,812 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpv0odudba' 2021-09-16:17:37:13,872 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpv0odudba' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:48<00:00, 18.13s/epoch, loss=0.00174, prev_loss=0.00175] Training epochs on cuda: 100%|██████████| 10/10 [01:48<00:00, 10.84s/epoch, loss=0.00174, prev_loss=0.00175] 2021-09-16:17:39:38,534 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpeu2is3u1' 2021-09-16:17:39:38,572 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpeu2is3u1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 19.65s/epoch, loss=6.05e-6, prev_loss=5.94e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 12.45s/epoch, loss=6.05e-6, prev_loss=5.94e-6] 2021-09-16:17:42:18,310 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_if_vuq9' 2021-09-16:17:42:18,349 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_if_vuq9' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:55<00:00, 12.65s/epoch, loss=0.000381, prev_loss=0.000381] Training epochs on cuda: 100%|██████████| 10/10 [00:55<00:00, 5.50s/epoch, loss=0.000381, prev_loss=0.000381] 2021-09-16:17:43:48,44 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpstd0773f' 2021-09-16:17:43:48,81 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpstd0773f' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:13<00:00, 14.60s/epoch, loss=7.84e-6, prev_loss=8.66e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:13<00:00, 7.38s/epoch, loss=7.84e-6, prev_loss=8.66e-6] 2021-09-16:17:45:36,913 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpgedjsj7b' 2021-09-16:17:45:36,949 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpgedjsj7b' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:22<00:00, 21.40s/epoch, loss=2.58e-5, prev_loss=2.83e-5] Training epochs on cuda: 100%|██████████| 10/10 [02:22<00:00, 14.25s/epoch, loss=2.58e-5, prev_loss=2.83e-5] 2021-09-16:17:48:34,660 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp7z0vick7' 2021-09-16:17:48:34,702 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp7z0vick7' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:37<00:00, 16.99s/epoch, loss=1.06e-5, prev_loss=1.31e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:37<00:00, 9.78s/epoch, loss=1.06e-5, prev_loss=1.31e-5] 2021-09-16:17:50:47,428 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpawrifpyg' 2021-09-16:17:50:47,465 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpawrifpyg' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:09<00:00, 14.11s/epoch, loss=4.2e-6, prev_loss=4.54e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:09<00:00, 6.91s/epoch, loss=4.2e-6, prev_loss=4.54e-6] 2021-09-16:17:52:31,448 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpy_mz9ql2' 2021-09-16:17:52:31,491 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpy_mz9ql2' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:35<00:00, 16.78s/epoch, loss=6.57e-6, prev_loss=5.43e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:35<00:00, 9.52s/epoch, loss=6.57e-6, prev_loss=5.43e-6] 2021-09-16:17:54:41,609 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpmkkomkd1' 2021-09-16:17:54:41,655 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpmkkomkd1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:02<00:00, 19.46s/epoch, loss=2.35e-5, prev_loss=2.85e-5] Training epochs on cuda: 100%|██████████| 10/10 [02:02<00:00, 12.29s/epoch, loss=2.35e-5, prev_loss=2.85e-5] 2021-09-16:17:57:19,421 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpfrws1kc6' 2021-09-16:17:57:19,473 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfrws1kc6' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:57<00:00, 18.89s/epoch, loss=1.41e-5, prev_loss=1.03e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:57<00:00, 11.77s/epoch, loss=1.41e-5, prev_loss=1.03e-5] 2021-09-16:17:59:52,55 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpw6568593' 2021-09-16:17:59:52,141 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpw6568593' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:24<00:00, 15.78s/epoch, loss=1.82e-5, prev_loss=1.93e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:24<00:00, 8.44s/epoch, loss=1.82e-5, prev_loss=1.93e-5] 2021-09-16:18:01:51,471 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp9re_7rzf' 2021-09-16:18:01:51,511 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp9re_7rzf' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 13.58s/epoch, loss=1.18e-5, prev_loss=1.02e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 6.43s/epoch, loss=1.18e-5, prev_loss=1.02e-5] 2021-09-16:18:03:30,691 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp3pi3f9jy' 2021-09-16:18:03:30,718 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp3pi3f9jy' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:39<00:00, 17.13s/epoch, loss=0.000165, prev_loss=0.000172] Training epochs on cuda: 100%|██████████| 10/10 [01:39<00:00, 9.91s/epoch, loss=0.000165, prev_loss=0.000172] 2021-09-16:18:05:44,568 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpfv49ztd3' 2021-09-16:18:05:44,593 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfv49ztd3' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [09:21<00:00, 63.12s/epoch, loss=0.0317, prev_loss=0.0318] Training epochs on cuda: 100%|██████████| 10/10 [09:21<00:00, 56.16s/epoch, loss=0.0317, prev_loss=0.0318] 2021-09-16:18:15:40,969 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_rizehmv' 2021-09-16:18:15:41,13 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_rizehmv' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:43<00:00, 23.56s/epoch, loss=0.000162, prev_loss=0.00016] Training epochs on cuda: 100%|██████████| 10/10 [02:43<00:00, 16.38s/epoch, loss=0.000162, prev_loss=0.00016] 2021-09-16:18:18:59,951 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpz_s3iefb' 2021-09-16:18:18:59,986 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpz_s3iefb' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00, 14.75s/epoch, loss=5.99e-6, prev_loss=6.13e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00, 7.66s/epoch, loss=5.99e-6, prev_loss=6.13e-6] 2021-09-16:18:20:51,329 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpkfvrp_yj' 2021-09-16:18:20:51,368 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpkfvrp_yj' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:20<00:00, 15.24s/epoch, loss=5.28e-6, prev_loss=5.24e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:20<00:00, 8.06s/epoch, loss=5.28e-6, prev_loss=5.24e-6] 2021-09-16:18:22:46,850 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp7y0zs05l' 2021-09-16:18:22:46,887 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp7y0zs05l' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00, 15.50s/epoch, loss=4.43e-6, prev_loss=4.61e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00, 8.30s/epoch, loss=4.43e-6, prev_loss=4.61e-6] 2021-09-16:18:24:44,850 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpjne4nlt_' 2021-09-16:18:24:44,887 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpjne4nlt_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 13.86s/epoch, loss=4.59e-5, prev_loss=5.67e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 6.66s/epoch, loss=4.59e-5, prev_loss=5.67e-5] 2021-09-16:18:26:26,498 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpciwyueh_' 2021-09-16:18:26:26,537 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpciwyueh_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:25<00:00, 15.72s/epoch, loss=4.35e-6, prev_loss=4.35e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:25<00:00, 8.50s/epoch, loss=4.35e-6, prev_loss=4.35e-6] 2021-09-16:18:28:26,419 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1ixspa_9' 2021-09-16:18:28:26,464 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1ixspa_9' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:55<00:00, 18.78s/epoch, loss=6.74e-6, prev_loss=4.32e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:55<00:00, 11.57s/epoch, loss=6.74e-6, prev_loss=4.32e-6] 2021-09-16:18:30:57,389 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpcwof6l99' 2021-09-16:18:30:57,423 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpcwof6l99' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:39<00:00, 17.08s/epoch, loss=2.8e-6, prev_loss=2.93e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:39<00:00, 9.90s/epoch, loss=2.8e-6, prev_loss=2.93e-6] 2021-09-16:18:33:11,372 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpaze5b2ra' 2021-09-16:18:33:11,405 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpaze5b2ra' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 13.54s/epoch, loss=6.51e-5, prev_loss=6.58e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 6.32s/epoch, loss=6.51e-5, prev_loss=6.58e-5] 2021-09-16:18:34:49,484 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpnbcbdpad' 2021-09-16:18:34:49,534 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpnbcbdpad' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:55<00:00, 12.78s/epoch, loss=7.5e-5, prev_loss=8.18e-5] Training epochs on cuda: 100%|██████████| 10/10 [00:55<00:00, 5.57s/epoch, loss=7.5e-5, prev_loss=8.18e-5] 2021-09-16:18:36:20,49 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpzyxlql82' 2021-09-16:18:36:20,106 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpzyxlql82' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 14.74s/epoch, loss=2.75e-6, prev_loss=2.77e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 7.46s/epoch, loss=2.75e-6, prev_loss=2.77e-6] 2021-09-16:18:38:09,593 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmph_nt2a1h' 2021-09-16:18:38:09,638 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmph_nt2a1h' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:22<00:00, 15.38s/epoch, loss=4.86e-6, prev_loss=4.87e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:22<00:00, 8.23s/epoch, loss=4.86e-6, prev_loss=4.87e-6] 2021-09-16:18:40:07,148 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpji8s1wyp' 2021-09-16:18:40:07,197 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpji8s1wyp' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:27<00:00, 15.97s/epoch, loss=3.48e-5, prev_loss=2.46e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:27<00:00, 8.73s/epoch, loss=3.48e-5, prev_loss=2.46e-5] 2021-09-16:18:42:09,782 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmphsnauj42' 2021-09-16:18:42:09,828 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmphsnauj42' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 13.29s/epoch, loss=1.97e-5, prev_loss=1.83e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.15s/epoch, loss=1.97e-5, prev_loss=1.83e-5] 2021-09-16:18:43:46,294 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpexbkbxf9' 2021-09-16:18:43:46,333 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpexbkbxf9' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 18.48s/epoch, loss=4.14e-6, prev_loss=4.27e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 11.27s/epoch, loss=4.14e-6, prev_loss=4.27e-6] 2021-09-16:18:46:14,44 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpcor9kmeh' 2021-09-16:18:46:14,83 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpcor9kmeh' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:27<00:00, 16.03s/epoch, loss=3.48e-5, prev_loss=4.9e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:27<00:00, 8.79s/epoch, loss=3.48e-5, prev_loss=4.9e-5] 2021-09-16:18:48:16,859 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpq8k8_cq_' 2021-09-16:18:48:16,904 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpq8k8_cq_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 14.71s/epoch, loss=2.05e-5, prev_loss=1.69e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 7.45s/epoch, loss=2.05e-5, prev_loss=1.69e-5] 2021-09-16:18:50:06,317 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp56zt5icf' 2021-09-16:18:50:06,364 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp56zt5icf' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 13.93s/epoch, loss=1.18e-5, prev_loss=1.05e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 6.79s/epoch, loss=1.18e-5, prev_loss=1.05e-5] 2021-09-16:18:51:49,101 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpod5385eq' 2021-09-16:18:51:49,151 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpod5385eq' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:17<00:00, 14.96s/epoch, loss=7.14e-7, prev_loss=4.38e-7] Training epochs on cuda: 100%|██████████| 10/10 [01:17<00:00, 7.77s/epoch, loss=7.14e-7, prev_loss=4.38e-7] 2021-09-16:18:53:41,764 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpt1q_ifsy' 2021-09-16:18:53:41,853 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpt1q_ifsy' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:38<00:00, 17.09s/epoch, loss=7.93e-5, prev_loss=7.87e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:38<00:00, 9.90s/epoch, loss=7.93e-5, prev_loss=7.87e-5] 2021-09-16:18:55:55,740 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpw1tmjuum' 2021-09-16:18:55:55,788 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpw1tmjuum' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:58<00:00, 18.99s/epoch, loss=3.81e-6, prev_loss=3.42e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:58<00:00, 11.82s/epoch, loss=3.81e-6, prev_loss=3.42e-6] 2021-09-16:18:58:29,265 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpte5puo0t' 2021-09-16:18:58:29,311 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpte5puo0t' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 20.44s/epoch, loss=6.67e-6, prev_loss=7.1e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 13.18s/epoch, loss=6.67e-6, prev_loss=7.1e-6] 2021-09-16:19:01:16,490 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpz_wbzc8w' 2021-09-16:19:01:16,538 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpz_wbzc8w' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 19.43s/epoch, loss=4e-6, prev_loss=3.8e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 12.19s/epoch, loss=4e-6, prev_loss=3.8e-6] 2021-09-16:19:03:53,995 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp8wd8i350' 2021-09-16:19:03:54,59 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp8wd8i350' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 19.39s/epoch, loss=9.15e-6, prev_loss=6.77e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 12.17s/epoch, loss=9.15e-6, prev_loss=6.77e-6] 2021-09-16:19:06:31,130 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpxp5g28rs' 2021-09-16:19:06:31,205 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpxp5g28rs' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:43<00:00, 17.63s/epoch, loss=5.76e-6, prev_loss=4.99e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:43<00:00, 10.40s/epoch, loss=5.76e-6, prev_loss=4.99e-6] 2021-09-16:19:08:50,444 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpgi27q8uq' 2021-09-16:19:08:50,507 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpgi27q8uq' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 19.72s/epoch, loss=5.45e-5, prev_loss=6.08e-5] Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 12.48s/epoch, loss=5.45e-5, prev_loss=6.08e-5] 2021-09-16:19:11:30,359 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_z35v4sv' 2021-09-16:19:11:30,408 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_z35v4sv' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:35<00:00, 16.80s/epoch, loss=1.31e-5, prev_loss=1.36e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:35<00:00, 9.55s/epoch, loss=1.31e-5, prev_loss=1.36e-5] 2021-09-16:19:13:41,12 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpppyzfvrr' 2021-09-16:19:13:41,78 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpppyzfvrr' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 19.88s/epoch, loss=1.29e-5, prev_loss=1.42e-5] Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 12.79s/epoch, loss=1.29e-5, prev_loss=1.42e-5] 2021-09-16:19:16:24,259 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmps32v7kbp' 2021-09-16:19:16:24,311 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmps32v7kbp' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 18.44s/epoch, loss=0.00135, prev_loss=0.00137] Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 11.21s/epoch, loss=0.00135, prev_loss=0.00137] 2021-09-16:19:18:51,438 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpxw00e1up' 2021-09-16:19:18:51,499 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpxw00e1up' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:29<00:00, 16.18s/epoch, loss=0.000433, prev_loss=0.00044] Training epochs on cuda: 100%|██████████| 10/10 [01:29<00:00, 8.92s/epoch, loss=0.000433, prev_loss=0.00044] 2021-09-16:19:20:55,632 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp2953zx7u' 2021-09-16:19:20:55,672 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp2953zx7u' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:58<00:00, 19.23s/epoch, loss=2.39e-5, prev_loss=2.87e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:58<00:00, 11.86s/epoch, loss=2.39e-5, prev_loss=2.87e-5] 2021-09-16:19:23:29,243 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpqm8h91h2' 2021-09-16:19:23:29,280 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpqm8h91h2' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:37<00:00, 16.97s/epoch, loss=1.59e-5, prev_loss=1.25e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:37<00:00, 9.79s/epoch, loss=1.59e-5, prev_loss=1.25e-5] 2021-09-16:19:25:42,403 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpcs_fvaik' 2021-09-16:19:25:42,444 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpcs_fvaik' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:38<00:00, 17.13s/epoch, loss=8.07e-6, prev_loss=6.84e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:38<00:00, 9.86s/epoch, loss=8.07e-6, prev_loss=6.84e-6] 2021-09-16:19:27:56,55 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpvsjn_szv' 2021-09-16:19:27:56,112 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpvsjn_szv' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 21.00s/epoch, loss=4.04e-6, prev_loss=4.29e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 13.76s/epoch, loss=4.04e-6, prev_loss=4.29e-6] 2021-09-16:19:30:48,850 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpj5h9jh68' 2021-09-16:19:30:48,919 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpj5h9jh68' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:41<00:00, 17.39s/epoch, loss=1.17e-5, prev_loss=2.36e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:41<00:00, 10.16s/epoch, loss=1.17e-5, prev_loss=2.36e-5] 2021-09-16:19:33:05,651 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpn08z4zf_' 2021-09-16:19:33:05,687 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpn08z4zf_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:55<00:00, 18.70s/epoch, loss=4.22e-6, prev_loss=4.55e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:55<00:00, 11.50s/epoch, loss=4.22e-6, prev_loss=4.55e-6] 2021-09-16:19:35:36,147 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpf89tdvq5' 2021-09-16:19:35:36,208 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpf89tdvq5' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:50<00:00, 18.42s/epoch, loss=0.000573, prev_loss=6.21e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:50<00:00, 11.09s/epoch, loss=0.000573, prev_loss=6.21e-5] 2021-09-16:19:38:02,75 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpwel6kmlr' 2021-09-16:19:38:02,162 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpwel6kmlr' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:45<00:00, 17.79s/epoch, loss=0.00119, prev_loss=0.00107] Training epochs on cuda: 100%|██████████| 10/10 [01:45<00:00, 10.56s/epoch, loss=0.00119, prev_loss=0.00107] 2021-09-16:19:40:22,820 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpxng7z7_0' 2021-09-16:19:40:22,883 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpxng7z7_0' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 14.52s/epoch, loss=0.000243, prev_loss=0.000234] Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 7.15s/epoch, loss=0.000243, prev_loss=0.000234] 2021-09-16:19:42:09,481 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpcyxhn9xf' 2021-09-16:19:42:09,526 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpcyxhn9xf' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:51<00:00, 18.34s/epoch, loss=2.6e-6, prev_loss=2.57e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:51<00:00, 11.12s/epoch, loss=2.6e-6, prev_loss=2.57e-6] 2021-09-16:19:44:35,997 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp9s3vo0fh' 2021-09-16:19:44:36,35 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp9s3vo0fh' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 18.47s/epoch, loss=2.62e-5, prev_loss=2.74e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 11.22s/epoch, loss=2.62e-5, prev_loss=2.74e-5] 2021-09-16:19:47:03,428 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpc5q2dasi' 2021-09-16:19:47:03,475 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpc5q2dasi' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:44<00:00, 17.71s/epoch, loss=2.1e-6, prev_loss=2.14e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:44<00:00, 10.44s/epoch, loss=2.1e-6, prev_loss=2.14e-6] 2021-09-16:19:49:23,297 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpt4dyt6a8' 2021-09-16:19:49:23,364 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpt4dyt6a8' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:46<00:00, 17.94s/epoch, loss=2.99e-6, prev_loss=3.32e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:46<00:00, 10.70s/epoch, loss=2.99e-6, prev_loss=3.32e-6] 2021-09-16:19:51:45,812 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpsib32skr' 2021-09-16:19:51:45,865 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpsib32skr' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:53<00:00, 18.57s/epoch, loss=2.35e-6, prev_loss=2.33e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:53<00:00, 11.34s/epoch, loss=2.35e-6, prev_loss=2.33e-6] 2021-09-16:19:54:14,915 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpwh2c5xqm' 2021-09-16:19:54:14,969 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpwh2c5xqm' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:48<00:00, 18.12s/epoch, loss=2.92e-5, prev_loss=6.66e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:48<00:00, 10.84s/epoch, loss=2.92e-5, prev_loss=6.66e-5] 2021-09-16:19:56:38,523 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_cgxhasw' 2021-09-16:19:56:38,564 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_cgxhasw' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:15<00:00, 20.90s/epoch, loss=4.5e-6, prev_loss=3.44e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:15<00:00, 13.56s/epoch, loss=4.5e-6, prev_loss=3.44e-6] 2021-09-16:19:59:29,984 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp3muwzkn7' 2021-09-16:19:59:30,20 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp3muwzkn7' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:54<00:00, 18.67s/epoch, loss=1.79e-6, prev_loss=1.78e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:54<00:00, 11.49s/epoch, loss=1.79e-6, prev_loss=1.78e-6] 2021-09-16:20:02:00,228 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpqwnuyx3_' 2021-09-16:20:02:00,282 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpqwnuyx3_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:42<00:00, 17.52s/epoch, loss=9.13e-6, prev_loss=8.96e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:42<00:00, 10.27s/epoch, loss=9.13e-6, prev_loss=8.96e-6] 2021-09-16:20:04:18,117 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpxrr33f46' 2021-09-16:20:04:18,155 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpxrr33f46' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:57<00:00, 19.09s/epoch, loss=1.74e-6, prev_loss=1.6e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:57<00:00, 11.77s/epoch, loss=1.74e-6, prev_loss=1.6e-6] 2021-09-16:20:06:51,279 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpryqo15jz' 2021-09-16:20:06:51,314 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpryqo15jz' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:55<00:00, 18.83s/epoch, loss=1.42e-5, prev_loss=1.79e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:55<00:00, 11.55s/epoch, loss=1.42e-5, prev_loss=1.79e-5] 2021-09-16:20:09:22,298 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpfjn2mtba' 2021-09-16:20:09:22,333 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfjn2mtba' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:49<00:00, 18.13s/epoch, loss=3.83e-6, prev_loss=3.36e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:49<00:00, 10.92s/epoch, loss=3.83e-6, prev_loss=3.36e-6] 2021-09-16:20:11:47,29 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpkjmhyhph' 2021-09-16:20:11:47,62 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpkjmhyhph' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:19<00:00, 21.16s/epoch, loss=4.13e-6, prev_loss=3.58e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:19<00:00, 13.93s/epoch, loss=4.13e-6, prev_loss=3.58e-6] 2021-09-16:20:14:41,569 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpkmab6sl1' 2021-09-16:20:14:41,605 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpkmab6sl1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:19<00:00, 21.15s/epoch, loss=2.05e-6, prev_loss=2.09e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:19<00:00, 13.95s/epoch, loss=2.05e-6, prev_loss=2.09e-6] 2021-09-16:20:17:36,785 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpu6tbjoao' 2021-09-16:20:17:36,832 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpu6tbjoao' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:18<00:00, 21.12s/epoch, loss=1.93e-6, prev_loss=1.97e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:18<00:00, 13.89s/epoch, loss=1.93e-6, prev_loss=1.97e-6] 2021-09-16:20:20:31,461 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp40ggy2kj' 2021-09-16:20:20:31,516 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp40ggy2kj' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:25<00:00, 21.74s/epoch, loss=2.11e-6, prev_loss=2.13e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:25<00:00, 14.51s/epoch, loss=2.11e-6, prev_loss=2.13e-6] 2021-09-16:20:23:32,64 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpw2ac0pjn' 2021-09-16:20:23:32,104 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpw2ac0pjn' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:22<00:00, 21.46s/epoch, loss=2.18e-6, prev_loss=2.3e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:22<00:00, 14.22s/epoch, loss=2.18e-6, prev_loss=2.3e-6] 2021-09-16:20:26:29,971 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpg2jrt5vb' 2021-09-16:20:26:30,3 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpg2jrt5vb' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:19<00:00, 21.15s/epoch, loss=1.18e-6, prev_loss=1.29e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:19<00:00, 13.99s/epoch, loss=1.18e-6, prev_loss=1.29e-6] 2021-09-16:20:29:25,383 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpsl9qyrp6' 2021-09-16:20:29:25,436 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpsl9qyrp6' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 20.08s/epoch, loss=6.16e-6, prev_loss=4.56e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 12.78s/epoch, loss=6.16e-6, prev_loss=4.56e-6] 2021-09-16:20:32:08,814 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp8ks3k157' 2021-09-16:20:32:08,879 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp8ks3k157' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:23<00:00, 21.64s/epoch, loss=2.23e-6, prev_loss=2.41e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:23<00:00, 14.38s/epoch, loss=2.23e-6, prev_loss=2.41e-6] 2021-09-16:20:35:08,125 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp8v9jcq6t' 2021-09-16:20:35:08,164 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp8v9jcq6t' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 19.99s/epoch, loss=2.1e-6, prev_loss=2.13e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 12.77s/epoch, loss=2.1e-6, prev_loss=2.13e-6] 2021-09-16:20:37:51,618 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpup5hlzrj' 2021-09-16:20:37:51,672 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpup5hlzrj' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:55<00:00, 12.79s/epoch, loss=1.97e-5, prev_loss=1.84e-5] Training epochs on cuda: 100%|██████████| 10/10 [00:55<00:00, 5.57s/epoch, loss=1.97e-5, prev_loss=1.84e-5] 2021-09-16:20:39:22,415 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp23fojl3q' 2021-09-16:20:39:22,452 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp23fojl3q' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 12.95s/epoch, loss=1.53e-5, prev_loss=1.4e-5] Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 5.67s/epoch, loss=1.53e-5, prev_loss=1.4e-5] 2021-09-16:20:40:54,235 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp4gjzz2ny' 2021-09-16:20:40:54,279 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp4gjzz2ny' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:13<00:00, 20.61s/epoch, loss=4.62e-6, prev_loss=3.58e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:13<00:00, 13.39s/epoch, loss=4.62e-6, prev_loss=3.58e-6] 2021-09-16:20:43:43,665 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpmeplkrqk' 2021-09-16:20:43:43,713 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpmeplkrqk' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:14<00:00, 20.59s/epoch, loss=2.55e-6, prev_loss=2.6e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:14<00:00, 13.41s/epoch, loss=2.55e-6, prev_loss=2.6e-6] 2021-09-16:20:46:33,198 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp8ld04cad' 2021-09-16:20:46:33,232 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp8ld04cad' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 18.94s/epoch, loss=1.32e-6, prev_loss=1.31e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 11.69s/epoch, loss=1.32e-6, prev_loss=1.31e-6] 2021-09-16:20:49:05,522 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp4_fy1g_w' 2021-09-16:20:49:05,556 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp4_fy1g_w' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 18.86s/epoch, loss=2.12e-6, prev_loss=1.96e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 11.64s/epoch, loss=2.12e-6, prev_loss=1.96e-6] 2021-09-16:20:51:37,404 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpr3xh28f9' 2021-09-16:20:51:37,489 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpr3xh28f9' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:40<00:00, 17.21s/epoch, loss=1.58e-6, prev_loss=1.72e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:40<00:00, 10.03s/epoch, loss=1.58e-6, prev_loss=1.72e-6] 2021-09-16:20:53:53,211 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpcm58ek1w' 2021-09-16:20:53:53,255 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpcm58ek1w' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:25<00:00, 21.76s/epoch, loss=1.72e-6, prev_loss=1.61e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:25<00:00, 14.58s/epoch, loss=1.72e-6, prev_loss=1.61e-6] 2021-09-16:20:56:54,351 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpegtwf01u' 2021-09-16:20:56:54,394 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpegtwf01u' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:15<00:00, 20.70s/epoch, loss=2.12e-6, prev_loss=1.75e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:15<00:00, 13.50s/epoch, loss=2.12e-6, prev_loss=1.75e-6] 2021-09-16:20:59:45,41 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpvse62jfk' 2021-09-16:20:59:45,104 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpvse62jfk' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 19.93s/epoch, loss=2.69e-6, prev_loss=2.25e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 12.80s/epoch, loss=2.69e-6, prev_loss=2.25e-6] 2021-09-16:21:02:28,579 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpi3319j7_' 2021-09-16:21:02:28,617 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpi3319j7_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:32<00:00, 22.44s/epoch, loss=4.04e-6, prev_loss=3.12e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:32<00:00, 15.22s/epoch, loss=4.04e-6, prev_loss=3.12e-6] 2021-09-16:21:05:36,214 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpl80tkyky' 2021-09-16:21:05:36,266 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpl80tkyky' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 19.16s/epoch, loss=1.33e-6, prev_loss=1.36e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 11.98s/epoch, loss=1.33e-6, prev_loss=1.36e-6] 2021-09-16:21:08:11,628 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpzsooiqk2' 2021-09-16:21:08:11,664 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpzsooiqk2' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:53<00:00, 18.62s/epoch, loss=1.77e-6, prev_loss=1.68e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:53<00:00, 11.38s/epoch, loss=1.77e-6, prev_loss=1.68e-6] 2021-09-16:21:10:40,918 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpzlzvsisa' 2021-09-16:21:10:40,982 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpzlzvsisa' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 20.31s/epoch, loss=1.43e-6, prev_loss=1.47e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 13.06s/epoch, loss=1.43e-6, prev_loss=1.47e-6] 2021-09-16:21:13:26,835 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp7sva8_0i' 2021-09-16:21:13:26,871 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp7sva8_0i' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:44<00:00, 17.67s/epoch, loss=2.25e-6, prev_loss=2.26e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:44<00:00, 10.49s/epoch, loss=2.25e-6, prev_loss=2.26e-6] 2021-09-16:21:15:47,161 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpki189i78' 2021-09-16:21:15:47,224 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpki189i78' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 19.09s/epoch, loss=2.33e-5, prev_loss=2.31e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 11.90s/epoch, loss=2.33e-5, prev_loss=2.31e-5] 2021-09-16:21:18:21,615 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmptbfuvngy' 2021-09-16:21:18:21,701 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmptbfuvngy' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:30<00:00, 22.34s/epoch, loss=2.83e-6, prev_loss=2.71e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:30<00:00, 15.08s/epoch, loss=2.83e-6, prev_loss=2.71e-6] 2021-09-16:21:21:28,100 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp2hiomaqd' 2021-09-16:21:21:28,134 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp2hiomaqd' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:30<00:00, 22.33s/epoch, loss=2.91e-6, prev_loss=2.66e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:30<00:00, 15.07s/epoch, loss=2.91e-6, prev_loss=2.66e-6] 2021-09-16:21:24:34,365 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpdunt5m3g' 2021-09-16:21:24:34,402 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdunt5m3g' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 20.32s/epoch, loss=5e-6, prev_loss=4.03e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 13.04s/epoch, loss=5e-6, prev_loss=4.03e-6] 2021-09-16:21:27:20,395 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpr1m8s0ez' 2021-09-16:21:27:20,456 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpr1m8s0ez' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 19.21s/epoch, loss=2.79e-6, prev_loss=2.66e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 11.92s/epoch, loss=2.79e-6, prev_loss=2.66e-6] 2021-09-16:21:29:55,199 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp2f98is2l' 2021-09-16:21:29:55,259 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp2f98is2l' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:14<00:00, 20.70s/epoch, loss=2.44e-5, prev_loss=1.16e-5] Training epochs on cuda: 100%|██████████| 10/10 [02:14<00:00, 13.46s/epoch, loss=2.44e-5, prev_loss=1.16e-5] 2021-09-16:21:32:45,609 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp8sfejjc5' 2021-09-16:21:32:45,646 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp8sfejjc5' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 13.29s/epoch, loss=9.9e-6, prev_loss=1e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.11s/epoch, loss=9.9e-6, prev_loss=1e-5] 2021-09-16:21:34:21,767 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpwkn2w7zv' 2021-09-16:21:34:21,825 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpwkn2w7zv' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:49<00:00, 18.22s/epoch, loss=2.17e-6, prev_loss=2.2e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:49<00:00, 10.95s/epoch, loss=2.17e-6, prev_loss=2.2e-6] 2021-09-16:21:36:46,921 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmptep0y3ey' 2021-09-16:21:36:46,976 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmptep0y3ey' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:50<00:00, 18.30s/epoch, loss=2.51e-6, prev_loss=2.49e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:50<00:00, 11.06s/epoch, loss=2.51e-6, prev_loss=2.49e-6] 2021-09-16:21:39:13,258 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpa1850hc8' 2021-09-16:21:39:13,312 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpa1850hc8' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:40<00:00, 17.44s/epoch, loss=3.07e-6, prev_loss=2.86e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:40<00:00, 10.09s/epoch, loss=3.07e-6, prev_loss=2.86e-6] 2021-09-16:21:41:29,727 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpgsmwhfqu' 2021-09-16:21:41:29,781 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpgsmwhfqu' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 19.17s/epoch, loss=4.39e-6, prev_loss=5.08e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 11.92s/epoch, loss=4.39e-6, prev_loss=5.08e-6] 2021-09-16:21:44:04,489 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpqddon6bt' 2021-09-16:21:44:04,527 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpqddon6bt' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:47<00:00, 18.04s/epoch, loss=2.52e-6, prev_loss=2.68e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:47<00:00, 10.79s/epoch, loss=2.52e-6, prev_loss=2.68e-6] 2021-09-16:21:46:27,949 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmprz0rornq' 2021-09-16:21:46:27,994 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmprz0rornq' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:13<00:00, 20.55s/epoch, loss=2.04e-6, prev_loss=2.07e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:13<00:00, 13.31s/epoch, loss=2.04e-6, prev_loss=2.07e-6] 2021-09-16:21:49:16,669 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpazj0lgyn' 2021-09-16:21:49:16,736 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpazj0lgyn' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:50<00:00, 18.33s/epoch, loss=2.27e-6, prev_loss=2.23e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:50<00:00, 11.06s/epoch, loss=2.27e-6, prev_loss=2.23e-6] 2021-09-16:21:51:42,878 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp5qpz6sk2' 2021-09-16:21:51:42,921 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp5qpz6sk2' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:08<00:00, 20.13s/epoch, loss=2.64e-6, prev_loss=2.9e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:08<00:00, 12.83s/epoch, loss=2.64e-6, prev_loss=2.9e-6] 2021-09-16:21:54:26,748 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpzd4210si' 2021-09-16:21:54:26,791 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpzd4210si' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:54<00:00, 18.64s/epoch, loss=2.85e-6, prev_loss=2.69e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:54<00:00, 11.43s/epoch, loss=2.85e-6, prev_loss=2.69e-6] 2021-09-16:21:56:56,751 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpy8ric3hw' 2021-09-16:21:56:56,803 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpy8ric3hw' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:54<00:00, 18.69s/epoch, loss=2.44e-6, prev_loss=2.42e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:54<00:00, 11.48s/epoch, loss=2.44e-6, prev_loss=2.42e-6] 2021-09-16:21:59:27,281 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpzoig0ab1' 2021-09-16:21:59:27,349 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpzoig0ab1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 18.90s/epoch, loss=3.11e-6, prev_loss=2.89e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 11.65s/epoch, loss=3.11e-6, prev_loss=2.89e-6] 2021-09-16:22:01:59,201 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpg7ny5bln' 2021-09-16:22:01:59,240 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpg7ny5bln' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 19.21s/epoch, loss=2.59e-6, prev_loss=2.59e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 11.95s/epoch, loss=2.59e-6, prev_loss=2.59e-6] 2021-09-16:22:04:34,346 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmppoha1_qk' 2021-09-16:22:04:34,402 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmppoha1_qk' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 20.16s/epoch, loss=7.07e-6, prev_loss=4.78e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 12.97s/epoch, loss=7.07e-6, prev_loss=4.78e-6] 2021-09-16:22:07:19,681 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp6iyc6gzi' 2021-09-16:22:07:19,717 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6iyc6gzi' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:57<00:00, 19.01s/epoch, loss=3.27e-6, prev_loss=3.11e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:57<00:00, 11.74s/epoch, loss=3.27e-6, prev_loss=3.11e-6] 2021-09-16:22:09:52,260 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp4tdsi1o5' 2021-09-16:22:09:52,309 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp4tdsi1o5' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:34<00:00, 22.68s/epoch, loss=5.83e-6, prev_loss=4.9e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:34<00:00, 15.44s/epoch, loss=5.83e-6, prev_loss=4.9e-6] 2021-09-16:22:13:02,185 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpz6p5erm0' 2021-09-16:22:13:02,217 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpz6p5erm0' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:12<00:00, 20.46s/epoch, loss=2.35e-6, prev_loss=2.36e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:12<00:00, 13.27s/epoch, loss=2.35e-6, prev_loss=2.36e-6] 2021-09-16:22:15:50,358 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpqjg24en3' 2021-09-16:22:15:50,417 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpqjg24en3' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 20.14s/epoch, loss=2.54e-6, prev_loss=2.54e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 13.02s/epoch, loss=2.54e-6, prev_loss=2.54e-6] 2021-09-16:22:18:36,431 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpls4hubau' 2021-09-16:22:18:36,470 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpls4hubau' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 20.36s/epoch, loss=2.62e-6, prev_loss=2.75e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 13.11s/epoch, loss=2.62e-6, prev_loss=2.75e-6] 2021-09-16:22:21:22,974 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpkyd_4qsd' 2021-09-16:22:21:23,26 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpkyd_4qsd' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 19.96s/epoch, loss=2.44e-6, prev_loss=2.43e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 12.72s/epoch, loss=2.44e-6, prev_loss=2.43e-6] 2021-09-16:22:24:05,664 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpcnmz2bxs' 2021-09-16:22:24:05,733 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpcnmz2bxs' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:57<00:00, 18.95s/epoch, loss=2.66e-6, prev_loss=2.63e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:57<00:00, 11.72s/epoch, loss=2.66e-6, prev_loss=2.63e-6] 2021-09-16:22:26:38,368 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp6jq4chi1' 2021-09-16:22:26:38,421 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6jq4chi1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 19.29s/epoch, loss=2.63e-6, prev_loss=2.69e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 12.03s/epoch, loss=2.63e-6, prev_loss=2.69e-6] 2021-09-16:22:29:14,251 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp6f7thphe' 2021-09-16:22:29:14,288 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6f7thphe' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 19.16s/epoch, loss=2.49e-6, prev_loss=2.58e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 12.00s/epoch, loss=2.49e-6, prev_loss=2.58e-6] 2021-09-16:22:31:49,885 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpmxuv6sel' 2021-09-16:22:31:49,948 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpmxuv6sel' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 20.19s/epoch, loss=4.87e-6, prev_loss=3.83e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 12.96s/epoch, loss=4.87e-6, prev_loss=3.83e-6] 2021-09-16:22:34:34,986 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpfd_7ie8b' 2021-09-16:22:34:35,23 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfd_7ie8b' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:14<00:00, 20.67s/epoch, loss=2.48e-6, prev_loss=2.54e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:14<00:00, 13.43s/epoch, loss=2.48e-6, prev_loss=2.54e-6] 2021-09-16:22:37:24,875 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpojgvqroq' 2021-09-16:22:37:24,919 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpojgvqroq' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:13<00:00, 20.57s/epoch, loss=2.85e-6, prev_loss=3.06e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:13<00:00, 13.33s/epoch, loss=2.85e-6, prev_loss=3.06e-6] 2021-09-16:22:40:13,863 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpkvb7zdeu' 2021-09-16:22:40:13,902 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpkvb7zdeu' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:13<00:00, 20.55s/epoch, loss=2.37e-6, prev_loss=2.5e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:13<00:00, 13.35s/epoch, loss=2.37e-6, prev_loss=2.5e-6] 2021-09-16:22:43:02,957 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp35t6bqgp' 2021-09-16:22:43:03,15 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp35t6bqgp' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:25<00:00, 21.81s/epoch, loss=4.68e-5, prev_loss=4.96e-5] Training epochs on cuda: 100%|██████████| 10/10 [02:25<00:00, 14.60s/epoch, loss=4.68e-5, prev_loss=4.96e-5] 2021-09-16:22:46:04,318 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp0e5qw53l' 2021-09-16:22:46:04,373 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp0e5qw53l' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:39<00:00, 23.12s/epoch, loss=1.6e-5, prev_loss=1.99e-5] Training epochs on cuda: 100%|██████████| 10/10 [02:39<00:00, 15.90s/epoch, loss=1.6e-5, prev_loss=1.99e-5] 2021-09-16:22:49:18,682 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmppdj8zj9c' 2021-09-16:22:49:18,718 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmppdj8zj9c' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 18.91s/epoch, loss=2.38e-6, prev_loss=2.58e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 11.66s/epoch, loss=2.38e-6, prev_loss=2.58e-6] 2021-09-16:22:51:50,992 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpwg1s09oq' 2021-09-16:22:51:51,30 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpwg1s09oq' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 19.32s/epoch, loss=2.56e-6, prev_loss=2.53e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 11.95s/epoch, loss=2.56e-6, prev_loss=2.53e-6] 2021-09-16:22:54:26,139 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpyyrnk__8' 2021-09-16:22:54:26,218 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpyyrnk__8' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 20.40s/epoch, loss=3.61e-6, prev_loss=3.31e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 13.11s/epoch, loss=3.61e-6, prev_loss=3.31e-6] 2021-09-16:22:57:12,884 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpb834i3aw' 2021-09-16:22:57:12,925 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpb834i3aw' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:51<00:00, 18.44s/epoch, loss=3.25e-6, prev_loss=3.06e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:51<00:00, 11.19s/epoch, loss=3.25e-6, prev_loss=3.06e-6] 2021-09-16:22:59:40,380 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpyazlax7p' 2021-09-16:22:59:40,432 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpyazlax7p' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:18<00:00, 21.11s/epoch, loss=2.33e-6, prev_loss=2.34e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:18<00:00, 13.86s/epoch, loss=2.33e-6, prev_loss=2.34e-6] 2021-09-16:23:02:34,726 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpptyi496b' 2021-09-16:23:02:34,768 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpptyi496b' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 20.11s/epoch, loss=2.45e-6, prev_loss=2.51e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 12.76s/epoch, loss=2.45e-6, prev_loss=2.51e-6] 2021-09-16:23:05:17,969 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpcrlf3eua' 2021-09-16:23:05:18,4 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpcrlf3eua' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 19.34s/epoch, loss=2.79e-6, prev_loss=2.68e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 12.15s/epoch, loss=2.79e-6, prev_loss=2.68e-6] 2021-09-16:23:07:55,114 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpbq6l7wh_' 2021-09-16:23:07:55,168 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpbq6l7wh_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:05<00:00, 19.83s/epoch, loss=3.24e-6, prev_loss=3.02e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:05<00:00, 12.54s/epoch, loss=3.24e-6, prev_loss=3.02e-6] 2021-09-16:23:10:36,156 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpdacid29g' 2021-09-16:23:10:36,225 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdacid29g' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:14<00:00, 20.66s/epoch, loss=1.11e-6, prev_loss=1.15e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:14<00:00, 13.45s/epoch, loss=1.11e-6, prev_loss=1.15e-6] 2021-09-16:23:13:26,1 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpk2vtu035' 2021-09-16:23:13:26,42 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpk2vtu035' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 18.81s/epoch, loss=6.59e-6, prev_loss=5.1e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 11.61s/epoch, loss=6.59e-6, prev_loss=5.1e-6] 2021-09-16:23:15:57,935 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp7bg538q5' 2021-09-16:23:15:57,988 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp7bg538q5' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 20.00s/epoch, loss=3.73e-6, prev_loss=4.07e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 12.74s/epoch, loss=3.73e-6, prev_loss=4.07e-6] 2021-09-16:23:18:41,115 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpha2ebkpn' 2021-09-16:23:18:41,154 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpha2ebkpn' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 19.27s/epoch, loss=2.65e-6, prev_loss=2.63e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 11.92s/epoch, loss=2.65e-6, prev_loss=2.63e-6] 2021-09-16:23:21:15,841 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpvyz7m1a8' 2021-09-16:23:21:15,881 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpvyz7m1a8' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 19.19s/epoch, loss=2.42e-6, prev_loss=2.48e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 11.94s/epoch, loss=2.42e-6, prev_loss=2.48e-6] 2021-09-16:23:23:50,756 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_i8ophn9' 2021-09-16:23:23:50,862 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_i8ophn9' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:03<00:00, 19.62s/epoch, loss=2.59e-6, prev_loss=2.63e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:03<00:00, 12.39s/epoch, loss=2.59e-6, prev_loss=2.63e-6] 2021-09-16:23:26:30,351 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpiwksk3ps' 2021-09-16:23:26:30,411 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpiwksk3ps' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 19.30s/epoch, loss=2.6e-6, prev_loss=2.62e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 12.04s/epoch, loss=2.6e-6, prev_loss=2.62e-6] 2021-09-16:23:29:06,267 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_wkcecs2' 2021-09-16:23:29:06,318 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_wkcecs2' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 20.93s/epoch, loss=2.79e-6, prev_loss=2.95e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 13.72s/epoch, loss=2.79e-6, prev_loss=2.95e-6] 2021-09-16:23:31:58,848 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmprzzheb34' 2021-09-16:23:31:58,885 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmprzzheb34' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:03<00:00, 19.62s/epoch, loss=3.05e-6, prev_loss=3.02e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:03<00:00, 12.35s/epoch, loss=3.05e-6, prev_loss=3.02e-6] 2021-09-16:23:34:37,503 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp0yzqgtl5' 2021-09-16:23:34:37,554 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp0yzqgtl5' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 19.28s/epoch, loss=3.09e-6, prev_loss=2.96e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 12.09s/epoch, loss=3.09e-6, prev_loss=2.96e-6] 2021-09-16:23:37:13,903 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmph393a2fp' 2021-09-16:23:37:13,947 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmph393a2fp' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 18.80s/epoch, loss=4.09e-6, prev_loss=3.67e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 11.60s/epoch, loss=4.09e-6, prev_loss=3.67e-6] 2021-09-16:23:39:45,475 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpbgt2m8oc' 2021-09-16:23:39:45,571 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpbgt2m8oc' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 19.49s/epoch, loss=1.24e-5, prev_loss=2.16e-5] Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 12.19s/epoch, loss=1.24e-5, prev_loss=2.16e-5] 2021-09-16:23:42:23,128 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmphem88t7g' 2021-09-16:23:42:23,166 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmphem88t7g' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:53<00:00, 18.55s/epoch, loss=2.38e-6, prev_loss=2.44e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:53<00:00, 11.31s/epoch, loss=2.38e-6, prev_loss=2.44e-6] 2021-09-16:23:44:51,847 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpdz7x27j0' 2021-09-16:23:44:51,885 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdz7x27j0' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 20.26s/epoch, loss=2.37e-6, prev_loss=2.42e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 13.05s/epoch, loss=2.37e-6, prev_loss=2.42e-6] 2021-09-16:23:47:37,948 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1_7_jg_m' 2021-09-16:23:47:37,984 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1_7_jg_m' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:08<00:00, 20.13s/epoch, loss=3.35e-6, prev_loss=3.14e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:08<00:00, 12.86s/epoch, loss=3.35e-6, prev_loss=3.14e-6] 2021-09-16:23:50:22,102 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp24ack053' 2021-09-16:23:50:22,143 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp24ack053' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:26<00:00, 21.94s/epoch, loss=5.2e-6, prev_loss=4.95e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:26<00:00, 14.70s/epoch, loss=5.2e-6, prev_loss=4.95e-6] 2021-09-16:23:53:24,392 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpnhcby259' 2021-09-16:23:53:24,427 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpnhcby259' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:06<00:00, 19.85s/epoch, loss=3.12e-6, prev_loss=3.02e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:06<00:00, 12.63s/epoch, loss=3.12e-6, prev_loss=3.02e-6] 2021-09-16:23:56:06,225 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpoxfe3bne' 2021-09-16:23:56:06,270 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpoxfe3bne' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 20.37s/epoch, loss=6.13e-6, prev_loss=4.73e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 13.16s/epoch, loss=6.13e-6, prev_loss=4.73e-6] 2021-09-16:23:58:53,307 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpzgkgujbr' 2021-09-16:23:58:53,352 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpzgkgujbr' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 19.25s/epoch, loss=2.65e-6, prev_loss=2.82e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 12.03s/epoch, loss=2.65e-6, prev_loss=2.82e-6] 2021-09-17:00:01:29,92 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpn2rm0dxm' 2021-09-17:00:01:29,170 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpn2rm0dxm' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:14<00:00, 20.72s/epoch, loss=3.02e-6, prev_loss=2.88e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:14<00:00, 13.49s/epoch, loss=3.02e-6, prev_loss=2.88e-6] 2021-09-17:00:04:19,349 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp2bh0vmwz' 2021-09-17:00:04:19,400 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp2bh0vmwz' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 19.07s/epoch, loss=3.37e-6, prev_loss=3.15e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 11.61s/epoch, loss=3.37e-6, prev_loss=3.15e-6] 2021-09-17:00:06:50,896 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp7d_786zb' 2021-09-17:00:06:50,931 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp7d_786zb' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:08<00:00, 14.09s/epoch, loss=1.67e-6, prev_loss=1.77e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:08<00:00, 6.83s/epoch, loss=1.67e-6, prev_loss=1.77e-6] 2021-09-17:00:08:34,315 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpiklxk_ts' 2021-09-17:00:08:34,357 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpiklxk_ts' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:06<00:00, 19.90s/epoch, loss=2.65e-6, prev_loss=2.66e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:06<00:00, 12.69s/epoch, loss=2.65e-6, prev_loss=2.66e-6] 2021-09-17:00:11:16,541 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpkg0dsjm_' 2021-09-17:00:11:16,592 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpkg0dsjm_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 20.26s/epoch, loss=2.28e-6, prev_loss=2.32e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 12.93s/epoch, loss=2.28e-6, prev_loss=2.32e-6] 2021-09-17:00:14:01,463 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpb2uujygz' 2021-09-17:00:14:01,503 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpb2uujygz' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:05<00:00, 19.75s/epoch, loss=2.43e-6, prev_loss=2.51e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:05<00:00, 12.52s/epoch, loss=2.43e-6, prev_loss=2.51e-6] 2021-09-17:00:16:42,162 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpjo9wxjs3' 2021-09-17:00:16:42,230 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpjo9wxjs3' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 20.50s/epoch, loss=2.59e-6, prev_loss=2.56e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 13.19s/epoch, loss=2.59e-6, prev_loss=2.56e-6] 2021-09-17:00:19:29,785 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpk_jig3s1' 2021-09-17:00:19:29,825 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpk_jig3s1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:12<00:00, 20.51s/epoch, loss=2.7e-6, prev_loss=2.62e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:12<00:00, 13.27s/epoch, loss=2.7e-6, prev_loss=2.62e-6] 2021-09-17:00:22:17,939 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp24et3j09' 2021-09-17:00:22:17,986 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp24et3j09' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 19.30s/epoch, loss=3.43e-6, prev_loss=3.28e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 12.11s/epoch, loss=3.43e-6, prev_loss=3.28e-6] 2021-09-17:00:24:54,607 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp7wgrp405' 2021-09-17:00:24:54,663 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp7wgrp405' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:22<00:00, 21.45s/epoch, loss=2.51e-6, prev_loss=2.69e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:22<00:00, 14.22s/epoch, loss=2.51e-6, prev_loss=2.69e-6] 2021-09-17:00:27:52,359 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpv6b1qt7h' 2021-09-17:00:27:52,396 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpv6b1qt7h' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:20<00:00, 21.29s/epoch, loss=7.86e-6, prev_loss=6.32e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:20<00:00, 14.06s/epoch, loss=7.86e-6, prev_loss=6.32e-6] 2021-09-17:00:30:48,87 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp7indlqra' 2021-09-17:00:30:48,119 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp7indlqra' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:29<00:00, 22.22s/epoch, loss=2.66e-6, prev_loss=2.74e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:29<00:00, 14.98s/epoch, loss=2.66e-6, prev_loss=2.74e-6] 2021-09-17:00:33:53,195 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp2ru4jj4o' 2021-09-17:00:33:53,244 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp2ru4jj4o' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:53<00:00, 18.66s/epoch, loss=4.3e-6, prev_loss=3.71e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:53<00:00, 11.40s/epoch, loss=4.3e-6, prev_loss=3.71e-6] 2021-09-17:00:36:22,616 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpjf1avp61' 2021-09-17:00:36:22,651 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpjf1avp61' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 20.35s/epoch, loss=3.38e-6, prev_loss=3.17e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 13.03s/epoch, loss=3.38e-6, prev_loss=3.17e-6] 2021-09-17:00:39:08,402 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpel3zi3ct' 2021-09-17:00:39:08,438 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpel3zi3ct' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 19.77s/epoch, loss=2.44e-6, prev_loss=2.46e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 12.45s/epoch, loss=2.44e-6, prev_loss=2.46e-6] 2021-09-17:00:41:48,337 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmppusrue5e' 2021-09-17:00:41:48,375 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmppusrue5e' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:06<00:00, 19.77s/epoch, loss=2.45e-6, prev_loss=2.42e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:06<00:00, 12.64s/epoch, loss=2.45e-6, prev_loss=2.42e-6] 2021-09-17:00:44:30,281 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp7_d5hr7g' 2021-09-17:00:44:30,316 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp7_d5hr7g' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 19.31s/epoch, loss=2.34e-6, prev_loss=2.39e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 12.10s/epoch, loss=2.34e-6, prev_loss=2.39e-6] 2021-09-17:00:47:06,765 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpidcj8v4g' 2021-09-17:00:47:06,830 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpidcj8v4g' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 19.96s/epoch, loss=3.13e-6, prev_loss=3.06e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 12.73s/epoch, loss=3.13e-6, prev_loss=3.06e-6] 2021-09-17:00:49:49,419 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp8fxk1z94' 2021-09-17:00:49:49,481 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp8fxk1z94' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 20.24s/epoch, loss=2.73e-6, prev_loss=2.68e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 12.98s/epoch, loss=2.73e-6, prev_loss=2.68e-6] 2021-09-17:00:52:34,632 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp840_7z40' 2021-09-17:00:52:34,668 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp840_7z40' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 13.68s/epoch, loss=1.36e-5, prev_loss=1.39e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 6.32s/epoch, loss=1.36e-5, prev_loss=1.39e-5] 2021-09-17:00:54:12,869 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp187ggxfw' 2021-09-17:00:54:12,906 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp187ggxfw' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 20.14s/epoch, loss=8.04e-6, prev_loss=5.27e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 12.98s/epoch, loss=8.04e-6, prev_loss=5.27e-6] 2021-09-17:00:56:58,402 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmppbwlcdm3' 2021-09-17:00:56:58,457 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmppbwlcdm3' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:24<00:00, 21.75s/epoch, loss=2.65e-6, prev_loss=2.46e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:24<00:00, 14.41s/epoch, loss=2.65e-6, prev_loss=2.46e-6] 2021-09-17:00:59:57,981 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpdnutpiya' 2021-09-17:00:59:58,56 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdnutpiya' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:22<00:00, 21.57s/epoch, loss=2.88e-6, prev_loss=2.7e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:22<00:00, 14.28s/epoch, loss=2.88e-6, prev_loss=2.7e-6] 2021-09-17:01:02:56,357 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1fyu68hu' 2021-09-17:01:02:56,405 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1fyu68hu' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:08<00:00, 20.03s/epoch, loss=9.89e-6, prev_loss=7.11e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:08<00:00, 12.80s/epoch, loss=9.89e-6, prev_loss=7.11e-6] 2021-09-17:01:05:39,935 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpb3s30aek' 2021-09-17:01:05:39,971 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpb3s30aek' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 20.29s/epoch, loss=2.58e-6, prev_loss=2.55e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 13.00s/epoch, loss=2.58e-6, prev_loss=2.55e-6] 2021-09-17:01:08:25,387 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpfoehhlrb' 2021-09-17:01:08:25,424 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfoehhlrb' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:22<00:00, 21.46s/epoch, loss=3.24e-6, prev_loss=2.97e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:22<00:00, 14.23s/epoch, loss=3.24e-6, prev_loss=2.97e-6] 2021-09-17:01:11:23,150 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp9nz3c28a' 2021-09-17:01:11:23,185 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp9nz3c28a' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:13<00:00, 20.65s/epoch, loss=2.16e-6, prev_loss=2.15e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:13<00:00, 13.38s/epoch, loss=2.16e-6, prev_loss=2.15e-6] 2021-09-17:01:14:12,533 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpzh27agwe' 2021-09-17:01:14:12,594 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpzh27agwe' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 18.88s/epoch, loss=5.9e-6, prev_loss=4.8e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 11.64s/epoch, loss=5.9e-6, prev_loss=4.8e-6] 2021-09-17:01:16:44,542 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmphwpniowm' 2021-09-17:01:16:44,578 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmphwpniowm' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:12<00:00, 20.67s/epoch, loss=2.79e-6, prev_loss=2.71e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:12<00:00, 13.26s/epoch, loss=2.79e-6, prev_loss=2.71e-6] 2021-09-17:01:19:32,719 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpsm5mk9jk' 2021-09-17:01:19:32,756 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpsm5mk9jk' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 20.12s/epoch, loss=2.97e-6, prev_loss=2.82e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 12.95s/epoch, loss=2.97e-6, prev_loss=2.82e-6] 2021-09-17:01:22:17,735 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp06vugmmb' 2021-09-17:01:22:17,783 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp06vugmmb' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 19.19s/epoch, loss=2.53e-6, prev_loss=2.62e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 12.00s/epoch, loss=2.53e-6, prev_loss=2.62e-6] 2021-09-17:01:24:53,33 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpfbspfy62' 2021-09-17:01:24:53,74 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfbspfy62' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:34<00:00, 22.67s/epoch, loss=2.67e-6, prev_loss=2.71e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:34<00:00, 15.46s/epoch, loss=2.67e-6, prev_loss=2.71e-6] 2021-09-17:01:28:02,673 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpahv5izdd' 2021-09-17:01:28:02,749 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpahv5izdd' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:35<00:00, 22.78s/epoch, loss=1.26e-6, prev_loss=1.26e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:35<00:00, 15.56s/epoch, loss=1.26e-6, prev_loss=1.26e-6] 2021-09-17:01:31:13,406 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpvawgci0p' 2021-09-17:01:31:13,440 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpvawgci0p' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 20.08s/epoch, loss=4.08e-6, prev_loss=3.59e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 12.90s/epoch, loss=4.08e-6, prev_loss=3.59e-6] 2021-09-17:01:33:57,713 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmptdrbkwsl' 2021-09-17:01:33:57,778 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmptdrbkwsl' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:12<00:00, 20.38s/epoch, loss=2.53e-6, prev_loss=2.52e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:12<00:00, 13.20s/epoch, loss=2.53e-6, prev_loss=2.52e-6] 2021-09-17:01:36:45,58 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpnzcea7lx' 2021-09-17:01:36:45,137 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpnzcea7lx' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:57<00:00, 18.97s/epoch, loss=2.57e-6, prev_loss=2.59e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:57<00:00, 11.77s/epoch, loss=2.57e-6, prev_loss=2.59e-6] 2021-09-17:01:39:18,160 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpy19c67zk' 2021-09-17:01:39:18,231 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpy19c67zk' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 19.69s/epoch, loss=2.88e-6, prev_loss=2.87e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 12.49s/epoch, loss=2.88e-6, prev_loss=2.87e-6] 2021-09-17:01:41:58,505 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp36pyqupd' 2021-09-17:01:41:58,558 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp36pyqupd' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 20.96s/epoch, loss=4.28e-5, prev_loss=4.42e-5] Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 13.74s/epoch, loss=4.28e-5, prev_loss=4.42e-5] 2021-09-17:01:44:51,251 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpz30o1yn7' 2021-09-17:01:44:51,288 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpz30o1yn7' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 18.96s/epoch, loss=3.38e-6, prev_loss=3.15e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 11.67s/epoch, loss=3.38e-6, prev_loss=3.15e-6] 2021-09-17:01:47:23,280 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpw1irtcsh' 2021-09-17:01:47:23,319 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpw1irtcsh' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:02<00:00, 19.53s/epoch, loss=2.38e-6, prev_loss=2.43e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:02<00:00, 12.30s/epoch, loss=2.38e-6, prev_loss=2.43e-6] 2021-09-17:01:50:01,703 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp9_uw388_' 2021-09-17:01:50:01,765 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp9_uw388_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 21.00s/epoch, loss=1.63e-5, prev_loss=3.47e-5] Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 13.79s/epoch, loss=1.63e-5, prev_loss=3.47e-5] 2021-09-17:01:52:55,24 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpdwnj460a' 2021-09-17:01:52:55,57 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdwnj460a' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:35<00:00, 22.99s/epoch, loss=2.98e-6, prev_loss=2.88e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:35<00:00, 15.54s/epoch, loss=2.98e-6, prev_loss=2.88e-6] 2021-09-17:01:56:05,524 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpa3cd5vyz' 2021-09-17:01:56:05,571 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpa3cd5vyz' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 19.85s/epoch, loss=2.57e-6, prev_loss=2.7e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 12.75s/epoch, loss=2.57e-6, prev_loss=2.7e-6] 2021-09-17:01:58:48,385 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpmfg0545g' 2021-09-17:01:58:48,434 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpmfg0545g' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:54<00:00, 18.68s/epoch, loss=3.31e-6, prev_loss=3.07e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:54<00:00, 11.49s/epoch, loss=3.31e-6, prev_loss=3.07e-6] 2021-09-17:02:01:18,656 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpgll45jgh' 2021-09-17:02:01:18,706 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpgll45jgh' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 20.28s/epoch, loss=2.64e-6, prev_loss=2.59e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 13.05s/epoch, loss=2.64e-6, prev_loss=2.59e-6] 2021-09-17:02:04:04,490 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpdqhnw1e8' 2021-09-17:02:04:04,524 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdqhnw1e8' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 20.14s/epoch, loss=2.39e-6, prev_loss=2.41e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 12.99s/epoch, loss=2.39e-6, prev_loss=2.41e-6] 2021-09-17:02:06:49,699 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp6chmidtu' 2021-09-17:02:06:49,755 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6chmidtu' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:03<00:00, 19.49s/epoch, loss=2.45e-6, prev_loss=2.42e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:03<00:00, 12.31s/epoch, loss=2.45e-6, prev_loss=2.42e-6] 2021-09-17:02:09:28,287 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpmi5put1m' 2021-09-17:02:09:28,322 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpmi5put1m' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 20.14s/epoch, loss=2.58e-6, prev_loss=2.51e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 12.97s/epoch, loss=2.58e-6, prev_loss=2.51e-6] 2021-09-17:02:12:13,256 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpr477_50a' 2021-09-17:02:12:13,300 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpr477_50a' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 20.94s/epoch, loss=3.01e-6, prev_loss=2.9e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 13.77s/epoch, loss=3.01e-6, prev_loss=2.9e-6] 2021-09-17:02:15:06,400 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp7i_ggo_u' 2021-09-17:02:15:06,434 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp7i_ggo_u' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 20.30s/epoch, loss=2.81e-6, prev_loss=2.66e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 13.14s/epoch, loss=2.81e-6, prev_loss=2.66e-6] 2021-09-17:02:17:53,296 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp6baimlnc' 2021-09-17:02:17:53,334 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6baimlnc' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:57<00:00, 19.02s/epoch, loss=2.96e-6, prev_loss=2.95e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:57<00:00, 11.78s/epoch, loss=2.96e-6, prev_loss=2.95e-6] 2021-09-17:02:20:26,496 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp3sr0jpkb' 2021-09-17:02:20:26,532 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp3sr0jpkb' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 20.42s/epoch, loss=2.23e-6, prev_loss=2.31e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 13.19s/epoch, loss=2.23e-6, prev_loss=2.31e-6] 2021-09-17:02:23:13,748 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpi3c4drbz' 2021-09-17:02:23:13,788 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpi3c4drbz' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:51<00:00, 18.29s/epoch, loss=2.9e-6, prev_loss=2.8e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:51<00:00, 11.12s/epoch, loss=2.9e-6, prev_loss=2.8e-6] 2021-09-17:02:25:40,332 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpr0qf896e' 2021-09-17:02:25:40,387 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpr0qf896e' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 19.20s/epoch, loss=4.73e-5, prev_loss=4.92e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 11.99s/epoch, loss=4.73e-5, prev_loss=4.92e-5] 2021-09-17:02:28:15,591 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpf8b5nic4' 2021-09-17:02:28:15,634 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpf8b5nic4' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 20.18s/epoch, loss=3.8e-6, prev_loss=3.3e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 13.02s/epoch, loss=3.8e-6, prev_loss=3.3e-6] 2021-09-17:02:31:01,113 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpfixp23l2' 2021-09-17:02:31:01,148 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfixp23l2' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 20.00s/epoch, loss=3.07e-6, prev_loss=2.93e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 12.97s/epoch, loss=3.07e-6, prev_loss=2.93e-6] 2021-09-17:02:33:46,249 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpfb85pueu' 2021-09-17:02:33:46,304 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfb85pueu' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:06<00:00, 19.83s/epoch, loss=2.49e-6, prev_loss=2.47e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:06<00:00, 12.64s/epoch, loss=2.49e-6, prev_loss=2.47e-6] 2021-09-17:02:36:27,992 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpsg5uu5w4' 2021-09-17:02:36:28,24 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpsg5uu5w4' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:05<00:00, 19.81s/epoch, loss=2.66e-6, prev_loss=2.75e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:05<00:00, 12.58s/epoch, loss=2.66e-6, prev_loss=2.75e-6] 2021-09-17:02:39:09,183 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_zzf_1h0' 2021-09-17:02:39:09,238 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_zzf_1h0' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 19.31s/epoch, loss=2.15e-6, prev_loss=2.2e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 12.11s/epoch, loss=2.15e-6, prev_loss=2.2e-6] 2021-09-17:02:41:45,683 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpge3fw5zd' 2021-09-17:02:41:45,722 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpge3fw5zd' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 19.94s/epoch, loss=2.68e-6, prev_loss=2.56e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 12.72s/epoch, loss=2.68e-6, prev_loss=2.56e-6] 2021-09-17:02:44:28,270 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpgr_g35df' 2021-09-17:02:44:28,323 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpgr_g35df' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:54<00:00, 18.65s/epoch, loss=2.78e-6, prev_loss=2.71e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:54<00:00, 11.45s/epoch, loss=2.78e-6, prev_loss=2.71e-6] 2021-09-17:02:46:58,21 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp9cowy8pv' 2021-09-17:02:46:58,61 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp9cowy8pv' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 20.24s/epoch, loss=2.87e-6, prev_loss=2.76e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 13.08s/epoch, loss=2.87e-6, prev_loss=2.76e-6] 2021-09-17:02:49:44,234 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpax4u07uf' 2021-09-17:02:49:44,292 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpax4u07uf' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 20.91s/epoch, loss=3.65e-6, prev_loss=3.39e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 13.75s/epoch, loss=3.65e-6, prev_loss=3.39e-6] 2021-09-17:02:52:36,908 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp0od0ymm1' 2021-09-17:02:52:36,950 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp0od0ymm1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:16<00:00, 20.89s/epoch, loss=3.67e-6, prev_loss=3.66e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:16<00:00, 13.69s/epoch, loss=3.67e-6, prev_loss=3.66e-6] 2021-09-17:02:55:29,0 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmptvwa0fqd' 2021-09-17:02:55:29,36 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmptvwa0fqd' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:26<00:00, 21.79s/epoch, loss=3.03e-6, prev_loss=3e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:26<00:00, 14.62s/epoch, loss=3.03e-6, prev_loss=3e-6] 2021-09-17:02:58:30,280 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpkq91pwag' 2021-09-17:02:58:30,353 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpkq91pwag' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:25<00:00, 21.75s/epoch, loss=3.42e-6, prev_loss=3.47e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:25<00:00, 14.54s/epoch, loss=3.42e-6, prev_loss=3.47e-6] 2021-09-17:03:01:30,846 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp2h4cikgj' 2021-09-17:03:01:30,889 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp2h4cikgj' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:08<00:00, 14.10s/epoch, loss=5.95e-5, prev_loss=6.06e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:08<00:00, 6.88s/epoch, loss=5.95e-5, prev_loss=6.06e-5] 2021-09-17:03:03:14,591 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpfnalr_1i' 2021-09-17:03:03:14,631 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfnalr_1i' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 19.18s/epoch, loss=2.63e-6, prev_loss=2.67e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 11.94s/epoch, loss=2.63e-6, prev_loss=2.67e-6] 2021-09-17:03:05:49,383 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpfe4djmrh' 2021-09-17:03:05:49,417 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfe4djmrh' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:15<00:00, 20.75s/epoch, loss=4.23e-6, prev_loss=4.31e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:15<00:00, 13.53s/epoch, loss=4.23e-6, prev_loss=4.31e-6] 2021-09-17:03:08:39,714 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpa_rpg7tl' 2021-09-17:03:08:39,763 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpa_rpg7tl' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:26<00:00, 21.80s/epoch, loss=2.91e-6, prev_loss=2.83e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:26<00:00, 14.63s/epoch, loss=2.91e-6, prev_loss=2.83e-6] 2021-09-17:03:11:41,499 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp63cbi5x_' 2021-09-17:03:11:41,551 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp63cbi5x_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:05<00:00, 19.77s/epoch, loss=2.49e-6, prev_loss=2.51e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:05<00:00, 12.57s/epoch, loss=2.49e-6, prev_loss=2.51e-6] 2021-09-17:03:14:22,498 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpi2b10vxw' 2021-09-17:03:14:22,550 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpi2b10vxw' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:54<00:00, 24.62s/epoch, loss=1.09e-6, prev_loss=1.09e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:54<00:00, 17.42s/epoch, loss=1.09e-6, prev_loss=1.09e-6] 2021-09-17:03:17:51,927 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp6o4_3arr' 2021-09-17:03:17:51,970 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6o4_3arr' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 19.23s/epoch, loss=2.41e-6, prev_loss=2.42e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 11.99s/epoch, loss=2.41e-6, prev_loss=2.42e-6] 2021-09-17:03:20:27,101 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1a1lzvzj' 2021-09-17:03:20:27,145 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1a1lzvzj' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00, 14.81s/epoch, loss=4.07e-6, prev_loss=4e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00, 7.61s/epoch, loss=4.07e-6, prev_loss=4e-6] 2021-09-17:03:22:18,71 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmprqu46fec' 2021-09-17:03:22:18,107 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmprqu46fec' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 19.29s/epoch, loss=2.41e-6, prev_loss=2.45e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 12.11s/epoch, loss=2.41e-6, prev_loss=2.45e-6] 2021-09-17:03:24:54,457 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpeprrm_r9' 2021-09-17:03:24:54,516 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpeprrm_r9' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 19.19s/epoch, loss=2.56e-6, prev_loss=2.64e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:59<00:00, 11.99s/epoch, loss=2.56e-6, prev_loss=2.64e-6] 2021-09-17:03:27:29,677 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpdan_bjd0' 2021-09-17:03:27:29,729 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdan_bjd0' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 19.62s/epoch, loss=2.46e-6, prev_loss=2.47e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 12.40s/epoch, loss=2.46e-6, prev_loss=2.47e-6] 2021-09-17:03:30:09,295 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpguuuvx40' 2021-09-17:03:30:09,340 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpguuuvx40' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:03<00:00, 19.58s/epoch, loss=2.5e-6, prev_loss=2.55e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:03<00:00, 12.39s/epoch, loss=2.5e-6, prev_loss=2.55e-6] 2021-09-17:03:32:48,533 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpborysih4' 2021-09-17:03:32:48,594 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpborysih4' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 13.60s/epoch, loss=1.26e-5, prev_loss=1.21e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 6.33s/epoch, loss=1.26e-5, prev_loss=1.21e-5] 2021-09-17:03:34:26,776 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpxyl132rb' 2021-09-17:03:34:26,804 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpxyl132rb' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 20.38s/epoch, loss=8.79e-6, prev_loss=7.46e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 13.19s/epoch, loss=8.79e-6, prev_loss=7.46e-6] 2021-09-17:03:37:13,623 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpluociyx0' 2021-09-17:03:37:13,651 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpluociyx0' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:34<00:00, 22.61s/epoch, loss=2.98e-6, prev_loss=3.19e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:34<00:00, 15.43s/epoch, loss=2.98e-6, prev_loss=3.19e-6] 2021-09-17:03:40:22,988 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpt8wl4ywx' 2021-09-17:03:40:23,31 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpt8wl4ywx' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:08<00:00, 20.06s/epoch, loss=2.45e-6, prev_loss=2.48e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:08<00:00, 12.83s/epoch, loss=2.45e-6, prev_loss=2.48e-6] 2021-09-17:03:43:06,609 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp6w2g4zo6' 2021-09-17:03:43:06,658 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6w2g4zo6' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:13<00:00, 20.64s/epoch, loss=2.51e-6, prev_loss=2.54e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:13<00:00, 13.34s/epoch, loss=2.51e-6, prev_loss=2.54e-6] 2021-09-17:03:45:55,385 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmppbi_0p65' 2021-09-17:03:45:55,417 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmppbi_0p65' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:16<00:00, 20.89s/epoch, loss=2.51e-6, prev_loss=2.61e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:16<00:00, 13.69s/epoch, loss=2.51e-6, prev_loss=2.61e-6] 2021-09-17:03:48:47,771 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmphbovugpg' 2021-09-17:03:48:47,806 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmphbovugpg' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 20.27s/epoch, loss=2.54e-6, prev_loss=2.55e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 13.05s/epoch, loss=2.54e-6, prev_loss=2.55e-6] 2021-09-17:03:51:33,700 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpm1k3jhnd' 2021-09-17:03:51:33,754 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpm1k3jhnd' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:02<00:00, 19.47s/epoch, loss=2.62e-6, prev_loss=2.66e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:02<00:00, 12.27s/epoch, loss=2.62e-6, prev_loss=2.66e-6] 2021-09-17:03:54:11,904 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmptcx2r3as' 2021-09-17:03:54:11,941 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmptcx2r3as' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:57<00:00, 18.96s/epoch, loss=2.75e-6, prev_loss=2.75e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:57<00:00, 11.72s/epoch, loss=2.75e-6, prev_loss=2.75e-6] 2021-09-17:03:56:44,633 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpio3x82fl' 2021-09-17:03:56:44,669 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpio3x82fl' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 19.73s/epoch, loss=2.45e-6, prev_loss=2.43e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 12.50s/epoch, loss=2.45e-6, prev_loss=2.43e-6] 2021-09-17:03:59:25,37 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpnzpirr_h' 2021-09-17:03:59:25,87 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpnzpirr_h' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 19.74s/epoch, loss=2.38e-6, prev_loss=2.5e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 12.45s/epoch, loss=2.38e-6, prev_loss=2.5e-6] 2021-09-17:04:02:04,938 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpn_m_i1ta' 2021-09-17:04:02:04,981 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpn_m_i1ta' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:55<00:00, 18.81s/epoch, loss=2.49e-6, prev_loss=2.5e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:55<00:00, 11.59s/epoch, loss=2.49e-6, prev_loss=2.5e-6] 2021-09-17:04:04:36,148 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp8tn7_kpi' 2021-09-17:04:04:36,186 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp8tn7_kpi' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 19.67s/epoch, loss=2.52e-6, prev_loss=2.63e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 12.46s/epoch, loss=2.52e-6, prev_loss=2.63e-6] 2021-09-17:04:07:16,73 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpnlitojss' 2021-09-17:04:07:16,120 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpnlitojss' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 18.77s/epoch, loss=2.44e-6, prev_loss=2.47e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:56<00:00, 11.61s/epoch, loss=2.44e-6, prev_loss=2.47e-6] 2021-09-17:04:09:47,579 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpbj1f0tpa' 2021-09-17:04:09:47,617 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpbj1f0tpa' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:05<00:00, 19.74s/epoch, loss=2.68e-6, prev_loss=2.59e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:05<00:00, 12.51s/epoch, loss=2.68e-6, prev_loss=2.59e-6] 2021-09-17:04:12:27,940 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpdbfj108a' 2021-09-17:04:12:27,986 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdbfj108a' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:13<00:00, 20.56s/epoch, loss=3.26e-6, prev_loss=3.09e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:13<00:00, 13.34s/epoch, loss=3.26e-6, prev_loss=3.09e-6] 2021-09-17:04:15:16,486 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpo8iqrvu3' 2021-09-17:04:15:16,549 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpo8iqrvu3' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:18<00:00, 21.07s/epoch, loss=2.67e-6, prev_loss=2.88e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:18<00:00, 13.84s/epoch, loss=2.67e-6, prev_loss=2.88e-6] 2021-09-17:04:18:10,278 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpiu5b5afc' 2021-09-17:04:18:10,314 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpiu5b5afc' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:06<00:00, 19.76s/epoch, loss=2.67e-6, prev_loss=2.69e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:06<00:00, 12.61s/epoch, loss=2.67e-6, prev_loss=2.69e-6] 2021-09-17:04:20:51,676 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpjxh0tej5' 2021-09-17:04:20:51,727 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpjxh0tej5' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 19.12s/epoch, loss=2.41e-6, prev_loss=2.59e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 12.03s/epoch, loss=2.41e-6, prev_loss=2.59e-6] 2021-09-17:04:23:27,697 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpy2fwu4t3' 2021-09-17:04:23:27,736 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpy2fwu4t3' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 18.56s/epoch, loss=2.69e-6, prev_loss=2.68e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 11.26s/epoch, loss=2.69e-6, prev_loss=2.68e-6] 2021-09-17:04:25:55,691 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpe87ov1nr' 2021-09-17:04:25:55,735 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpe87ov1nr' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [09:13<00:00, 62.36s/epoch, loss=0.0193, prev_loss=0.0194] Training epochs on cuda: 100%|██████████| 10/10 [09:13<00:00, 55.40s/epoch, loss=0.0193, prev_loss=0.0194] 2021-09-17:04:35:44,616 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp21k1pcjb' 2021-09-17:04:35:44,692 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp21k1pcjb' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:02<00:00, 19.45s/epoch, loss=2.91e-6, prev_loss=2.86e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:02<00:00, 12.21s/epoch, loss=2.91e-6, prev_loss=2.86e-6] 2021-09-17:04:38:22,201 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmptowd_r6z' 2021-09-17:04:38:22,247 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmptowd_r6z' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:29<00:00, 22.10s/epoch, loss=3.62e-6, prev_loss=3.66e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:29<00:00, 14.91s/epoch, loss=3.62e-6, prev_loss=3.66e-6] 2021-09-17:04:41:26,422 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpepv03855' 2021-09-17:04:41:26,454 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpepv03855' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 19.26s/epoch, loss=2.54e-6, prev_loss=2.6e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 12.03s/epoch, loss=2.54e-6, prev_loss=2.6e-6] 2021-09-17:04:44:02,71 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpyw1za5y_' 2021-09-17:04:44:02,105 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpyw1za5y_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:54<00:00, 18.65s/epoch, loss=4.17e-6, prev_loss=4.7e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:54<00:00, 11.45s/epoch, loss=4.17e-6, prev_loss=4.7e-6] 2021-09-17:04:46:31,909 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpvcml72l2' 2021-09-17:04:46:31,971 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpvcml72l2' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 19.27s/epoch, loss=9.74e-7, prev_loss=9.86e-7] Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 12.09s/epoch, loss=9.74e-7, prev_loss=9.86e-7] 2021-09-17:04:49:08,214 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp3v8jukxv' 2021-09-17:04:49:08,248 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp3v8jukxv' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:58<00:00, 19.05s/epoch, loss=2.63e-6, prev_loss=2.75e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:58<00:00, 11.82s/epoch, loss=2.63e-6, prev_loss=2.75e-6] 2021-09-17:04:51:41,644 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp3o03_9s6' 2021-09-17:04:51:41,696 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp3o03_9s6' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:05<00:00, 19.80s/epoch, loss=2.29e-6, prev_loss=2.37e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:05<00:00, 12.55s/epoch, loss=2.29e-6, prev_loss=2.37e-6] 2021-09-17:04:54:22,556 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_v78_qn3' 2021-09-17:04:54:22,598 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_v78_qn3' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 18.45s/epoch, loss=2.7e-6, prev_loss=2.64e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 11.25s/epoch, loss=2.7e-6, prev_loss=2.64e-6] 2021-09-17:04:56:50,382 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp8rup2tx4' 2021-09-17:04:56:50,427 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp8rup2tx4' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:53<00:00, 18.44s/epoch, loss=2.64e-6, prev_loss=2.66e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:53<00:00, 11.36s/epoch, loss=2.64e-6, prev_loss=2.66e-6] 2021-09-17:04:59:19,472 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpo6c2r3km' 2021-09-17:04:59:19,526 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpo6c2r3km' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:48<00:00, 18.05s/epoch, loss=2.93e-6, prev_loss=2.98e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:48<00:00, 10.87s/epoch, loss=2.93e-6, prev_loss=2.98e-6] 2021-09-17:05:01:43,529 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpw01ibhtj' 2021-09-17:05:01:43,578 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpw01ibhtj' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 20.37s/epoch, loss=2.89e-6, prev_loss=2.78e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 13.13s/epoch, loss=2.89e-6, prev_loss=2.78e-6] 2021-09-17:05:04:29,982 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpyje_00ox' 2021-09-17:05:04:30,26 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpyje_00ox' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:50<00:00, 18.26s/epoch, loss=2.32e-6, prev_loss=2.4e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:50<00:00, 11.08s/epoch, loss=2.32e-6, prev_loss=2.4e-6] 2021-09-17:05:06:56,127 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpqdcvq9eb' 2021-09-17:05:06:56,216 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpqdcvq9eb' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 18.46s/epoch, loss=2.65e-6, prev_loss=2.66e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 11.28s/epoch, loss=2.65e-6, prev_loss=2.66e-6] 2021-09-17:05:09:24,310 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp6mavit7s' 2021-09-17:05:09:24,367 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6mavit7s' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:54<00:00, 18.67s/epoch, loss=2.92e-6, prev_loss=2.9e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:54<00:00, 11.47s/epoch, loss=2.92e-6, prev_loss=2.9e-6] 2021-09-17:05:11:54,349 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpwb152tbp' 2021-09-17:05:11:54,391 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpwb152tbp' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:43<00:00, 17.50s/epoch, loss=2.64e-5, prev_loss=2.01e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:43<00:00, 10.33s/epoch, loss=2.64e-5, prev_loss=2.01e-5] 2021-09-17:05:14:12,663 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmptoul0525' 2021-09-17:05:14:12,706 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmptoul0525' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:15<00:00, 20.66s/epoch, loss=3.89e-6, prev_loss=3.44e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:15<00:00, 13.50s/epoch, loss=3.89e-6, prev_loss=3.44e-6] 2021-09-17:05:17:02,740 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpqmd3fy4m' 2021-09-17:05:17:02,778 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpqmd3fy4m' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:20<00:00, 21.21s/epoch, loss=2.52e-6, prev_loss=2.56e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:20<00:00, 14.01s/epoch, loss=2.52e-6, prev_loss=2.56e-6] 2021-09-17:05:19:57,806 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmps3_e_yhi' 2021-09-17:05:19:57,865 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmps3_e_yhi' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 19.23s/epoch, loss=2.43e-6, prev_loss=2.39e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 12.02s/epoch, loss=2.43e-6, prev_loss=2.39e-6] 2021-09-17:05:22:33,327 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp34ldwyk7' 2021-09-17:05:22:33,380 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp34ldwyk7' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:54<00:00, 18.59s/epoch, loss=2.73e-5, prev_loss=2.77e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:54<00:00, 11.44s/epoch, loss=2.73e-5, prev_loss=2.77e-5] 2021-09-17:05:25:02,943 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmprn7kqrkx' 2021-09-17:05:25:02,986 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmprn7kqrkx' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 19.31s/epoch, loss=2.91e-6, prev_loss=2.76e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:00<00:00, 12.00s/epoch, loss=2.91e-6, prev_loss=2.76e-6] 2021-09-17:05:27:38,120 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmprgrx4pmc' 2021-09-17:05:27:38,162 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmprgrx4pmc' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 19.88s/epoch, loss=3.92e-5, prev_loss=3.9e-5] Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 12.70s/epoch, loss=3.92e-5, prev_loss=3.9e-5] 2021-09-17:05:30:20,705 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpndb9t2i7' 2021-09-17:05:30:20,743 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpndb9t2i7' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:19<00:00, 21.23s/epoch, loss=2.33e-6, prev_loss=2.35e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:19<00:00, 13.97s/epoch, loss=2.33e-6, prev_loss=2.35e-6] 2021-09-17:05:33:15,911 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpcd0jbm93' 2021-09-17:05:33:15,949 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpcd0jbm93' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:05<00:00, 19.56s/epoch, loss=4.25e-6, prev_loss=3.99e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:05<00:00, 12.52s/epoch, loss=4.25e-6, prev_loss=3.99e-6] 2021-09-17:05:35:56,208 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpkbgdkxkv' 2021-09-17:05:35:56,255 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpkbgdkxkv' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:20<00:00, 21.28s/epoch, loss=2.23e-6, prev_loss=2.32e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:20<00:00, 14.00s/epoch, loss=2.23e-6, prev_loss=2.32e-6] 2021-09-17:05:38:51,768 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpo233r3pz' 2021-09-17:05:38:51,810 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpo233r3pz' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 20.86s/epoch, loss=3.09e-6, prev_loss=3.41e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 13.73s/epoch, loss=3.09e-6, prev_loss=3.41e-6] 2021-09-17:05:41:44,420 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1j5vn_vr' 2021-09-17:05:41:44,458 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1j5vn_vr' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:06<00:00, 19.79s/epoch, loss=2.54e-6, prev_loss=2.64e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:06<00:00, 12.63s/epoch, loss=2.54e-6, prev_loss=2.64e-6] 2021-09-17:05:44:25,993 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpax619yln' 2021-09-17:05:44:26,68 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpax619yln' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:02<00:00, 19.43s/epoch, loss=2.64e-6, prev_loss=2.63e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:02<00:00, 12.25s/epoch, loss=2.64e-6, prev_loss=2.63e-6] 2021-09-17:05:47:03,849 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpnsj5a31r' 2021-09-17:05:47:03,890 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpnsj5a31r' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 20.26s/epoch, loss=2.57e-6, prev_loss=2.49e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:10<00:00, 13.10s/epoch, loss=2.57e-6, prev_loss=2.49e-6] 2021-09-17:05:49:50,142 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpkw7vh8d4' 2021-09-17:05:49:50,190 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpkw7vh8d4' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 20.14s/epoch, loss=2.79e-6, prev_loss=2.73e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 12.97s/epoch, loss=2.79e-6, prev_loss=2.73e-6] 2021-09-17:05:52:35,179 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmprap_z7dr' 2021-09-17:05:52:35,224 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmprap_z7dr' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:35<00:00, 22.74s/epoch, loss=4.79e-6, prev_loss=4.28e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:35<00:00, 15.59s/epoch, loss=4.79e-6, prev_loss=4.28e-6] 2021-09-17:05:55:46,122 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp6aqcu57f' 2021-09-17:05:55:46,170 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6aqcu57f' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:13<00:00, 20.52s/epoch, loss=3.45e-6, prev_loss=3.3e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:13<00:00, 13.34s/epoch, loss=3.45e-6, prev_loss=3.3e-6] 2021-09-17:05:58:34,886 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpibmt5zlk' 2021-09-17:05:58:34,927 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpibmt5zlk' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:58<00:00, 19.07s/epoch, loss=3.1e-6, prev_loss=2.9e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:58<00:00, 11.89s/epoch, loss=3.1e-6, prev_loss=2.9e-6] 2021-09-17:06:01:09,143 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpfpl4qu4y' 2021-09-17:06:01:09,185 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfpl4qu4y' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:02<00:00, 19.46s/epoch, loss=1.45e-5, prev_loss=3.05e-5] Training epochs on cuda: 100%|██████████| 10/10 [02:02<00:00, 12.29s/epoch, loss=1.45e-5, prev_loss=3.05e-5] 2021-09-17:06:03:47,443 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp7arzf7iu' 2021-09-17:06:03:47,481 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp7arzf7iu' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:15<00:00, 20.75s/epoch, loss=2.58e-6, prev_loss=2.5e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:15<00:00, 13.58s/epoch, loss=2.58e-6, prev_loss=2.5e-6] 2021-09-17:06:06:38,550 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpii0a6uu9' 2021-09-17:06:06:38,586 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpii0a6uu9' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 19.95s/epoch, loss=7.92e-6, prev_loss=6.39e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 12.76s/epoch, loss=7.92e-6, prev_loss=6.39e-6] 2021-09-17:06:09:21,90 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp4wmgm2mb' 2021-09-17:06:09:21,154 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp4wmgm2mb' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:29<00:00, 16.37s/epoch, loss=1.25e-5, prev_loss=1.09e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:29<00:00, 8.90s/epoch, loss=1.25e-5, prev_loss=1.09e-5] 2021-09-17:06:11:25,171 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpmixnefot' 2021-09-17:06:11:25,242 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpmixnefot' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:15<00:00, 20.73s/epoch, loss=3.06e-6, prev_loss=2.95e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:15<00:00, 13.55s/epoch, loss=3.06e-6, prev_loss=2.95e-6] 2021-09-17:06:14:15,914 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp4gnbls2o' 2021-09-17:06:14:15,965 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp4gnbls2o' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 20.93s/epoch, loss=3e-6, prev_loss=2.87e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 13.71s/epoch, loss=3e-6, prev_loss=2.87e-6] 2021-09-17:06:17:08,229 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmppxwx9iyr' 2021-09-17:06:17:08,265 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmppxwx9iyr' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:12<00:00, 20.45s/epoch, loss=2.88e-6, prev_loss=2.75e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:12<00:00, 13.27s/epoch, loss=2.88e-6, prev_loss=2.75e-6] 2021-09-17:06:19:56,197 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_8vkoup1' 2021-09-17:06:19:56,291 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_8vkoup1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 20.16s/epoch, loss=3.41e-6, prev_loss=3.11e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:09<00:00, 12.99s/epoch, loss=3.41e-6, prev_loss=3.11e-6] 2021-09-17:06:22:41,496 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpd2kf8_i3' 2021-09-17:06:22:41,539 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpd2kf8_i3' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:15<00:00, 20.76s/epoch, loss=6.12e-6, prev_loss=4.29e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:15<00:00, 13.53s/epoch, loss=6.12e-6, prev_loss=4.29e-6] 2021-09-17:06:25:32,123 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp096b39o_' 2021-09-17:06:25:32,191 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp096b39o_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:03<00:00, 19.61s/epoch, loss=3.17e-6, prev_loss=3.07e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:03<00:00, 12.40s/epoch, loss=3.17e-6, prev_loss=3.07e-6] 2021-09-17:06:28:11,431 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpr4pzfmvj' 2021-09-17:06:28:11,469 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpr4pzfmvj' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 20.37s/epoch, loss=3.49e-6, prev_loss=3.54e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 13.20s/epoch, loss=3.49e-6, prev_loss=3.54e-6] 2021-09-17:06:30:58,702 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1ok102f4' 2021-09-17:06:30:58,736 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1ok102f4' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:13<00:00, 14.54s/epoch, loss=4.76e-6, prev_loss=4.1e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:13<00:00, 7.31s/epoch, loss=4.76e-6, prev_loss=4.1e-6] 2021-09-17:06:32:46,844 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpkx9e87f5' 2021-09-17:06:32:46,886 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpkx9e87f5' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 synset) to (word -> word) Number of edges with words as nodes = 388301 triples' shape: (388301, 3) TriplesFactory(num_entities=45182, num_relations=31, num_triples=388301, inverse_triples=False) HpoPipelineResult(study=, objective=Objective(dataset=None, model=, loss=, optimizer=, training_loop=, stopper=, evaluator=, result_tracker=, metric='adjusted_arithmetic_mean_rank_index', dataset_kwargs=None, training=TriplesFactory(num_entities=45182, num_relations=31, num_triples=310640, inverse_triples=False), testing=TriplesFactory(num_entities=45182, num_relations=31, num_triples=38830, inverse_triples=False), validation=TriplesFactory(num_entities=45182, num_relations=31, num_triples=38831, inverse_triples=False), evaluation_entity_whitelist=None, evaluation_relation_whitelist=None, model_kwargs={'embedding_dim': 100}, model_kwargs_ranges=None, loss_kwargs=None, loss_kwargs_ranges=None, regularizer=None, regularizer_kwargs=None, regularizer_kwargs_ranges=None, optimizer_kwargs=None, optimizer_kwargs_ranges=None, training_loop_kwargs=None, negative_sampler=, negative_sampler_kwargs=None, negative_sampler_kwargs_ranges=None, training_kwargs={'num_epochs': 10}, training_kwargs_ranges=None, stopper_kwargs=None, evaluator_kwargs=None, evaluation_kwargs=None, filter_validation_when_testing=True, result_tracker_kwargs=None, device=None, save_model_directory=None))