2021-09-16:17:22:35,504 INFO [iwn.py:43] Loading bengali language synsets... 2021-09-16:17:22:49,208 WARNING [utils.py:429] using automatically assigned random_state=3179480306 2021-09-16:17:22:49,361 INFO [splitting.py:60] done splitting triples to groups of sizes [267018, 38830, 38831] /srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release. from numpy.core.umath_tests import inner1d /srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/gradient_boosting.py:34: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations from ._gradient_boosting import predict_stages /srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/gradient_boosting.py:34: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations from ._gradient_boosting import predict_stages [I 2021-09-16 17:22:53,483] A new study created in memory with name: no-name-8e955b25-a1f3-4a57-b42e-1f7594053d64 2021-09-16:17:22:53,565 INFO [hpo.py:622] Using model: 2021-09-16:17:22:53,565 INFO [hpo.py:626] Using loss: 2021-09-16:17:22:53,565 INFO [hpo.py:637] Using regularizer: 2021-09-16:17:22:53,566 INFO [hpo.py:641] Using optimizer: 2021-09-16:17:22:53,566 INFO [hpo.py:645] Using training loop: 2021-09-16:17:22:53,566 INFO [hpo.py:651] Using negative sampler: 2021-09-16:17:22:53,566 INFO [hpo.py:662] Using evaluator: 2021-09-16:17:22:53,566 INFO [hpo.py:666] Attempting to maximize adjusted_arithmetic_mean_rank_index 2021-09-16:17:22:53,566 INFO [hpo.py:668] Filter validation triples when testing: True /srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932]. low=low, old_high=old_high, high=high, step=step 2021-09-16:17:22:53,584 WARNING [api.py:823] No random seed is specified. Setting to 1437431581. Training epochs on cuda: 0%| | 0/10 [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:35<00:00, 6.81s/epoch, loss=0.000503, prev_loss=0.000503] Training epochs on cuda: 100%|██████████| 10/10 [00:35<00:00, 3.58s/epoch, loss=0.000503, prev_loss=0.000503] 2021-09-16:17:23:43,771 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpicvbza94' 2021-09-16:17:23:43,840 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpicvbza94' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 9.68s/epoch, loss=0.00237, prev_loss=0.00238] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.16s/epoch, loss=0.00237, prev_loss=0.00238] 2021-09-16:17:25:00,556 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp5so3dixq' 2021-09-16:17:25:00,616 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp5so3dixq' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:38<00:00, 7.16s/epoch, loss=0.0014, prev_loss=0.0014] Training epochs on cuda: 100%|██████████| 10/10 [00:38<00:00, 3.87s/epoch, loss=0.0014, prev_loss=0.0014] 2021-09-16:17:25:54,491 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp8tqfivgw' 2021-09-16:17:25:54,567 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp8tqfivgw' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 8.94s/epoch, loss=0.000416, prev_loss=0.000416] Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 5.76s/epoch, loss=0.000416, prev_loss=0.000416] 2021-09-16:17:27:07,371 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpm7mr68zi' 2021-09-16:17:27:07,445 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpm7mr68zi' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:48<00:00, 8.11s/epoch, loss=2.1e-5, prev_loss=2.04e-5] Training epochs on cuda: 100%|██████████| 10/10 [00:48<00:00, 4.86s/epoch, loss=2.1e-5, prev_loss=2.04e-5] 2021-09-16:17:28:11,407 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpi0em8w3k' 2021-09-16:17:28:11,461 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpi0em8w3k' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 9.49s/epoch, loss=4.02e-5, prev_loss=4.12e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 6.25s/epoch, loss=4.02e-5, prev_loss=4.12e-5] 2021-09-16:17:29:29,254 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpd843idb0' 2021-09-16:17:29:29,306 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpd843idb0' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:50<00:00, 8.25s/epoch, loss=0.0016, prev_loss=0.0016] Training epochs on cuda: 100%|██████████| 10/10 [00:50<00:00, 5.08s/epoch, loss=0.0016, prev_loss=0.0016] 2021-09-16:17:30:35,401 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp75i58x68' 2021-09-16:17:30:35,463 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp75i58x68' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:41<00:00, 7.42s/epoch, loss=0.000638, prev_loss=0.000638] Training epochs on cuda: 100%|██████████| 10/10 [00:41<00:00, 4.17s/epoch, loss=0.000638, prev_loss=0.000638] 2021-09-16:17:31:32,401 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp6bxnzszd' 2021-09-16:17:31:32,454 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6bxnzszd' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:14<00:00, 16.67s/epoch, loss=0.000312, prev_loss=0.000311] Training epochs on cuda: 100%|██████████| 10/10 [02:14<00:00, 13.41s/epoch, loss=0.000312, prev_loss=0.000311] 2021-09-16:17:34:02,117 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_3auerc5' 2021-09-16:17:34:02,184 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_3auerc5' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00, 7.19s/epoch, loss=0.000261, prev_loss=0.000262] Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00, 3.95s/epoch, loss=0.000261, prev_loss=0.000262] 2021-09-16:17:34:57,160 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpioae44tt' 2021-09-16:17:34:57,214 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpioae44tt' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [07:57<00:00, 50.93s/epoch, loss=0.096, prev_loss=0.096] Training epochs on cuda: 100%|██████████| 10/10 [07:57<00:00, 47.77s/epoch, loss=0.096, prev_loss=0.096] 2021-09-16:17:43:10,196 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp70n966h3' 2021-09-16:17:43:10,238 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp70n966h3' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:41<00:00, 7.36s/epoch, loss=0.00149, prev_loss=0.00149] Training epochs on cuda: 100%|██████████| 10/10 [00:41<00:00, 4.11s/epoch, loss=0.00149, prev_loss=0.00149] 2021-09-16:17:44:06,635 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpj6xa5enm' 2021-09-16:17:44:06,693 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpj6xa5enm' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:41<00:00, 7.41s/epoch, loss=0.000553, prev_loss=0.000761] Training epochs on cuda: 100%|██████████| 10/10 [00:41<00:00, 4.18s/epoch, loss=0.000553, prev_loss=0.000761] 2021-09-16:17:45:03,922 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmphnhnsj1x' 2021-09-16:17:45:03,999 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmphnhnsj1x' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.15s/epoch, loss=0.00371, prev_loss=0.00483] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.95s/epoch, loss=0.00371, prev_loss=0.00483] 2021-09-16:17:46:18,663 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp2kmtguac' 2021-09-16:17:46:18,737 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp2kmtguac' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:54<00:00, 8.65s/epoch, loss=0.00368, prev_loss=0.00368] Training epochs on cuda: 100%|██████████| 10/10 [00:54<00:00, 5.40s/epoch, loss=0.00368, prev_loss=0.00368] 2021-09-16:17:47:28,31 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpdazeuw3e' 2021-09-16:17:47:28,103 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdazeuw3e' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 9.64s/epoch, loss=0.000741, prev_loss=0.000759] Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 6.44s/epoch, loss=0.000741, prev_loss=0.000759] 2021-09-16:17:48:48,27 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpoi3e_u99' 2021-09-16:17:48:48,104 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpoi3e_u99' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 16.99s/epoch, loss=0.0162, prev_loss=0.0162] Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 13.74s/epoch, loss=0.0162, prev_loss=0.0162] 2021-09-16:17:51:20,752 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp84iue329' 2021-09-16:17:51:20,815 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp84iue329' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 9.84s/epoch, loss=0.00188, prev_loss=0.00188] Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 6.68s/epoch, loss=0.00188, prev_loss=0.00188] 2021-09-16:17:52:43,42 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpwouicja9' 2021-09-16:17:52:43,109 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpwouicja9' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:49<00:00, 8.12s/epoch, loss=0.00218, prev_loss=0.00218] Training epochs on cuda: 100%|██████████| 10/10 [00:49<00:00, 4.95s/epoch, loss=0.00218, prev_loss=0.00218] 2021-09-16:17:53:47,860 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpw1jz0hpv' 2021-09-16:17:53:47,924 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpw1jz0hpv' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.27s/epoch, loss=0.00709, prev_loss=0.00709] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.99s/epoch, loss=0.00709, prev_loss=0.00709] 2021-09-16:17:55:03,61 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpeew_qx4l' 2021-09-16:17:55:03,122 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpeew_qx4l' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:37<00:00, 7.05s/epoch, loss=0.00111, prev_loss=0.00111] Training epochs on cuda: 100%|██████████| 10/10 [00:37<00:00, 3.77s/epoch, loss=0.00111, prev_loss=0.00111] 2021-09-16:17:55:56,124 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp78i8ncr9' 2021-09-16:17:55:56,196 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp78i8ncr9' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 9.34s/epoch, loss=0.00132, prev_loss=0.00128] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.13s/epoch, loss=0.00132, prev_loss=0.00128] 2021-09-16:17:57:12,715 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp4uuewxq9' 2021-09-16:17:57:12,807 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp4uuewxq9' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00, 7.65s/epoch, loss=0.00117, prev_loss=0.00117] Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00, 4.56s/epoch, loss=0.00117, prev_loss=0.00117] 2021-09-16:17:58:13,604 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmphmqxb2fx' 2021-09-16:17:58:13,671 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmphmqxb2fx' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 9.36s/epoch, loss=0.00459, prev_loss=0.00464] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.17s/epoch, loss=0.00459, prev_loss=0.00464] 2021-09-16:17:59:30,608 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpuhrjpz00' 2021-09-16:17:59:30,673 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpuhrjpz00' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:48<00:00, 8.17s/epoch, loss=0.00107, prev_loss=0.00107] Training epochs on cuda: 100%|██████████| 10/10 [00:48<00:00, 4.89s/epoch, loss=0.00107, prev_loss=0.00107] 2021-09-16:18:00:34,870 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpj4al1fpr' 2021-09-16:18:00:34,941 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpj4al1fpr' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:38<00:00, 12.84s/epoch, loss=0.0132, prev_loss=0.0132] Training epochs on cuda: 100%|██████████| 10/10 [01:38<00:00, 9.86s/epoch, loss=0.0132, prev_loss=0.0132] 2021-09-16:18:02:28,887 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpbrzg67nk' 2021-09-16:18:02:28,976 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpbrzg67nk' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:46<00:00, 7.90s/epoch, loss=0.00241, prev_loss=0.00241] Training epochs on cuda: 100%|██████████| 10/10 [00:46<00:00, 4.68s/epoch, loss=0.00241, prev_loss=0.00241] 2021-09-16:18:03:31,5 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp5qfwdv3_' 2021-09-16:18:03:31,76 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp5qfwdv3_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00, 7.15s/epoch, loss=0.000731, prev_loss=0.000731] Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00, 3.92s/epoch, loss=0.000731, prev_loss=0.000731] 2021-09-16:18:04:25,612 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpuw15t_80' 2021-09-16:18:04:25,679 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpuw15t_80' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 9.20s/epoch, loss=0.00292, prev_loss=0.00292] Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 5.87s/epoch, loss=0.00292, prev_loss=0.00292] 2021-09-16:18:05:39,698 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp9pmhgsvy' 2021-09-16:18:05:39,793 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp9pmhgsvy' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 10.22s/epoch, loss=0.00919, prev_loss=0.00919] Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 7.15s/epoch, loss=0.00919, prev_loss=0.00919] 2021-09-16:18:07:06,635 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp2ozx8a8f' 2021-09-16:18:07:06,712 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp2ozx8a8f' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:43<00:00, 7.54s/epoch, loss=0.00027, prev_loss=0.0003] Training epochs on cuda: 100%|██████████| 10/10 [00:43<00:00, 4.31s/epoch, loss=0.00027, prev_loss=0.0003] 2021-09-16:18:08:05,122 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpfb01bqtj' 2021-09-16:18:08:05,187 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfb01bqtj' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00, 7.97s/epoch, loss=0.000312, prev_loss=0.000314] Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00, 4.58s/epoch, loss=0.000312, prev_loss=0.000314] 2021-09-16:18:09:06,232 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpjvpd8fg7' 2021-09-16:18:09:06,310 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpjvpd8fg7' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00, 7.62s/epoch, loss=0.000244, prev_loss=0.000244] Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00, 3.98s/epoch, loss=0.000244, prev_loss=0.000244] 2021-09-16:18:10:01,612 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp3cfi4zk1' 2021-09-16:18:10:01,686 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp3cfi4zk1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:37<00:00, 7.03s/epoch, loss=0.000593, prev_loss=0.000593] Training epochs on cuda: 100%|██████████| 10/10 [00:37<00:00, 3.79s/epoch, loss=0.000593, prev_loss=0.000593] 2021-09-16:18:10:54,950 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp6z6v7rzw' 2021-09-16:18:10:55,5 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6z6v7rzw' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:38<00:00, 6.95s/epoch, loss=0.000456, prev_loss=0.000469] Training epochs on cuda: 100%|██████████| 10/10 [00:38<00:00, 3.81s/epoch, loss=0.000456, prev_loss=0.000469] 2021-09-16:18:11:48,457 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_scve8v2' 2021-09-16:18:11:48,516 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_scve8v2' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:52<00:00, 8.57s/epoch, loss=0.000438, prev_loss=0.000438] Training epochs on cuda: 100%|██████████| 10/10 [00:52<00:00, 5.30s/epoch, loss=0.000438, prev_loss=0.000438] 2021-09-16:18:12:56,920 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpxcob2smy' 2021-09-16:18:12:56,987 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpxcob2smy' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00, 7.25s/epoch, loss=0.000809, prev_loss=0.000809] Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00, 3.98s/epoch, loss=0.000809, prev_loss=0.000809] 2021-09-16:18:13:52,90 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpyw0zyqmk' 2021-09-16:18:13:52,164 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpyw0zyqmk' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:40<00:00, 7.21s/epoch, loss=0.000455, prev_loss=0.000455] Training epochs on cuda: 100%|██████████| 10/10 [00:40<00:00, 4.03s/epoch, loss=0.000455, prev_loss=0.000455] 2021-09-16:18:14:47,858 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpxgw2z1sg' 2021-09-16:18:14:47,938 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpxgw2z1sg' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:35<00:00, 6.81s/epoch, loss=0.000198, prev_loss=0.000198] Training epochs on cuda: 100%|██████████| 10/10 [00:35<00:00, 3.55s/epoch, loss=0.000198, prev_loss=0.000198] 2021-09-16:18:15:38,639 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp6yot038t' 2021-09-16:18:15:38,708 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6yot038t' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 9.30s/epoch, loss=0.00023, prev_loss=0.000239] Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 6.05s/epoch, loss=0.00023, prev_loss=0.000239] 2021-09-16:18:16:54,563 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp6vrhc8fa' 2021-09-16:18:16:54,627 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6vrhc8fa' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:49<00:00, 8.19s/epoch, loss=0.000987, prev_loss=0.000987] Training epochs on cuda: 100%|██████████| 10/10 [00:49<00:00, 4.93s/epoch, loss=0.000987, prev_loss=0.000987] 2021-09-16:18:17:59,201 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_l9wtbo7' 2021-09-16:18:17:59,255 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_l9wtbo7' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 9.33s/epoch, loss=0.000484, prev_loss=0.000484] Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 6.06s/epoch, loss=0.000484, prev_loss=0.000484] 2021-09-16:18:19:15,276 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp10smzs4c' 2021-09-16:18:19:15,330 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp10smzs4c' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:17<00:00, 11.01s/epoch, loss=0.000189, prev_loss=0.000201] Training epochs on cuda: 100%|██████████| 10/10 [01:17<00:00, 7.73s/epoch, loss=0.000189, prev_loss=0.000201] 2021-09-16:18:20:48,65 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpf2gpvh7r' 2021-09-16:18:20:48,120 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpf2gpvh7r' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:34<00:00, 12.71s/epoch, loss=0.000434, prev_loss=0.000436] Training epochs on cuda: 100%|██████████| 10/10 [01:34<00:00, 9.48s/epoch, loss=0.000434, prev_loss=0.000436] 2021-09-16:18:22:38,831 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpv135yece' 2021-09-16:18:22:38,875 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpv135yece' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:18<00:00, 11.12s/epoch, loss=0.000205, prev_loss=0.000205] Training epochs on cuda: 100%|██████████| 10/10 [01:18<00:00, 7.82s/epoch, loss=0.000205, prev_loss=0.000205] 2021-09-16:18:24:12,647 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpg7hm02c7' 2021-09-16:18:24:12,717 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpg7hm02c7' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00, 10.92s/epoch, loss=0.00056, prev_loss=0.00056] Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00, 7.68s/epoch, loss=0.00056, prev_loss=0.00056] 2021-09-16:18:25:44,924 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpctdnlnfl' 2021-09-16:18:25:44,986 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpctdnlnfl' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00, 11.54s/epoch, loss=0.000557, prev_loss=0.000556] Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00, 8.35s/epoch, loss=0.000557, prev_loss=0.000556] 2021-09-16:18:27:23,968 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpzdeooob6' 2021-09-16:18:27:24,41 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpzdeooob6' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:40<00:00, 13.38s/epoch, loss=0.00024, prev_loss=0.000247] Training epochs on cuda: 100%|██████████| 10/10 [01:40<00:00, 10.09s/epoch, loss=0.00024, prev_loss=0.000247] 2021-09-16:18:29:20,450 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpdrj2aaak' 2021-09-16:18:29:20,522 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdrj2aaak' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:49<00:00, 14.16s/epoch, loss=0.00069, prev_loss=0.00069] Training epochs on cuda: 100%|██████████| 10/10 [01:49<00:00, 10.97s/epoch, loss=0.00069, prev_loss=0.00069] 2021-09-16:18:31:25,828 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpbcyu4cdy' 2021-09-16:18:31:25,889 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpbcyu4cdy' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:51<00:00, 14.41s/epoch, loss=0.000619, prev_loss=0.000621] Training epochs on cuda: 100%|██████████| 10/10 [01:51<00:00, 11.13s/epoch, loss=0.000619, prev_loss=0.000621] 2021-09-16:18:33:33,85 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1fcdmkuq' 2021-09-16:18:33:33,154 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1fcdmkuq' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00, 11.56s/epoch, loss=0.000595, prev_loss=0.000595] Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00, 8.31s/epoch, loss=0.000595, prev_loss=0.000595] 2021-09-16:18:35:11,917 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpmyk26b1t' 2021-09-16:18:35:12,13 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpmyk26b1t' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:44<00:00, 13.62s/epoch, loss=0.000182, prev_loss=0.000189] Training epochs on cuda: 100%|██████████| 10/10 [01:44<00:00, 10.41s/epoch, loss=0.000182, prev_loss=0.000189] 2021-09-16:18:37:11,776 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmph21u1w6r' 2021-09-16:18:37:11,862 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmph21u1w6r' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:41<00:00, 13.40s/epoch, loss=0.000433, prev_loss=0.000437] Training epochs on cuda: 100%|██████████| 10/10 [01:41<00:00, 10.18s/epoch, loss=0.000433, prev_loss=0.000437] 2021-09-16:18:39:09,299 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp3o9zr_be' 2021-09-16:18:39:09,368 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp3o9zr_be' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:58<00:00, 15.09s/epoch, loss=0.000781, prev_loss=0.000781] Training epochs on cuda: 100%|██████████| 10/10 [01:58<00:00, 11.82s/epoch, loss=0.000781, prev_loss=0.000781] 2021-09-16:18:41:23,280 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpovhi97wr' 2021-09-16:18:41:23,335 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpovhi97wr' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:28<00:00, 12.10s/epoch, loss=0.000409, prev_loss=0.000424] Training epochs on cuda: 100%|██████████| 10/10 [01:28<00:00, 8.85s/epoch, loss=0.000409, prev_loss=0.000424] 2021-09-16:18:43:07,532 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpja0k108e' 2021-09-16:18:43:07,604 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpja0k108e' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00, 7.29s/epoch, loss=0.000646, prev_loss=0.000646] Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00, 3.96s/epoch, loss=0.000646, prev_loss=0.000646] 2021-09-16:18:44:02,590 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpt4iopz6q' 2021-09-16:18:44:02,649 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpt4iopz6q' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:19<00:00, 11.21s/epoch, loss=0.000504, prev_loss=0.000554] Training epochs on cuda: 100%|██████████| 10/10 [01:19<00:00, 7.95s/epoch, loss=0.000504, prev_loss=0.000554] 2021-09-16:18:45:37,616 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp12yfdry2' 2021-09-16:18:45:37,727 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp12yfdry2' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:43<00:00, 13.61s/epoch, loss=0.000835, prev_loss=0.000833] Training epochs on cuda: 100%|██████████| 10/10 [01:43<00:00, 10.34s/epoch, loss=0.000835, prev_loss=0.000833] 2021-09-16:18:47:36,938 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp3au78081' 2021-09-16:18:47:36,993 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp3au78081' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:29<00:00, 12.21s/epoch, loss=0.00128, prev_loss=0.00128] Training epochs on cuda: 100%|██████████| 10/10 [01:29<00:00, 8.92s/epoch, loss=0.00128, prev_loss=0.00128] 2021-09-16:18:49:21,696 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp91kf4hrs' 2021-09-16:18:49:21,795 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp91kf4hrs' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00, 7.81s/epoch, loss=7.65e-7, prev_loss=7.68e-7] Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00, 4.58s/epoch, loss=7.65e-7, prev_loss=7.68e-7] 2021-09-16:18:50:23,81 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpy38v1hbk' 2021-09-16:18:50:23,146 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpy38v1hbk' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:42<00:00, 7.47s/epoch, loss=0.000289, prev_loss=0.000306] Training epochs on cuda: 100%|██████████| 10/10 [00:42<00:00, 4.24s/epoch, loss=0.000289, prev_loss=0.000306] 2021-09-16:18:51:20,996 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpcxp85mh1' 2021-09-16:18:51:21,56 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpcxp85mh1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00, 7.86s/epoch, loss=0.000261, prev_loss=0.000271] Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00, 4.56s/epoch, loss=0.000261, prev_loss=0.000271] 2021-09-16:18:52:22,122 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmptbohw040' 2021-09-16:18:52:22,203 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmptbohw040' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00, 7.89s/epoch, loss=0.000649, prev_loss=0.000649] Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00, 4.59s/epoch, loss=0.000649, prev_loss=0.000649] 2021-09-16:18:53:23,588 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpq6v99qyr' 2021-09-16:18:53:23,673 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpq6v99qyr' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:08<00:00, 10.20s/epoch, loss=0.000808, prev_loss=0.00081] Training epochs on cuda: 100%|██████████| 10/10 [01:08<00:00, 6.82s/epoch, loss=0.000808, prev_loss=0.00081] 2021-09-16:18:54:47,396 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp0gsdus6b' 2021-09-16:18:54:47,456 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp0gsdus6b' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:42<00:00, 7.52s/epoch, loss=0.000569, prev_loss=0.000569] Training epochs on cuda: 100%|██████████| 10/10 [00:42<00:00, 4.26s/epoch, loss=0.000569, prev_loss=0.000569] 2021-09-16:18:55:45,729 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmppf8gv8lj' 2021-09-16:18:55:45,818 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmppf8gv8lj' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 10.08s/epoch, loss=0.000238, prev_loss=0.000244] Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 6.77s/epoch, loss=0.000238, prev_loss=0.000244] 2021-09-16:18:57:08,898 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmplffo9t_1' 2021-09-16:18:57:08,965 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmplffo9t_1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.19s/epoch, loss=0.000906, prev_loss=0.000916] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.96s/epoch, loss=0.000906, prev_loss=0.000916] 2021-09-16:18:58:24,100 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp5s7jcf8q' 2021-09-16:18:58:24,180 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp5s7jcf8q' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [08:04<00:00, 51.88s/epoch, loss=0.0417, prev_loss=0.0417] Training epochs on cuda: 100%|██████████| 10/10 [08:04<00:00, 48.44s/epoch, loss=0.0417, prev_loss=0.0417] 2021-09-16:19:06:44,1 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmproeqa94r' 2021-09-16:19:06:44,85 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmproeqa94r' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:52<00:00, 8.53s/epoch, loss=0.00117, prev_loss=0.00117] Training epochs on cuda: 100%|██████████| 10/10 [00:52<00:00, 5.23s/epoch, loss=0.00117, prev_loss=0.00117] 2021-09-16:19:07:51,798 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpnu7tbiro' 2021-09-16:19:07:51,851 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpnu7tbiro' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:15<00:00, 10.78s/epoch, loss=0.000424, prev_loss=0.000464] Training epochs on cuda: 100%|██████████| 10/10 [01:15<00:00, 7.57s/epoch, loss=0.000424, prev_loss=0.000464] 2021-09-16:19:09:22,995 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpzrlxr41m' 2021-09-16:19:09:23,41 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpzrlxr41m' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 9.57s/epoch, loss=0.00062, prev_loss=0.00062] Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 6.30s/epoch, loss=0.00062, prev_loss=0.00062] 2021-09-16:19:10:41,482 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpu_0u9hse' 2021-09-16:19:10:41,546 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpu_0u9hse' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:15<00:00, 10.79s/epoch, loss=0.000357, prev_loss=0.00037] Training epochs on cuda: 100%|██████████| 10/10 [01:15<00:00, 7.56s/epoch, loss=0.000357, prev_loss=0.00037] 2021-09-16:19:12:12,661 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpayd4mcx1' 2021-09-16:19:12:12,715 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpayd4mcx1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00, 11.55s/epoch, loss=0.000484, prev_loss=0.000498] Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00, 8.33s/epoch, loss=0.000484, prev_loss=0.000498] 2021-09-16:19:13:51,403 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpk_enymmy' 2021-09-16:19:13:51,463 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpk_enymmy' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 10.30s/epoch, loss=0.0004, prev_loss=0.000405] Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 7.17s/epoch, loss=0.0004, prev_loss=0.000405] 2021-09-16:19:15:18,638 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpbn9z831o' 2021-09-16:19:15:18,689 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpbn9z831o' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 10.00s/epoch, loss=0.00107, prev_loss=0.00107] Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 6.72s/epoch, loss=0.00107, prev_loss=0.00107] 2021-09-16:19:16:41,405 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp83idqwnh' 2021-09-16:19:16:41,443 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp83idqwnh' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00, 10.90s/epoch, loss=0.00338, prev_loss=0.00338] Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00, 7.63s/epoch, loss=0.00338, prev_loss=0.00338] 2021-09-16:19:18:13,118 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmppc1h_qva' 2021-09-16:19:18:13,180 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmppc1h_qva' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:47<00:00, 8.03s/epoch, loss=0.000138, prev_loss=0.000144] Training epochs on cuda: 100%|██████████| 10/10 [00:47<00:00, 4.70s/epoch, loss=0.000138, prev_loss=0.000144] 2021-09-16:19:19:15,587 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpdrls3fg5' 2021-09-16:19:19:15,649 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdrls3fg5' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:48<00:00, 8.14s/epoch, loss=0.000338, prev_loss=0.00034] Training epochs on cuda: 100%|██████████| 10/10 [00:48<00:00, 4.88s/epoch, loss=0.000338, prev_loss=0.00034] 2021-09-16:19:20:19,895 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpv3xoesyb' 2021-09-16:19:20:19,939 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpv3xoesyb' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:44<00:00, 7.71s/epoch, loss=0.000471, prev_loss=0.000471] Training epochs on cuda: 100%|██████████| 10/10 [00:44<00:00, 4.47s/epoch, loss=0.000471, prev_loss=0.000471] 2021-09-16:19:21:20,36 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpycxe2ljd' 2021-09-16:19:21:20,118 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpycxe2ljd' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:53<00:00, 8.57s/epoch, loss=0.000732, prev_loss=0.000754] Training epochs on cuda: 100%|██████████| 10/10 [00:53<00:00, 5.33s/epoch, loss=0.000732, prev_loss=0.000754] 2021-09-16:19:22:28,851 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpex7zmk75' 2021-09-16:19:22:28,906 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpex7zmk75' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:47<00:00, 7.95s/epoch, loss=5.76e-7, prev_loss=5.6e-7] Training epochs on cuda: 100%|██████████| 10/10 [00:47<00:00, 4.70s/epoch, loss=5.76e-7, prev_loss=5.6e-7] 2021-09-16:19:23:31,301 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpw9y9gdp1' 2021-09-16:19:23:31,356 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpw9y9gdp1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:40<00:00, 7.29s/epoch, loss=0.000167, prev_loss=0.00017] Training epochs on cuda: 100%|██████████| 10/10 [00:40<00:00, 4.06s/epoch, loss=0.000167, prev_loss=0.00017] 2021-09-16:19:24:27,237 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp4736tqos' 2021-09-16:19:24:27,297 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp4736tqos' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:40<00:00, 7.25s/epoch, loss=0.000232, prev_loss=0.000269] Training epochs on cuda: 100%|██████████| 10/10 [00:40<00:00, 4.02s/epoch, loss=0.000232, prev_loss=0.000269] 2021-09-16:19:25:22,727 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpi3kugdct' 2021-09-16:19:25:22,789 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpi3kugdct' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00, 7.09s/epoch, loss=0.000522, prev_loss=0.000522] Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00, 3.90s/epoch, loss=0.000522, prev_loss=0.000522] 2021-09-16:19:26:17,98 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmplzc2gheg' 2021-09-16:19:26:17,155 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmplzc2gheg' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:46<00:00, 7.87s/epoch, loss=0.000206, prev_loss=0.000207] Training epochs on cuda: 100%|██████████| 10/10 [00:46<00:00, 4.63s/epoch, loss=0.000206, prev_loss=0.000207] 2021-09-16:19:27:18,755 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_8vk5k79' 2021-09-16:19:27:18,831 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_8vk5k79' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:47<00:00, 8.01s/epoch, loss=0.000654, prev_loss=0.000657] Training epochs on cuda: 100%|██████████| 10/10 [00:47<00:00, 4.74s/epoch, loss=0.000654, prev_loss=0.000657] 2021-09-16:19:28:21,540 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpe2j_yfvt' 2021-09-16:19:28:21,641 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpe2j_yfvt' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:44<00:00, 7.66s/epoch, loss=0.000542, prev_loss=0.000543] Training epochs on cuda: 100%|██████████| 10/10 [00:44<00:00, 4.40s/epoch, loss=0.000542, prev_loss=0.000543] 2021-09-16:19:29:20,893 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpa942yjpq' 2021-09-16:19:29:20,967 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpa942yjpq' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00, 7.75s/epoch, loss=0.000175, prev_loss=0.000252] Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00, 4.50s/epoch, loss=0.000175, prev_loss=0.000252] 2021-09-16:19:30:21,399 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpax7o0sxa' 2021-09-16:19:30:21,474 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpax7o0sxa' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:54<00:00, 8.64s/epoch, loss=0.000387, prev_loss=0.000387] Training epochs on cuda: 100%|██████████| 10/10 [00:54<00:00, 5.42s/epoch, loss=0.000387, prev_loss=0.000387] 2021-09-16:19:31:30,926 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpk3pak348' 2021-09-16:19:31:30,997 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpk3pak348' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:44<00:00, 7.64s/epoch, loss=0.000584, prev_loss=0.000584] Training epochs on cuda: 100%|██████████| 10/10 [00:44<00:00, 4.41s/epoch, loss=0.000584, prev_loss=0.000584] 2021-09-16:19:32:30,360 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmplf8lgr11' 2021-09-16:19:32:30,452 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmplf8lgr11' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 8.88s/epoch, loss=0.000124, prev_loss=0.000125] Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 5.63s/epoch, loss=0.000124, prev_loss=0.000125] 2021-09-16:19:33:42,107 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpoy41auci' 2021-09-16:19:33:42,189 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpoy41auci' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:54<00:00, 8.66s/epoch, loss=0.000129, prev_loss=0.000131] Training epochs on cuda: 100%|██████████| 10/10 [00:54<00:00, 5.47s/epoch, loss=0.000129, prev_loss=0.000131] 2021-09-16:19:34:52,206 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp9dw9zlch' 2021-09-16:19:34:52,281 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp9dw9zlch' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 8.89s/epoch, loss=0.000248, prev_loss=0.000246] Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 5.72s/epoch, loss=0.000248, prev_loss=0.000246] 2021-09-16:19:36:04,795 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmph5x2xxsj' 2021-09-16:19:36:04,848 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmph5x2xxsj' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 9.38s/epoch, loss=0.000113, prev_loss=0.000113] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.16s/epoch, loss=0.000113, prev_loss=0.000113] 2021-09-16:19:37:21,796 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmppdllbqqa' 2021-09-16:19:37:21,874 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmppdllbqqa' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 9.27s/epoch, loss=0.000462, prev_loss=0.000463] Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 6.07s/epoch, loss=0.000462, prev_loss=0.000463] 2021-09-16:19:38:37,821 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1d9pn36v' 2021-09-16:19:38:37,897 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1d9pn36v' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:55<00:00, 8.74s/epoch, loss=0.000232, prev_loss=0.000253] Training epochs on cuda: 100%|██████████| 10/10 [00:55<00:00, 5.50s/epoch, loss=0.000232, prev_loss=0.000253] 2021-09-16:19:39:48,273 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpz1v182gv' 2021-09-16:19:39:48,343 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpz1v182gv' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 9.97s/epoch, loss=0.000363, prev_loss=0.000364] Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 6.71s/epoch, loss=0.000363, prev_loss=0.000364] 2021-09-16:19:41:10,779 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp65cy9v2u' 2021-09-16:19:41:10,873 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp65cy9v2u' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 9.38s/epoch, loss=0.000171, prev_loss=0.00018] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.16s/epoch, loss=0.000171, prev_loss=0.00018] 2021-09-16:19:42:27,802 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpbfpse_30' 2021-09-16:19:42:27,883 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpbfpse_30' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:52<00:00, 8.49s/epoch, loss=0.000342, prev_loss=0.000343] Training epochs on cuda: 100%|██████████| 10/10 [00:52<00:00, 5.28s/epoch, loss=0.000342, prev_loss=0.000343] 2021-09-16:19:43:36,35 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpl8f8aq1t' 2021-09-16:19:43:36,84 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpl8f8aq1t' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 9.59s/epoch, loss=0.000328, prev_loss=0.000328] Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 6.35s/epoch, loss=0.000328, prev_loss=0.000328] 2021-09-16:19:44:54,996 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpd14dxwlk' 2021-09-16:19:44:55,77 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpd14dxwlk' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 10.70s/epoch, loss=3.36e-7, prev_loss=3.28e-7] Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 7.47s/epoch, loss=3.36e-7, prev_loss=3.28e-7] 2021-09-16:19:46:25,167 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpcdv83kjm' 2021-09-16:19:46:25,245 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpcdv83kjm' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 9.41s/epoch, loss=0.000242, prev_loss=0.000247] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.16s/epoch, loss=0.000242, prev_loss=0.000247] 2021-09-16:19:47:42,180 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp8sc9dick' 2021-09-16:19:47:42,256 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp8sc9dick' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:53<00:00, 8.60s/epoch, loss=0.000194, prev_loss=0.000213] Training epochs on cuda: 100%|██████████| 10/10 [00:53<00:00, 5.33s/epoch, loss=0.000194, prev_loss=0.000213] 2021-09-16:19:48:50,866 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_4ftbm77' 2021-09-16:19:48:50,908 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_4ftbm77' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:51<00:00, 8.41s/epoch, loss=0.000208, prev_loss=0.000222] Training epochs on cuda: 100%|██████████| 10/10 [00:51<00:00, 5.20s/epoch, loss=0.000208, prev_loss=0.000222] 2021-09-16:19:49:58,187 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpjvpda6ic' 2021-09-16:19:49:58,241 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpjvpda6ic' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:19<00:00, 11.21s/epoch, loss=0.000104, prev_loss=0.000103] Training epochs on cuda: 100%|██████████| 10/10 [01:19<00:00, 7.96s/epoch, loss=0.000104, prev_loss=0.000103] 2021-09-16:19:51:33,137 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpnxor8kvb' 2021-09-16:19:51:33,184 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpnxor8kvb' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:18<00:00, 11.11s/epoch, loss=0.000258, prev_loss=0.000283] Training epochs on cuda: 100%|██████████| 10/10 [01:18<00:00, 7.86s/epoch, loss=0.000258, prev_loss=0.000283] 2021-09-16:19:53:07,133 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpy7ogmb5a' 2021-09-16:19:53:07,187 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpy7ogmb5a' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:30<00:00, 12.23s/epoch, loss=0.000189, prev_loss=0.000182] Training epochs on cuda: 100%|██████████| 10/10 [01:30<00:00, 9.06s/epoch, loss=0.000189, prev_loss=0.000182] 2021-09-16:19:54:53,195 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpgyl3gad9' 2021-09-16:19:54:53,246 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpgyl3gad9' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:17<00:00, 10.99s/epoch, loss=0.000309, prev_loss=0.000312] Training epochs on cuda: 100%|██████████| 10/10 [01:17<00:00, 7.72s/epoch, loss=0.000309, prev_loss=0.000312] 2021-09-16:19:56:25,815 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp81sq1he6' 2021-09-16:19:56:25,867 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp81sq1he6' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:28<00:00, 12.11s/epoch, loss=0.000284, prev_loss=0.000306] Training epochs on cuda: 100%|██████████| 10/10 [01:28<00:00, 8.90s/epoch, loss=0.000284, prev_loss=0.000306] 2021-09-16:19:58:10,302 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmptwgg14_k' 2021-09-16:19:58:10,353 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmptwgg14_k' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 9.66s/epoch, loss=0.000414, prev_loss=0.000414] Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 6.51s/epoch, loss=0.000414, prev_loss=0.000414] 2021-09-16:19:59:30,870 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpqe50_b_h' 2021-09-16:19:59:30,922 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpqe50_b_h' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:30<00:00, 18.41s/epoch, loss=0.00762, prev_loss=0.00764] Training epochs on cuda: 100%|██████████| 10/10 [02:30<00:00, 15.07s/epoch, loss=0.00762, prev_loss=0.00764] 2021-09-16:20:02:16,785 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpkuyyqjkc' 2021-09-16:20:02:16,844 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpkuyyqjkc' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 9.58s/epoch, loss=0.000304, prev_loss=0.00031] Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 6.34s/epoch, loss=0.000304, prev_loss=0.00031] 2021-09-16:20:03:35,439 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpm_tukck7' 2021-09-16:20:03:35,465 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpm_tukck7' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:12<00:00, 10.44s/epoch, loss=0.00126, prev_loss=0.00119] Training epochs on cuda: 100%|██████████| 10/10 [01:12<00:00, 7.30s/epoch, loss=0.00126, prev_loss=0.00119] 2021-09-16:20:05:03,580 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpi2jgs5f6' 2021-09-16:20:05:03,632 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpi2jgs5f6' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 9.80s/epoch, loss=0.000284, prev_loss=0.000287] Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 6.52s/epoch, loss=0.000284, prev_loss=0.000287] 2021-09-16:20:06:24,83 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpi74t2_bm' 2021-09-16:20:06:24,135 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpi74t2_bm' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 10.40s/epoch, loss=0.00108, prev_loss=0.00108] Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 7.15s/epoch, loss=0.00108, prev_loss=0.00108] 2021-09-16:20:07:50,847 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpty73_krf' 2021-09-16:20:07:50,898 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpty73_krf' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:21<00:00, 11.38s/epoch, loss=0.000657, prev_loss=0.000676] Training epochs on cuda: 100%|██████████| 10/10 [01:21<00:00, 8.17s/epoch, loss=0.000657, prev_loss=0.000676] 2021-09-16:20:09:27,853 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpl8fwo0rf' 2021-09-16:20:09:27,905 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpl8fwo0rf' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 9.85s/epoch, loss=0.000293, prev_loss=0.000283] Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 6.67s/epoch, loss=0.000293, prev_loss=0.000283] 2021-09-16:20:10:49,872 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpmhc3jpiu' 2021-09-16:20:10:49,924 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpmhc3jpiu' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 9.72s/epoch, loss=0.00138, prev_loss=0.00138] Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 6.56s/epoch, loss=0.00138, prev_loss=0.00138] 2021-09-16:20:12:10,730 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpj32p1irg' 2021-09-16:20:12:10,782 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpj32p1irg' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00, 10.89s/epoch, loss=0.000439, prev_loss=0.000441] Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00, 7.69s/epoch, loss=0.000439, prev_loss=0.000441] 2021-09-16:20:13:43,42 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpz3obmpzn' 2021-09-16:20:13:43,89 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpz3obmpzn' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 9.64s/epoch, loss=0.00161, prev_loss=0.00161] Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 6.35s/epoch, loss=0.00161, prev_loss=0.00161] 2021-09-16:20:15:01,872 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1xs2a8zs' 2021-09-16:20:15:01,956 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1xs2a8zs' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 9.98s/epoch, loss=0.00089, prev_loss=0.00089] Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 6.69s/epoch, loss=0.00089, prev_loss=0.00089] 2021-09-16:20:16:24,147 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpcxyjgg56' 2021-09-16:20:16:24,207 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpcxyjgg56' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:28<00:00, 12.14s/epoch, loss=0.000228, prev_loss=0.000234] Training epochs on cuda: 100%|██████████| 10/10 [01:28<00:00, 8.83s/epoch, loss=0.000228, prev_loss=0.000234] 2021-09-16:20:18:07,828 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp4ym0ds2n' 2021-09-16:20:18:07,898 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp4ym0ds2n' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:20<00:00, 11.27s/epoch, loss=0.00031, prev_loss=0.000312] Training epochs on cuda: 100%|██████████| 10/10 [01:20<00:00, 8.01s/epoch, loss=0.00031, prev_loss=0.000312] 2021-09-16:20:19:43,264 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpljj7riix' 2021-09-16:20:19:43,329 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpljj7riix' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:26<00:00, 11.91s/epoch, loss=0.000282, prev_loss=0.000288] Training epochs on cuda: 100%|██████████| 10/10 [01:26<00:00, 8.70s/epoch, loss=0.000282, prev_loss=0.000288] 2021-09-16:20:21:25,644 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpjqzhcdw4' 2021-09-16:20:21:25,715 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpjqzhcdw4' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:25<00:00, 11.73s/epoch, loss=0.000592, prev_loss=0.000649] Training epochs on cuda: 100%|██████████| 10/10 [01:25<00:00, 8.57s/epoch, loss=0.000592, prev_loss=0.000649] 2021-09-16:20:23:06,747 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpt3to03h7' 2021-09-16:20:23:06,839 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpt3to03h7' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:37<00:00, 12.99s/epoch, loss=0.000354, prev_loss=0.000367] Training epochs on cuda: 100%|██████████| 10/10 [01:37<00:00, 9.75s/epoch, loss=0.000354, prev_loss=0.000367] 2021-09-16:20:24:59,690 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpu2bw8p04' 2021-09-16:20:24:59,765 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpu2bw8p04' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:21<00:00, 11.33s/epoch, loss=0.000278, prev_loss=0.000273] Training epochs on cuda: 100%|██████████| 10/10 [01:21<00:00, 8.12s/epoch, loss=0.000278, prev_loss=0.000273] 2021-09-16:20:26:36,359 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1sj2r_tx' 2021-09-16:20:26:36,425 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1sj2r_tx' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:22<00:00, 11.81s/epoch, loss=0.00124, prev_loss=0.00124] Training epochs on cuda: 100%|██████████| 10/10 [01:22<00:00, 8.30s/epoch, loss=0.00124, prev_loss=0.00124] 2021-09-16:20:28:14,618 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1kfoh8p1' 2021-09-16:20:28:14,684 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1kfoh8p1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:22<00:00, 11.42s/epoch, loss=0.000697, prev_loss=0.000861] Training epochs on cuda: 100%|██████████| 10/10 [01:22<00:00, 8.21s/epoch, loss=0.000697, prev_loss=0.000861] 2021-09-16:20:29:52,168 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpoo5r_5xc' 2021-09-16:20:29:52,263 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpoo5r_5xc' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00, 10.77s/epoch, loss=0.000677, prev_loss=0.000791] Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00, 7.62s/epoch, loss=0.000677, prev_loss=0.000791] 2021-09-16:20:31:23,737 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpocwbtupg' 2021-09-16:20:31:23,820 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpocwbtupg' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:12<00:00, 10.44s/epoch, loss=0.00108, prev_loss=0.00108] Training epochs on cuda: 100%|██████████| 10/10 [01:12<00:00, 7.24s/epoch, loss=0.00108, prev_loss=0.00108] 2021-09-16:20:32:51,405 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpddfq_icd' 2021-09-16:20:32:51,471 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpddfq_icd' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 9.96s/epoch, loss=0.00031, prev_loss=0.000298] Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 6.74s/epoch, loss=0.00031, prev_loss=0.000298] 2021-09-16:20:34:14,185 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpfx5e74kp' 2021-09-16:20:34:14,256 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfx5e74kp' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:28<00:00, 12.03s/epoch, loss=0.000315, prev_loss=0.000311] Training epochs on cuda: 100%|██████████| 10/10 [01:28<00:00, 8.81s/epoch, loss=0.000315, prev_loss=0.000311] 2021-09-16:20:35:57,790 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmprynmc929' 2021-09-16:20:35:57,863 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmprynmc929' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:27<00:00, 12.06s/epoch, loss=0.000516, prev_loss=0.000684] Training epochs on cuda: 100%|██████████| 10/10 [01:27<00:00, 8.77s/epoch, loss=0.000516, prev_loss=0.000684] 2021-09-16:20:37:40,929 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp91spm278' 2021-09-16:20:37:40,988 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp91spm278' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:20<00:00, 11.24s/epoch, loss=0.000257, prev_loss=0.000251] Training epochs on cuda: 100%|██████████| 10/10 [01:20<00:00, 8.01s/epoch, loss=0.000257, prev_loss=0.000251] 2021-09-16:20:39:16,291 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpa5w_dnwo' 2021-09-16:20:39:16,363 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpa5w_dnwo' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:21<00:00, 11.41s/epoch, loss=0.000209, prev_loss=0.000208] Training epochs on cuda: 100%|██████████| 10/10 [01:21<00:00, 8.17s/epoch, loss=0.000209, prev_loss=0.000208] 2021-09-16:20:40:53,354 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpp2oab7mt' 2021-09-16:20:40:53,427 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpp2oab7mt' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00, 11.53s/epoch, loss=0.000394, prev_loss=0.000398] Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00, 8.32s/epoch, loss=0.000394, prev_loss=0.000398] 2021-09-16:20:42:31,924 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpoekb9w_s' 2021-09-16:20:42:32,1 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpoekb9w_s' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:15<00:00, 10.84s/epoch, loss=0.000317, prev_loss=0.000319] Training epochs on cuda: 100%|██████████| 10/10 [01:15<00:00, 7.58s/epoch, loss=0.000317, prev_loss=0.000319] 2021-09-16:20:44:03,138 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpilxrbj95' 2021-09-16:20:44:03,215 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpilxrbj95' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:27<00:00, 12.06s/epoch, loss=0.00266, prev_loss=0.00266] Training epochs on cuda: 100%|██████████| 10/10 [01:27<00:00, 8.74s/epoch, loss=0.00266, prev_loss=0.00266] 2021-09-16:20:45:45,936 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpqv0nfi1r' 2021-09-16:20:45:46,13 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpqv0nfi1r' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 10.68s/epoch, loss=0.000757, prev_loss=0.000807] Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 7.43s/epoch, loss=0.000757, prev_loss=0.000807] 2021-09-16:20:47:15,654 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmptebzywhm' 2021-09-16:20:47:15,739 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmptebzywhm' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:13<00:00, 10.53s/epoch, loss=0.000889, prev_loss=0.000889] Training epochs on cuda: 100%|██████████| 10/10 [01:13<00:00, 7.33s/epoch, loss=0.000889, prev_loss=0.000889] 2021-09-16:20:48:44,345 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpkm7_16x0' 2021-09-16:20:48:44,407 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpkm7_16x0' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00, 11.56s/epoch, loss=0.000421, prev_loss=0.000417] Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00, 8.31s/epoch, loss=0.000421, prev_loss=0.000417] 2021-09-16:20:50:22,963 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmptzlel7ju' 2021-09-16:20:50:23,39 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmptzlel7ju' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 9.49s/epoch, loss=0.000284, prev_loss=0.000283] Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 6.24s/epoch, loss=0.000284, prev_loss=0.000283] 2021-09-16:20:51:40,700 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpkflr0nhy' 2021-09-16:20:51:40,755 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpkflr0nhy' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 10.02s/epoch, loss=0.000291, prev_loss=0.0003] Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 6.78s/epoch, loss=0.000291, prev_loss=0.0003] 2021-09-16:20:53:03,859 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpr_up0g42' 2021-09-16:20:53:03,913 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpr_up0g42' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:26<00:00, 11.86s/epoch, loss=0.000411, prev_loss=0.000408] Training epochs on cuda: 100%|██████████| 10/10 [01:26<00:00, 8.65s/epoch, loss=0.000411, prev_loss=0.000408] 2021-09-16:20:54:45,809 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp9hacx102' 2021-09-16:20:54:45,881 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp9hacx102' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 10.32s/epoch, loss=0.000628, prev_loss=0.00063] Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 7.17s/epoch, loss=0.000628, prev_loss=0.00063] 2021-09-16:20:56:12,846 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpd0yef468' 2021-09-16:20:56:12,908 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpd0yef468' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:32<00:00, 12.47s/epoch, loss=0.00111, prev_loss=0.00112] Training epochs on cuda: 100%|██████████| 10/10 [01:32<00:00, 9.20s/epoch, loss=0.00111, prev_loss=0.00112] 2021-09-16:20:58:00,322 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpl4c55kpk' 2021-09-16:20:58:00,379 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpl4c55kpk' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:18<00:00, 11.17s/epoch, loss=0.000256, prev_loss=0.000257] Training epochs on cuda: 100%|██████████| 10/10 [01:18<00:00, 7.89s/epoch, loss=0.000256, prev_loss=0.000257] 2021-09-16:20:59:34,694 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpq5eor345' 2021-09-16:20:59:34,766 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpq5eor345' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:32<00:00, 12.50s/epoch, loss=0.000315, prev_loss=0.000403] Training epochs on cuda: 100%|██████████| 10/10 [01:32<00:00, 9.27s/epoch, loss=0.000315, prev_loss=0.000403] 2021-09-16:21:01:23,34 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpp7g3z73v' 2021-09-16:21:01:23,98 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpp7g3z73v' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:13<00:00, 10.60s/epoch, loss=0.000818, prev_loss=0.000818] Training epochs on cuda: 100%|██████████| 10/10 [01:13<00:00, 7.36s/epoch, loss=0.000818, prev_loss=0.000818] 2021-09-16:21:02:52,82 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpmgxstjyf' 2021-09-16:21:02:52,163 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpmgxstjyf' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:48<00:00, 14.10s/epoch, loss=0.000227, prev_loss=0.000229] Training epochs on cuda: 100%|██████████| 10/10 [01:48<00:00, 10.81s/epoch, loss=0.000227, prev_loss=0.000229] 2021-09-16:21:04:55,690 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpwfgn8jhu' 2021-09-16:21:04:55,751 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpwfgn8jhu' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:25<00:00, 11.81s/epoch, loss=0.00027, prev_loss=0.000263] Training epochs on cuda: 100%|██████████| 10/10 [01:25<00:00, 8.59s/epoch, loss=0.00027, prev_loss=0.000263] 2021-09-16:21:06:36,981 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp3mqpdwol' 2021-09-16:21:06:37,47 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp3mqpdwol' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:27<00:00, 12.12s/epoch, loss=0.000343, prev_loss=0.000344] Training epochs on cuda: 100%|██████████| 10/10 [01:27<00:00, 8.79s/epoch, loss=0.000343, prev_loss=0.000344] 2021-09-16:21:08:20,465 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpmtty87ub' 2021-09-16:21:08:20,533 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpmtty87ub' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 10.60s/epoch, loss=0.000532, prev_loss=0.000506] Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 7.41s/epoch, loss=0.000532, prev_loss=0.000506] 2021-09-16:21:09:49,940 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpr0ywf035' 2021-09-16:21:09:50,14 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpr0ywf035' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 9.32s/epoch, loss=0.000302, prev_loss=0.000302] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.19s/epoch, loss=0.000302, prev_loss=0.000302] 2021-09-16:21:11:07,275 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpytzrxxvt' 2021-09-16:21:11:07,327 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpytzrxxvt' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:30<00:00, 12.28s/epoch, loss=0.000696, prev_loss=0.00087] Training epochs on cuda: 100%|██████████| 10/10 [01:30<00:00, 9.02s/epoch, loss=0.000696, prev_loss=0.00087] 2021-09-16:21:12:53,2 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp55t0tzvo' 2021-09-16:21:12:53,42 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp55t0tzvo' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 9.93s/epoch, loss=0.000406, prev_loss=0.000428] Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 6.66s/epoch, loss=0.000406, prev_loss=0.000428] 2021-09-16:21:14:15,16 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpeh5tjudh' 2021-09-16:21:14:15,86 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpeh5tjudh' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:51<00:00, 8.42s/epoch, loss=0.000344, prev_loss=0.00035] Training epochs on cuda: 100%|██████████| 10/10 [00:51<00:00, 5.18s/epoch, loss=0.000344, prev_loss=0.00035] 2021-09-16:21:15:22,297 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpfmb7evsi' 2021-09-16:21:15:22,376 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfmb7evsi' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:49<00:00, 14.19s/epoch, loss=9.07e-5, prev_loss=9.14e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:49<00:00, 10.94s/epoch, loss=9.07e-5, prev_loss=9.14e-5] 2021-09-16:21:17:27,310 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpqoij25_o' 2021-09-16:21:17:27,377 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpqoij25_o' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 10.79s/epoch, loss=0.00121, prev_loss=0.00121] Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 7.44s/epoch, loss=0.00121, prev_loss=0.00121] 2021-09-16:21:18:57,147 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpx3yna5fu' 2021-09-16:21:18:57,235 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpx3yna5fu' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:15<00:00, 10.79s/epoch, loss=0.000424, prev_loss=0.000424] Training epochs on cuda: 100%|██████████| 10/10 [01:15<00:00, 7.58s/epoch, loss=0.000424, prev_loss=0.000424] 2021-09-16:21:20:28,430 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp70hlaugp' 2021-09-16:21:20:28,495 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp70hlaugp' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:40<00:00, 13.17s/epoch, loss=0.000151, prev_loss=0.000172] Training epochs on cuda: 100%|██████████| 10/10 [01:40<00:00, 10.03s/epoch, loss=0.000151, prev_loss=0.000172] 2021-09-16:21:22:24,177 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpxr5cghko' 2021-09-16:21:22:24,248 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpxr5cghko' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 16.83s/epoch, loss=7.48e-5, prev_loss=7.66e-5] Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 13.70s/epoch, loss=7.48e-5, prev_loss=7.66e-5] 2021-09-16:21:24:57,4 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpzlre6y44' 2021-09-16:21:24:57,72 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpzlre6y44' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 8.83s/epoch, loss=9.59e-5, prev_loss=0.000101] Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 5.62s/epoch, loss=9.59e-5, prev_loss=0.000101] 2021-09-16:21:26:08,664 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmptssmtks5' 2021-09-16:21:26:08,725 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmptssmtks5' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 9.61s/epoch, loss=8.42e-5, prev_loss=8.8e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 6.35s/epoch, loss=8.42e-5, prev_loss=8.8e-5] 2021-09-16:21:27:27,600 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_gizx7t2' 2021-09-16:21:27:27,650 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_gizx7t2' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 9.50s/epoch, loss=0.000323, prev_loss=0.000357] Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 6.26s/epoch, loss=0.000323, prev_loss=0.000357] 2021-09-16:21:28:45,602 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpzkw59aea' 2021-09-16:21:28:45,740 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpzkw59aea' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 9.06s/epoch, loss=0.000217, prev_loss=0.000246] Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 5.76s/epoch, loss=0.000217, prev_loss=0.000246] 2021-09-16:21:29:58,647 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpzhktam_0' 2021-09-16:21:29:58,727 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpzhktam_0' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 9.03s/epoch, loss=0.000364, prev_loss=0.000391] Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 5.74s/epoch, loss=0.000364, prev_loss=0.000391] 2021-09-16:21:31:11,578 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpv3d9vte_' 2021-09-16:21:31:11,661 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpv3d9vte_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 8.87s/epoch, loss=0.000336, prev_loss=0.000336] Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 5.65s/epoch, loss=0.000336, prev_loss=0.000336] 2021-09-16:21:32:23,574 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp4ffdnate' 2021-09-16:21:32:23,637 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp4ffdnate' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:08<00:00, 10.15s/epoch, loss=0.000206, prev_loss=0.000244] Training epochs on cuda: 100%|██████████| 10/10 [01:08<00:00, 6.87s/epoch, loss=0.000206, prev_loss=0.000244] 2021-09-16:21:33:47,691 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpt_2ftx0d' 2021-09-16:21:33:47,761 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpt_2ftx0d' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 10.71s/epoch, loss=0.000153, prev_loss=0.000168] Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 7.50s/epoch, loss=0.000153, prev_loss=0.000168] 2021-09-16:21:35:18,179 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp4bbq0haf' 2021-09-16:21:35:18,242 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp4bbq0haf' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 9.24s/epoch, loss=0.000104, prev_loss=0.000107] Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 6.06s/epoch, loss=0.000104, prev_loss=0.000107] 2021-09-16:21:36:34,213 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp5hhaxbk3' 2021-09-16:21:36:34,281 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp5hhaxbk3' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 9.34s/epoch, loss=0.000179, prev_loss=0.000199] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.13s/epoch, loss=0.000179, prev_loss=0.000199] 2021-09-16:21:37:50,954 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp3hsb3356' 2021-09-16:21:37:51,19 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp3hsb3356' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 8.93s/epoch, loss=0.000108, prev_loss=0.000111] Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 5.75s/epoch, loss=0.000108, prev_loss=0.000111] 2021-09-16:21:39:03,811 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp79s1vdz7' 2021-09-16:21:39:03,879 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp79s1vdz7' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 8.94s/epoch, loss=0.000322, prev_loss=0.000405] Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 5.73s/epoch, loss=0.000322, prev_loss=0.000405] 2021-09-16:21:40:16,515 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp6hl2t3v3' 2021-09-16:21:40:16,607 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6hl2t3v3' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 8.68s/epoch, loss=0.000116, prev_loss=0.000119] Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 5.67s/epoch, loss=0.000116, prev_loss=0.000119] 2021-09-16:21:41:28,724 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmphgw1bwv8' 2021-09-16:21:41:28,776 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmphgw1bwv8' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:53<00:00, 8.55s/epoch, loss=0.00041, prev_loss=0.000424] Training epochs on cuda: 100%|██████████| 10/10 [00:53<00:00, 5.38s/epoch, loss=0.00041, prev_loss=0.000424] 2021-09-16:21:42:37,905 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_4lm1ig1' 2021-09-16:21:42:37,957 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_4lm1ig1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.17s/epoch, loss=0.000194, prev_loss=0.000236] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.94s/epoch, loss=0.000194, prev_loss=0.000236] 2021-09-16:21:43:52,690 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp5ac5z387' 2021-09-16:21:43:52,741 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp5ac5z387' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:52<00:00, 8.45s/epoch, loss=0.000111, prev_loss=0.000127] Training epochs on cuda: 100%|██████████| 10/10 [00:52<00:00, 5.24s/epoch, loss=0.000111, prev_loss=0.000127] 2021-09-16:21:45:00,457 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpebofqgog' 2021-09-16:21:45:00,508 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpebofqgog' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:52<00:00, 8.54s/epoch, loss=0.000426, prev_loss=0.000442] Training epochs on cuda: 100%|██████████| 10/10 [00:52<00:00, 5.29s/epoch, loss=0.000426, prev_loss=0.000442] 2021-09-16:21:46:08,682 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1z98smg9' 2021-09-16:21:46:08,732 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1z98smg9' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 9.08s/epoch, loss=0.000494, prev_loss=0.000506] Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 5.87s/epoch, loss=0.000494, prev_loss=0.000506] 2021-09-16:21:47:22,641 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpl94f6pnc' 2021-09-16:21:47:22,684 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpl94f6pnc' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:54<00:00, 8.66s/epoch, loss=0.000119, prev_loss=0.000119] Training epochs on cuda: 100%|██████████| 10/10 [00:54<00:00, 5.42s/epoch, loss=0.000119, prev_loss=0.000119] 2021-09-16:21:48:32,169 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp3geonrih' 2021-09-16:21:48:32,221 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp3geonrih' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:47<00:00, 7.97s/epoch, loss=0.000421, prev_loss=0.000431] Training epochs on cuda: 100%|██████████| 10/10 [00:47<00:00, 4.72s/epoch, loss=0.000421, prev_loss=0.000431] 2021-09-16:21:49:34,809 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpdqqfm_33' 2021-09-16:21:49:34,860 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdqqfm_33' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:51<00:00, 8.34s/epoch, loss=0.000501, prev_loss=0.000501] Training epochs on cuda: 100%|██████████| 10/10 [00:51<00:00, 5.11s/epoch, loss=0.000501, prev_loss=0.000501] 2021-09-16:21:50:41,331 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpfshrqolw' 2021-09-16:21:50:41,370 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfshrqolw' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:54<00:00, 8.70s/epoch, loss=0.000104, prev_loss=0.000109] Training epochs on cuda: 100%|██████████| 10/10 [00:54<00:00, 5.49s/epoch, loss=0.000104, prev_loss=0.000109] 2021-09-16:21:51:51,675 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmps7x3jzaz' 2021-09-16:21:51:51,726 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmps7x3jzaz' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 8.89s/epoch, loss=0.000197, prev_loss=0.00025] Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 5.68s/epoch, loss=0.000197, prev_loss=0.00025] 2021-09-16:21:53:03,877 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpj1ghn0cj' 2021-09-16:21:53:03,928 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpj1ghn0cj' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:53<00:00, 8.61s/epoch, loss=0.000468, prev_loss=0.000469] Training epochs on cuda: 100%|██████████| 10/10 [00:53<00:00, 5.37s/epoch, loss=0.000468, prev_loss=0.000469] 2021-09-16:21:54:13,17 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1qacy_jj' 2021-09-16:21:54:13,69 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1qacy_jj' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:51<00:00, 8.41s/epoch, loss=0.000183, prev_loss=0.000213] Training epochs on cuda: 100%|██████████| 10/10 [00:51<00:00, 5.20s/epoch, loss=0.000183, prev_loss=0.000213] 2021-09-16:21:55:20,392 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp9dxftxyi' 2021-09-16:21:55:20,443 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp9dxftxyi' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 9.54s/epoch, loss=0.000115, prev_loss=0.000126] Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 6.28s/epoch, loss=0.000115, prev_loss=0.000126] 2021-09-16:21:56:38,659 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp7g6jsobv' 2021-09-16:21:56:38,711 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp7g6jsobv' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.20s/epoch, loss=0.000456, prev_loss=0.000456] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.90s/epoch, loss=0.000456, prev_loss=0.000456] 2021-09-16:21:57:53,61 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmptwh2jmp3' 2021-09-16:21:57:53,121 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmptwh2jmp3' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 9.40s/epoch, loss=0.000121, prev_loss=0.000128] Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 6.22s/epoch, loss=0.000121, prev_loss=0.000128] 2021-09-16:21:59:10,662 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1m8g0k0p' 2021-09-16:21:59:10,718 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1m8g0k0p' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 9.23s/epoch, loss=0.000194, prev_loss=0.000205] Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 6.03s/epoch, loss=0.000194, prev_loss=0.000205] 2021-09-16:22:00:26,348 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp52ap9myp' 2021-09-16:22:00:26,400 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp52ap9myp' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 9.71s/epoch, loss=0.000115, prev_loss=0.000118] Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 6.46s/epoch, loss=0.000115, prev_loss=0.000118] 2021-09-16:22:01:46,420 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpobzd_xbw' 2021-09-16:22:01:46,472 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpobzd_xbw' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 9.57s/epoch, loss=0.000466, prev_loss=0.000471] Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 6.31s/epoch, loss=0.000466, prev_loss=0.000471] 2021-09-16:22:03:04,902 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpjv8z_omn' 2021-09-16:22:03:04,959 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpjv8z_omn' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.10s/epoch, loss=0.000238, prev_loss=0.000252] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.93s/epoch, loss=0.000238, prev_loss=0.000252] 2021-09-16:22:04:19,593 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpra54zbe0' 2021-09-16:22:04:19,645 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpra54zbe0' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.15s/epoch, loss=0.000103, prev_loss=0.000106] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.91s/epoch, loss=0.000103, prev_loss=0.000106] 2021-09-16:22:05:34,58 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpu564b93z' 2021-09-16:22:05:34,88 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpu564b93z' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.13s/epoch, loss=0.000106, prev_loss=0.000112] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.95s/epoch, loss=0.000106, prev_loss=0.000112] 2021-09-16:22:06:48,966 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_o42a4yr' 2021-09-16:22:06:49,18 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_o42a4yr' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 9.12s/epoch, loss=0.000384, prev_loss=0.000406] Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 5.87s/epoch, loss=0.000384, prev_loss=0.000406] 2021-09-16:22:08:03,125 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpy_255mtm' 2021-09-16:22:08:03,168 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpy_255mtm' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 9.57s/epoch, loss=0.000483, prev_loss=0.000476] Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 6.34s/epoch, loss=0.000483, prev_loss=0.000476] 2021-09-16:22:09:21,937 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpuqhzf5lp' 2021-09-16:22:09:21,990 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpuqhzf5lp' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 9.77s/epoch, loss=0.000139, prev_loss=0.000147] Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 6.55s/epoch, loss=0.000139, prev_loss=0.000147] 2021-09-16:22:10:42,870 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpz5dnt9tk' 2021-09-16:22:10:42,925 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpz5dnt9tk' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 9.57s/epoch, loss=0.000496, prev_loss=0.000508] Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 6.37s/epoch, loss=0.000496, prev_loss=0.000508] 2021-09-16:22:12:02,98 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmphu3o61lp' 2021-09-16:22:12:02,150 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmphu3o61lp' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 9.91s/epoch, loss=0.000108, prev_loss=0.000112] Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 6.76s/epoch, loss=0.000108, prev_loss=0.000112] 2021-09-16:22:13:25,92 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpl9rong3k' 2021-09-16:22:13:25,148 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpl9rong3k' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 9.72s/epoch, loss=0.000148, prev_loss=0.000159] Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 6.48s/epoch, loss=0.000148, prev_loss=0.000159] 2021-09-16:22:14:45,350 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpxab92e3a' 2021-09-16:22:14:45,402 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpxab92e3a' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 9.74s/epoch, loss=0.000479, prev_loss=0.000505] Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 6.47s/epoch, loss=0.000479, prev_loss=0.000505] 2021-09-16:22:16:05,524 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpdbnza09z' 2021-09-16:22:16:05,576 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdbnza09z' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 10.19s/epoch, loss=0.000547, prev_loss=0.000547] Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 6.76s/epoch, loss=0.000547, prev_loss=0.000547] 2021-09-16:22:17:28,468 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpoo10f0g0' 2021-09-16:22:17:28,519 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpoo10f0g0' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 9.06s/epoch, loss=0.000138, prev_loss=0.000144] Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 5.87s/epoch, loss=0.000138, prev_loss=0.000144] 2021-09-16:22:18:42,525 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpbp__cqk3' 2021-09-16:22:18:42,576 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpbp__cqk3' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 9.45s/epoch, loss=0.000423, prev_loss=0.000436] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.19s/epoch, loss=0.000423, prev_loss=0.000436] 2021-09-16:22:19:59,738 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmprpu0x9dh' 2021-09-16:22:19:59,789 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmprpu0x9dh' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 9.63s/epoch, loss=0.000134, prev_loss=0.000146] Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 6.41s/epoch, loss=0.000134, prev_loss=0.000146] 2021-09-16:22:21:19,197 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpdcft036k' 2021-09-16:22:21:19,249 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdcft036k' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 9.60s/epoch, loss=0.000492, prev_loss=0.000492] Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 6.41s/epoch, loss=0.000492, prev_loss=0.000492] 2021-09-16:22:22:38,693 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpt19x9ox2' 2021-09-16:22:22:38,746 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpt19x9ox2' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 9.74s/epoch, loss=0.000577, prev_loss=0.000578] Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 6.51s/epoch, loss=0.000577, prev_loss=0.000578] 2021-09-16:22:23:59,162 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpg4svglov' 2021-09-16:22:23:59,213 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpg4svglov' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.19s/epoch, loss=0.000126, prev_loss=0.000136] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.97s/epoch, loss=0.000126, prev_loss=0.000136] 2021-09-16:22:25:14,243 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp9i_8f_ze' 2021-09-16:22:25:14,288 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp9i_8f_ze' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 9.09s/epoch, loss=0.000116, prev_loss=0.000125] Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 5.89s/epoch, loss=0.000116, prev_loss=0.000125] 2021-09-16:22:26:28,542 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpbb7j5fln' 2021-09-16:22:26:28,595 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpbb7j5fln' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 8.99s/epoch, loss=0.000278, prev_loss=0.000319] Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 5.76s/epoch, loss=0.000278, prev_loss=0.000319] 2021-09-16:22:27:41,540 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmppt2kgflz' 2021-09-16:22:27:41,594 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmppt2kgflz' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 9.37s/epoch, loss=0.000119, prev_loss=0.000127] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.17s/epoch, loss=0.000119, prev_loss=0.000127] 2021-09-16:22:28:58,667 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpx6rp1wg_' 2021-09-16:22:28:58,719 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpx6rp1wg_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 9.34s/epoch, loss=0.00011, prev_loss=0.000114] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.10s/epoch, loss=0.00011, prev_loss=0.000114] 2021-09-16:22:30:15,131 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpaqvml7xf' 2021-09-16:22:30:15,183 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpaqvml7xf' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 9.35s/epoch, loss=0.000422, prev_loss=0.000432] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.15s/epoch, loss=0.000422, prev_loss=0.000432] 2021-09-16:22:31:32,73 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpujcaoj38' 2021-09-16:22:31:32,128 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpujcaoj38' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 9.30s/epoch, loss=0.000291, prev_loss=0.000298] Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 6.04s/epoch, loss=0.000291, prev_loss=0.000298] 2021-09-16:22:32:47,892 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpe1wcal2y' 2021-09-16:22:32:47,944 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpe1wcal2y' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 9.07s/epoch, loss=0.000491, prev_loss=0.000492] Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 5.86s/epoch, loss=0.000491, prev_loss=0.000492] 2021-09-16:22:34:01,800 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp26ljg3b9' 2021-09-16:22:34:01,850 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp26ljg3b9' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 9.66s/epoch, loss=0.000392, prev_loss=0.000399] Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 6.48s/epoch, loss=0.000392, prev_loss=0.000399] 2021-09-16:22:35:22,21 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpi6z93x0i' 2021-09-16:22:35:22,72 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpi6z93x0i' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 8.83s/epoch, loss=0.000514, prev_loss=0.000516] Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 5.63s/epoch, loss=0.000514, prev_loss=0.000516] 2021-09-16:22:36:33,721 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpconyn6si' 2021-09-16:22:36:33,771 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpconyn6si' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 9.31s/epoch, loss=0.0002, prev_loss=0.000229] Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 6.09s/epoch, loss=0.0002, prev_loss=0.000229] 2021-09-16:22:37:49,967 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpnf0zmlz1' 2021-09-16:22:37:50,19 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpnf0zmlz1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 8.86s/epoch, loss=0.000132, prev_loss=0.000187] Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 5.66s/epoch, loss=0.000132, prev_loss=0.000187] 2021-09-16:22:39:02,52 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpapunhhhn' 2021-09-16:22:39:02,107 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpapunhhhn' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 9.07s/epoch, loss=0.000122, prev_loss=0.000127] Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 5.77s/epoch, loss=0.000122, prev_loss=0.000127] 2021-09-16:22:40:15,202 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1fzpxkux' 2021-09-16:22:40:15,255 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1fzpxkux' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 8.95s/epoch, loss=0.00014, prev_loss=0.000158] Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 5.70s/epoch, loss=0.00014, prev_loss=0.000158] 2021-09-16:22:41:27,600 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpg0jv2hdx' 2021-09-16:22:41:27,652 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpg0jv2hdx' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.12s/epoch, loss=0.000474, prev_loss=0.000486] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.91s/epoch, loss=0.000474, prev_loss=0.000486] 2021-09-16:22:42:42,166 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp2xupnl86' 2021-09-16:22:42:42,218 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp2xupnl86' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 8.84s/epoch, loss=0.000486, prev_loss=0.000488] Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 5.64s/epoch, loss=0.000486, prev_loss=0.000488] 2021-09-16:22:43:53,932 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpgal2lani' 2021-09-16:22:43:53,982 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpgal2lani' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 9.44s/epoch, loss=0.000142, prev_loss=0.000147] Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 6.21s/epoch, loss=0.000142, prev_loss=0.000147] 2021-09-16:22:45:11,463 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpucxmltaa' 2021-09-16:22:45:11,515 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpucxmltaa' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 9.28s/epoch, loss=0.000588, prev_loss=0.000606] Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 6.07s/epoch, loss=0.000588, prev_loss=0.000606] 2021-09-16:22:46:27,540 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1nvm4m9q' 2021-09-16:22:46:27,592 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1nvm4m9q' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 8.94s/epoch, loss=0.000147, prev_loss=0.000158] Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 5.71s/epoch, loss=0.000147, prev_loss=0.000158] 2021-09-16:22:47:40,31 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp7bz1ci5v' 2021-09-16:22:47:40,83 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp7bz1ci5v' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 8.97s/epoch, loss=0.000578, prev_loss=0.000578] Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 5.70s/epoch, loss=0.000578, prev_loss=0.000578] 2021-09-16:22:48:52,387 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpuqtl_e9z' 2021-09-16:22:48:52,440 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpuqtl_e9z' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.09s/epoch, loss=0.000136, prev_loss=0.000145] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.90s/epoch, loss=0.000136, prev_loss=0.000145] 2021-09-16:22:50:06,730 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpdppyydru' 2021-09-16:22:50:06,781 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdppyydru' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.17s/epoch, loss=0.000203, prev_loss=0.000226] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.91s/epoch, loss=0.000203, prev_loss=0.000226] 2021-09-16:22:51:21,298 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpbzrsi674' 2021-09-16:22:51:21,350 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpbzrsi674' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 9.34s/epoch, loss=0.000522, prev_loss=0.000524] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.11s/epoch, loss=0.000522, prev_loss=0.000524] 2021-09-16:22:52:37,749 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpzdvfu3d0' 2021-09-16:22:52:37,801 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpzdvfu3d0' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.36s/epoch, loss=0.000464, prev_loss=0.000476] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.91s/epoch, loss=0.000464, prev_loss=0.000476] 2021-09-16:22:53:52,246 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp5ukz6hky' 2021-09-16:22:53:52,298 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp5ukz6hky' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 9.25s/epoch, loss=0.000101, prev_loss=0.000107] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.12s/epoch, loss=0.000101, prev_loss=0.000107] 2021-09-16:22:55:08,838 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpccyg5voe' 2021-09-16:22:55:08,889 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpccyg5voe' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 9.95s/epoch, loss=0.000476, prev_loss=0.000498] Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 6.64s/epoch, loss=0.000476, prev_loss=0.000498] 2021-09-16:22:56:30,709 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpyy7w5e4x' 2021-09-16:22:56:30,760 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpyy7w5e4x' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 9.02s/epoch, loss=0.000309, prev_loss=0.000349] Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 5.77s/epoch, loss=0.000309, prev_loss=0.000349] 2021-09-16:22:57:43,785 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpp5av56id' 2021-09-16:22:57:43,836 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpp5av56id' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 9.68s/epoch, loss=0.000145, prev_loss=0.000158] Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 6.43s/epoch, loss=0.000145, prev_loss=0.000158] 2021-09-16:22:59:03,537 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpms8yl4bk' 2021-09-16:22:59:03,592 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpms8yl4bk' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 9.69s/epoch, loss=0.00047, prev_loss=0.00047] Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 6.51s/epoch, loss=0.00047, prev_loss=0.00047] 2021-09-16:23:00:24,58 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpnvv5_20e' 2021-09-16:23:00:24,112 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpnvv5_20e' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 9.04s/epoch, loss=0.000154, prev_loss=0.000169] Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 5.88s/epoch, loss=0.000154, prev_loss=0.000169] 2021-09-16:23:01:38,164 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpnqagr8pf' 2021-09-16:23:01:38,217 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpnqagr8pf' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:09<00:00, 10.11s/epoch, loss=0.000514, prev_loss=0.000515] Training epochs on cuda: 100%|██████████| 10/10 [01:09<00:00, 6.97s/epoch, loss=0.000514, prev_loss=0.000515] 2021-09-16:23:03:03,233 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_p0v4sli' 2021-09-16:23:03:03,286 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_p0v4sli' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 9.07s/epoch, loss=0.000542, prev_loss=0.000543] Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 5.78s/epoch, loss=0.000542, prev_loss=0.000543] 2021-09-16:23:04:16,435 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp98w_atp1' 2021-09-16:23:04:16,496 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp98w_atp1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 9.65s/epoch, loss=0.000427, prev_loss=0.00043] Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 6.42s/epoch, loss=0.000427, prev_loss=0.00043] 2021-09-16:23:05:36,0 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmppf1stg0l' 2021-09-16:23:05:36,47 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmppf1stg0l' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 9.40s/epoch, loss=0.000136, prev_loss=0.000144] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.12s/epoch, loss=0.000136, prev_loss=0.000144] 2021-09-16:23:06:52,583 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpw5synddd' 2021-09-16:23:06:52,637 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpw5synddd' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 9.53s/epoch, loss=0.000579, prev_loss=0.000579] Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 6.32s/epoch, loss=0.000579, prev_loss=0.000579] 2021-09-16:23:08:11,101 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmptifva_ga' 2021-09-16:23:08:11,153 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmptifva_ga' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 8.84s/epoch, loss=0.000117, prev_loss=0.000124] Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 5.66s/epoch, loss=0.000117, prev_loss=0.000124] 2021-09-16:23:09:22,975 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp90b0wan0' 2021-09-16:23:09:23,30 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp90b0wan0' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 9.11s/epoch, loss=0.000126, prev_loss=0.000133] Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 5.84s/epoch, loss=0.000126, prev_loss=0.000133] 2021-09-16:23:10:36,726 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpung8rux4' 2021-09-16:23:10:36,782 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpung8rux4' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 8.98s/epoch, loss=0.000124, prev_loss=0.000129] Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00, 5.77s/epoch, loss=0.000124, prev_loss=0.000129] 2021-09-16:23:11:49,872 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmphdjfd12s' 2021-09-16:23:11:49,926 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmphdjfd12s' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 8.90s/epoch, loss=0.000515, prev_loss=0.000516] Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00, 5.69s/epoch, loss=0.000515, prev_loss=0.000516] 2021-09-16:23:13:02,211 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmppfnmvkjb' 2021-09-16:23:13:02,264 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmppfnmvkjb' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 9.08s/epoch, loss=0.000505, prev_loss=0.000511] Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 5.83s/epoch, loss=0.000505, prev_loss=0.000511] 2021-09-16:23:14:15,949 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp3fbee35i' 2021-09-16:23:14:16,4 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp3fbee35i' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:34<00:00, 6.68s/epoch, loss=0.000491, prev_loss=0.000492] Training epochs on cuda: 100%|██████████| 10/10 [00:34<00:00, 3.49s/epoch, loss=0.000491, prev_loss=0.000492] 2021-09-16:23:15:06,265 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpq_bcp7ex' 2021-09-16:23:15:06,317 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpq_bcp7ex' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 9.17s/epoch, loss=0.000123, prev_loss=0.000134] Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 6.05s/epoch, loss=0.000123, prev_loss=0.000134] 2021-09-16:23:16:22,129 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1a6h55ji' 2021-09-16:23:16:22,184 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1a6h55ji' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.14s/epoch, loss=0.000549, prev_loss=0.000549] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.94s/epoch, loss=0.000549, prev_loss=0.000549] 2021-09-16:23:17:36,934 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmptoskhlx2' 2021-09-16:23:17:36,988 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmptoskhlx2' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 9.07s/epoch, loss=0.00013, prev_loss=0.000142] Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 5.85s/epoch, loss=0.00013, prev_loss=0.000142] 2021-09-16:23:18:50,829 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmppiupffd_' 2021-09-16:23:18:50,883 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmppiupffd_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.15s/epoch, loss=0.000493, prev_loss=0.000493] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.90s/epoch, loss=0.000493, prev_loss=0.000493] 2021-09-16:23:20:05,221 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpwfvo1nus' 2021-09-16:23:20:05,273 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpwfvo1nus' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.22s/epoch, loss=0.000518, prev_loss=0.000518] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.99s/epoch, loss=0.000518, prev_loss=0.000518] 2021-09-16:23:21:20,495 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmptmzrsmea' 2021-09-16:23:21:20,549 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmptmzrsmea' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 9.58s/epoch, loss=0.000344, prev_loss=0.000491] Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 6.39s/epoch, loss=0.000344, prev_loss=0.000491] 2021-09-16:23:22:39,661 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpjrwil7o4' 2021-09-16:23:22:39,713 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpjrwil7o4' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.09s/epoch, loss=0.000492, prev_loss=0.000492] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.90s/epoch, loss=0.000492, prev_loss=0.000492] 2021-09-16:23:23:54,155 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpcszf988l' 2021-09-16:23:23:54,206 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpcszf988l' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 9.51s/epoch, loss=0.000484, prev_loss=0.000502] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.18s/epoch, loss=0.000484, prev_loss=0.000502] 2021-09-16:23:25:11,385 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpo9zuktj6' 2021-09-16:23:25:11,437 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpo9zuktj6' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 9.68s/epoch, loss=0.000112, prev_loss=0.000114] Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 6.44s/epoch, loss=0.000112, prev_loss=0.000114] 2021-09-16:23:26:31,191 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp5d5al5mn' 2021-09-16:23:26:31,245 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp5d5al5mn' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 9.19s/epoch, loss=0.000479, prev_loss=0.00049] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.93s/epoch, loss=0.000479, prev_loss=0.00049] 2021-09-16:23:27:45,837 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp5nnux4ou' 2021-09-16:23:27:45,889 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp5nnux4ou' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 9.41s/epoch, loss=0.000277, prev_loss=0.000287] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.11s/epoch, loss=0.000277, prev_loss=0.000287] 2021-09-16:23:29:02,395 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp8n1jruzu' 2021-09-16:23:29:02,445 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp8n1jruzu' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 10.60s/epoch, loss=0.00016, prev_loss=0.000176] Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 7.13s/epoch, loss=0.00016, prev_loss=0.000176] 2021-09-16:23:30:29,111 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpvq2unev_' 2021-09-16:23:30:29,162 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpvq2unev_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 9.99s/epoch, loss=0.000579, prev_loss=0.000579] Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 6.78s/epoch, loss=0.000579, prev_loss=0.000579] 2021-09-16:23:31:52,242 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpwzs3oom9' 2021-09-16:23:31:52,294 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpwzs3oom9' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 9.80s/epoch, loss=0.000534, prev_loss=0.000536] Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 6.59s/epoch, loss=0.000534, prev_loss=0.000536] 2021-09-16:23:33:13,537 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmprc1d_kdx' 2021-09-16:23:33:13,588 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmprc1d_kdx' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 9.53s/epoch, loss=0.000475, prev_loss=0.000494] Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 6.22s/epoch, loss=0.000475, prev_loss=0.000494] 2021-09-16:23:34:31,84 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp9eif68xl' 2021-09-16:23:34:31,135 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp9eif68xl' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 9.11s/epoch, loss=0.000506, prev_loss=0.000513] Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 5.84s/epoch, loss=0.000506, prev_loss=0.000513] 2021-09-16:23:35:44,930 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpubormpdc' 2021-09-16:23:35:44,980 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpubormpdc' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 10.00s/epoch, loss=0.000154, prev_loss=0.000169] Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 6.70s/epoch, loss=0.000154, prev_loss=0.000169] 2021-09-16:23:37:07,362 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp92rh2hw5' 2021-09-16:23:37:07,413 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp92rh2hw5' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 9.98s/epoch, loss=0.000615, prev_loss=0.000615] Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 6.70s/epoch, loss=0.000615, prev_loss=0.000615] 2021-09-16:23:38:29,712 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpr0tcjc2l' 2021-09-16:23:38:29,766 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpr0tcjc2l' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:09<00:00, 10.10s/epoch, loss=0.000534, prev_loss=0.000539] Training epochs on cuda: 100%|██████████| 10/10 [01:09<00:00, 6.91s/epoch, loss=0.000534, prev_loss=0.000539] 2021-09-16:23:39:54,236 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpmrmtimfb' 2021-09-16:23:39:54,288 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpmrmtimfb' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 9.73s/epoch, loss=0.000515, prev_loss=0.000526] Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 6.52s/epoch, loss=0.000515, prev_loss=0.000526] 2021-09-16:23:41:14,757 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp22vs8r8g' 2021-09-16:23:41:14,808 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp22vs8r8g' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 9.47s/epoch, loss=0.00053, prev_loss=0.000542] Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 6.28s/epoch, loss=0.00053, prev_loss=0.000542] 2021-09-16:23:42:32,913 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpk1qmwvhx' 2021-09-16:23:42:32,971 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpk1qmwvhx' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 9.80s/epoch, loss=0.00012, prev_loss=0.000129] Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 6.57s/epoch, loss=0.00012, prev_loss=0.000129] 2021-09-16:23:43:54,80 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpmr3ongwf' 2021-09-16:23:43:54,132 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpmr3ongwf' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:10<00:00, 10.33s/epoch, loss=9.36e-7, prev_loss=8.97e-7] Training epochs on cuda: 100%|██████████| 10/10 [01:10<00:00, 7.07s/epoch, loss=9.36e-7, prev_loss=8.97e-7] 2021-09-16:23:45:20,199 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpilyucp_b' 2021-09-16:23:45:20,251 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpilyucp_b' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 9.52s/epoch, loss=0.000546, prev_loss=0.000558] Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 6.29s/epoch, loss=0.000546, prev_loss=0.000558] 2021-09-16:23:46:38,477 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp56xjyn_t' 2021-09-16:23:46:38,529 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp56xjyn_t' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 9.61s/epoch, loss=0.000536, prev_loss=0.000545] Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 6.40s/epoch, loss=0.000536, prev_loss=0.000545] 2021-09-16:23:47:57,803 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpigs9ubec' 2021-09-16:23:47:57,856 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpigs9ubec' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:18<00:00, 11.02s/epoch, loss=0.000249, prev_loss=0.000245] Training epochs on cuda: 100%|██████████| 10/10 [01:18<00:00, 7.87s/epoch, loss=0.000249, prev_loss=0.000245] 2021-09-16:23:49:32,15 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmppfvot7js' 2021-09-16:23:49:32,67 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmppfvot7js' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:10<00:00, 10.33s/epoch, loss=0.000135, prev_loss=0.000148] Training epochs on cuda: 100%|██████████| 10/10 [01:10<00:00, 7.08s/epoch, loss=0.000135, prev_loss=0.000148] 2021-09-16:23:50:58,313 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpi0baiaqv' 2021-09-16:23:50:58,365 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpi0baiaqv' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:09<00:00, 10.16s/epoch, loss=0.000137, prev_loss=0.000154] Training epochs on cuda: 100%|██████████| 10/10 [01:09<00:00, 6.97s/epoch, loss=0.000137, prev_loss=0.000154] 2021-09-16:23:52:23,479 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmph8oufwko' 2021-09-16:23:52:23,545 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmph8oufwko' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 9.66s/epoch, loss=0.000597, prev_loss=0.000616] Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 6.44s/epoch, loss=0.000597, prev_loss=0.000616] 2021-09-16:23:53:43,256 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp5z7tbvzt' 2021-09-16:23:53:43,327 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp5z7tbvzt' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:09<00:00, 10.13s/epoch, loss=0.000186, prev_loss=0.000206] Training epochs on cuda: 100%|██████████| 10/10 [01:09<00:00, 6.99s/epoch, loss=0.000186, prev_loss=0.000206] 2021-09-16:23:55:08,496 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp0v6hb674' 2021-09-16:23:55:08,578 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp0v6hb674' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 10.36s/epoch, loss=0.000489, prev_loss=0.000491] Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 7.18s/epoch, loss=0.000489, prev_loss=0.000491] 2021-09-16:23:56:35,735 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmprtoquhvr' 2021-09-16:23:56:35,798 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmprtoquhvr' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 9.96s/epoch, loss=0.000464, prev_loss=0.000474] Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 6.71s/epoch, loss=0.000464, prev_loss=0.000474] 2021-09-16:23:57:58,343 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpqtidlb_m' 2021-09-16:23:57:58,425 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpqtidlb_m' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 9.79s/epoch, loss=0.000547, prev_loss=0.000547] Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 6.54s/epoch, loss=0.000547, prev_loss=0.000547] 2021-09-16:23:59:19,212 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpg6vuwap7' 2021-09-16:23:59:19,314 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpg6vuwap7' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:17<00:00, 10.96s/epoch, loss=0.000484, prev_loss=0.000488] Training epochs on cuda: 100%|██████████| 10/10 [01:17<00:00, 7.71s/epoch, loss=0.000484, prev_loss=0.000488] 2021-09-17:00:00:51,740 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpzau72ymj' 2021-09-17:00:00:51,802 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpzau72ymj' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 9.49s/epoch, loss=0.000503, prev_loss=0.0005] Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00, 6.21s/epoch, loss=0.000503, prev_loss=0.0005] 2021-09-17:00:02:09,357 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpq19kwp6o' 2021-09-17:00:02:09,436 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpq19kwp6o' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:10<00:00, 10.19s/epoch, loss=0.000131, prev_loss=0.000138] Training epochs on cuda: 100%|██████████| 10/10 [01:10<00:00, 7.01s/epoch, loss=0.000131, prev_loss=0.000138] 2021-09-17:00:03:34,991 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpf56sbp63' 2021-09-17:00:03:35,60 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpf56sbp63' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:09<00:00, 10.29s/epoch, loss=0.000555, prev_loss=0.000562] Training epochs on cuda: 100%|██████████| 10/10 [01:09<00:00, 6.94s/epoch, loss=0.000555, prev_loss=0.000562] 2021-09-17:00:04:59,856 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpatb3fk1u' 2021-09-17:00:04:59,918 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpatb3fk1u' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:08<00:00, 10.26s/epoch, loss=0.000579, prev_loss=0.000579] Training epochs on cuda: 100%|██████████| 10/10 [01:08<00:00, 6.86s/epoch, loss=0.000579, prev_loss=0.000579] 2021-09-17:00:06:23,856 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpffwtewjd' 2021-09-17:00:06:23,944 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpffwtewjd' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 9.98s/epoch, loss=0.000132, prev_loss=0.000139] Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 6.66s/epoch, loss=0.000132, prev_loss=0.000139] 2021-09-17:00:07:46,28 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpawk9r36q' 2021-09-17:00:07:46,106 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpawk9r36q' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 9.99s/epoch, loss=0.00055, prev_loss=0.00055] Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 6.68s/epoch, loss=0.00055, prev_loss=0.00055] 2021-09-17:00:09:08,307 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp648gu8ke' 2021-09-17:00:09:08,369 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp648gu8ke' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:08<00:00, 10.04s/epoch, loss=0.00058, prev_loss=0.00058] Training epochs on cuda: 100%|██████████| 10/10 [01:08<00:00, 6.80s/epoch, loss=0.00058, prev_loss=0.00058] 2021-09-17:00:10:31,701 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpx7ln3zdo' 2021-09-17:00:10:31,789 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpx7ln3zdo' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:15<00:00, 10.73s/epoch, loss=0.000548, prev_loss=0.000548] Training epochs on cuda: 100%|██████████| 10/10 [01:15<00:00, 7.52s/epoch, loss=0.000548, prev_loss=0.000548] 2021-09-17:00:12:02,333 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpjs3c2rzw' 2021-09-17:00:12:02,390 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpjs3c2rzw' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 9.96s/epoch, loss=0.000518, prev_loss=0.000518] Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 6.72s/epoch, loss=0.000518, prev_loss=0.000518] 2021-09-17:00:13:24,963 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpdjlwvtfs' 2021-09-17:00:13:25,40 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdjlwvtfs' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 9.26s/epoch, loss=0.000567, prev_loss=0.000577] Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00, 6.06s/epoch, loss=0.000567, prev_loss=0.000577] 2021-09-17:00:14:40,947 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpj7eahlix' 2021-09-17:00:14:41,33 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpj7eahlix' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:12<00:00, 10.60s/epoch, loss=0.00164, prev_loss=0.00164] Training epochs on cuda: 100%|██████████| 10/10 [01:12<00:00, 7.29s/epoch, loss=0.00164, prev_loss=0.00164] 2021-09-17:00:16:09,349 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpc20i9ibd' 2021-09-17:00:16:09,396 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpc20i9ibd' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 9.77s/epoch, loss=0.000126, prev_loss=0.000135] Training epochs on cuda: 100%|██████████| 10/10 [01:05<00:00, 6.53s/epoch, loss=0.000126, prev_loss=0.000135] 2021-09-17:00:17:29,990 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp9bupj215' 2021-09-17:00:17:30,72 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp9bupj215' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 9.70s/epoch, loss=0.000151, prev_loss=0.000168] Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 6.49s/epoch, loss=0.000151, prev_loss=0.000168] 2021-09-17:00:18:50,435 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpnoy7rw6l' 2021-09-17:00:18:50,496 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpnoy7rw6l' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:08<00:00, 10.08s/epoch, loss=0.000615, prev_loss=0.000615] Training epochs on cuda: 100%|██████████| 10/10 [01:08<00:00, 6.83s/epoch, loss=0.000615, prev_loss=0.000615] 2021-09-17:00:20:14,130 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp2palalzs' 2021-09-17:00:20:14,196 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp2palalzs' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 9.89s/epoch, loss=0.000579, prev_loss=0.000579] Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 6.60s/epoch, loss=0.000579, prev_loss=0.000579] 2021-09-17:00:21:35,580 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp8gem3d41' 2021-09-17:00:21:35,664 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp8gem3d41' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 synset) to (word -> word) Number of edges with words as nodes = 388301 triples' shape: (388301, 3) TriplesFactory(num_entities=45182, num_relations=31, num_triples=388301, inverse_triples=False) HpoPipelineResult(study=, objective=Objective(dataset=None, model=, loss=, optimizer=, training_loop=, stopper=, evaluator=, result_tracker=, metric='adjusted_arithmetic_mean_rank_index', dataset_kwargs=None, training=TriplesFactory(num_entities=45182, num_relations=31, num_triples=310640, inverse_triples=False), testing=TriplesFactory(num_entities=45182, num_relations=31, num_triples=38830, inverse_triples=False), validation=TriplesFactory(num_entities=45182, num_relations=31, num_triples=38831, inverse_triples=False), evaluation_entity_whitelist=None, evaluation_relation_whitelist=None, model_kwargs={'embedding_dim': 100}, model_kwargs_ranges=None, loss_kwargs=None, loss_kwargs_ranges=None, regularizer=, regularizer_kwargs=None, regularizer_kwargs_ranges=None, optimizer_kwargs=None, optimizer_kwargs_ranges=None, training_loop_kwargs=None, negative_sampler=, negative_sampler_kwargs=None, negative_sampler_kwargs_ranges=None, training_kwargs={'num_epochs': 10}, training_kwargs_ranges=None, stopper_kwargs=None, evaluator_kwargs=None, evaluation_kwargs=None, filter_validation_when_testing=True, result_tracker_kwargs=None, device=None, save_model_directory=None))