2021-09-16:17:27:07,185 INFO [iwn.py:43] Loading bengali language synsets... 2021-09-16:17:27:20,93 WARNING [utils.py:429] using automatically assigned random_state=3048715693 2021-09-16:17:27:20,169 INFO [splitting.py:60] done splitting triples to groups of sizes [267005, 38830, 38831] /srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release. from numpy.core.umath_tests import inner1d /srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/gradient_boosting.py:34: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations from ._gradient_boosting import predict_stages /srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/gradient_boosting.py:34: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations from ._gradient_boosting import predict_stages [32m[I 2021-09-16 17:27:20,924][0m A new study created in memory with name: no-name-51080656-f2e2-4ebe-abb4-f172fe38107f[0m 2021-09-16:17:27:20,973 INFO [hpo.py:622] Using model: 2021-09-16:17:27:20,973 INFO [hpo.py:626] Using loss: 2021-09-16:17:27:20,974 INFO [hpo.py:641] Using optimizer: 2021-09-16:17:27:20,974 INFO [hpo.py:645] Using training loop: 2021-09-16:17:27:20,974 INFO [hpo.py:651] Using negative sampler: 2021-09-16:17:27:20,974 INFO [hpo.py:662] Using evaluator: 2021-09-16:17:27:20,974 INFO [hpo.py:666] Attempting to maximize adjusted_arithmetic_mean_rank_index 2021-09-16:17:27:20,974 INFO [hpo.py:668] Filter validation triples when testing: True /srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932]. low=low, old_high=old_high, high=high, step=step 2021-09-16:17:27:20,978 WARNING [api.py:823] No random seed is specified. Setting to 2218594265. Training epochs on cuda: 0%| | 0/10 [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:13<00:00, 14.56s/epoch, loss=0.00134, prev_loss=0.00134] Training epochs on cuda: 100%|██████████| 10/10 [01:13<00:00, 7.32s/epoch, loss=0.00134, prev_loss=0.00134] 2021-09-16:17:28:40,29 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpyjl8c6h3' 2021-09-16:17:28:40,73 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpyjl8c6h3' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 13.91s/epoch, loss=0.000635, prev_loss=0.00064] Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 6.65s/epoch, loss=0.000635, prev_loss=0.00064] 2021-09-16:17:30:21,603 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpptxttqya' 2021-09-16:17:30:21,660 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpptxttqya' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 13.25s/epoch, loss=0.000232, prev_loss=0.000232] Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00, 5.88s/epoch, loss=0.000232, prev_loss=0.000232] 2021-09-16:17:31:55,423 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpplw8krx1' 2021-09-16:17:31:55,485 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpplw8krx1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 13.19s/epoch, loss=2.37e-5, prev_loss=2.11e-5] Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00, 5.99s/epoch, loss=2.37e-5, prev_loss=2.11e-5] 2021-09-16:17:33:30,248 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmphbd2i_4l' 2021-09-16:17:33:30,310 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmphbd2i_4l' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 14.67s/epoch, loss=0.00026, prev_loss=0.000265] Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 7.43s/epoch, loss=0.00026, prev_loss=0.000265] 2021-09-16:17:35:19,246 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp55wrbiom' 2021-09-16:17:35:19,305 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp55wrbiom' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:19<00:00, 15.18s/epoch, loss=0.000134, prev_loss=0.000143] Training epochs on cuda: 100%|██████████| 10/10 [01:19<00:00, 7.95s/epoch, loss=0.000134, prev_loss=0.000143] 2021-09-16:17:37:13,812 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpv0odudba' 2021-09-16:17:37:13,872 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpv0odudba' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:48<00:00, 18.13s/epoch, loss=0.00174, prev_loss=0.00175] Training epochs on cuda: 100%|██████████| 10/10 [01:48<00:00, 10.84s/epoch, loss=0.00174, prev_loss=0.00175] 2021-09-16:17:39:38,534 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpeu2is3u1' 2021-09-16:17:39:38,572 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpeu2is3u1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 19.65s/epoch, loss=6.05e-6, prev_loss=5.94e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 12.45s/epoch, loss=6.05e-6, prev_loss=5.94e-6] 2021-09-16:17:42:18,310 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_if_vuq9' 2021-09-16:17:42:18,349 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_if_vuq9' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:55<00:00, 12.65s/epoch, loss=0.000381, prev_loss=0.000381] Training epochs on cuda: 100%|██████████| 10/10 [00:55<00:00, 5.50s/epoch, loss=0.000381, prev_loss=0.000381] 2021-09-16:17:43:48,44 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpstd0773f' 2021-09-16:17:43:48,81 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpstd0773f' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:13<00:00, 14.60s/epoch, loss=7.84e-6, prev_loss=8.66e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:13<00:00, 7.38s/epoch, loss=7.84e-6, prev_loss=8.66e-6] 2021-09-16:17:45:36,913 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpgedjsj7b' 2021-09-16:17:45:36,949 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpgedjsj7b' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:22<00:00, 21.40s/epoch, loss=2.58e-5, prev_loss=2.83e-5] Training epochs on cuda: 100%|██████████| 10/10 [02:22<00:00, 14.25s/epoch, loss=2.58e-5, prev_loss=2.83e-5] 2021-09-16:17:48:34,660 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp7z0vick7' 2021-09-16:17:48:34,702 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp7z0vick7' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:37<00:00, 16.99s/epoch, loss=1.06e-5, prev_loss=1.31e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:37<00:00, 9.78s/epoch, loss=1.06e-5, prev_loss=1.31e-5] 2021-09-16:17:50:47,428 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpawrifpyg' 2021-09-16:17:50:47,465 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpawrifpyg' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:09<00:00, 14.11s/epoch, loss=4.2e-6, prev_loss=4.54e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:09<00:00, 6.91s/epoch, loss=4.2e-6, prev_loss=4.54e-6] 2021-09-16:17:52:31,448 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpy_mz9ql2' 2021-09-16:17:52:31,491 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpy_mz9ql2' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:35<00:00, 16.78s/epoch, loss=6.57e-6, prev_loss=5.43e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:35<00:00, 9.52s/epoch, loss=6.57e-6, prev_loss=5.43e-6] 2021-09-16:17:54:41,609 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpmkkomkd1' 2021-09-16:17:54:41,655 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpmkkomkd1' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:02<00:00, 19.46s/epoch, loss=2.35e-5, prev_loss=2.85e-5] Training epochs on cuda: 100%|██████████| 10/10 [02:02<00:00, 12.29s/epoch, loss=2.35e-5, prev_loss=2.85e-5] 2021-09-16:17:57:19,421 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpfrws1kc6' 2021-09-16:17:57:19,473 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfrws1kc6' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:57<00:00, 18.89s/epoch, loss=1.41e-5, prev_loss=1.03e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:57<00:00, 11.77s/epoch, loss=1.41e-5, prev_loss=1.03e-5] 2021-09-16:17:59:52,55 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpw6568593' 2021-09-16:17:59:52,141 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpw6568593' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:24<00:00, 15.78s/epoch, loss=1.82e-5, prev_loss=1.93e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:24<00:00, 8.44s/epoch, loss=1.82e-5, prev_loss=1.93e-5] 2021-09-16:18:01:51,471 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp9re_7rzf' 2021-09-16:18:01:51,511 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp9re_7rzf' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 13.58s/epoch, loss=1.18e-5, prev_loss=1.02e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00, 6.43s/epoch, loss=1.18e-5, prev_loss=1.02e-5] 2021-09-16:18:03:30,691 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp3pi3f9jy' 2021-09-16:18:03:30,718 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp3pi3f9jy' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:39<00:00, 17.13s/epoch, loss=0.000165, prev_loss=0.000172] Training epochs on cuda: 100%|██████████| 10/10 [01:39<00:00, 9.91s/epoch, loss=0.000165, prev_loss=0.000172] 2021-09-16:18:05:44,568 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpfv49ztd3' 2021-09-16:18:05:44,593 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfv49ztd3' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [09:21<00:00, 63.12s/epoch, loss=0.0317, prev_loss=0.0318] Training epochs on cuda: 100%|██████████| 10/10 [09:21<00:00, 56.16s/epoch, loss=0.0317, prev_loss=0.0318] 2021-09-16:18:15:40,969 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_rizehmv' 2021-09-16:18:15:41,13 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_rizehmv' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:43<00:00, 23.56s/epoch, loss=0.000162, prev_loss=0.00016] Training epochs on cuda: 100%|██████████| 10/10 [02:43<00:00, 16.38s/epoch, loss=0.000162, prev_loss=0.00016] 2021-09-16:18:18:59,951 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpz_s3iefb' 2021-09-16:18:18:59,986 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpz_s3iefb' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00, 14.75s/epoch, loss=5.99e-6, prev_loss=6.13e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00, 7.66s/epoch, loss=5.99e-6, prev_loss=6.13e-6] 2021-09-16:18:20:51,329 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpkfvrp_yj' 2021-09-16:18:20:51,368 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpkfvrp_yj' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:20<00:00, 15.24s/epoch, loss=5.28e-6, prev_loss=5.24e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:20<00:00, 8.06s/epoch, loss=5.28e-6, prev_loss=5.24e-6] 2021-09-16:18:22:46,850 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp7y0zs05l' 2021-09-16:18:22:46,887 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp7y0zs05l' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00, 15.50s/epoch, loss=4.43e-6, prev_loss=4.61e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00, 8.30s/epoch, loss=4.43e-6, prev_loss=4.61e-6] 2021-09-16:18:24:44,850 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpjne4nlt_' 2021-09-16:18:24:44,887 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpjne4nlt_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 13.86s/epoch, loss=4.59e-5, prev_loss=5.67e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00, 6.66s/epoch, loss=4.59e-5, prev_loss=5.67e-5] 2021-09-16:18:26:26,498 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpciwyueh_' 2021-09-16:18:26:26,537 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpciwyueh_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:25<00:00, 15.72s/epoch, loss=4.35e-6, prev_loss=4.35e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:25<00:00, 8.50s/epoch, loss=4.35e-6, prev_loss=4.35e-6] 2021-09-16:18:28:26,419 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp1ixspa_9' 2021-09-16:18:28:26,464 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1ixspa_9' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:55<00:00, 18.78s/epoch, loss=6.74e-6, prev_loss=4.32e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:55<00:00, 11.57s/epoch, loss=6.74e-6, prev_loss=4.32e-6] 2021-09-16:18:30:57,389 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpcwof6l99' 2021-09-16:18:30:57,423 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpcwof6l99' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:39<00:00, 17.08s/epoch, loss=2.8e-6, prev_loss=2.93e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:39<00:00, 9.90s/epoch, loss=2.8e-6, prev_loss=2.93e-6] 2021-09-16:18:33:11,372 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpaze5b2ra' 2021-09-16:18:33:11,405 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpaze5b2ra' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 13.54s/epoch, loss=6.51e-5, prev_loss=6.58e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:03<00:00, 6.32s/epoch, loss=6.51e-5, prev_loss=6.58e-5] 2021-09-16:18:34:49,484 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpnbcbdpad' 2021-09-16:18:34:49,534 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpnbcbdpad' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [00:55<00:00, 12.78s/epoch, loss=7.5e-5, prev_loss=8.18e-5] Training epochs on cuda: 100%|██████████| 10/10 [00:55<00:00, 5.57s/epoch, loss=7.5e-5, prev_loss=8.18e-5] 2021-09-16:18:36:20,49 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpzyxlql82' 2021-09-16:18:36:20,106 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpzyxlql82' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 14.74s/epoch, loss=2.75e-6, prev_loss=2.77e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 7.46s/epoch, loss=2.75e-6, prev_loss=2.77e-6] 2021-09-16:18:38:09,593 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmph_nt2a1h' 2021-09-16:18:38:09,638 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmph_nt2a1h' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:22<00:00, 15.38s/epoch, loss=4.86e-6, prev_loss=4.87e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:22<00:00, 8.23s/epoch, loss=4.86e-6, prev_loss=4.87e-6] 2021-09-16:18:40:07,148 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpji8s1wyp' 2021-09-16:18:40:07,197 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpji8s1wyp' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:27<00:00, 15.97s/epoch, loss=3.48e-5, prev_loss=2.46e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:27<00:00, 8.73s/epoch, loss=3.48e-5, prev_loss=2.46e-5] 2021-09-16:18:42:09,782 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmphsnauj42' 2021-09-16:18:42:09,828 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmphsnauj42' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 13.29s/epoch, loss=1.97e-5, prev_loss=1.83e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00, 6.15s/epoch, loss=1.97e-5, prev_loss=1.83e-5] 2021-09-16:18:43:46,294 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpexbkbxf9' 2021-09-16:18:43:46,333 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpexbkbxf9' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 18.48s/epoch, loss=4.14e-6, prev_loss=4.27e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 11.27s/epoch, loss=4.14e-6, prev_loss=4.27e-6] 2021-09-16:18:46:14,44 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpcor9kmeh' 2021-09-16:18:46:14,83 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpcor9kmeh' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:27<00:00, 16.03s/epoch, loss=3.48e-5, prev_loss=4.9e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:27<00:00, 8.79s/epoch, loss=3.48e-5, prev_loss=4.9e-5] 2021-09-16:18:48:16,859 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpq8k8_cq_' 2021-09-16:18:48:16,904 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpq8k8_cq_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 14.71s/epoch, loss=2.05e-5, prev_loss=1.69e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:14<00:00, 7.45s/epoch, loss=2.05e-5, prev_loss=1.69e-5] 2021-09-16:18:50:06,317 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp56zt5icf' 2021-09-16:18:50:06,364 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp56zt5icf' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 13.93s/epoch, loss=1.18e-5, prev_loss=1.05e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 6.79s/epoch, loss=1.18e-5, prev_loss=1.05e-5] 2021-09-16:18:51:49,101 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpod5385eq' 2021-09-16:18:51:49,151 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpod5385eq' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:17<00:00, 14.96s/epoch, loss=7.14e-7, prev_loss=4.38e-7] Training epochs on cuda: 100%|██████████| 10/10 [01:17<00:00, 7.77s/epoch, loss=7.14e-7, prev_loss=4.38e-7] 2021-09-16:18:53:41,764 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpt1q_ifsy' 2021-09-16:18:53:41,853 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpt1q_ifsy' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:38<00:00, 17.09s/epoch, loss=7.93e-5, prev_loss=7.87e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:38<00:00, 9.90s/epoch, loss=7.93e-5, prev_loss=7.87e-5] 2021-09-16:18:55:55,740 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpw1tmjuum' 2021-09-16:18:55:55,788 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpw1tmjuum' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:58<00:00, 18.99s/epoch, loss=3.81e-6, prev_loss=3.42e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:58<00:00, 11.82s/epoch, loss=3.81e-6, prev_loss=3.42e-6] 2021-09-16:18:58:29,265 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpte5puo0t' 2021-09-16:18:58:29,311 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpte5puo0t' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 20.44s/epoch, loss=6.67e-6, prev_loss=7.1e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:11<00:00, 13.18s/epoch, loss=6.67e-6, prev_loss=7.1e-6] 2021-09-16:19:01:16,490 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpz_wbzc8w' 2021-09-16:19:01:16,538 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpz_wbzc8w' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 19.43s/epoch, loss=4e-6, prev_loss=3.8e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 12.19s/epoch, loss=4e-6, prev_loss=3.8e-6] 2021-09-16:19:03:53,995 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp8wd8i350' 2021-09-16:19:03:54,59 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp8wd8i350' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 19.39s/epoch, loss=9.15e-6, prev_loss=6.77e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:01<00:00, 12.17s/epoch, loss=9.15e-6, prev_loss=6.77e-6] 2021-09-16:19:06:31,130 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpxp5g28rs' 2021-09-16:19:06:31,205 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpxp5g28rs' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:43<00:00, 17.63s/epoch, loss=5.76e-6, prev_loss=4.99e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:43<00:00, 10.40s/epoch, loss=5.76e-6, prev_loss=4.99e-6] 2021-09-16:19:08:50,444 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpgi27q8uq' 2021-09-16:19:08:50,507 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpgi27q8uq' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 19.72s/epoch, loss=5.45e-5, prev_loss=6.08e-5] Training epochs on cuda: 100%|██████████| 10/10 [02:04<00:00, 12.48s/epoch, loss=5.45e-5, prev_loss=6.08e-5] 2021-09-16:19:11:30,359 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_z35v4sv' 2021-09-16:19:11:30,408 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_z35v4sv' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:35<00:00, 16.80s/epoch, loss=1.31e-5, prev_loss=1.36e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:35<00:00, 9.55s/epoch, loss=1.31e-5, prev_loss=1.36e-5] 2021-09-16:19:13:41,12 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpppyzfvrr' 2021-09-16:19:13:41,78 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpppyzfvrr' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 19.88s/epoch, loss=1.29e-5, prev_loss=1.42e-5] Training epochs on cuda: 100%|██████████| 10/10 [02:07<00:00, 12.79s/epoch, loss=1.29e-5, prev_loss=1.42e-5] 2021-09-16:19:16:24,259 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmps32v7kbp' 2021-09-16:19:16:24,311 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmps32v7kbp' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 18.44s/epoch, loss=0.00135, prev_loss=0.00137] Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 11.21s/epoch, loss=0.00135, prev_loss=0.00137] 2021-09-16:19:18:51,438 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpxw00e1up' 2021-09-16:19:18:51,499 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpxw00e1up' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:29<00:00, 16.18s/epoch, loss=0.000433, prev_loss=0.00044] Training epochs on cuda: 100%|██████████| 10/10 [01:29<00:00, 8.92s/epoch, loss=0.000433, prev_loss=0.00044] 2021-09-16:19:20:55,632 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp2953zx7u' 2021-09-16:19:20:55,672 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp2953zx7u' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:58<00:00, 19.23s/epoch, loss=2.39e-5, prev_loss=2.87e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:58<00:00, 11.86s/epoch, loss=2.39e-5, prev_loss=2.87e-5] 2021-09-16:19:23:29,243 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpqm8h91h2' 2021-09-16:19:23:29,280 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpqm8h91h2' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:37<00:00, 16.97s/epoch, loss=1.59e-5, prev_loss=1.25e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:37<00:00, 9.79s/epoch, loss=1.59e-5, prev_loss=1.25e-5] 2021-09-16:19:25:42,403 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpcs_fvaik' 2021-09-16:19:25:42,444 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpcs_fvaik' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:38<00:00, 17.13s/epoch, loss=8.07e-6, prev_loss=6.84e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:38<00:00, 9.86s/epoch, loss=8.07e-6, prev_loss=6.84e-6] 2021-09-16:19:27:56,55 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpvsjn_szv' 2021-09-16:19:27:56,112 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpvsjn_szv' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 21.00s/epoch, loss=4.04e-6, prev_loss=4.29e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 13.76s/epoch, loss=4.04e-6, prev_loss=4.29e-6] 2021-09-16:19:30:48,850 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpj5h9jh68' 2021-09-16:19:30:48,919 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpj5h9jh68' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:41<00:00, 17.39s/epoch, loss=1.17e-5, prev_loss=2.36e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:41<00:00, 10.16s/epoch, loss=1.17e-5, prev_loss=2.36e-5] 2021-09-16:19:33:05,651 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpn08z4zf_' 2021-09-16:19:33:05,687 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpn08z4zf_' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:55<00:00, 18.70s/epoch, loss=4.22e-6, prev_loss=4.55e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:55<00:00, 11.50s/epoch, loss=4.22e-6, prev_loss=4.55e-6] 2021-09-16:19:35:36,147 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpf89tdvq5' 2021-09-16:19:35:36,208 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpf89tdvq5' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:50<00:00, 18.42s/epoch, loss=0.000573, prev_loss=6.21e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:50<00:00, 11.09s/epoch, loss=0.000573, prev_loss=6.21e-5] 2021-09-16:19:38:02,75 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpwel6kmlr' 2021-09-16:19:38:02,162 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpwel6kmlr' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:45<00:00, 17.79s/epoch, loss=0.00119, prev_loss=0.00107] Training epochs on cuda: 100%|██████████| 10/10 [01:45<00:00, 10.56s/epoch, loss=0.00119, prev_loss=0.00107] 2021-09-16:19:40:22,820 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpxng7z7_0' 2021-09-16:19:40:22,883 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpxng7z7_0' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 14.52s/epoch, loss=0.000243, prev_loss=0.000234] Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 7.15s/epoch, loss=0.000243, prev_loss=0.000234] 2021-09-16:19:42:09,481 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpcyxhn9xf' 2021-09-16:19:42:09,526 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpcyxhn9xf' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:51<00:00, 18.34s/epoch, loss=2.6e-6, prev_loss=2.57e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:51<00:00, 11.12s/epoch, loss=2.6e-6, prev_loss=2.57e-6] 2021-09-16:19:44:35,997 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp9s3vo0fh' 2021-09-16:19:44:36,35 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp9s3vo0fh' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 18.47s/epoch, loss=2.62e-5, prev_loss=2.74e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:52<00:00, 11.22s/epoch, loss=2.62e-5, prev_loss=2.74e-5] 2021-09-16:19:47:03,428 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpc5q2dasi' 2021-09-16:19:47:03,475 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpc5q2dasi' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:44<00:00, 17.71s/epoch, loss=2.1e-6, prev_loss=2.14e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:44<00:00, 10.44s/epoch, loss=2.1e-6, prev_loss=2.14e-6] 2021-09-16:19:49:23,297 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpt4dyt6a8' 2021-09-16:19:49:23,364 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpt4dyt6a8' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:46<00:00, 17.94s/epoch, loss=2.99e-6, prev_loss=3.32e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:46<00:00, 10.70s/epoch, loss=2.99e-6, prev_loss=3.32e-6] 2021-09-16:19:51:45,812 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpsib32skr' 2021-09-16:19:51:45,865 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpsib32skr' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:53<00:00, 18.57s/epoch, loss=2.35e-6, prev_loss=2.33e-6] Training epochs on cuda: 100%|██████████| 10/10 [01:53<00:00, 11.34s/epoch, loss=2.35e-6, prev_loss=2.33e-6] 2021-09-16:19:54:14,915 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmpwh2c5xqm' 2021-09-16:19:54:14,969 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmpwh2c5xqm' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [01:48<00:00, 18.12s/epoch, loss=2.92e-5, prev_loss=6.66e-5] Training epochs on cuda: 100%|██████████| 10/10 [01:48<00:00, 10.84s/epoch, loss=2.92e-5, prev_loss=6.66e-5] 2021-09-16:19:56:38,523 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp_cgxhasw' 2021-09-16:19:56:38,564 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_cgxhasw' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00 Saved checkpoint after having finished epoch 10. Training epochs on cuda: 100%|██████████| 10/10 [02:15<00:00, 20.90s/epoch, loss=4.5e-6, prev_loss=3.44e-6] Training epochs on cuda: 100%|██████████| 10/10 [02:15<00:00, 13.56s/epoch, loss=4.5e-6, prev_loss=3.44e-6] 2021-09-16:19:59:29,984 INFO [training_loop.py:1092] => loading checkpoint '/tmp/tmp3muwzkn7' 2021-09-16:19:59:30,20 INFO [training_loop.py:1135] => loaded checkpoint '/tmp/tmp3muwzkn7' stopped after having finished epoch 10 Evaluating on cuda: 0%| | 0.00/38.8k [00:00